Neural Networks and the Simplist XOR Problem

20. Neural Networks and the Simplist XOR Problem#

This was adopted from the PyTorch Tutorials.
Simple supervised machine learning.
http://pytorch.org/tutorials/beginner/pytorch_with_examples.html

20.1. Neural Networks#

Neural networks are the foundation of deep learning, which has revolutionized the

In the mathematical theory of artificial neural networks, the universal approximation theorem states[1] that a feed-forward network with a single hidden layer containing a finite number of neurons (i.e., a multilayer perceptron), can approximate continuous functions on compact subsets of Rn, under mild assumptions on the activation function.

A simple task that Neural Networks can do but simple linear models cannot is called the XOR problem.
The XOR problem involves an output being 1 if either of two inputs is 1, but not both.

20.1.1. Generate Fake Data#

D_in is the number of dimensions of an input varaible.
D_out is the number of dimentions of an output variable.
Here we are learning some special “fake” data that represents the xor problem.
Here, the dv is 1 if either the first or second variable is

# -*- coding: utf-8 -*-
import numpy as np

#This is our independent and dependent variables. 
x = np.array([ [0,0,0],[1,0,0],[0,1,0],[0,0,0] ])
y = np.array([[0,1,1,0]]).T
print("Input data:\n",x,"\n Output data:\n",y)

Input data:
 [[0 0 0]
 [1 0 0]
 [0 1 0]
 [0 0 0]] 
 Output data:
 [[0]
 [1]
 [1]
 [0]]

20.1.2. A Simple Neural Network#

Here we are going to build a neural network.
First layer (D_in)has to be the length of the input.
H is the length of the output.
D_out is 1 as it will be the probability it is a 1.

np.random.seed(seed=83832)
#D_in is the number of input variables. 
#H is the hidden dimension.
#D_out is the number of dimensions for the output. 
D_in, H, D_out = 3, 2, 1

# Randomly initialize weights og out 2 hidden layer network.
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)
bias = np.random.randn(H, 1)

20.1.3. But “Hidden Layers” Aren’t Hidden#

Let’s take a look
These are just random numbers.

print(w1, w2)

[[-0.20401151  0.62388689]
 [-0.10186284  1.47372825]
 [ 1.07856887  0.01873049]] [[ 0.49346731]
 [-1.19376828]]

20.1.4. Update the Weights using Gradient Decent#

Calculate the predited value
Calculate the loss function
Compute the gradients of w1 and w2 with respect to the loss function
Update the weights using the learning rate

learning_rate = .01
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)

    #A relu is just the activation.
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

10.65792615907139
9.10203339892777
7.928225580610054
7.016030709608875
6.289798199184453
5.699847385692147
5.2123305302347624
4.803466247932402
4.456102755004962
4.1575876890269665
3.898402733982808
3.671262676836925
3.4705056296083194
3.291670966818706
3.1312013137273507
2.9862283397788603
2.854416299096229
2.733846078586037
2.622928124188624
2.5203362600714687
2.4249568284296723
2.335849203166264
2.2522148435722413
2.173372827242625
2.0987403459205147
2.0278170362586616
1.9601722976944669
1.8954349540796849
1.8332847664299674
1.773445416375481
1.7156786642283006
1.6597794495384952
1.6055717509418743
1.5529050598636533
1.5016513520352168
1.4517024638575724
1.402967798918812
1.3553723045677533
1.3088546702007282
1.2633657084618402
1.2188668883615361
1.175328995740111
1.1327309018102432
1.0910584249086992
1.0503032742251954
1.0104620672725408
0.9715354153057676
0.9335270728583867
0.8964431490967778
0.8602913798451839
0.8250804599443017
0.7908194361131566
0.7575171607226662
0.725181806895671
0.6938204451583221
0.6634386815182467
0.6340403563729178
0.6056273030939351
0.5781991645253675
0.5517532650109411
0.5262845349569829
0.5017854843733736
0.4782462213367543
0.45565451090747644
0.4339958697177296
0.4132536912411343
0.39340939665695995
0.3744426062333802
0.3563313262679901
0.339052146830798
0.3225804458429691
0.3068905953796952
0.2919561664924972
0.27775012928952875
0.26424504547683114
0.25141325103470336
0.23922702716849936
0.2276587581210479
0.2166810748552383
0.20626698400286633
0.1963899818243421
0.1870241532299702
0.17814425617563434
0.16972579196376805
0.1617450621557261
0.1541792129363326
0.147006267868564
0.140205150039638
0.13375569463319448
0.1276386529698942
0.1218356890447229
0.11632936955756723
0.11110314838793904
0.1061413464085187
0.10142912746853275
0.09695247130958222
0.09269814410571503
0.08865366724822979
0.08480728492549391
0.0811479309802164
0.07766519546208477
0.07434929123315356
0.0711910209273244
0.06818174451394585
0.06531334766910445
0.06257821111654908
0.05996918106326466
0.05747954082229276
0.05510298368722062
0.052833587098539955
0.050665788121483174
0.04859436023766026
0.046614391438508374
0.044721263596905425
0.042910633083986426
0.04117841259093982
0.0395207541100794
0.03793403302553889
0.03641483326129067
0.03495993343264216
0.03356629394672944
0.0322310449976488
0.03095147540259279
0.029725022226571782
0.028549261144892105
0.027421897494436413
0.0263407579668736
0.025303782899147254
0.02430901911889927
0.023354613304830955
0.0224388058243556
0.021559925013216318
0.02071638186401746
0.019906665092819995
0.019129336555072554
0.01838302698417915
0.017666432027935646
0.016978308559893886
0.016317471244437356
0.015682789335971285
0.015073183694146064
0.014487623998449207
0.01392512614681838
0.013384749824153785
0.012865596227742457
0.01236680593765744
0.011887556921165793
0.01142706266107241
0.010984570398752825
0.010559359483383497
0.010150739819576832
0.009758050406265562
0.009380657960268358
0.009017955618505363
0.00866936171332396
0.008334318615846222
0.008012291642660496
0.007702768021557011
0.0074052559123513875
0.007119283479154988
0.00684439801073854
0.006580165085898312
0.006326167780975099
0.0060820059168944065
0.005847295343298466
0.005621667257523059
0.00540476755634021
0.00519625621854179
0.004995806716578387
0.004803105455597478
0.004617851238342064
0.004439754754478814
0.004268538093024131
0.004103934276626908
0.003945686816550616
0.0037935492872733776
0.0036472849196956558
0.003506666212010064
0.0033714745573471235
0.0032414998873664088
0.0031165403310131914
0.002996401887707673
0.0028808981142773157
0.0027698498249834596
0.002663084804030042
0.002560437529977551
0.0024617489115170118
0.0023668660340890185
0.0022756419168604115
0.002187935279597463
0.002103610318998506
0.0020225364940712155
0.0019445883201617943
0.0018696451712621129
0.0017975910902400137
0.0017283146066554844
0.0016617085618411765
0.0015976699409418664
0.0015360997116213772
0.001476902669159437
0.0014199872876736073
0.0013652655772137235
0.0013126529464877815
0.0012620680709887451
0.0012134327663025218
0.0011666718663866913
0.0011217131066188356
0.0010784870114223695
0.0010369267862857035
0.000996968213998663
0.0009585495549376858
0.0009216114512381271
0.0008860968346992506
0.0008519508382735161
0.0008191207109983951
0.0007875557362343561
0.000757207153078624
0.0007280280808296751
0.0006999734463822273
0.000672999914437815
0.0006470658204202973
0.000622131105990457
0.0005981572570579552
0.0005751072441928266
0.0005529454653431726
0.0005316376907686722
0.0005111510101038712
0.0004914537814680439
0.0004725155825422385
0.0004543071635367196
0.0004368004019756303
0.00041996825922807537
0.0004037847387179752
0.00038822484574746864
0.000373264548871404
0.0003588807427627314
0.0003450512125110722
0.00033175459929894827
0.00031897036740236116
0.00030667877246449906
0.00029486083099329336
0.00028349829103548963
0.0002725736039817843
0.0002620698974593076
0.0002519709492694341
0.00024226116233050105
0.00023292554058671085
0.00022394966584582068
0.00021531967550979456
0.00020702224116398246
0.00019904454799161983
0.0001913742749818817
0.0001839995759008063
0.00017690906099565552
0.00017009177940448836
0.00016353720224369502
0.00015723520634729167
0.00015117605863298604
0.00014535040107069527
0.00013974923623037293
0.00013436391338677672
0.0001291861151597235
0.00012420784466916755
0.00011942141318526729
0.00011481942825437294
0.00011039478228253808
0.00010614064155901145
0.0001020504357026523
9.811784751502961e-05
9.433680322452288e-05
9.07014631063634e-05
8.720621246405432e-05
8.384565295836376e-05
8.06145942704182e-05
7.750804608602226e-05
7.45212103888556e-05
7.16494740506235e-05
6.888840170673525e-05
6.623372890648624e-05
6.36813555272088e-05
6.122733944212446e-05
5.8867890432272306e-05
5.659936433295092e-05
5.4418257405766915e-05
5.232120092748597e-05
5.030495598744348e-05
4.8366408485379654e-05
4.650256432203307e-05
4.4710544775062853e-05
4.2987582053132594e-05
4.1331015021312185e-05
3.9738285091188996e-05
3.8206932269354095e-05
3.673459135812327e-05
3.531898830268317e-05
3.3957936678981286e-05
3.2649334316941296e-05
3.139116005380407e-05
3.0181470612568156e-05
2.901839760070956e-05
2.790014462456529e-05
2.6824984514883835e-05
2.579125665930827e-05
2.479736443763488e-05
2.3841772755898297e-05
2.2923005675484053e-05
2.2039644133597908e-05
2.119032375156838e-05
2.0373732727611678e-05
1.958860981078675e-05
1.8833742353025982e-05
1.8107964436237772e-05
1.7410155071560296e-05
1.673923646802122e-05
1.6094172367903756e-05
1.547396644626215e-05
1.4877660772109866e-05
1.430433432890405e-05
1.3753101592043518e-05
1.3223111161177204e-05
1.271354444522436e-05
1.2223614398052364e-05
1.1752564302897218e-05
1.1299666603598998e-05
1.0864221780898391e-05
1.0445557272020895e-05
1.0043026431896353e-05
9.656007534414035e-06
9.283902812157043e-06
8.926137533144259e-06
8.582159113149512e-06
8.251436262223122e-06
7.933458164113636e-06
7.627733687293064e-06
7.33379062640477e-06
7.051174972921215e-06
6.779450213931473e-06
6.518196657925287e-06
6.267010786575047e-06
6.025504631492694e-06
5.793305174999138e-06
5.570053773989272e-06
5.355405605998847e-06
5.149029136614468e-06
4.9506056074123485e-06
4.759828543619497e-06
4.576403280761419e-06
4.4000465095373554e-06
4.230485838242033e-06
4.0674593720484435e-06
3.91071530849992e-06
3.760011548592869e-06
3.615115322845847e-06
3.4758028317824277e-06
3.3418589002573495e-06
3.2130766451173086e-06
3.0892571556569434e-06
2.9702091863957054e-06
2.8557488616903765e-06
2.7456993917342825e-06
2.6398907994953916e-06
2.538159658179872e-06
2.4403488388159694e-06
2.3463072675564664e-06
2.2558896923420854e-06
2.1689564585528806e-06
2.0853732933059644e-06
2.0050110980633565e-06
1.9277457492352228e-06
1.8534579064655777e-06
1.7820328283025376e-06
1.7133601949808364e-06
1.6473339380218593e-06
1.5838520764096164e-06
1.5228165590753771e-06
1.464133113451991e-06
1.40771109986799e-06
1.3534633715480287e-06
1.3013061400118565e-06
1.2511588456580528e-06
1.202944033337219e-06
1.156587232716991e-06
1.1120168432603495e-06
1.0691640236341087e-06
1.0279625853816399e-06
9.883488906893815e-07
9.502617540972218e-07
9.136423479925375e-07
8.784341117501692e-07
8.445826643692572e-07
8.120357204815001e-07
7.807430095922277e-07
7.506561984347445e-07
7.217288163193481e-07
6.939161833544208e-07
6.671753414368786e-07
6.41464987902722e-07
6.167454117308101e-07
5.929784322084336e-07
5.701273399586136e-07
5.481568402408807e-07
5.270329984367319e-07
5.06723187636041e-07
4.871960382424293e-07
4.684213895220894e-07
4.5037024301790184e-07
4.33014717761703e-07
4.16328007208233e-07
4.002843378327513e-07
3.848589293226123e-07
3.7002795630127346e-07
3.5576851152987916e-07
3.4205857052566243e-07
3.288769575438337e-07
3.162033128703886e-07
3.040180613761368e-07
2.9230238228100927e-07
2.8103818008466216e-07
2.7020805661710233e-07
2.5979528416658235e-07
2.4978377964218444e-07
2.4015807973288856e-07
2.3090331702399554e-07
2.2200519703291648e-07
2.1344997613110857e-07
2.0522444031625843e-07
1.9731588480219806e-07
1.897120943962158e-07
1.8240132463183187e-07
1.7537228362828715e-07
1.6861411465060232e-07
1.6211637934017274e-07
1.5586904159212865e-07
1.498624520541246e-07
1.4408733322270431e-07
1.3853476511318796e-07
1.331961714828652e-07
1.2806330658368513e-07
1.2312824242717036e-07
1.1838335653864165e-07
1.1382132018416605e-07
1.0943508705079577e-07
1.0521788236287128e-07
1.0116319241884456e-07
9.726475452980349e-08
9.351654734641305e-08
8.991278155889296e-08
8.64478909551478e-08
8.311652382328898e-08
7.991353468584456e-08
7.683397635167653e-08
7.387309227562576e-08
7.102630921117364e-08
6.828923014717936e-08
6.565762751588525e-08
6.312743666418616e-08
6.06947495745769e-08
5.835580882951291e-08
5.6107001807907785e-08
5.394485510481166e-08
5.1866029167092425e-08
4.986731313500184e-08
4.7945619882884734e-08
4.6097981250896086e-08
4.432154346061449e-08
4.2613562707360456e-08
4.097140092171029e-08
3.939252169562097e-08
3.7874486364041686e-08
3.6414950238919145e-08
3.501165898714879e-08
3.366244514904944e-08
3.236522479040909e-08
3.1117994283806455e-08
2.991882721375856e-08
2.8765871401598003e-08
2.7657346044208938e-08
2.6591538963808025e-08
2.5566803963398962e-08
2.4581558284015407e-08
2.3634280160019428e-08
2.2723506468796883e-08
2.1847830470759422e-08
2.100589963668148e-08
2.0196413558541738e-08
1.9418121941025346e-08
1.8669822670309218e-08
1.7950359957501043e-08
1.7258622553235455e-08
1.65935420314299e-08
1.5954091138984373e-08
1.5339282209192226e-08
1.4748165636183706e-08
1.4179828408244494e-08
1.3633392697563574e-08
1.3108014504384369e-08
1.2602882353487569e-08
1.2117216040820382e-08
1.1650265428291116e-08
1.1201309285195922e-08

Fully connected

20.1.5. Verify the Predictions#

Obtained a predicted value from our model and compare to origional.

pred = np.maximum(x.dot(w1),0).dot(w2)

print (pred, "\n", y)

[[0.        ]
 [0.99992661]
 [1.00007337]
 [0.        ]] 
 [[0]
 [1]
 [1]
 [0]]

array([[0],
       [1],
       [1],
       [0]])

#We can see that the weights have been updated. 
w1

array([[-0.20401151,  1.01377406],
       [-0.10186284,  1.01392285],
       [ 1.07856887,  0.01873049]])

w2

array([[0.49346731],
       [0.98634069]])

# Relu just removes the negative numbers.  
h_relu

array([[0.        , 0.        ],
       [0.        , 1.01377258],
       [0.        , 1.01392433],
       [0.        , 0.        ]])