Neural Networks

63. Neural Networks#

This was adopted from the PyTorch Tutorials.
http://pytorch.org/tutorials/beginner/pytorch_with_examples.html

63.1. Neural Networks#

Neural networks are the foundation of deep learning, which has revolutionized the

In the mathematical theory of artificial neural networks, the universal approximation theorem states[1] that a feed-forward network with a single hidden layer containing a finite number of neurons (i.e., a multilayer perceptron), can approximate continuous functions on compact subsets of Rn, under mild assumptions on the activation function.

63.1.1. Generate Fake Data#

D_in is the number of dimensions of an input varaible.
D_out is the number of dimentions of an output variable.
Here we are learning some special “fake” data that represents the xor problem.
Here, the dv is 1 if either the first or second variable is

# -*- coding: utf-8 -*-
import numpy as np

#This is our independent and dependent variables. 
x = np.array([ [0,0,0],[1,0,0],[0,1,0],[0,0,0] ])
y = np.array([[0,1,1,0]]).T
print("Input data:\n",x,"\n Output data:\n",y)

Input data:
 [[0 0 0]
 [1 0 0]
 [0 1 0]
 [0 0 0]] 
 Output data:
 [[0]
 [1]
 [1]
 [0]]

63.1.2. A Simple Neural Network#

Here we are going to build a neural network with 2 hidden layers.

np.random.seed(seed=83832)
#D_in is the number of input variables. 
#H is the hidden dimension.
#D_out is the number of dimensions for the output. 
D_in, H, D_out = 3, 2, 1

# Randomly initialize weights og out 2 hidden layer network.
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)
bias = np.random.randn(H, 1)

63.1.3. Learn the Appropriate Weights via Backpropogation#

Learning rate adjust how quickly the model will adjust parameters.

# -*- coding: utf-8 -*-

learning_rate = .01
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)

    #A relu is just the activation.
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

10.6579261591
9.10203339893
7.92822558061
7.01603070961
6.28979819918
5.69984738569
5.21233053023
4.80346624793
4.456102755
4.15758768903
3.89840273398
3.67126267684
3.47050562961
3.29167096682
3.13120131373
2.98622833978
2.8544162991
2.73384607859
2.62292812419
2.52033626007
2.42495682843
2.33584920317
2.25221484357
2.17337282724
2.09874034592
2.02781703626
1.96017229769
1.89543495408
1.83328476643
1.77344541638
1.71567866423
1.65977944954
1.60557175094
1.55290505986
1.50165135204
1.45170246386
1.40296779892
1.35537230457
1.3088546702
1.26336570846
1.21886688836
1.17532899574
1.13273090181
1.09105842491
1.05030327423
1.01046206727
0.971535415306
0.933527072858
0.896443149097
0.860291379845
0.825080459944
0.790819436113
0.757517160723
0.725181806896
0.693820445158
0.663438681518
0.634040356373
0.605627303094
0.578199164525
0.551753265011
0.526284534957
0.501785484373
0.478246221337
0.455654510907
0.433995869718
0.413253691241
0.393409396657
0.374442606233
0.356331326268
0.339052146831
0.322580445843
0.30689059538
0.291956166492
0.27775012929
0.264245045477
0.251413251035
0.239227027168
0.227658758121
0.216681074855
0.206266984003
0.196389981824
0.18702415323
0.178144256176
0.169725791964
0.161745062156
0.154179212936
0.147006267869
0.14020515004
0.133755694633
0.12763865297
0.121835689045
0.116329369558
0.111103148388
0.106141346409
0.101429127469
0.0969524713096
0.0926981441057
0.0886536672482
0.0848072849255
0.0811479309802
0.0776651954621
0.0743492912332
0.0711910209273
0.0681817445139
0.0653133476691
0.0625782111165
0.0599691810633
0.0574795408223
0.0551029836872
0.0528335870985
0.0506657881215
0.0485943602377
0.0466143914385
0.0447212635969
0.042910633084
0.0411784125909
0.0395207541101
0.0379340330255
0.0364148332613
0.0349599334326
0.0335662939467
0.0322310449976
0.0309514754026
0.0297250222266
0.0285492611449
0.0274218974944
0.0263407579669
0.0253037828991
0.0243090191189
0.0233546133048
0.0224388058244
0.0215599250132
0.020716381864
0.0199066650928
0.0191293365551
0.0183830269842
0.0176664320279
0.0169783085599
0.0163174712444
0.015682789336
0.0150731836941
0.0144876239984
0.0139251261468
0.0133847498242
0.0128655962277
0.0123668059377
0.0118875569212
0.0114270626611
0.0109845703988
0.0105593594834
0.0101507398196
0.00975805040627
0.00938065796027
0.00901795561851
0.00866936171332
0.00833431861585
0.00801229164266
0.00770276802156
0.00740525591235
0.00711928347915
0.00684439801074
0.0065801650859
0.00632616778098
0.00608200591689
0.0058472953433
0.00562166725752
0.00540476755634
0.00519625621854
0.00499580671658
0.0048031054556
0.00461785123834
0.00443975475448
0.00426853809302
0.00410393427663
0.00394568681655
0.00379354928727
0.0036472849197
0.00350666621201
0.00337147455735
0.00324149988737
0.00311654033101
0.00299640188771
0.00288089811428
0.00276984982498
0.00266308480403
0.00256043752998
0.00246174891152
0.00236686603409
0.00227564191686
0.0021879352796
0.002103610319
0.00202253649407
0.00194458832016
0.00186964517126
0.00179759109024
0.00172831460666
0.00166170856184
0.00159766994094
0.00153609971162
0.00147690266916
0.00141998728767
0.00136526557721
0.00131265294649
0.00126206807099
0.0012134327663
0.00116667186639
0.00112171310662
0.00107848701142
0.00103692678629
0.000996968213999
0.000958549554938
0.000921611451238
0.000886096834699
0.000851950838274
0.000819120710998
0.000787555736234
0.000757207153079
0.00072802808083
0.000699973446382
0.000672999914438
0.00064706582042
0.00062213110599
0.000598157257058
0.000575107244193
0.000552945465343
0.000531637690769
0.000511151010104
0.000491453781468
0.000472515582542
0.000454307163537
0.000436800401976
0.000419968259228
0.000403784738718
0.000388224845747
0.000373264548871
0.000358880742763
0.000345051212511
0.000331754599299
0.000318970367402
0.000306678772464
0.000294860830993
0.000283498291035
0.000272573603982
0.000262069897459
0.000251970949269
0.000242261162331
0.000232925540587
0.000223949665846
0.00021531967551
0.000207022241164
0.000199044547992
0.000191374274982
0.000183999575901
0.000176909060996
0.000170091779404
0.000163537202244
0.000157235206347
0.000151176058633
0.000145350401071
0.00013974923623
0.000134363913387
0.00012918611516
0.000124207844669
0.000119421413185
0.000114819428254
0.000110394782283
0.000106140641559
0.000102050435703
9.8117847515e-05
9.43368032245e-05
9.07014631064e-05
8.72062124641e-05
8.38456529584e-05
8.06145942704e-05
7.7508046086e-05
7.45212103889e-05
7.16494740506e-05
6.88884017067e-05
6.62337289065e-05
6.36813555272e-05
6.12273394421e-05
5.88678904323e-05
5.6599364333e-05
5.44182574058e-05
5.23212009275e-05
5.03049559874e-05
4.83664084854e-05
4.6502564322e-05
4.47105447751e-05
4.29875820531e-05
4.13310150213e-05
3.97382850912e-05
3.82069322694e-05
3.67345913581e-05
3.53189883027e-05
3.3957936679e-05
3.26493343169e-05
3.13911600538e-05
3.01814706126e-05
2.90183976007e-05
2.79001446246e-05
2.68249845149e-05
2.57912566593e-05
2.47973644376e-05
2.38417727559e-05
2.29230056755e-05
2.20396441336e-05
2.11903237516e-05
2.03737327276e-05
1.95886098108e-05
1.8833742353e-05
1.81079644362e-05
1.74101550716e-05
1.6739236468e-05
1.60941723679e-05
1.54739664463e-05
1.48776607721e-05
1.43043343289e-05
1.3753101592e-05
1.32231111612e-05
1.27135444452e-05
1.22236143981e-05
1.17525643029e-05
1.12996666036e-05
1.08642217809e-05
1.0445557272e-05
1.00430264319e-05
9.65600753441e-06
9.28390281216e-06
8.92613753314e-06
8.58215911315e-06
8.25143626222e-06
7.93345816411e-06
7.62773368729e-06
7.3337906264e-06
7.05117497292e-06
6.77945021393e-06
6.51819665793e-06
6.26701078658e-06
6.02550463149e-06
5.793305175e-06
5.57005377399e-06
5.355405606e-06
5.14902913661e-06
4.95060560741e-06
4.75982854362e-06
4.57640328076e-06
4.40004650954e-06
4.23048583824e-06
4.06745937205e-06
3.9107153085e-06
3.76001154859e-06
3.61511532285e-06
3.47580283178e-06
3.34185890026e-06
3.21307664512e-06
3.08925715566e-06
2.9702091864e-06
2.85574886169e-06
2.74569939173e-06
2.6398907995e-06
2.53815965818e-06
2.44034883882e-06
2.34630726756e-06
2.25588969234e-06
2.16895645855e-06
2.08537329331e-06
2.00501109806e-06
1.92774574924e-06
1.85345790647e-06
1.7820328283e-06
1.71336019498e-06
1.64733393802e-06
1.58385207641e-06
1.52281655908e-06
1.46413311345e-06
1.40771109987e-06
1.35346337155e-06
1.30130614001e-06
1.25115884566e-06
1.20294403334e-06
1.15658723272e-06
1.11201684326e-06
1.06916402363e-06
1.02796258538e-06
9.88348890689e-07
9.50261754097e-07
9.13642347993e-07
8.7843411175e-07
8.44582664369e-07
8.12035720482e-07
7.80743009592e-07
7.50656198435e-07
7.21728816319e-07
6.93916183354e-07
6.67175341437e-07
6.41464987903e-07
6.16745411731e-07
5.92978432208e-07
5.70127339959e-07
5.48156840241e-07
5.27032998437e-07
5.06723187636e-07
4.87196038242e-07
4.68421389522e-07
4.50370243018e-07
4.33014717762e-07
4.16328007208e-07
4.00284337833e-07
3.84858929323e-07
3.70027956301e-07
3.5576851153e-07
3.42058570526e-07
3.28876957544e-07
3.1620331287e-07
3.04018061376e-07
2.92302382281e-07
2.81038180085e-07
2.70208056617e-07
2.59795284167e-07
2.49783779642e-07
2.40158079733e-07
2.30903317024e-07
2.22005197033e-07
2.13449976131e-07
2.05224440316e-07
1.97315884802e-07
1.89712094396e-07
1.82401324632e-07
1.75372283628e-07
1.68614114651e-07
1.6211637934e-07
1.55869041592e-07
1.49862452054e-07
1.44087333223e-07
1.38534765113e-07
1.33196171483e-07
1.28063306584e-07
1.23128242427e-07
1.18383356539e-07
1.13821320184e-07
1.09435087051e-07
1.05217882363e-07
1.01163192419e-07
9.72647545298e-08
9.35165473464e-08
8.99127815589e-08
8.64478909551e-08
8.31165238233e-08
7.99135346858e-08
7.68339763517e-08
7.38730922756e-08
7.10263092112e-08
6.82892301472e-08
6.56576275159e-08
6.31274366642e-08
6.06947495746e-08
5.83558088295e-08
5.61070018079e-08
5.39448551048e-08
5.18660291671e-08
4.9867313135e-08
4.79456198829e-08
4.60979812509e-08
4.43215434606e-08
4.26135627074e-08
4.09714009217e-08
3.93925216956e-08
3.7874486364e-08
3.64149502389e-08
3.50116589871e-08
3.3662445149e-08
3.23652247904e-08
3.11179942838e-08
2.99188272138e-08
2.87658714016e-08
2.76573460442e-08
2.65915389638e-08
2.55668039634e-08
2.4581558284e-08
2.363428016e-08
2.27235064688e-08
2.18478304708e-08
2.10058996367e-08
2.01964135585e-08
1.9418121941e-08
1.86698226703e-08
1.79503599575e-08
1.72586225532e-08
1.65935420314e-08
1.5954091139e-08
1.53392822092e-08
1.47481656362e-08
1.41798284082e-08
1.36333926976e-08
1.31080145044e-08
1.26028823535e-08
1.21172160408e-08
1.16502654283e-08
1.12013092852e-08

#CFully connected

pred = np.maximum(x.dot(w1),0).dot(w2)

print (pred, "\n", y)

[[ 0.        ]
 [ 0.99992661]
 [ 1.00007337]
 [ 0.        ]] 
 [[0]
 [1]
 [1]
 [0]]

63.1.4. Hidden Layers are Often Viewed as Unknown#

Just a weighting matrix

#However
w1

array([[-0.20401151,  1.01377406],
       [-0.10186284,  1.01392285],
       [ 1.07856887,  0.01873049]])

w2

array([[ 0.49346731],
       [ 0.98634069]])

# Relu just removes the negative numbers.  
h_relu

array([[ 0.        ,  0.        ,  0.        ],
       [ 0.72108356,  0.        ,  0.        ],
       [ 0.72753913,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ]])