The next section of the course I'm taking covers some theoretical topics (perceptrons, activation and cost functions, gradient descent, and back propagation) before diving into the manual creation of a Neural Network using Python. Here is the code I wrote while following this section of the course, without using Tensorflow or any other machine learning specific libraries. Additionally, here are some notes I took during the lectures:
Introduction to Neural Networks
- A “perceptron” is an artificial attempt to mimic biological neurons
- biological neurons have a body that accepts input through many Dendrites and pushes output through a single Axon, which connects to another neuron
- artificial neurons (perceptrons) have a similar structure:
- only 2 inputs allowed
- each input has a feature value
- each input is multiplied by a “weight”, the value of which is typically random
- the result of input*weight are passed to an activation function in the body
- e.g. if sum of inputs*weights is pos, output 1, else output 0
- However, if both inputs = 0, then the output will be neither pos nor neg.
- Add a “Bias” input, e.g. 1, to account for this.
- Input to activation function looks like this mathematically:
- Hidden layers: in between input layer and output layer
- 3 or more hidden layers => a “deep network”
- z = wx + b
Activation Functions:
- sigmoid function:
- plug z in for x
- hyperbolic tangent: tanh(z)
- note lowest output is -1, not 0
- Rectified Linear Unit (ReLU): max(0, z)
- ReLU and tanh() tend to have best performance, so focus on those
Cost Functions
- aka “loss”
- used to evaluate performance of a neuron
- use cost functions to measure how far off we are from expected value
- variables:
- y to represent true value
- a to represent neuron’s prediction
- z = w*x + b
- pass z into activation function sigma(z) = a
- Quadratic Cost
- larger errors are more prominent due to squaring
- can cause slowdown in learning speed
- Cross Entropy
)
- allows for faster learning than quadratic
- the larger the difference, the faster the neuron can learn
Gradient Descent
- optimization algorithm for minding minimum of a function
- Use gradient descent to figure out the best parameters for minimizing the cost (e.g. the best values for the weights of neuron inputs)
- to find min, take steps proportional to the negative of the gradient
- Want to choose a weight W that minimizes cost C
- finding min is difficult for multiple parameters, so use deep learning library in practice
- a gradient is the derivative of the function at a point
Backpropagation
- use backpropagation to quickly adjust optimal parameters across entire network
- used to calculate error contribution of each neuron after a batch of data is processed
- relies on chain rule to go through network and calculate these errors
- calculates error at output, then distributes back through network’s layers
- requires known desired output for each input (supervised learning)
Tensorflow Playground
- “loss” == “cost”
- epoch: how many times the data was fed through the network
- dense network: every neuron is connected to every other neuron in the next layer
- The higher the learning rate, the faster the model will learn, but the less accurate it will be.
- Tensorflow Playground can support up to 6 hidden layers with 8 neurons per layer
- the more layers and neurons a model has, the longer each epoch takes
---------------------------
It's fascinating to watch as the model becomes more accurate as more epochs are completed. Here is an example of a simple data set being fed through a network with 3 hidden layers consisting of a total of 8 neurons. The accuracy of the model can be visualized by the rightmost diagrams; the closer the background color aligns with those of the data points, the more accurate the model is. This accuracy is quantified by the test loss and training loss scores, with 0 being a perfect model and 1 being the worst model. Notice the number of epochs in each representation (see upper left corner), and know that this environment ran this particular model at roughly 50 epochs per second. So in only a couple of seconds, the model improved tremendously.

-----------------------
It's fascinating to watch as the model becomes more accurate as more epochs are completed. Here is an example of a simple data set being fed through a network with 3 hidden layers consisting of a total of 8 neurons. The accuracy of the model can be visualized by the rightmost diagrams; the closer the background color aligns with those of the data points, the more accurate the model is. This accuracy is quantified by the test loss and training loss scores, with 0 being a perfect model and 1 being the worst model. Notice the number of epochs in each representation (see upper left corner), and know that this environment ran this particular model at roughly 50 epochs per second. So in only a couple of seconds, the model improved tremendously.
However, when given a more complex data set, the same model performs significantly worse. Notice the number of epochs is 1,270 compared to 176 in the more evolved previous test. This run has a much lower accuracy than that one, since the background color in the rightmost diagram doesn't line up with the data points at all.
The Tensorflow Playground can support up to 6 hidden layers with 8 neurons per layer, so next I maxed that out to see the results. Note that the more layers and neurons a model has, the longer each epoch takes. With this model, the environment ran about 12 epochs per second -- much slower than the first model. I also changed the activation function to ReLU.

Even with all of these layers and neurons and running for several minutes, the model still struggles to find a good fit for the more complex spiral data set. The loss here is 0.037, much worse than the simple data set model previously shown.
Manual Creation of Neural Network - Part 1
- reviewed classes and objects in Python, emphasizing ‘super’ keyword
- JN
Manual Creation of Neural Network - Part 2: Operation
- Operation Class will have input and output nodes, global default graph variable, and a compute method
Manual Creation of Neural Network - Part 3
- placeholder: an empty node that needs a value to be provided before it can compute an output
- variable: changeable parameter of Graph
- Graph: global variable connecting variables and placeholders to operations
Overall, this was an interesting section with many topics that will be essential to understanding how TensorFlow and other deep learning libraries work behind the scenes. In the next section of the course, I'll be learning the basics of TensorFlow, Google's Python library for Machine/Deep Learning.



Comments
Post a Comment