Skip to main content

Deep Learning - Intro to Neural Networks

The next section of the course I'm taking covers some theoretical topics (perceptrons, activation and cost functions, gradient descent, and back propagation) before diving into the manual creation of a Neural Network using Python. Here is the code I wrote while following this section of the course, without using Tensorflow or any other machine learning specific libraries. Additionally, here are some notes I took during the lectures:

Introduction to Neural Networks

  • A “perceptron” is an artificial attempt to mimic biological neurons
  • biological neurons have a body that accepts input through many Dendrites and pushes output through a single Axon, which connects to another neuron
  • artificial neurons (perceptrons) have a similar structure:
    • only 2 inputs allowed
    • each input has a feature value
    • each input is multiplied by a “weight”, the value of which is typically random
    • the result of input*weight are passed to an activation function in the body
      • e.g. if sum of inputs*weights is pos, output 1, else output 0
      • However, if both inputs = 0, then the output will be neither pos nor neg.
      • Add a “Bias” input, e.g. 1, to account for this.
      • Input to activation function looks like this mathematically:
  • Hidden layers: in between input layer and output layer
  • 3 or more hidden layers => a “deep network”
  • z = wx + b

Activation Functions:
    • sigmoid function:
      • plug z in for x
    • hyperbolic tangent: tanh(z)
      • note lowest output is -1, not 0
    • Rectified Linear Unit (ReLU): max(0, z)
  • ReLU and tanh() tend to have best performance, so focus on those

Cost Functions
  • aka “loss”
  • used to evaluate performance of a neuron
  • use cost functions to measure how far off we are from expected value
  • variables:
    • y to represent true value
    • a to represent neuron’s prediction
    • z = w*x + b
    • pass z into activation function sigma(z) = a
  • Quadratic Cost
    • larger errors are more prominent due to squaring
    • can cause slowdown in learning speed
  • Cross Entropy
    • )
    • allows for faster learning than quadratic
    • the larger the difference, the faster the neuron can learn

Gradient Descent
  • optimization algorithm for minding minimum of a function
  • Use gradient descent to figure out the best parameters for minimizing the cost (e.g. the best values for the weights of neuron inputs)
  • to find min, take steps proportional to the negative of the gradient
    • Want to choose a weight W that minimizes cost C
    • finding min is difficult for multiple parameters, so use deep learning library in practice
  • a gradient is the derivative of the function at a point

Backpropagation
  • use backpropagation to quickly adjust optimal parameters across entire network
  • used to calculate error contribution of each neuron after a batch of data is processed
    • relies on chain rule to go through network and calculate these errors
  • calculates error at output, then distributes back through network’s layers
  • requires known desired output for each input (supervised learning)

Tensorflow Playground
  • “loss” == “cost”
  • epoch: how many times the data was fed through the network
  • dense network: every neuron is connected to every other neuron in the next layer
  • The higher the learning rate, the faster the model will learn, but the less accurate it will be.
  • Tensorflow Playground can support up to 6 hidden layers with 8 neurons per layer
  • the more layers and neurons a model has, the longer each epoch takes

---------------------------

It's fascinating to watch as the model becomes more accurate as more epochs are completed. Here is an example of a simple data set being fed through a network with 3 hidden layers consisting of a total of 8 neurons. The accuracy of the model can be visualized by the rightmost diagrams; the closer the background color aligns with those of the data points, the more accurate the model is. This accuracy is quantified by the test loss and training loss scores, with 0 being a perfect model and 1 being the worst model. Notice the number of epochs in each representation (see upper left corner), and know that this environment ran this particular model at roughly 50 epochs per second. So in only a couple of seconds, the model improved tremendously.





However, when given a more complex data set, the same model performs significantly worse. Notice the number of epochs is 1,270 compared to 176 in the more evolved previous test. This run has a much lower accuracy than that one, since the background color in the rightmost diagram doesn't line up with the data points at all. 


The Tensorflow Playground can support up to 6 hidden layers with 8 neurons per layer, so next I maxed that out to see the results. Note that the more layers and neurons a model has, the longer each epoch takes. With this model, the environment ran about 12 epochs per second -- much slower than the first model. I also changed the activation function to ReLU. 




Even with all of these layers and neurons and running for several minutes, the model still struggles to find a good fit for the more complex spiral data set. The loss here is 0.037, much worse than the simple data set model previously shown. 

-----------------------

Manual Creation of Neural Network - Part 1
  • reviewed classes and objects in Python, emphasizing ‘super’ keyword
  • JN

Manual Creation of Neural Network - Part 2: Operation
  • Operation Class will have input and output nodes, global default graph variable, and a compute method


Manual Creation of Neural Network - Part 3

  • placeholder: an empty node that needs a value to be provided before it can compute an output
  • variable: changeable parameter of Graph
  • Graph: global variable connecting variables and placeholders to operations


Overall, this was an interesting section with many topics that will be essential to understanding how TensorFlow and other deep learning libraries work behind the scenes. In the next section of the course, I'll be learning the basics of TensorFlow, Google's Python library for Machine/Deep Learning.

Comments

Popular posts from this blog

Reading from and Modifying a DynamoDB Table using API Gateway

In many IoT (Internet of Things) projects, it is necessary to read from and write to a database. AWS has an application called DynamoDB that is a NoSQL database, exactly what we need. While it is very simple to add/modify/delete entries in this database via the user interface, it's not so obvious how to do so via HTTP API calls. So, in this post I'll continue the AWS portion of my journey into IoT to learn how to do just that. I'll be following  this  tutorial. The first thing I encountered is I wasn't sure if I was supposed to follow the steps under "Defining APIs", or if that was just knowledge for later. Turns out, that was knowledge for later. If I had any prior experience with APIs I probably would have realized this. Basically, the "API definition"s are just a high-level plan for what you want your API requests to look like in the end. In my opinion this information should go in the "Creating the APIs" section. Moving through "Cr...

Arduino Project: Sound Alert via Twitter

For my second IoT project using Arduino MKR1000, I'll employ a sound sensor to send out alerts via Twitter when a loud sound is detected. This could have many uses, ranging from informing the inhabitants of a house that there is someone at the door, informing a car owner that their alarm is going off, or residents that a gunshot has gone off in their neighborhood. See my previous posts to learn about the setup process for the Arduino MKR1000, which is wifi-enabled without any additional extensions. I will be using the Arduino IDE to compile the code. For hardware, I'll be using the sound sensor I purchased here , as well as my Arduino, three solder-free wires, a generic breadboard, and a Kindle power cable (this probably isn't the safest way to power Arduino, but it's what I had). I'll be following this  tutorial to set up the sound detector, as well as some code from here  to utilize Twitter. The sound setup tutorial was very simple and one of the clearest wiring g...

Deep Learning - Recurrent Neural Networks (RNNs)

The latest Deep Learning topic I've been exploring is the Recurrent Neural Network (RNN). It has many use cases, including: time series extrapolation (i.e. given historical data, predict future output); object trajectory; and natural language processing (NLP). The code I wrote during this section of the course I'm taking can be found here . For my summary of RNNs, click here . Here are the notes I took while watching the lectures: RNN Theory Examples of Sequences Time Series Data (e.g. stock prices, sales rates) Sentences (i.e. natural language) Audio/Music Car Trajectories (where a car will go in a situation) [1,2,3] shifted one time step into the future would be [2,3,4] will train a network to predict y-values for this A normal neuron has input, activation function, then output A Recurrent Neuron sends output back to itself this process is called “unrolling” the neuron note that the input of each neuron is a summation from both ...