Skip to main content

Deep Learning - Convolutional Neural Networks (CNN's) using TensorFlow

In this next section of the machine learning course I'm taking, I'll be introduced to Convolutional Neural Networks (CNNs -- not to be mistaken for the news company). I'm especially interested in this topic because CNNs are the primary tool used in image-based learning. From self-driving cars making sense of their surroundings, to better predicting if a tumor is cancerous, the possibilities in this field are endless!

In this post, I'll document my theoretical learnings from the lectures I watch, then note things from the code-along example CNN in which I build a model to estimate handwritten numbers using the well-known MNIST data set. Last, I'll present my inquiries and results from a CNN project that attempts recognize a variety of objects in the CIFAR dataset.

My summary of CNNs can be found here.

My code for this section of the course can be found here.

Here are the notes I took during the lectures and code-alongs:

Convolutional Neural Networks
  • Review
    • Within a neuron: z=wx+b, a = activation_function(z)
    • Activation functions: perceptrons (pos. or neg.), sigmoid, tanh, ReLu
    • connecting neurons creates a network
    • a neural net has input/output layers, hidden layers in between; more layers = more abstraction
    • measurements of error: cost/loss function, which can be quadratic or cross-entropy
    • Once we have error, minimize it by choose better weight and bias values; use gradient descent to find optimal values
  • New Theory Topics
    • Initialization of Weights Options
      • Xavier (Glorot) Initialization
        • Uniform / Normal
      • Draw Weights from a distribution with zero mean and a specific variance
        • W: initialization distribution for the neuron in question
        • nin : number of neurons feeding into the neuron in question
        • If mean of weights and x’s = 0, the 2nd eqn simplifies to the 3rd
    • Gradient Descent
      • Learning Rate: defines step size during gradient descent
        • smaller -> longer to train model
        • larger -> chance of overstepping minimum and never converging
      • Batch Size: allow us to use stochastic (randomized) gradient descent
        • smaller batch size -> less representative of overall data
        • larger batch size -> longer time needed to train
      • Second-Order Behavior of gradient descent allows us to adjust learning rate based on rate of descent
        • Error is large when descent is first starting training, nice to take large steps in beginning, smaller as it gets closer to the minimum
          • AdaGrad
          • RMSProp
          • Adam <- we’ll focus on this one, later (built into TF)
      • Unstable / Vanishing Gradients
        • As number of layers in a network is increased, the layers towards input are less affected by the error calculated at the output as you go backwards through the network
        • Initialization and normalization help mitigate these issues
        • Will discuss more when covering RNN’s
      • Underfitting
        • If getting large error on both training and evaluation data sets
      • Overfitting
        • Very low error on training data, high error on evaluation data
      • How to mitigate overfitting
        • L1/L2 Regularization
          • adds penalty for larger weights in the model so you don’t end up with one neuron/feature in training set with a large weight
        • Dropout
          • randomly remove neurons during training so the network doesn’t rely too much on any particular neuron
        • Expanding Data
          • add noise, tilt images, adding white noise to sound data, etc.  to training data so the model doesn’t overfit
  • MNIST Data Overview
    • TF has 55k training images, 10k test images, 5k validation (holdout) images ready to download
    • Dataset contains handwritten digits (0-9)
    • A single-digit image can be represented as an array on a 2-D plane with 28x28 pixels
      • array can be flattened into a 1D vector of 784 number
      • flattening removes some information (e.g. relationship of a pixel to its neighbors), come back to this later
    • Can think of the entire group of 55k images as a tensor (n-dimensional array) with dimensions 784x55,000
    • Use One-Hot encoding for labels
      • have a single array for the digit represented by each image instead of labels like “one”, “two”...
      • label is represented based on index position in the label array
      • corresponding label will be a 1 at the index location and a 0 elsewhere
        • ex. 4 would have this label array: [0,0,0,0,1,0,0,0,0,0]
        • index starts at 0 and goes to 9
      • Therefore, labels for training data end up being a 2D array 10 by 55k
  • MNIST “Basic” Approach
    • We’ll use the Softmax Regression Approach before using a CNN
      • returns list of values between 0 and 1 that add up to = 1
      • can think of these as probabilities
      • use Softmax as the activation function
      • exp(z) means e^z
  • Convolutional Neural Network (CNN) Theory
    • tensors make it easier to feed in sets of images to the model (I,H,W,C)
      • I: images
      • H: height of image (in pixels)
      • W: width of image (in pixels)
      • C: color channels: 1-grayscale, 3- RGB
    • Recall: a densely connected network has every neuron connected to every other neuron
    • A CNN has each unit connected to a smaller number of nearby units in the next layer
    • Why use a CNN instead of a DNN?
      • most images are 256x256 pixels or more
      • too many parameters, unscalable to new images
    • In image processing, pixels near to each other are more correlated
      • CNN’s are great for this
    • Each CNN layer looks @ increasingly larger part of image
    • CNN helps w/ regularization, limits search of weights to size of the convolution
    • Convolutional layers are only connected to pixels in their respective fields
    • Issue: there may not be an input available for edge neurons
      • Fix by adding “padding” of 0’s around the image
    • ^^ this is an example of edge detection (dark vs light pixel)
    • Neurons that aren’t connected -> add 0 edges
    • Each filter detects a different feature
    • For 2-D grayscale:
    • For 2-D colored:
    • Filters are commonly visualized with grids. E.g. :
    • Pooling (AKA subsampling) layers subsample input image, reducing memory use and computer load; also reduces number of parameters
      • create a 2x2 pool of pixels & evaluate the max value
      • only max val makes it to next layer
      • next, move over by a stride and repeat.
      • pooling removes some information
      • small pooling kernel of 2x2 w/ stride of 2 removes 75% of input data!
      • an image typically has too much information to not have a pooling layer; too computationally expensive
    • Dropout is a form of regularization to help prevent overfitting
      • during training, units & their connections are randomly dropped
“Logit is nowadays used in ML community for any non-normalised probability distribution (basically anything that gets mapped to a probability distribution by a parameter-less transformation, like sigmoid function for a binary variable or softmax for multinomial one). It is not a strict mathematical term, but gained enough popularity to be included in TF documentation.” - source



Code-Along

My code for this section can be found in the link posted above

While testing the model in the session, it took about 30 minutes for the process to complete on my laptop (2.7 GHz i5 MPB). To decrease the amount of time, simply decrease the number of steps from 5000 to perhaps 2000. By step 4900, my model had an accuracy of 98.93% (it had an accuracy of 11% after the 1st step, and 94.25% by the 100th step).

Exercise

My task was to create a CNN that can correctly label/identify images from the widely-used CIFAR-10 data set. This set contains 50,000 training images and 10,000 test images, and can be found here. There are 10 possible image labels: airplane, automobile, bird,  cat, deer, dog, frog, horse, ship, and truck. My code can be found here.

From here on,  I'll document any findings or issues I have, as well as my results.

I found out that a "pickling" Python object is used "for serializing and de-serializing a Python object structure. 'Pickling' is the process whereby a Python object hierarchy is converted into a byte stream, and 'unpickling' is the inverse operation, whereby a byte stream is converted back into an object hierarchy." [Source.]

When asked for the directory path of the cifar-10-batches-py, I initially inserted the wrong path format (I used the ~ notation). The program actually requires a /Users/... format. I found an easy way to copy a directory's file path on Mac OS X here.

While using the example I coded during the lectures, I remembered that I had never figured out what a dimension of -1 meant when reshaping an object. I found this in the TensorFlow documentation:

# tensor 't' is [[[1, 1, 1],
#                 [2, 2, 2]],
#                [[3, 3, 3],
#                 [4, 4, 4]],
#                [[5, 5, 5],
#                 [6, 6, 6]]]
# tensor 't' has shape [3, 2, 3]
# pass '[-1]' to flatten 't'
reshape(t, [-1]) ==> [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6]
# -1 can also be used to infer the shape
# -1 is inferred to be 9:
reshape(t, [2, -1]) ==> [[1, 1, 1, 2, 2, 2, 3, 3, 3],
                         [4, 4, 4, 5, 5, 5, 6, 6, 6]]
I found this example extremely helpful and it cleared up my confusion. A dimension of -1 is either used to infer the dimension (note that when one dimension isn't given, as in the top example above, that dimension is assumed to be 0), or if given as the only parameter it reshapes the objects into a 1-D array.

One aspect of the project I'm unsure about is that when I use imshow, my images appear much less intelligible than in my instructor's example. My instructor's images look like this:



While mine look like this:


I'm not sure why this is, but I'll post here if I find out anything. 

In general, I found myself relying heavily on the previous CNN example using the MNIST data set to see the steps to take, as well as the documentation to remind me what certain TensorFlow methods do. I suppose this is to be expected, considering how new I am to the process. My goal is to gain a more solid and comprehensive understanding of CNNs as I gain more experience in the field.

The training and testing process took quite a while on my laptop, around 30 minutes. In the end, my model achieved an accuracy of around 69%. The highest printed accuracy was 70.13% at step 4200, but the final printed accuracy was 67.02% at step 4900.





















Comments

Popular posts from this blog

Reading from and Modifying a DynamoDB Table using API Gateway

In many IoT (Internet of Things) projects, it is necessary to read from and write to a database. AWS has an application called DynamoDB that is a NoSQL database, exactly what we need. While it is very simple to add/modify/delete entries in this database via the user interface, it's not so obvious how to do so via HTTP API calls. So, in this post I'll continue the AWS portion of my journey into IoT to learn how to do just that. I'll be following  this  tutorial. The first thing I encountered is I wasn't sure if I was supposed to follow the steps under "Defining APIs", or if that was just knowledge for later. Turns out, that was knowledge for later. If I had any prior experience with APIs I probably would have realized this. Basically, the "API definition"s are just a high-level plan for what you want your API requests to look like in the end. In my opinion this information should go in the "Creating the APIs" section. Moving through "Cr...

Arduino Project: Sound Alert via Twitter

For my second IoT project using Arduino MKR1000, I'll employ a sound sensor to send out alerts via Twitter when a loud sound is detected. This could have many uses, ranging from informing the inhabitants of a house that there is someone at the door, informing a car owner that their alarm is going off, or residents that a gunshot has gone off in their neighborhood. See my previous posts to learn about the setup process for the Arduino MKR1000, which is wifi-enabled without any additional extensions. I will be using the Arduino IDE to compile the code. For hardware, I'll be using the sound sensor I purchased here , as well as my Arduino, three solder-free wires, a generic breadboard, and a Kindle power cable (this probably isn't the safest way to power Arduino, but it's what I had). I'll be following this  tutorial to set up the sound detector, as well as some code from here  to utilize Twitter. The sound setup tutorial was very simple and one of the clearest wiring g...

Deep Learning - Recurrent Neural Networks (RNNs)

The latest Deep Learning topic I've been exploring is the Recurrent Neural Network (RNN). It has many use cases, including: time series extrapolation (i.e. given historical data, predict future output); object trajectory; and natural language processing (NLP). The code I wrote during this section of the course I'm taking can be found here . For my summary of RNNs, click here . Here are the notes I took while watching the lectures: RNN Theory Examples of Sequences Time Series Data (e.g. stock prices, sales rates) Sentences (i.e. natural language) Audio/Music Car Trajectories (where a car will go in a situation) [1,2,3] shifted one time step into the future would be [2,3,4] will train a network to predict y-values for this A normal neuron has input, activation function, then output A Recurrent Neuron sends output back to itself this process is called “unrolling” the neuron note that the input of each neuron is a summation from both ...