In this next section of the machine learning course I'm taking, I'll be introduced to Convolutional Neural Networks (CNNs -- not to be mistaken for the news company). I'm especially interested in this topic because CNNs are the primary tool used in image-based learning. From self-driving cars making sense of their surroundings, to better predicting if a tumor is cancerous, the possibilities in this field are endless!
In this post, I'll document my theoretical learnings from the lectures I watch, then note things from the code-along example CNN in which I build a model to estimate handwritten numbers using the well-known MNIST data set. Last, I'll present my inquiries and results from a CNN project that attempts recognize a variety of objects in the CIFAR dataset.
My summary of CNNs can be found here.
My code for this section of the course can be found here.
Here are the notes I took during the lectures and code-alongs:
Code-Along
My code for this section can be found in the link posted above
While testing the model in the session, it took about 30 minutes for the process to complete on my laptop (2.7 GHz i5 MPB). To decrease the amount of time, simply decrease the number of steps from 5000 to perhaps 2000. By step 4900, my model had an accuracy of 98.93% (it had an accuracy of 11% after the 1st step, and 94.25% by the 100th step).
Exercise
My task was to create a CNN that can correctly label/identify images from the widely-used CIFAR-10 data set. This set contains 50,000 training images and 10,000 test images, and can be found here. There are 10 possible image labels: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. My code can be found here.
From here on, I'll document any findings or issues I have, as well as my results.
I found out that a "pickling" Python object is used "for serializing and de-serializing a Python object structure. 'Pickling' is the process whereby a Python object hierarchy is converted into a byte stream, and 'unpickling' is the inverse operation, whereby a byte stream is converted back into an object hierarchy." [Source.]
When asked for the directory path of the cifar-10-batches-py, I initially inserted the wrong path format (I used the ~ notation). The program actually requires a /Users/... format. I found an easy way to copy a directory's file path on Mac OS X here.
While using the example I coded during the lectures, I remembered that I had never figured out what a dimension of -1 meant when reshaping an object. I found this in the TensorFlow documentation:
# tensor 't' is [[[1, 1, 1],
One aspect of the project I'm unsure about is that when I use imshow, my images appear much less intelligible than in my instructor's example. My instructor's images look like this:
In general, I found myself relying heavily on the previous CNN example using the MNIST data set to see the steps to take, as well as the documentation to remind me what certain TensorFlow methods do. I suppose this is to be expected, considering how new I am to the process. My goal is to gain a more solid and comprehensive understanding of CNNs as I gain more experience in the field.
The training and testing process took quite a while on my laptop, around 30 minutes. In the end, my model achieved an accuracy of around 69%. The highest printed accuracy was 70.13% at step 4200, but the final printed accuracy was 67.02% at step 4900.
In this post, I'll document my theoretical learnings from the lectures I watch, then note things from the code-along example CNN in which I build a model to estimate handwritten numbers using the well-known MNIST data set. Last, I'll present my inquiries and results from a CNN project that attempts recognize a variety of objects in the CIFAR dataset.
My summary of CNNs can be found here.
My code for this section of the course can be found here.
Here are the notes I took during the lectures and code-alongs:
Convolutional Neural Networks
- Review
- Within a neuron: z=wx+b, a = activation_function(z)
- Activation functions: perceptrons (pos. or neg.), sigmoid, tanh, ReLu
- connecting neurons creates a network
- a neural net has input/output layers, hidden layers in between; more layers = more abstraction
- measurements of error: cost/loss function, which can be quadratic or cross-entropy
- Once we have error, minimize it by choose better weight and bias values; use gradient descent to find optimal values
- New Theory Topics
- Initialization of Weights Options
- Xavier (Glorot) Initialization
- Uniform / Normal
- Draw Weights from a distribution with zero mean and a specific variance
- W: initialization distribution for the neuron in question
- nin : number of neurons feeding into the neuron in question
- If mean of weights and x’s = 0, the 2nd eqn simplifies to the 3rd
- Gradient Descent
- Learning Rate: defines step size during gradient descent
- smaller -> longer to train model
- larger -> chance of overstepping minimum and never converging
- Batch Size: allow us to use stochastic (randomized) gradient descent
- smaller batch size -> less representative of overall data
- larger batch size -> longer time needed to train
- Second-Order Behavior of gradient descent allows us to adjust learning rate based on rate of descent
- Error is large when descent is first starting training, nice to take large steps in beginning, smaller as it gets closer to the minimum
- AdaGrad
- RMSProp
- Adam <- we’ll focus on this one, later (built into TF)
- Unstable / Vanishing Gradients
- As number of layers in a network is increased, the layers towards input are less affected by the error calculated at the output as you go backwards through the network
- Initialization and normalization help mitigate these issues
- Will discuss more when covering RNN’s
- Underfitting
- If getting large error on both training and evaluation data sets
- Overfitting
- Very low error on training data, high error on evaluation data
- How to mitigate overfitting
- L1/L2 Regularization
- adds penalty for larger weights in the model so you don’t end up with one neuron/feature in training set with a large weight
- Dropout
- randomly remove neurons during training so the network doesn’t rely too much on any particular neuron
- Expanding Data
- add noise, tilt images, adding white noise to sound data, etc. to training data so the model doesn’t overfit
- MNIST Data Overview
- TF has 55k training images, 10k test images, 5k validation (holdout) images ready to download
- Dataset contains handwritten digits (0-9)
- A single-digit image can be represented as an array on a 2-D plane with 28x28 pixels
- array can be flattened into a 1D vector of 784 number
- flattening removes some information (e.g. relationship of a pixel to its neighbors), come back to this later
- Can think of the entire group of 55k images as a tensor (n-dimensional array) with dimensions 784x55,000
- Use One-Hot encoding for labels
- have a single array for the digit represented by each image instead of labels like “one”, “two”...
- label is represented based on index position in the label array
- corresponding label will be a 1 at the index location and a 0 elsewhere
- ex. 4 would have this label array: [0,0,0,0,1,0,0,0,0,0]
- index starts at 0 and goes to 9
- Therefore, labels for training data end up being a 2D array 10 by 55k
- MNIST “Basic” Approach
- We’ll use the Softmax Regression Approach before using a CNN
- returns list of values between 0 and 1 that add up to = 1
- can think of these as probabilities
- use Softmax as the activation function
- exp(z) means e^z
- Convolutional Neural Network (CNN) Theory
- tensors make it easier to feed in sets of images to the model (I,H,W,C)
- I: images
- H: height of image (in pixels)
- W: width of image (in pixels)
- C: color channels: 1-grayscale, 3- RGB
- Recall: a densely connected network has every neuron connected to every other neuron
- A CNN has each unit connected to a smaller number of nearby units in the next layer
- Why use a CNN instead of a DNN?
- most images are 256x256 pixels or more
- too many parameters, unscalable to new images
- In image processing, pixels near to each other are more correlated
- CNN’s are great for this
- Each CNN layer looks @ increasingly larger part of image
- CNN helps w/ regularization, limits search of weights to size of the convolution
- Convolutional layers are only connected to pixels in their respective fields
- Issue: there may not be an input available for edge neurons
- Fix by adding “padding” of 0’s around the image
- ^^ this is an example of edge detection (dark vs light pixel)
- Neurons that aren’t connected -> add 0 edges
- Each filter detects a different feature
- For 2-D grayscale:
- For 2-D colored:
- Filters are commonly visualized with grids. E.g. :
- Pooling (AKA subsampling) layers subsample input image, reducing memory use and computer load; also reduces number of parameters
- create a 2x2 pool of pixels & evaluate the max value
- only max val makes it to next layer
- next, move over by a stride and repeat.
- pooling removes some information
- small pooling kernel of 2x2 w/ stride of 2 removes 75% of input data!
- an image typically has too much information to not have a pooling layer; too computationally expensive
- Dropout is a form of regularization to help prevent overfitting
- during training, units & their connections are randomly dropped
Code-Along
My code for this section can be found in the link posted above
While testing the model in the session, it took about 30 minutes for the process to complete on my laptop (2.7 GHz i5 MPB). To decrease the amount of time, simply decrease the number of steps from 5000 to perhaps 2000. By step 4900, my model had an accuracy of 98.93% (it had an accuracy of 11% after the 1st step, and 94.25% by the 100th step).
Exercise
My task was to create a CNN that can correctly label/identify images from the widely-used CIFAR-10 data set. This set contains 50,000 training images and 10,000 test images, and can be found here. There are 10 possible image labels: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. My code can be found here.
From here on, I'll document any findings or issues I have, as well as my results.
I found out that a "pickling" Python object is used "for serializing and de-serializing a Python object structure. 'Pickling' is the process whereby a Python object hierarchy is converted into a byte stream, and 'unpickling' is the inverse operation, whereby a byte stream is converted back into an object hierarchy." [Source.]
When asked for the directory path of the cifar-10-batches-py, I initially inserted the wrong path format (I used the ~ notation). The program actually requires a /Users/... format. I found an easy way to copy a directory's file path on Mac OS X here.
While using the example I coded during the lectures, I remembered that I had never figured out what a dimension of -1 meant when reshaping an object. I found this in the TensorFlow documentation:
# tensor 't' is [[[1, 1, 1],
# [2, 2, 2]],
# [[3, 3, 3],
# [4, 4, 4]],
# [[5, 5, 5],
# [6, 6, 6]]]
# tensor 't' has shape [3, 2, 3]
# pass '[-1]' to flatten 't'
reshape(t, [-1]) ==> [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6]
# -1 can also be used to infer the shape
# -1 is inferred to be 9:
reshape(t, [2, -1]) ==> [[1, 1, 1, 2, 2, 2, 3, 3, 3],
[4, 4, 4, 5, 5, 5, 6, 6, 6]]
I found this example extremely helpful and it cleared up my confusion. A dimension of -1 is either used to infer the dimension (note that when one dimension isn't given, as in the top example above, that dimension is assumed to be 0), or if given as the only parameter it reshapes the objects into a 1-D array.One aspect of the project I'm unsure about is that when I use imshow, my images appear much less intelligible than in my instructor's example. My instructor's images look like this:
While mine look like this:
I'm not sure why this is, but I'll post here if I find out anything.
In general, I found myself relying heavily on the previous CNN example using the MNIST data set to see the steps to take, as well as the documentation to remind me what certain TensorFlow methods do. I suppose this is to be expected, considering how new I am to the process. My goal is to gain a more solid and comprehensive understanding of CNNs as I gain more experience in the field.
The training and testing process took quite a while on my laptop, around 30 minutes. In the end, my model achieved an accuracy of around 69%. The highest printed accuracy was 70.13% at step 4200, but the final printed accuracy was 67.02% at step 4900.


Comments
Post a Comment