Skip to main content

Deep Learning - TensorFlow Basics

The next section of my course will cover the basics of TensorFlow, Google's open-source machine learning library for Python. It is quite common, and is the successor to the Theano framework. It is one of the most popular machine learning frameworks.

Essentially, TensorFlow (TF) provides built-in methods to easily perform common machine learning tasks. It also sets up all of the classes and functions that I learned to manually (without TF) create in the previous section (see blog post).

Here is all of the code I wrote while following this section. I encourage the reader to use this code as reference while reading through this post, as it contains crucial information to understanding many of the topics I refer to below (particularly in the Issues/Observations section). The Classification and Regression exercises were basically "homework", while the other files were code-alongs from the class. There are many comments explaining what the code does, as well as some notes/reminders.

Here are the notes I took while going through this section:

Introduction to TensorFlow 

  • a constant is the most basic type of tensor
  • Here is a resource on how to multiply two matrices
  • Here’s a diagram for the layout of a matrix:
  • The number of columns of the 1st matrix must equal the number of rows of the 2nd matrix
TensorFlow Graphs
  • graphs are sets of connected nodes (AKA vertices)
  • connections <==> edges
  • in TensorFlow (TF) each node is an operation with inputs and an output
  • First coding this graph:
  • TF automatically creates a default graph, which can be set to be a different graph


Variables and Placeholders (in TF)
  • Two main types of tensor objects in a Graph: Variables and Placeholders
  • TF tunes parameters of model during optimization process
  • Variables
    • can hold values of weights and biases throughout a session
    • need to be initialized
  • Placeholders
    • are initially empty and are used to feed in training examples
    • need to be declared (but not initialized)


TF Neural Net

  • Steps:
    • build a graph
    • initiate session
    • feed data in and get some output
  • It’s very costly to feed in many data points (e.g. 1 million) at once, so normally feed in batches of data for training
  • Estimator API
    • tf.estimator
    • Types:
      • .LinearClassifier: constructs linear classification model
      • .LinearRegressor: constructs linear regression model
      • .DNNClassifier: constructs NN classification model
      • .DNNRegressor: constructs NN regression model
      • also combined versions between DNN and Linear
    • To use Estimator API:
      • define list of feature columns
      • create Estimator model
      • create data input function
      • call train, evaluate, and predict methods on est. object
    • To Save/Restore A Model:
      • saver = tf.train.Saver()
      • when making a new session, save it with saver.save(...)
      • to restore a previously saved model, use saver.restore(...)

Issues and Observations

Here are some miscellaneous observations and difficulties I had while following through this section, both while coding along with the instructor and while completing the exercises and doing my own experiments.

...

When I first tried "import tensorflow" in my Jupyter Notebook I got an error saying that the module didn't exist, so I went to my command line and used "pip install tensorflow==1.2.0 --ignore-installed" to fix the issue. 

...

It's important to use parentheses properly when using methods. I was confused of my output at first, until I realized that I had forgotten parentheses. Notice the different memory locations and types (function vs. Graph):




... 

The first neural net I made using TensorFlow involved creating a neuron that performs a simple linear fit to some 2-dimensional data. 

...

While coding along the TF Neural Network, I wanted to see how the model would perform using just one training step. It actually performed better than I expected, as the test data (red line) generally follows the "real" data (blue dots). 



And here's the same example but with 100 training steps instead of 1:


While there wasn't too much of a visual difference between the two, keep in mind that this data set (the generation of which can be found in the Jupyter Notebook linked to in the intro) is very simple and easy to fit. With a more complex data set, the difference between these two tests would be much more noticeable. 


After running prediction using tf.estimator.inputs.pandas_input_fn, dictionary-like objects are returned:

{'class_ids': array([1]),
  'classes': array([b'1'], dtype=object),
  'logistic': array([ 0.57128888], dtype=float32),
  'logits': array([ 0.28711164], dtype=float32),
  'probabilities': array([ 0.42871115,  0.57128888], dtype=float32)}

One of these objects is returned for each x-value given (the model tries to predict the y-value, which implies which class the data should belong to. In this example, the class is 1, which means the model has predicted that the person matching that data point does have diabetes (or at least is more likely to have diabetes than to not have diabetes). The 'probabilities' field shows the likelihood of the data belonging to each class. In this case, the model gives that there's a 43% chance the data should belong to the no-diabetes class, and a 57% chance the data should belong to the diabetes-positive class.

A densely connected neural network can also be used for predicting output with tf.estimator.DNNClassifier. Here's the result with 3 layers of 10 neurons:

{'accuracy': 0.74891776,
 'accuracy_baseline': 0.64935064,
 'auc': 0.8228395,
 'auc_precision_recall': 0.69959283,
 'average_loss': 0.50522906,
 'global_step': 1000,
 'label/mean': 0.35064936,
 'loss': 4.8628297,
 'prediction/mean': 0.31253079
}

Notice the accuracy is 74%.

Here's the same experiment, but with 5 layers of 10, 20, 20, 20, and 10 neurons, respectively.

{'accuracy': 0.73593074,
 'accuracy_baseline': 0.64935064,
 'auc': 0.80596703,
 'auc_precision_recall': 0.66074276,
 'average_loss': 0.52225369,
 'global_step': 1000,
 'label/mean': 0.35064936,
 'loss': 5.0266919,
 'prediction/mean': 0.34027791
}

We can tell that using this many layers and neurons with our data (which has only 10 features) actually resulted in a LESS accurate model (73% compared to 74% before). This likely means that this new model is overfitting to the data, which means that it can't predict new data as well as it could with fewer layers and neurons.

...

Regression Exercise

The first "homework" exercise that I completed involved creating a TF neural network (NN) using Regression to predict the median home value of each block in a small geographic area. The features given are:
  • housingMedianAge: continuous. 
  • totalRooms: continuous. 
  • totalBedrooms: continuous. 
  • population: continuous. 
  • households: continuous. 
  • medianIncome: continuous. 
  • medianHouseValue: continuous (trying to predict this value)

While working through this, I performed a short experiment to see how the number of layers, number of neurons per layer, and number of steps affects the loss of the final step of the model. The results are below. Note that there are six features, so having more than six neurons per network shouldn't improve the accuracy of model. [6,6,6] means the model is running with three layers, each layer having six neurons. 

NN's: [6,6,6]








NN's: [6,6,6,6,6,6]









Notice that the loss for both cases (three layers and six layers) was the least with 10,000 steps, which means that, somewhere between 10,000 and 100,000 lies the optimal number of steps for the lowest loss. Also notice that the test case with the lowest overall loss was 10,000 steps and 3 layers, which I suppose means that, at a certain point, more layers and more steps can each lead to a model that overfits to the data during training. This results in a less accurate model when evaluating with new data that it hasn't seen before. Fascinating. 

In case you're wondering how loss and accuracy relate: "Unlike accuracy, loss is not a percentage. It is a summation of the errors made for each example in training or validation sets." [Source]
...

Classification Exercise


The second exercise involved creating a NN with TF to determine whether or not an individual has an annual salary of more than $50,000 or less than $50,000 given the following information:

age Continuous 
workclass Categorical
education Categorical
education_num Continuous
marital_status Categorical
occupation Categorical
relationship Categorical
race Categorical 
gender Categorical 
capital_gain Continuous
capital_loss Continuous
hours_per_week Continuous
native_country Categorical
income Categorical

My code can be found here.

After completing the (linear classification) model, here's how it performed:




Learning the basics of the TensorFlow framework has laid the groundwork for future projects and learning new topics in Deep Learning. In the next section, I'll learn about Convolutional Neural Networks (CNN's), which are fundamental to deep learning with images. On to the next section!







Comments

Popular posts from this blog

Reading from and Modifying a DynamoDB Table using API Gateway

In many IoT (Internet of Things) projects, it is necessary to read from and write to a database. AWS has an application called DynamoDB that is a NoSQL database, exactly what we need. While it is very simple to add/modify/delete entries in this database via the user interface, it's not so obvious how to do so via HTTP API calls. So, in this post I'll continue the AWS portion of my journey into IoT to learn how to do just that. I'll be following  this  tutorial. The first thing I encountered is I wasn't sure if I was supposed to follow the steps under "Defining APIs", or if that was just knowledge for later. Turns out, that was knowledge for later. If I had any prior experience with APIs I probably would have realized this. Basically, the "API definition"s are just a high-level plan for what you want your API requests to look like in the end. In my opinion this information should go in the "Creating the APIs" section. Moving through "Cr...

Arduino Project: Sound Alert via Twitter

For my second IoT project using Arduino MKR1000, I'll employ a sound sensor to send out alerts via Twitter when a loud sound is detected. This could have many uses, ranging from informing the inhabitants of a house that there is someone at the door, informing a car owner that their alarm is going off, or residents that a gunshot has gone off in their neighborhood. See my previous posts to learn about the setup process for the Arduino MKR1000, which is wifi-enabled without any additional extensions. I will be using the Arduino IDE to compile the code. For hardware, I'll be using the sound sensor I purchased here , as well as my Arduino, three solder-free wires, a generic breadboard, and a Kindle power cable (this probably isn't the safest way to power Arduino, but it's what I had). I'll be following this  tutorial to set up the sound detector, as well as some code from here  to utilize Twitter. The sound setup tutorial was very simple and one of the clearest wiring g...

Deep Learning - Recurrent Neural Networks (RNNs)

The latest Deep Learning topic I've been exploring is the Recurrent Neural Network (RNN). It has many use cases, including: time series extrapolation (i.e. given historical data, predict future output); object trajectory; and natural language processing (NLP). The code I wrote during this section of the course I'm taking can be found here . For my summary of RNNs, click here . Here are the notes I took while watching the lectures: RNN Theory Examples of Sequences Time Series Data (e.g. stock prices, sales rates) Sentences (i.e. natural language) Audio/Music Car Trajectories (where a car will go in a situation) [1,2,3] shifted one time step into the future would be [2,3,4] will train a network to predict y-values for this A normal neuron has input, activation function, then output A Recurrent Neuron sends output back to itself this process is called “unrolling” the neuron note that the input of each neuron is a summation from both ...