The next section of my course will cover the basics of TensorFlow, Google's open-source machine learning library for Python. It is quite common, and is the successor to the Theano framework. It is one of the most popular machine learning frameworks.
Essentially, TensorFlow (TF) provides built-in methods to easily perform common machine learning tasks. It also sets up all of the classes and functions that I learned to manually (without TF) create in the previous section (see blog post).
Here is all of the code I wrote while following this section. I encourage the reader to use this code as reference while reading through this post, as it contains crucial information to understanding many of the topics I refer to below (particularly in the Issues/Observations section). The Classification and Regression exercises were basically "homework", while the other files were code-alongs from the class. There are many comments explaining what the code does, as well as some notes/reminders.
Here are the notes I took while going through this section:
Here are some miscellaneous observations and difficulties I had while following through this section, both while coding along with the instructor and while completing the exercises and doing my own experiments.
...
When I first tried "import tensorflow" in my Jupyter Notebook I got an error saying that the module didn't exist, so I went to my command line and used "pip install tensorflow==1.2.0 --ignore-installed" to fix the issue.
...
It's important to use parentheses properly when using methods. I was confused of my output at first, until I realized that I had forgotten parentheses. Notice the different memory locations and types (function vs. Graph):
...
The first neural net I made using TensorFlow involved creating a neuron that performs a simple linear fit to some 2-dimensional data.
...
While coding along the TF Neural Network, I wanted to see how the model would perform using just one training step. It actually performed better than I expected, as the test data (red line) generally follows the "real" data (blue dots).
After running prediction using tf.estimator.inputs.pandas_input_fn, dictionary-like objects are returned:
One of these objects is returned for each x-value given (the model tries to predict the y-value, which implies which class the data should belong to. In this example, the class is 1, which means the model has predicted that the person matching that data point does have diabetes (or at least is more likely to have diabetes than to not have diabetes). The 'probabilities' field shows the likelihood of the data belonging to each class. In this case, the model gives that there's a 43% chance the data should belong to the no-diabetes class, and a 57% chance the data should belong to the diabetes-positive class.
A densely connected neural network can also be used for predicting output with tf.estimator.DNNClassifier. Here's the result with 3 layers of 10 neurons:
Notice the accuracy is 74%.
Here's the same experiment, but with 5 layers of 10, 20, 20, 20, and 10 neurons, respectively.
We can tell that using this many layers and neurons with our data (which has only 10 features) actually resulted in a LESS accurate model (73% compared to 74% before). This likely means that this new model is overfitting to the data, which means that it can't predict new data as well as it could with fewer layers and neurons.
...
Regression Exercise
The first "homework" exercise that I completed involved creating a TF neural network (NN) using Regression to predict the median home value of each block in a small geographic area. The features given are:
NN's: [6,6,6]
Classification Exercise
The second exercise involved creating a NN with TF to determine whether or not an individual has an annual salary of more than $50,000 or less than $50,000 given the following information:
age Continuous
workclass Categorical
education Categorical
education_num Continuous
marital_status Categorical
occupation Categorical
relationship Categorical
race Categorical
gender Categorical
capital_gain Continuous
capital_loss Continuous
hours_per_week Continuous
native_country Categorical
income Categorical
My code can be found here.
After completing the (linear classification) model, here's how it performed:
Learning the basics of the TensorFlow framework has laid the groundwork for future projects and learning new topics in Deep Learning. In the next section, I'll learn about Convolutional Neural Networks (CNN's), which are fundamental to deep learning with images. On to the next section!
Essentially, TensorFlow (TF) provides built-in methods to easily perform common machine learning tasks. It also sets up all of the classes and functions that I learned to manually (without TF) create in the previous section (see blog post).
Here is all of the code I wrote while following this section. I encourage the reader to use this code as reference while reading through this post, as it contains crucial information to understanding many of the topics I refer to below (particularly in the Issues/Observations section). The Classification and Regression exercises were basically "homework", while the other files were code-alongs from the class. There are many comments explaining what the code does, as well as some notes/reminders.
Here are the notes I took while going through this section:
Introduction to TensorFlow
- a constant is the most basic type of tensor
- Here is a resource on how to multiply two matrices
- Here’s a diagram for the layout of a matrix:
- The number of columns of the 1st matrix must equal the number of rows of the 2nd matrix
TensorFlow Graphs
- graphs are sets of connected nodes (AKA vertices)
- connections <==> edges
- in TensorFlow (TF) each node is an operation with inputs and an output
- First coding this graph:
- TF automatically creates a default graph, which can be set to be a different graph
Variables and Placeholders (in TF)
- Two main types of tensor objects in a Graph: Variables and Placeholders
- TF tunes parameters of model during optimization process
- Variables
- can hold values of weights and biases throughout a session
- need to be initialized
- Placeholders
- are initially empty and are used to feed in training examples
- need to be declared (but not initialized)
TF Neural Net
- Steps:
- build a graph
- initiate session
- feed data in and get some output
- It’s very costly to feed in many data points (e.g. 1 million) at once, so normally feed in batches of data for training
- Estimator API
- tf.estimator
- Types:
- .LinearClassifier: constructs linear classification model
- .LinearRegressor: constructs linear regression model
- .DNNClassifier: constructs NN classification model
- .DNNRegressor: constructs NN regression model
- also combined versions between DNN and Linear
- To use Estimator API:
- define list of feature columns
- create Estimator model
- create data input function
- call train, evaluate, and predict methods on est. object
- To Save/Restore A Model:
- saver = tf.train.Saver()
- when making a new session, save it with saver.save(...)
- to restore a previously saved model, use saver.restore(...)
Here are some miscellaneous observations and difficulties I had while following through this section, both while coding along with the instructor and while completing the exercises and doing my own experiments.
...
When I first tried "import tensorflow" in my Jupyter Notebook I got an error saying that the module didn't exist, so I went to my command line and used "pip install tensorflow==1.2.0 --ignore-installed" to fix the issue.
...
It's important to use parentheses properly when using methods. I was confused of my output at first, until I realized that I had forgotten parentheses. Notice the different memory locations and types (function vs. Graph):
...
The first neural net I made using TensorFlow involved creating a neuron that performs a simple linear fit to some 2-dimensional data.
...
While coding along the TF Neural Network, I wanted to see how the model would perform using just one training step. It actually performed better than I expected, as the test data (red line) generally follows the "real" data (blue dots).
And here's the same example but with 100 training steps instead of 1:
While there wasn't too much of a visual difference between the two, keep in mind that this data set (the generation of which can be found in the Jupyter Notebook linked to in the intro) is very simple and easy to fit. With a more complex data set, the difference between these two tests would be much more noticeable.
After running prediction using tf.estimator.inputs.pandas_input_fn, dictionary-like objects are returned:
{'class_ids': array([1]),
'classes': array([b'1'], dtype=object),
'logistic': array([ 0.57128888], dtype=float32),
'logits': array([ 0.28711164], dtype=float32),
'probabilities': array([ 0.42871115, 0.57128888], dtype=float32)}
One of these objects is returned for each x-value given (the model tries to predict the y-value, which implies which class the data should belong to. In this example, the class is 1, which means the model has predicted that the person matching that data point does have diabetes (or at least is more likely to have diabetes than to not have diabetes). The 'probabilities' field shows the likelihood of the data belonging to each class. In this case, the model gives that there's a 43% chance the data should belong to the no-diabetes class, and a 57% chance the data should belong to the diabetes-positive class.
A densely connected neural network can also be used for predicting output with tf.estimator.DNNClassifier. Here's the result with 3 layers of 10 neurons:
{'accuracy': 0.74891776,
'accuracy_baseline': 0.64935064,
'auc': 0.8228395,
'auc_precision_recall': 0.69959283,
'average_loss': 0.50522906,
'global_step': 1000,
'label/mean': 0.35064936,
'loss': 4.8628297,
'prediction/mean': 0.31253079
}
Notice the accuracy is 74%.
Here's the same experiment, but with 5 layers of 10, 20, 20, 20, and 10 neurons, respectively.
{'accuracy': 0.73593074,
'accuracy_baseline': 0.64935064,
'auc': 0.80596703,
'auc_precision_recall': 0.66074276,
'average_loss': 0.52225369,
'global_step': 1000,
'label/mean': 0.35064936,
'loss': 5.0266919,
'prediction/mean': 0.34027791
}
We can tell that using this many layers and neurons with our data (which has only 10 features) actually resulted in a LESS accurate model (73% compared to 74% before). This likely means that this new model is overfitting to the data, which means that it can't predict new data as well as it could with fewer layers and neurons.
...
Regression Exercise
The first "homework" exercise that I completed involved creating a TF neural network (NN) using Regression to predict the median home value of each block in a small geographic area. The features given are:
- housingMedianAge: continuous.
- totalRooms: continuous.
- totalBedrooms: continuous.
- population: continuous.
- households: continuous.
- medianIncome: continuous.
- medianHouseValue: continuous (trying to predict this value)
While working through this, I performed a short experiment to see how the number of layers, number of neurons per layer, and number of steps affects the loss of the final step of the model. The results are below. Note that there are six features, so having more than six neurons per network shouldn't improve the accuracy of model. [6,6,6] means the model is running with three layers, each layer having six neurons.
NN's: [6,6,6]
NN's: [6,6,6,6,6,6]
Notice that the loss for both cases (three layers and six layers) was the least with 10,000 steps, which means that, somewhere between 10,000 and 100,000 lies the optimal number of steps for the lowest loss. Also notice that the test case with the lowest overall loss was 10,000 steps and 3 layers, which I suppose means that, at a certain point, more layers and more steps can each lead to a model that overfits to the data during training. This results in a less accurate model when evaluating with new data that it hasn't seen before. Fascinating.
In case you're wondering how loss and accuracy relate: "Unlike accuracy, loss is not a percentage. It is a summation of the errors made for each example in training or validation sets." [Source]
...
Classification Exercise
The second exercise involved creating a NN with TF to determine whether or not an individual has an annual salary of more than $50,000 or less than $50,000 given the following information:
age Continuous
workclass Categorical
education Categorical
education_num Continuous
marital_status Categorical
occupation Categorical
relationship Categorical
race Categorical
gender Categorical
capital_gain Continuous
capital_loss Continuous
hours_per_week Continuous
native_country Categorical
income Categorical
My code can be found here.
After completing the (linear classification) model, here's how it performed:
Learning the basics of the TensorFlow framework has laid the groundwork for future projects and learning new topics in Deep Learning. In the next section, I'll learn about Convolutional Neural Networks (CNN's), which are fundamental to deep learning with images. On to the next section!










Comments
Post a Comment