These are the notes that I wrote while attending the first three courses from Deep Learning Specialization program at Coursera. I you have attended/are attending the Deep Learning Specialization program, this note would straight away look familiar to you.

So here it goes.

~

Representation and Notations

---

These are the conventions used throughout the course.

L

Number of layers in the network. In the example diagram above, there are four layers in the network (L=4), on which three are hidden layers. By convention the input layer (X) is not counted as layer.

l

Layer indices. For example “l=1” means the first layer, “l=L” means the last layer or the output layer. As additional convention, l=0 is used to call the input layer.

m

Number of (training) samples.

_^[l]

Superscript square bracket indicates the layer number. For example, a^[2] means the activation values (a vector) for the second layer.

_⁽ⁱ⁾

Superscript round bracket indicates the sample number. For example, x⁽ⁱ⁾ means the i-th input sample (which itself is a vector)

.._i

Subscript index indicates the feature number or unit number. For example, x₁ indicates the first input feature, a₂ indicates the activation of the second unit in a particular layer, etc.

n^[l]

n denotes the number of units, and square bracket superscript is used to indicate the layer number. Hence n^[0] means the number of units in the input layer (three in the example above),  n^[2] means the number of units in layer 2 (five in the diagram), and so on.

x

The input vector containing one training sample:

X

X is a matrix containing all input samples. The dimension is n_x by m, where n_x is the number of features (technically it’s more correct to call this n^[0], but the prof is not too consistent about this either) and m is the number of samples that we have.

It will look like this:

a^[l]

Activation units in layer l. For one particular layer, this is a column vector of n^[l] dimension:

A^[l]

For vectorized implementation across all samples, A^[l] represents the activation values for all units in layer l across all samples:

z^[l]

Logits in layer l. The dimension is the same as a^[l].

Z^[l]

Vectorized data of Z across all samples. Similar format and dimension to A^[l].

b^[l]

Bias in layer l. The dimension is the same as a^[l].

W^[l]

Matrix of weights in layer l, which dimension is n^[l] by n^[l-1].

~

Forward Propagation

---

Initialization

Initialize W with small random variables. Initialize b with zeros.

There are more advanced initialization techniques in Weight Initialization section.

Forward Propagation

For each layer l, the vectorized forward propagation formulas are:

where g^[l](z) is the activation function for layer l. See below.

Activation Functions

At each node, an activation function is used to turn Z, which is a linear value, into A, a non-linear value. Without activation functions, the whole neural network no matter how large or deep would just be a silly linear function!

Each layer can use different activation functions, hence the activation function is indexed by the layer number (the g^[l](z) notation).

Sigmoid

Sigmoid activation function is typically used as the activation function of the output layer where the output is binary classification (1 and 0), because g(z) natively outputs values between 0 and 1.

For other layers, ReLU or tanh are prefered (see below), and for multiclass output, softmax is used (also see below).

tanh

ReLU (Rectified Linear Unit)

One of the drawback of sigmoid and tanh functions is the gradient becomes very small if the value of Z is positively large or negatively small, making the descent slow. The ReLU activation function solves this.

Leaky ReLU

Softmax

See Multiclass Classification.

Cost Function

Loss Function

The loss function for logistic regression for binary classification measures how well we’re doing with respect to single training example:

Cost Function

The cost function measures how well we’re doing over the whole training set:

~

Backward Propagation

---

Initialization

For logistic regression/classification problems (binary or multiclass), we calculate dZ^[L] as:

For those who are interested in how dZ turns out like that, see the calculus notes below.

Backprop for Layer l

Calculate the following for each layer, starting with layer L down to layer 1.

For layer L, calculate dZ as explained in Initialization section above. For other layers,  calculate dZ as:

Or if you prefer, we can calculate dZ in two steps, by first calculating dA:

And then dZ:

Once we have dZ, we can calculate the gradient of the parameters and bias as follows:

Calculus Notes

For each variable dz, da, dW, and db, dx means the derivative of the loss function L with regard to x. So:

• dz means the derivative of the loss function L with regard to z, i.e. 
• da means the derivative of the loss function L with regard to a, i.e. 
• and so on.

Note that the capital letter version (dZ, dA) just means that it is the vectorized version (for all samples).

The loss function L(a,y) for binary classification is defined above in Loss Function section above, and for multi-class classification it will be defined differently later in Multiclass Classification section.

From L(a, y) for binary classification, we calculate da as follows.

dz is calculated as:

dW is calculated as:

Note that there is a subtle note in dW = dz • a formula above, in that a is the activation vector of the previous layer, hence the exact formula with layer index is:

Similarly for db:

Updating the Parameters

Once we get dW and db we can update the parameters:

The learning rate α is a hyperparameter which value need to be fine tuned. This will be explained in the Tuning section below.

~

Multi-class Classification

---

With multi-class classification, the output layer now has more than 1 outputs. Suppose we have three classes:

• class 1: cat
• class 2: bird
• class 3: other

The neural network may look like this. Just notice that the output layer has three outputs:

Note:

Do not confuse multi-class classification and multi-label classification. In multi-class classification, we have multiple classes and the model predicts one class that is the most likely representation of the input. In multi-label classification, we also have multiple classes, but now the model can predict more than one classes that may be present in the input (e.g. an image may contain both cat and bird).

The multi-label model is also called multi-task learning.

Forward Propagation

For the training labels in multiclass classification setup, we need to convert y (a scalar number) into y as vector. For example, for three classes above, each value is converted into vector as follows:

For forward propagation, calculate Z as usual:

Then calculate vector t as:

Visually, t look like this (for a three class output):

With this we can calculate the activation vector a^[L] as:

Note that the above is a vector operation (t is a vector).

Visually, a looks like this (for a three class output):

And once we’re done with the calculation, each value corresponds to the probability that the input is the same class as the unit. Note that the sum of the probabilities is 1. So we may end up with values like this:

which means that the most likely output is class 2 since it has the biggest probability.

Cost Function

Softmax Loss Function

Softmax Cost Function

Backward Propagation

Since multiclass classification only differs with the binary classification NN in the last layer/output layer (i.e. the output has more than one nodes and has softmax activation function instead), then only the first part of gradient descent calculation is different. Specifically, the dZ^[L] calculation is different.

Unfortunately the prof didn’t say how it should be calculated, because he said it will be done by the framework (Tensorflow). Will update once I have more info.

~

Tuning

---

Optimizing Bias vs Variance 

High bias means the model underfits the data. High variance means the model overfits the data.

We need to monitor our training set error vs dev set error. Establish the optimal error, which is also called the Bayes error, which is the theoretical best error that can be achieved. Alternatively, since establishing Bayes error is difficult, we can also compare with human level performance.

Don’t just look at the raw error numbers, but rather compare them with the baseline error (either Bayes or human level error). The difference between training error and the baseline is called “avoidable bias”. The difference between dev error and training error is called the “variance“.

Here are some examples:

• suppose the baseline error is 1%
  • if training error is 7% and dev set error is 10%, that means the avoidable bias is 6% and the variance is 3%. We should fix the bias since it is larger.
• on the other hand, suppose the baseline error is 6% for the same problem:
  • this means the avoidable bias is 1% and the variance is (still) 3%. We should optimize the variance instead.

The basic recipe for training or optimizing the model is as follows:

1. Does it have high bias? If yes, then fix it (always fix high bias problem first).
   1. Traing bigger model
   2. Train longer/use better optimization algorithms (momentum, RMSprop, Adam, etc.)
   3. Maybe use different NN architecture/hyperparameters search.
2. One the bias is good, fix any high variance issues.
   1. Get more data, becaues training on more data will help generalize better
   2. Use regularization (L2, dropout, data augmentation, etc.)
   3. Maybe use different NN architecture/hyperparameters search.

Regularization

Regularization is a way to fix high variance (overfit) problem. The intuition for regularization is it penalizes W for being too large, and this makes the network simpler.

Cost Function

For neural network, the formula to add regularization to the cost function is:

or:

where is called Frobenius norm:

Practically, this simply just adds all the weights in the network:

Note that the bias is omitted from the regularization, but you can include it if you want, by adding:

Gradient Descent

Updated gradient descent with regularization:

The update formula is the same:

L2 regularization is also called “weight decay” because the way it “decays” the W in the update formula above by  factor.

Dropout Regularization

The idea is to drop units randomly during training (but not during test time). The intuition is that by dropping the nodes it makes the subsequent nodes less dependent on that node; nodes can’t rely on specific nodes because they may be dropped at any time.

We define a probability ratio called keep_prob to indicate the probability that a unit is kept. The keep_prob can be set differently for each layer, for example, bigger layer can be set to have more probability for being dropped than smaller layer. Then randomly “drop” units during the iteration with probability (1 – keep_prob). We drop different units on different iteration/mini-batch.

Forward Propagation

For example we’re working on layer 3 (this is reflected in variable names).

1) Define the keep_prob, say 80% probability:

keep_prob = 0.8

2) Create array containing “mask” to drop units. Note that the “< keep_prob” part of the expression will convert array of probability values into array of booleans (True or False).

d3 = np.random.rand(a3.shape[0], a3.shape[1]) < keep_prob

3) Drop the units. In this expression, Numpy treats these booleans as 0 or 1, effectively zeroing (“dropping”) the a3 nodes when the corresponding d3 nodes are zero.

 a3 = a3 * d3

4) Scale the value back to the original scale. This is called “inverted dropout“:

a3 /= keep_prob

We can also merge step 4 into step 2 to make the process slightly  shorter:

d3 = (np.random.rand(a3.shape[0], a3.shape[1]) < keep_prob) / keep_prob

Backward Propagation

It’s not explained in the course. Need to trace the programming exercise. TBD.

Test Time

Do not drop units during test time! Dropout is only done during training.

Data Augmentation

Another way to fix overfitting is to add more data. If this is not possible (or is too expensive), we can augment the data to make new data, but this is not as good as getting fresh data of course:

Early Stopping

Plot cost function from training set vs dev set, and stop the training if we detect that we are starting to overfit the training data.

Critic for early stopping method is that it mixes two processes which should otherwise be independent: optimize cost function and not overfit. Optimizing cost function should have been the primary objective of our training, and stopping early messes up with this objective. The solution is to use regularization.

Normalizing Input

Normalizing the input (x) makes the features lay at the same scale and this makes the training goes faster.

Note: should we divide by the standard deviation (i.e. square root of the variance) instead?

Weight Initialization

Large weight values make Z value large, which slow down gradient descent, because with sigmoid or tanh activation functions, the gradient of the activation function becomes very small as Z becomes very big (positive or negative).

More units in the layer and more layers in the network make this problem worse  too. This is known as vanishing/exploding gradient problem, because if we have weight that is >1, that will be multiplied by as many times as the number of layers, making it “explode”. And similarly when it’s small, making it “vanish”. So choosing good initial weight values is desirable.

Typical Initialization

One typical weight initialization is to scale them down with some constants:

W[l] = np.random.rand(shape) * 0.01

Divide by Variance

A reasonable implementation is to scale with the variance of number of features:

W[l] = np.random.rand(shape) * np.sqrt( 1 / n[l-1] ) 

Where n[l-1] indicates the number of units in the previous layer. This is used especially when the activation function is tanh, and this method is called Xavier initialization.

If ReLU activation is used, this is said to work better (based on a paper by He at all):

W[l] = np.random.rand(shape) * np.sqrt( 2 / n[l-1] ) 

Gradient Checking

Gradient checking is used to double-check that our gradient calculation is correct. Use it sparingly as it is very slow.

For each feature i, calculate the approximate gradient (remember that gradient is the slope of the function):

Epsilon is a very small value, say 1e-7.

Then calculate the difference with the actual gradient calculated during gradient descent:

For a small epsilon (~1e-7), the difference is expected to be small (< 1e-7).

The algorithm to do this with Numpy is as follows.

for each i:
    Jplus = J(θi+ε)
    Jminus = J(θi-ε)
    gradapprox = (Jplus - Jminus) / (2*ε)
    
    grad = backward_propagation(x, theta)

    numerator = np.linalg.norm(grad - gradapprox)
    denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox)
    diff = numerator / denominator    

Mini Batch Gradient Descent

Divide the training examples into mini-batches of say 64, 128, 512, 1000 samples each, and run one iteration on these mini-batches instead of the whole sample. If we have very large number of samples this should make the training converge faster because otherwise (with the batch method) the weight is only updated once we process the whole sample.

At another extreme, we can train one sample at a time (called stochastic gradient descent), but this would be too slow as it defeats the benefit of vectorization.

Gradient Descent with Momentum

With this method, we update W (and b) with the exponentially weighted running average of dW (and db) instead. This will work faster than the standard gradient descent. The intuition is that by averaging the values, it smoothens the oscilation in the gradient descent (say if dW value of one particular feature in one iteration is +10, and the next iteration is +10, the average will be 0 and it cancels out the oscilation on that particular feature, the the descent will “focus” on the right direction towards the minimum).

Here VdW is an exponentially weighted moving average of dW. It’s initialized to 0 and updated on every iteration:

Similarly for Vdb:

The hyperparameter β is usually 0.9.

Updating the parameters then are done with VdW and Vdb:

Optionally, bias correction may be applied to fix the first few values of the moving average:

RMSprop (Root Mean Square Prop)

The intuition is to dampen large oscilation in some particular directions by penalizing movements that are large (dividing it by square root of itself). The formulas are:

where β is typically 0.999 and epsilon ϵ is very small (e.g. 10^-8) and is used to avoid division by zero.

Adam Optimization

Adam (adaptive moment estimation) optimization combines momentum and RMSprop optimizations above.

This is the momentum part with bias correction:

This is the RMSprop part with bias correction:

The updates are defined as:

The typical hyperparameter values are β1 = 0.9, β2 = 0.999, learning rate α needs to be tuned, and epsilon ϵ is 10^-8.

Learning Rate Decay

Learning rate decay can be used to speed up learning but typically it’s not that important (lower down in priority list).

There are many ways to decay the learning rate. For example it can be calculated as follows, with α0=0.2 and decay_rate=1:

Another one called exponential decay:

Another one:

Another one (t is mini-batch number):

Or alternatively, do manual decay (manually decrease learning rate based on manual observation).

Batch Normalization

The idea is to normalize the value of Z in any hidden layers so that the values don’t fluctuate much between iterations, and this makes the subsequent layers happier.

Training Stage

For each hidden layer l in the network, for each iteration:

Calculate the mean:

where m can be the number of samples or the number of samples in a mini-batch. In fact batch normalization is probably used more with mini-batches.

Then calculate the variance:

And normalize z:

By now we have z which values are centered around 0 and with variance σ². But this is not desirable, so we modify them as follows:

where γ and β are learnable parameters.

Test Time

Since usually we predict one sample at the time during test time, there is specific calculation that needs to be done to get the mean and variance.

Usually people estimate the mean and σ² as the exponentially weighted average of the previous mini-batches.

~

Structuring Machine Learning Projects

Splitting the Data

Split the samples into training set, development/dev set (a.k.a hold-out cross validation set), and test set. It’s important to have separate dev and test set. The dev set is used as the target for the optimization training. Once training is finished and the model is done, the test set is then used as the unbiased measurement for the model’s performance. Since we are iterating to get the best performance on the dev set, it is possible to overfit the dev set, and the test set is used to detect this. If there is overfit to the dev set, one way to fix is to have larger dev set.

Traditionally the data is split in 60:20:20 proportion, or 70:30 if we don’t have test set. But in this big data era, if we have very big data (>1 million samples), maybe 1% (=10,000) is enough for both dev and test sets.

When splitting the dataset, make sure the data in dev and test sets come from the same distribution. It is not so much of a problem when the training and dev set distributions are different. The problem is when dev and test set distributions are different. It’s like optimizing the model to hit a certain target (the dev set), and then use different target to test it.

Test Set from Different Distribution

Suppose we’re building cat classifier for mobile app. We have 200,000 (high resolution) cat images downloaded from the web. We also have 10,000 images of cats taken by using mobile camera and uploaded by the users. Obviously these 10,000 images are the data that we care most since it represents the actual use case of the app. How to split the data effectively?

One way to do it is to merge both data randomly and split them into training, dev, and test sets normally as before. But this is a bad way to do it because our 10,000 images will have very little impact on the training and the test set will have little resemblance with the actual use case (we have >95% of high quality web images which do not exist in the real use case).

A better way to do it is to split the data as follows:

• training set consists of 200,000 images from the web + 10,000 images from users
• dev set consists of 5,000 images from users
• test set consists of 5,000 images from users

But then our training will have different distribution than the dev/test sets. The bias and variance analysis will need to change (see below).

Bias and Variance with Mismatched Data Distribution

When our training set and dev set have different distribution, we may have the following results:

• training error: 1%
• dev error: 10%

It is difficult to conclude if the error difference is due to variance or because of different distribution. To fix this, add another set, training-dev set, which has the same distribution as the training set, but the model is not trained on this set.

Then we can analyze the errors more accurately.

For example, continuing with the previous sample (where we have 10% dev error):

• case 1:
  • human level error: 0%
  • training error: 1%
  • training-dev error: 9%
  • dev error: 10%
  • With these errors, we can conclude that we have high variance.
• case 2:
  • human level error: 0%
  • training error: 1%
  • training-dev error: 1.5%
  • dev error: 10%
  • With these errors, we can conclude that the error is due to data mismatch

Fixing Data Mismatch

• Carry out manual error analysis to try to understand difference between training and dev/test sets
• Make training data more similar
  • collect more data similar to dev/test sets
  • synthetize training data, but be careful so that the training doesn’t overfit this synthetized data (for example, we may be able to synthetize thousands of car images from a game, but maybe there are only 20 types of cars in the game)

Orthogonalization

When fixing machine learning problem (e.g. fixing the bias, fixing the variance), pick a method that solves that specific problem, rather a method that affects a lot of aspects of the learning at the same time. This will confuse our results.

Single Evaluation Metric

If we have many evaluation metrics (e.g. precision and recall, results across many countries, etc.), it’s better to combine them into one single metric (e.g. F1 score, the weighted average, etc.) so that the results are easier to compare.

The metric must reflect the real world objective for the product (e.g. user satisfaction). If the metric no longer reflects that, it must be changed.

Satisficing and Optimizing Metrics

Satisficing metric means metric that cannot be violated. Optimizing metric means metric that is optimized to work better.

For example, we want classifier with the best accuracy which run-time cannot exceed 100ms. The 100ms restriction is the satisficing metric, while the accuracy is the optimizing metric.

Error Analysis

To improve accuracy, analyse the samples that are predicted wrongly by our model. Find the reason why they are wrongly predicted, and focus on fixing error that will improve the accuracy most significantly.

For example, we have 90% accuracy to our cat detection model, which is equal to 10% error. By doing error analysis, suppose we counted that 10% of this error is because the model is misclassifying dogs, 50% is because the image is blurry, 10% is because it is misclassifying big cats, and 30% is due to other reasons. By having this error analysis, we now know where we will make significant improvement to our accuracy.

Incorrectly Labelled Data

During error analysis, if we find that we have many incorrectly labelled data, we may decide to fix those samples (depending on how much improvement we can expect by doing it). If we do so, make sure to also fix the test set, so that the dev and test sets continue to have the same distribution.

Build Something Quickly, then Iterate

The general advise on building new machine learning project is to set up dev/test sets and metric, and build the initial system quickly (don’t overthink it), and then iterate on it based on bias/variance analysis and error analysis above to prioritize what to fix next.

For application areas on which we have significant prior experience, or significant body of academic literature for the exact same problem, then it is okay to build a more complex system based on these literature.

Transfer Learning

Sometimes it is possible to reuse a model trained to do one task to do another task. For example, reuse a model that has been trained to do cat detection to do radiology image classification.

This kind of reuse is called transfer learning. The way to do it is to delete the last layer of the network (and all the weights) and replace with a fresh new layer and new set of random weights.

If the new dataset is small, it’s best to only train the last layer and keep all the other layers fixed. If the new dataset is large, we can retrain the whole network. If we do the latter, then the training for the initial task (the cat detection) is called pre-training, while the training for the new task is called fine tuning.

Rough guidelines on when transfer learning (from task A to task B) makes sense:

• task A and B have the same input x
• we have a lot more data for task A than task B
• low level features from task A may be helpful for learning B

Multi-task Learning

In multi-task learning, we start off simultaneously trying to have the neural network do several things at the same time. Then each of this task hopefully helps all of the other task.

In multi-task learning, the network can detect more than one labels in a particular input.

Rough guidelines on when multi-task learning makes sense:

• the tasks could benefit from having shared lower-level features
• the data in each of the task helps making the whole dataset bigger
•  usually the neural network is big enough

That’s it for now!