chapter 5

Question 1

why using DL framework?

Answer 1

Because we do not need to implement all these new techniques from scratch in our neural network, but we focus on the big picture of solving real-world problems.

Question 2

examples of DL framework

Answer 2

• TensorFlow
• PyTorch

Question 3

steps foe using DL framework

Answer 3

1- write our program as a Python program.
2- import the framework of choice as
a library.
3 - use DL functions from the framework that fit in our program.

Question 4

what are the important functions we use to build our neural networks?

Answer 4

• keras.Sequential: to build the network.
• keras.layers.Flatten: to handle our inputs.
• keras.layers.Dense: to handle the hidden layers, number or nurons, and the activation function.
• keras.optimizer.SGD: to apply the gradient descent and the learning rate.
• compile: prepare the model for training.
• fit: to start the trainig.

Question 5

Epochs

Answer 5

refers to one cycle through the full training dataset.

Question 6

Batch Size

Answer 6

The number of training samples used in one iteration.

Question 7

Saturated neurons

Answer 7

when the neuron be insensitive to input changes because their derivative is 0 in the saturated region, that can cause learning to stop completely.

Question 8

vanishing gradient problem

Answer 8

problem where the backpropagated error is 0 and the weights are not adjusted. Saturated neuron is one of the causes.

Question 9

How to avoid Saturated neurons?

Answer 9

Three common techniques:
- weight initialization.
- input standardization.
- batch normalization.

Question 10

weight initialization.

Answer 10

The way to ensure that our neurons are not saturated to begin with, so, if a neuron has a large number of inputs, then we want to initialize the weights to a smaller value to have a reasonable probability of still keeping the input to the activation function close to 0 to avoid saturation.

Question 11

weight initialization strategies

Answer 11

• Glorot initialization: recommended for tanh- and sigmoid-based neurons.
• He initialization: recommended for ReLU-based
  neurons.

Question 12

Input standardization.

Answer 12

Standardizing the input data to be centered around 0 and with most values close to 0, By subtracting the mean and dividing by the standard deviation.

Question 13

Batch normalization

Answer 13

The idea is to normalize values inside of the network as well and thereby prevent hidden neurons from becoming saturated.

Question 14

The two strategies to apply batch normalization.

Answer 14

• apply the normalization before the activation function.
• apply the normalization after the activation function.

Question 15

Examples of activation functions to avoid saturated neurons.

Answer 15

• ReLU.
• leaky ReLU.
• maxout.
• elu.
• softPlus.

Question 16

what are the variations on gradient descent that help in faster learning?

Answer 16

1- Momentum.
2-Adaptive learning rate.
3- Adaptive moments.

Question 17

Gradients exploding

Answer 17

where the gradient becomes too big in
some point, causing a huge step size.

Question 18

Gradient clipping

Answer 18

is a technique to avoid exploding gradients
by simply not allowing overly large values of the gradient in the weight update
step.

Question 19

How to avoid overfitting?

Answer 19

split the dataset to :
1- training dataset»»»» to train the model.
2-validating dataset»»»» to tune hyperparameters.
3- testing dataset»»»» to test the model.