Deep neural networks

Question 1

What are the applications of deep neural networks?

Answer 1

• Medical, stocks, self driving cars, computer vission, medical diagnosis

Question 2

What are neural networks?

Answer 2

• Vaguely mimic the process in which the brain operates, neurons that fire bits of info.
• Simple explanation: There are some red and blue shells in the floor, the neural network draws the line that best separates them. Neural networks given some data in the form of red and blue points it find the line that best separates them

Question 3

boundary line general equation 2d

Answer 3

w1x1+w2x2+b+0

Question 4

What is a percepton?

Answer 4

• Building block of a neural network
• Encoding of the equation into a small graph
• small nodes x1,…, xn, 1 and their edges w1,…,wn, b
• then linear function node calculates the sum of the multiplication of each small node going into it by its edge.
• then the step function returns a 1 or 0 depending if the result is negative or positive linear function
• Perceptrons as Logical Operators

Question 5

Why are neural networks called like that?

Answer 5

• Because it works similarly to a neuron. The percepton gets an input and based on the inputs it returns an output.
• A neuron gets electrical inputs through its dendrites and then decides to emit an electrical pulse or not

Question 6

Perceptron trick?

Answer 6

• Punto (4,5) bias = 1

- w1 - 4, w2 -5, bias - 1

Question 7

Perceptron algorithm

Answer 7

1. Start with random weights: w1, …, wn, b
2. For every misclassified point (x1, …, xn): If prediction = 0:
   - For i = 1 …n
   - Change wi + a xi
   - Change b to b + a
3. For every misclassified point (x1, …, xn): If prediction = 1:
   - For i = 1 …n
   - Change wi - a xi
   - Change b to b - a

Question 8

What is an error function?

Answer 8

• Tells us how far we are from the solution
• For Neural nets the result should be continuos not discrete
• The error function should assign a penalty value to each point. More to the incorrectly classified
• We need to move from discrete predictions to continious

Question 9

What is the new perceptron?

Answer 9

• Replace the step function with the sigmond function.
• 1 / (1 + e ^ (-x))
• The sigmond will give a greater probability to a big number, 0.5 to 0 and very small probabilities to negative numbers

Question 10

How to do multi-class classification?

Answer 10

• We use the softmax function.

Question 11

What is the softmax function?

Answer 11

• It helps to translate a score from a function (Linear for example) to a probability.
• You have scores Z1, …, Zn
• P(class i) = e^Zi/(e^Z1+…+e^Zn)

Question 12

What is cross-entropy?

Answer 12

• It is used for calculating the maximum likelihood of a model.
• -ln(p1)-ln(p2)
• Goal is to minimize the cross-entropy. Better if the number is small
  Formula
  yi = 1 if present on box i
• sum yiln(pi) + (1-yi)ln(1-pi)

Question 13

Multi-class cross-entropy?

Answer 13

• -sum sum yij*ln(pij)

Question 14

Error function

Answer 14

• 1/m*sum(i in m) (1-yi)(1-ln(^yi)) + yiln(^yi)

- m is the number of points

Question 15

Why and how to calculate the gradient descent?

Answer 15

• The negative of delta E (gradient descent) is the direction to move to decrease the most error
• It is the derivative of the error function by each weight and bias

Question 16

How to combine two preceptons?

Answer 16

• Calculate the probability of the first model
• Calculate the probability of the second model
• Add the two probabilities and apply the sigmond equation

Question 17

Layer in a neural network?

Answer 17

• Input layer: The inputs of the model
• Hidden layer: The first models
• Output layer: The combination of the models

Question 18

How to do multiclass classification with a neural network?

Answer 18

• Add more nodes to the output layer that gives a probability for each class

Question 19

What is to train a neural network?

Answer 19

• It means to estimate the most appropriate weights

Question 20

What is feedforward?

Answer 20

• Feedforward is the process neural networks use to turn the input into an output

Question 21

What is backpropagation?

Answer 21

Doing a feedforward operation.
Comparing the output of the model with the desired output.
Calculating the error.
Running the feedforward operation backwards (backpropagation) to spread the error to each of the weights.
Use this to update the weights, and get a better model.
Continue this until we have a model that is good.

Question 22

What is Keras?

Answer 22

• It is a library to code neural networks
• The general idea for this example is that you’ll first load the data, then define the network, and then finally train the network.

Question 23

Simple neural net with Keras

Answer 23

from keras.models import Sequential
from keras.layers.core import Dense, Activation, Flatten

    #Create the Sequential model
    model = Sequential()

    #1st Layer - Add an input layer of 32 nodes
    model.add(Dense, input_dim=32)

    #2nd Layer - Add a fully connected layer of 128 nodes
    model.add(Dense(128))

    #3rd Layer - Add a softmax activation layer
    model.add(Activation('softmax'))

    #4th Layer - Add a fully connected layer
    model.add(Dense(10))

    #5th Layer - Add a Sigmoid activation layer
    model.add(Activation('sigmoid'))

model. compile(loss=”categorical_crossentropy”, optimizer=”adam”, metrics = [‘accuracy’])
model. summary()

model. fit(X, y, nb_epoch=1000, verbose=0)
model. evaluate()

Question 24

What is an epoch?

Answer 24

It is an step in the direction of the gradient to decrease the error

Question 25

Batch and stochastic gradient descent

Answer 25

• To do feedforward and backpropagation to recalculate the weights and improve them split the data into batches and train into those batches. This is more efficient
• This instead of always using all the data each step you do it with each small batch

Question 26

Learning rate rule of thumb

Answer 26

• If your model is not working decrease the learning rate

Question 27

Learning rate decay?

Answer 27

• If you are approaching the best solution decrease learning rate.
• If steep long steeps
• If plain short steeps

Question 28

Why to test a model?

Answer 28

• Train with a set of data and then test with a different set of data will allow the model to avoid overfitting

Question 29

How to determine the number of epochs?

Answer 29

• With a model complexity graph. A good number of epochs is one where the number of training errors and testing errors are low

Question 30

What is regularization?

Answer 30

Is the way to avoid model overfitting. You penalize high weight coefficients. Add to the error function Lamda*(w1^2 … w2^2)

Question 31

what is dropout?

Answer 31

• it is not using some nodes in the training.

Question 32

activation functions?

Answer 32

• Sigmond
• hyperbolic tangent. Similar to sigmond but it is between 1 and -1. It allows bigger steps.
• relu. if x>=0 then x, else 0. Or the max between 1 and 0
• The last activation function needs to be a sigmond, because the output needs to be a probability

Question 33

how to avoid local minimum?

Answer 33

• Restart from different places. This means to change the initial weight values in each restart

Question 34

What is momentum?

Answer 34

• It is a different way to estimate the step.

- The step is a linear combination of the past steps

Question 35

Keras optimizers

Answer 35

SGD
This is Stochastic Gradient Descent. It uses the following parameters: Learning rate, Momentum, Nesterov Momentum

Adam
Adam (Adaptive Moment Estimation) uses a more complicated exponential decay that consists of not just considering the average (first moment), but also the variance (second moment) of the previous steps.

RMSProp
RMSProp (RMS stands for Root Mean Squared Error) decreases the learning rate by dividing it by an exponentially decaying average of squared gradients.

Question 36

Good tips to avoid overfitting

Answer 36

• Increase the batch size. The number of batches represents the number of groups in which the data is divided. It increases the number of iterations, but make it quicker to do each
• Use dropout