Lec 5 | Neural Networks

Question 1

It is inspired by neuroscience

Answer 1

AI Neural Networks

Question 2

• These are connected to and can receive and send electrical signals from other ____________ .
• They process input signals and can be activated which will send its electrical signal forward.
• A unit connected to other units

Answer 2

Neuron

Question 3

• It is mathematical model for learning inspired by biological neural network
• It models mathematical functions that map inputs to outputs based on the structure and parameters of the network.
• Allows for learning the network’s parameters based on data

Answer 3

Artificial Neural Network

Question 4

w₀ is a constant, also called [???] , modifying the value of the whole expression

Answer 4

Bias

Question 5

Activation Functions

gives 0 before a certain threshold is reached and 1 after the threshold is reached.

Answer 5

Step Function

Question 6

Activation Functions

Give the formula for the step function

Answer 6

g(x) = 1 if x ≥ 0, else 0

Question 7

Activation Functions

gives as output any real number from 0 to 1, thus expressing graded confidence in its judgment

Answer 7

Logistic Function/Sigmoid

Question 8

Activation Functions

What is the formula for the logistic function/sigmoid?

Answer 8

g(x) = e^x / ((e^x)+1)

Question 9

Activation functions

Allows the output to be any positive value. If the value is negative, it sets it to 0.

Answer 9

Rectified Linear Unit (ReLU)

Question 10

Activation Functions

Formula for ReLU?

Answer 10

g(x) = max(0, x)

Question 11

It is an algorithm for minimizing loss when training neural networks

Answer 11

Gradient Descent

Question 12

PSEUDOCODE

Gradient Descent

Answer 12

Start with a random choice of weights
Repeat:
     Calculate the gradient based on all data points:  that will lead to decreasing loss. 
     Update weights according to the gradient.

Question 13

What is a drawback or a problem of the gradient descent and how do you solve or minimize the problem?

Answer 13

• It requires to calculate the gradient based on all data points. It is computationally costly
• Use Stochastic Gradient Descent or Mini-Batch Gradient Descent to minimize the problem.

Question 14

The gradient is calculated based on one point chosen at random.

Answer 14

Stochastic Gradient Descent

Question 15

PSEUDOCODE

Stochastic Gradient Descent

Answer 15

Start with a random choice of weights
Repeat:
     Calculate the gradient based on one data point:  that will lead to decreasing loss. 
     Update weights according to the gradient.

Question 16

What is a drawback or problem with Stochastic Gradient Descent and how can it be solve?

Answer 16

It can be inacurrate. A way to solve this is by using Mini-Batch Gradient Descent.

Question 17

• This computes the gradient based on on a few points selected at random
• Finds a compromise between computation cost and accuracy

Answer 17

Mini-batch Gradient Descent

Question 18

Pseudocode

Mini-batch Gradient Descent

Answer 18

Start with a random choice of weights
Repeat:
    Calculate the gradient based on one small batch:  that will lead to decreasing loss. 
    Update weights according to the gradient.

Question 19

Main takeaway for Gradient Descents?

Answer 19

None of these solutions is perfect, and different solutions might be employed in different situations.

Question 20

• Only capable of learning linearly separable decision boundary.
• Uses a straight line to separate data
• It could classify an input to be one type or another

Answer 20

Perceptron

Question 21

Some data are not linearly separatable. What do we do use for data that are non-linearly separable?

Answer 21

multilayer neural networks

Question 22

an artificial neural network with an input layer, an output layer, and at least one hidden layer.

Answer 22

Multilayer Neural Networks

Question 23

It processes weighted inputs. It receives weights, performs an action on it and passes outputs to the next layer, until the output layer(final layer ?) is reached. It enables modeling of non-linear data.

Answer 23

Hidden Layer

Question 24

It is the main algorithm used for training neural networks with hidden layers.

Answer 24

Backpropagation

Question 25

How does backpropagation work?

Answer 25

It does so by starting with the errors in the output units, calculating the gradient descent for the weights of the previous layer, and repeating the process until the input layer is reached.

Question 26

# PSEUDOCODE Backpropagation

Answer 26

``` * Calculate error for output layer * For each layer, starting with output layer and moving inwards towards earliest hidden layer: * Propagate error back one layer. In other words, the current layer that’s being considered sends the errors to the preceding layer. * Update weights. ```

Question 27

Can you extend the backpropagation algorithm?

Answer 27

This can be extended to any number of hidden layers, creating deep neural networks, which are neural networks that have more than one hidden layer.

Question 28

neural network with multiple hidden layers

Answer 28

Deep Neural Network

Question 29

It is the danger of modeling the training data too closely, thus failing to generalize to new data.

Answer 29

Overfitting

Question 30

How to combat overfitting?

Answer 30

Use Dropout

Question 31

The temporary removing of units — selected at random — from a neural network to prevent over-reliance on certain units. Throughout training, the neural network will assume different forms, each time dropping some other units and then using them again.

Answer 31

Dropout

Question 32

A python library that has an an implementation for neural networks using the backpropagation algorithm

Answer 32

TensorFlow

Question 33

It encompasses the different computational methods for analyzing and understanding digital images, and it is often achieved using neural networks.

Answer 33

Computer Vision

Question 34

What do images consist of?

Answer 34

pixels with RGB (3 values from 0-255)

Question 35

What are the drawbacks of using Computer Vision?

Answer 35

1. The breaking down of the image into pixels and the values of their colors, we can’t use the structure of the image as an aid. 2. The sheer number of inputs is very big, which means that we will have to calculate a lot of weights.

Question 36

Applys a filter that adds each pixel value of an image to its neighbors, weighted according to a kernel matrix. Doing so alters the image and can help the neural network process it.

Answer 36

Image Convolution

Question 37

A drawback of Image Convolution?

Answer 37

It is computationally expensive due to the number of pixels that serve as input to the neural network..

Question 38

It reduces the size of an input by sampling from regions in the input

Answer 38

Pooling

Question 39

Pooling by choosing the maximum value in each region

Answer 39

Max-pooling

Question 40

Neural networks that use convolution, usually for analyzing images

Answer 40

Convolutional Neural Network

Question 41

Explain how Convolution Neural Networks work.

Answer 41

starts by applying filters that can help distill some features of the image using different kernels. These filters can be improved in the same way as other weights in the neural network, by adjusting their kernels based on the error of the output. Then, the resulting images are pooled, after which the pixels are fed to a traditional neural network as inputs

Question 42

Give a benefit of Convolutional Networks

Answer 42

One of the benefits of these processes is that, by convoluting and pooling, the neural network becomes less sensitive to variation. That is, if the same picture is taken from slightly different angles, the input for convolutional neural network will be similar, whereas, without convolution and pooling, the input from each image would be vastly different.

Question 43

* Neural network that has connections only in one direction * An input data is provided to the network, which eventually produces some output.

Answer 43

Feed-forward neural Network

Question 44

What is a limitation of the Feed-forward Neural Network

Answer 44

Input needs to be in a fixed shape/fixed number of neurons and has a fixed number of output

Question 45

It consists of a non-linear structure, where the network uses its own output as input.

Answer 45

Recurrent Neural Network

Question 46

Explain the difference between Recurrent Neural Networks and Feed-Forward Neural Networks

Answer 46

Feed-Forward Neural Network * Uses input to get output * Incapable of varying the number of outputs Recurrent Neural Network * Uses output as input * Capable of varying the number of outputs * Helpful in cases where the network deals with sequences and not a single individual object

Question 47

# CS50 QUIZ Consider the below neural network, where we set: * w0 = -5 * w1 = 2 * w2 = -1 and * w3 = 3. * x1, x2, and x3 represent input neurons, and y represents the output neuron. What value will this network compute for y given inputs x1 = 3, x2 = 2, and x3 = 4 if we use a step activation function? What if we use a ReLU activation function? * 0 for step activation function, 0 for ReLU activation function * 0 for step activation function, 1 for ReLU activation function * 1 for step activation function, 0 for ReLU activation function * 1 for step activation function, 1 for ReLU activation function * 1 for step activation function, 11 for ReLU activation function * 1 for step activation function, 16 for ReLU activation function * 11 for step activation function, 11 for ReLU activation function * 16 for step activation function, 16 for ReLU activation function

Answer 47

1 for step activation function, 11 for ReLU activation function

Question 48

# CS50 QUIZ How many total weights (including biases) will there be for a fully connected neural network with a single input layer with 3 units, a single hidden layer with 5 units, and a single output layer with 4 units?

Answer 48

44

Question 49

# CS50 QUIZ Consider a recurrent neural network that listens to a audio speech sample, and classifies it according to whose voice it is. What network architecture is the best fit for this problem? * One-to-one (single input, single output) * Many-to-one (multiple inputs, single output) * One-to-many (single input, multiple outputs) * Many-to-many (multiple inputs, multiple outputs)

Answer 49

Many-to-one (multiple inputs, single output)

Question 50

# CS50 QUIZ Consider a 4x4 grayscale image with the following pixel values. ``` 2 4 6 8 16 14 12 10 18 20 22 24 32 30 28 26 ``` What would be the result of applying a 2x2 max-pool to the original image? * [[16, 12], [32, 28]] * [[16, 14], [32, 30]] * [[22, 24], [32, 30]] * [[14, 12], [30, 28]] * [[16, 14], [22, 24]] * [[16, 12], [32, 30]] ## Footnote Answers are formatted as a matrix [[a, b], [c, d]] where [a, b] is the first row and [c, d] is the second row.)

Answer 50

[[16, 12], [32, 28]]