08_neural networks

Question 1

How do kNN, linear models and tree-based models really learn?

Answer 1

not iteratively

knn: computed distances and compares distribution of unseen data points with distribution of seen data points

linear: fitted to seen data based on the task

tree-based: identify and memorize patterns relevant to the task

Question 2

With which three components does the human brain work?

Answer 2

Neurons (nerve cells)

Dendrites (connects neurons)

Axons (long-distance connections)

–> neurons are inter-connected forming a dense network

Question 3

How is information passed through neurons in the human brain?

Answer 3

through electrical signals

connected neurons absorb the incoming signals and process them. some of them will fire, but not all.

–> cascade of signals

Question 4

What do we need for neural networks to represent the deep cascade of the layers of neurons in a human brain?

Answer 4

input data, which is processed in its hidden layers, and generates output data

Question 5

What is a fully connected network?

Answer 5

a neural network where each neuron is connected to all neurons in the previous layers and all the neurons in the following layer

Question 6

How can a fully connected neural network be characterized?

Answer 6

• number of layers (depth)
• number of neurons in each layer
• number of input variables (= number of neurons in the first layer)
• number of output variables (= number of neurons in the final layer)

Question 7

How does a fully connected neural network work?

Answer 7

• vectorial input data provided to the network, one value per neuron in the input layer
• all inputs are seen by each neuron in the underlying layer
• each neuron will process the incoming info, firing (1) under some conditions and (0) otherwise
• repeat
• output is generated in the final layer

Question 8

How does a neural network act in general terms?

Answer 8

acts as a function approximator

• any mathematical function can be approcimated

Question 9

Can we implement artificial neural networks to learn specific tasks?

Answer 9

through connectionism, everything is connected with everything

Question 10

What are two problems we have to solve before we can implement artificial neural networks?

Answer 10

1) how to implement neurons?

2) how to train the network?

Question 11

How does a general neuron work?

Answer 11

number of inputs might differ from number of outputs - what is the function?

takes in a vector of values, processes them and returns a binary signal based on its learned behavior, which is then passed on to all neurons in the following layer

Question 12

What is part of the function of a perceptron?

Answer 12

input variable x,
weight w
bias value b

–> if the resulting value is greater zero, perceptron neuron fires, otherwise not

step function is called activation function: introduces non-linearity into the output of the perceptron

Question 13

What can a single perceptron be considered as?

Answer 13

a linear classifier

Question 14

How do we train a perceptron?

Answer 14

perceptron learning rule, weights are adjusted by a step size that is called the LEARNING RATE

by iteratively running this algorithm over training data multiple times, weights can be learned so that the model perform properly

Question 15

What is a major limitation of individual perceptrons?

Answer 15

inability to reproduce a logical exclusive-or (XOR) function!

• bc are simply linear functions

multi-layer perceptrons concatenate layers of perceptrons, which makes them much more powerful

Question 16

What does MLP stand for?

Answer 16

multi-layer perceptron

Question 17

What are MLPs?

Answer 17

simple feed-forward neural networks (info traverses graph in only one direction

• fully-connected
• can learn more complex relations from data than single perceptrons, each layer adds NON-LINEARITIES that increase the model’s capacity
• modern MLPs utilize additional layers and other non-linear activation functions that support the learning process

Question 18

What is the function behind a neuron?

Answer 18

x * w + b > 0

Question 19

What do artificial neurons compute?

Answer 19

dot-product between input vectors and learned weights
and produce an output signal that propagates through all deep layers

Question 20

What is a perceptron?

Answer 20

simple artificial neuron that produces a binary output

Question 21

What is a multi-layer perceptron?

Answer 21

an early fully-connected neural network

Question 22

What does an activation function do?

Answer 22

defines when a neuron “fires”

Non-linearity increases in the model’s capacity

Question 23

What is a simple step function?

Answer 23

g(x) = geschwungene Klammer entweder 1 if <condition> or 0 else</condition>

to define whether a neuron fires or not

Question 24

What are advantages and disadvantages of the step function?

Answer 24

+ simple to implement

+ computationally inexpensive

• only binary (discrete) output
• no gradient

Question 25

What is the sigmoid function?

Answer 25

o(x) = exp(x) / 1 + exp(x)

Question 26

What are advantages and disadvantages of the sigmoid function?

Answer 26

+ continuous non-linear function

+ gradient defined

• asymmetric output value range [0, 1]
• computationally expensive

Question 27

What is the tanh function?

Answer 27

tanh (x) = sinh (x) / cosh (x)

Question 28

What are advantages and disadvantages for the tanh function?

Answer 28

+ continuous non-linear function

+ gradient defined

• symmetric output value range [-1, 1]
• computationally expensive

Question 29

What is the ReLu function?

Answer 29

rectified linear unit function

ReLU(x) = geschwungene Klammer x if x>0; = else

Question 30

What are advantages of the ReLU function?

Answer 30

+ continuous non-linear function

+ gradient defined, and simple to compute

+ computationally inexpensive

Question 31

Why is it important for the activation function to be differentiable?

Answer 31

we need the gradient to be computable.
therefore, the step function is not a good choice as it has no gradient

Question 32

Why is the ReLU used most often?

Answer 32

Sigmoid, Tanh and ReLU roughly lead to similar results, but the ReLU is computationally the most efficient

Question 33

What should a good activation function be?

Answer 33

continuously differentiable

non-linear

computationally inexpensive

Question 34

What enables deep neural networks to learn complex tasks?

Answer 34

the non-linearity of activation functions

Question 35

What is the Least squares fitting in linear regression?

Answer 35

a convex optimization problem:
there is only one solution to the problem and it is per definition the best solution

Question 36

How do we modify the neural networks weights to reduce the loss?

Answer 36

• random changes (possible but not very goal-oriented)
• backpropagation (we check for every single weights how changing it would affect the loss)

Question 37

How can we modify each individual weight parameter?

Answer 37

based on computed gradients

wi = wi - alpha upsidedowntriangle wi

Question 38

What is a learning rate?

Answer 38

alpha (definition of step size for the modifications to the weights)

Question 39

What is stochastic gradient descent?

Answer 39

iterative process, depends on the random selection of mini-batches

following the gradients in the weight space to the lowest loss value

–> allows us to find the minimum of the loss in an iterative process

Question 40

What happens if we use a small learning rate?

Answer 40

it will take a long time to reach the global minimum, we could also possibly get stuck in a local minimum

Question 41

What happens if we use a large learning rate?

Answer 41

it is possible that we miss the global minimum,

also convergence is unlikely

Question 42

How do neural networks learn?

Answer 42

learn patterns from data to perform specific tasks

early layers extract low-level signals with spatial significance

later layers interpret these signals and provide semantic significance

–> end-to-end learning

Question 43

What does Stochastic gradient descent (SGD) do?

Answer 43

it uses the gradients computed with backpropagation to update network weight parameters iteratively to reduce the model’s loss

Question 44

What is key to a meaningful training process in neural networks?

Answer 44

ability to compute the gradient of the loss function
with respect to every single network weight parameter

this is achieved through a process called backpropagation

Question 45

What is the neural network training pipeline?

Answer 45

1) sample batch (input data x and target data y) from training dataset

1 epoch:
- evaluate model on batch input data (prediction) in forward pass

• compute loss on prediction and target y
• compute weight gradients with backprop.
• modify weights based on gradients and learning rate
• repeat for all batches

2) repeat for a number of epochs, monitor training and validation loss + metrics

3) stop before overfitting sets in

Question 46

What do you see in the curves of the training and the validation loss in well-trained neural network models?

Answer 46

If the validation loss sinks less fast than the training loss but still does not go up after some iterations, the model is well-trained