Model Training, Tuning and Evaluation [up to RNNs]

Question 1

What is an activation function?

Answer 1

The function inside a neuron that takes in the inputs and uses that to figure out the output.

Question 2

What is a rectified linear unit activation function (ReLU)?

Answer 2

An activation function where you have a linear function above 0 and below 0 it just outputs whatever was input.
This is easy and fast to compute.

Question 3

What is a parametric ReLU activation function?

Answer 3

The same as ReLU, except of just outputting the inputs on the back half the slope is learned through back-propagation.
Complicated.

Question 4

What is a Swish activation function?

Answer 4

Developed by Google. Works very well for deep neural networks.

Question 5

What activation functions do RNNs tend to use?

Answer 5

Non-linear activation functions w/ TanH

Question 6

What are convolutional neural networks mainly used for?

Answer 6

Image analysis

Question 7

What does feature location invariant mean? And which type of neural network is it?

Answer 7

Means it doesn’t matter where within an image the key object is. Convolutional neural networks.

Question 8

How does a convolutional neural network work?

Answer 8

Takes a source image, breaks it into chunks called convolutions. Then slowly layers these convolutions and increases the complexity processed.
For example, start with lines, then with shapes, then recognising and assembling these shapes, then recognising objects, etc.

Question 9

What is something important to note about the source data for CNNs?

Answer 9

It must be of the appropriate dimensions - width x length x colour channels.