General

Question 1

What is Layer in Neural Network

Answer 1

A layer is a sequence of operators (BatchNorm, Conv etc) plus 1 activation (ReLU, Sigmoid etc)

1 Layer = 1 Activation

Question 2

How and Benefits of Normalize RGB value to defined mean and std values, i.e.

Normalize(
mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])

Answer 2

1. Find the mean and standard variance of the training set

2. use that as the mean and std value to normalize every input image during prediction.

Question 3

What are the usual image Preprocessing in machine learning?

Answer 3

Scale/Resize
Crop
To Tensor
Normalization

Question 4

in ipython, why wall time is smaller than CPU time?

In [52]: %time out = resnet(batch_t)
CPU times: user 619 ms, sys: 0 ns, total: 619 ms
Wall time: 313 ms

Answer 4

On a N-cpu/core computer, if a task is run in parallel on all N cores, then the wall time is a 1/N of the total CPU time

Question 5

Python lists or tuples of numbers are collections of Python objects that are individually
allocated in memory

Answer 5

Thus list and tuples are not efficient storage of matrix. Use numpy array or tensor

Question 6

Use “range indexing” for each of the tensor’s dimensions CREATES A VIEW only (not a copy)

Assume tensor T of shape 3x4x5
T[0:2, 1:4, 2:5].shape == 2x3x3

Answer 6

Slicing creates another tensor that presents a different view of the same underlying data.

I.e. Slicing generates VIEWs, not COPIES

Question 7

Transposing without copying

Answer 7

Matrix can be transposed in “view” by creating a view of tensor with different tensor metadata:

Offset, shape and stride

Question 8

Neural networks exhibit the best training performance when the input data ranges roughly from 0 to 1, or from -1 to 1. Thus transforming input data from any range into [0,1] or [-1, 1] is called Normalization

Why?

Answer 8

Because activation functions are only sensitive/linear around [0,1] or [-1, 1] depends on the activation function used.

In other ranges, the activation function saturates at min/max value when input changes, which does not contribute to the network anymore.