Machine Learning

Question 1

What is learning?

Answer 1

Learning is the acquisition of new information or knowledge or the process of doing so by trial and error.

Question 2

What is Machine Learning?

Answer 2

Machine Learning is the field of study that gives computers the ability to learn without being explicitly programmed.

Question 3

What are the four components of machine learning?

Answer 3

• Dataset S: set of samples generated by some process; either labelled or unlabelled
• Model M: an representation of a input/output relationships intended to model the process that generates S
• Objective Function L: a function that encodes the current performance of M (e.g. loss or reward)
• Algorithm A: the learning algorithm that adjusts M based on S and L

Question 4

What is the cognitive function of Autonomy within a cognitive system?

Answer 4

The ability to dynamically adapt to changes in the environment (e.g. continuous online learning from a live data stream)

Question 5

What is the cognitive function of Perception within a cognitive system?

Answer 5

The ability to learn how to detect and categorize perceptual stimuli (e.g. unsupervised learning of visual features)

Question 6

Why are conventional programming methods not possible in complex dynamic environments?

Answer 6

• The environment is changing continuously
• Dynamics and objects are too complex to be modelled explicitly
• The system itself is subject to change

Question 7

What is a feature?

Answer 7

A feature is an individual measurable property or characteristic of a phenomenon being observed.

They abstract from raw data and represent semantic information in the data.

Question 8

What is feature engineering?

Answer 8

Feature engineering is the process of using domain knowledge to extract features from raw data.

Selected features are grouped into a feature vector.

Designing features is one of the most challenging tasks in machine learning. Manual feature engineering can become extremely complex for real-world datasets.

Question 9

What is the definition of the Machine Learning Task?

Answer 9

Train a model M in a hypothesis space H using a learning algorithm A so that M minimizes loss L for dataset S

Question 10

Why can many real-world tasks not be modelled as pairs of input and desired outputs?

Answer 10

Many real-world tasks have complex dynamics and unknown goal representations.

Question 11

What is the goal of Unsupervised Learning?

Answer 11

To discover structural features in the data set, given solely unlabelled data. Often applied as a pre-processing tool for initial data analysis

Question 12

What is semi-supervised learning?

Answer 12

Semi-supervised learning combines a small amount of labelled data with a large amount of unlabelled data during training. A priori assumptions on input data is required

Question 13

What is a Model?

Answer 13

The model M of a machine learning system encapsulates the outcome of the learning process. The learning algorithm A’s goal is to find an optimal model M* = argmin_(M ∈ H) of L(M).

It is often not possible to find a global minimum; therefore local optima have to suffice.

Question 14

What is the hypothesis space H?

Answer 14

All possible models.

E.g. decision trees, polynomials, neural networks, …

Question 15

What are ensemble methods?

Answer 15

Ensemble methods combine multiple learning algorithms to obtain better predictive performance.

Question 16

What is Boosting?

Answer 16

Boosting is an ensemble method in supervised learning. By combining several weak learners, it is possible to build a strong learner.

Boosting algorithms compute a strong learner incrementally by constructing an ensemble of hypothesis and increasing the weights of incorrectly learned samples. The training of new hypotheses focuses on samples with large weights.

Question 17

What is a weak learner?

Answer 17

A classifier that is just slightly better than randomly guessing.

Question 18

How can the performance of a hypothesis be judged?

Answer 18

By how well it predicts seen data from the training set (“fit”) and unseen data (generalization).

Question 19

What is Occam’s Razor?

Answer 19

Is is desirable to choose a model that is a simple as possible (fewer parameters). “Of two competing theories, the simpler explanation of an entity is to be preferred”

Question 20

What are generative and discriminative models?

Answer 20

• Discriminative models learn the boundary between classes so they can discriminate between different kinds of data instances.
• Based on the posterior probabilities P(y|x)
• Generative models learn the distribution of data so they can generate new data instances.
• Based on the prior probabilities P(x|y); predictions can be computed by applying Bayes’
• compact representations of the training data set that have considerably less parameters than the dataset S

Question 21

How can overfitting be detected?

Answer 21

By training on a training set while continuously assessing the performance and adjusting parameters based on a validation set. Finally, a test set is used to assess the performance of the final model.

Question 22

What is cross-validation?

Answer 22

The dataset is partitioned into k subsets and learned on in k iterations, with a different subset chosen each time. The overall performance corresponds to the averaged performance of the k iterations.

Question 23

How can overfitting be avoided?

Answer 23

• Regularization: include a regularization term in the objective function L that punishes complexity
• More training data: increase complexity of the dataset
• Dataset augmentation: transform training samples (add noise, shifts, rotations, etc.)

Question 24

How is an artificial neuron defined?

Answer 24

By its synaptic weights and an activation function.

Question 25

What is the output n(x) of a neuron n?

Answer 25

n(x) = A(w₁x₁ + … + wₙxₙ+ b)

Question 26

What is a perceptron?

Answer 26

A perceptron is a linear classifier that is based on a single neuron with a digital threshold function. The perceptron criterion punishes only incorrectly classified samples.

Invented by Frank Rosenblatt in 1958.

Question 27

What does the perceptron convergence theorem state?

Answer 27

If a solution exists (i.e. the data set is linearly separable), then the perceptron learning algorithm finds a solution within a finite number of steps.

Question 28

What is the desired output of a regression task?

Answer 28

A regressor h that predicts yₙ for a given xₙ with minimum loss L.

Question 29

What is the difference between interpolation and regression?

Answer 29

The interpolation function f must be consistent with the samples S: f(xᵢ)=yᵢ

In contrast, in regression the hypothesis h should minimize L and generalize well to new samples.

Question 30

What does Maximum Likelihood mean?

Answer 30

For a given dataset, we want to optimize the weights so that they maximize the likelihood of the observations (target values yₙ) for the input xₙ.

Question 31

What is the 1-of-K coding scheme?

Answer 31

𝐾 classes are represented using a 1-of-𝐾 coding scheme where

𝐶ₗ = (0,0, … , 1, … , 0) ∈ {0,1}ᴷ

Question 32

What are generalized linear models for classification?

Answer 32

y_cls(xₙ, w) = f(y_reg(xₙ, w))

A generalized linear model adds an nonlinear activation function f to a linear regression task.

Subspaces y_cls(xₙ, w) = c for a constant c are called decision surfaces.

Question 33

What is a linear discriminant function?

Answer 33

A linear discriminant function divides the feature space by a hyperplane decision surface.

Question 34

What is the Heaviside step function?

Answer 34

f(x)=0 if n<0, 1 if n≥0

The resulting decision surface is wᵀxₙ = 0

Question 35

What is the One-Versus-the-Rest Classifier?

Answer 35

We separate K classes with K-1 binary discriminant functions. Every discriminant function separates one class from all others.

Some regions may be classified ambiguously!

Question 36

What is the One-Versus-One Classifier?

Answer 36

We separate K classes pairwise with K(K-1)/2 binary discriminant functions. The class of the data sample is assigned by a majority vote.

Some regions may be classified ambiguously!

Question 37

What is the problem with one-versus-the-rest and one-versus-one classifiers? How can that be fixed?

Answer 37

The input space regions defined by the discriminant functions may overlap or do not cover the whole input space.

The solution is to determine a partition of the input space. Every class is assigned a linear function. A sample is assigned to a class if the function returns a larger value than all other classes.

Question 38

What is the problem with drawing class identifiers from ordered sets? How can that be fixed?

Answer 38

Ordered sets implicitly impose a non-existing order on the classes that might be captured by the learned model.

Use one-hot encoding (0,0,…,1,0,..)

Question 39

What are the issues with Least Squares Linear Classification?

Answer 39

Least squares classification is sensitive to outliers.

Also, if the input is linearly separable, the algorithm computes one of infinitely many solutions.

Question 40

What are Support Vector Machines?

Answer 40

SVMs minimize the generalization error by computing a hyperplane that maximizes the margin of the classifier.

Only support vectors define the decision boundary of the SVM and must be saved.

In general, the optimization function includes slack variables to allow for training samples that lie inside the margin (-> better generalization performance)

Question 41

What was ADALINE?

Answer 41

ADALINE was an early (1960) single-layer artificial neural network with multiple nodes, where each node accepts multiple inputs and generates one output.

Question 42

What was the AI winter?

Answer 42

The AI winter was period of reduced funding and interest in artificial intelligence research. It began in 1969 with the proof that the perceptron only works for simple data sets that are linearly separable.

The AI winter ended with the development of new neural network architectures and the backpropagation algorithm.

Question 43

What is the universal approximation theorem?

Answer 43

A neural network with only a single hidden layer can approximate any function
arbitrarily well.

Question 44

How does backpropagation work?

Answer 44

Backpropagation optimizes the loss by varying the weights based on the chain rule.

Question 45

What is SIFT?

Answer 45

The scale-invariant feature transform (SIFT) is a feature detection algorithm in computer vision to detect and describe local features in images.

It was published by David Lowe in 1999.

The SIFT features are designed to be invariant to scale, translation and partially invariant to illumination changes and projections. Specific neurons in the inferior temporal cortex share similarities with the SIFT features.

Question 46

What is a Deep Neural Network?

Answer 46

A deep neural network (DNN) is an artificial neural network (ANN) with multiple (hidden) layers between the input and output layers.

Question 47

What is the advantage of deep networks?

Answer 47

Deep neural networks can learn suitable feature space representations automatically.

Layers in deep neural networks correspond to features at different levels of abstraction.

Question 48

What do the “convolutions” in CNNs imply?

Answer 48

Convolutions are filters on discrete (=sampled) signals (images, audio, sensor data etc).

Discrete convolutions can be achieved with kernels.

The synaptic connectivity pattern of convolutional neural networks implements a neural version of the discrete convolution.

Question 49

What are convolutional kernels?

Answer 49

Matrices that define a signal filter. The entries of the kernel determine the influence of neighboring data points (e.g. pixels) of a sample.

The filtered data sample is computed by sliding the kernel along the input sample.

Kernels encode position-invariant features.

Question 50

What are Gabor filters?

Answer 50

Gabor filters can detect image content with a specific frequency and direction.

Deep neural networks typically learn features similar to Gabor filters in the first layer.

Question 51

What was LeNet-5 (1998)?

Answer 51

A CNN for digit recognition

32 x 32 pixel input
60,000 trainable parameters

Question 52

What was AlexNet (2012)?

Answer 52

First modern deep CNN.

• accelerated training on GPUs
• outperformed all competitors in 2012

Question 53

What is weight sharing?

Answer 53

Weight sharing is a way to reduce the number of parameters (e.g. weights of a filter) while allowing for more robust feature detection.

Question 54

What is max-pooling?

Answer 54

Max-pooling is a sample-based discretization process. The objective is to down-sample an input image, reducing its dimensionality and allowing for assumptions to be made about features contained in the sub-regions binned.

• reduces number of weights
• makes network more robust against translations in the input image

Question 55

How can CNNs benefit from the Fourier transform?

Answer 55

Convolutions are computationally expensive.

When taking the Fourier transform (converting inputs and kernels into Fourier space), convolutions become simple multiplications: ℱ(𝑓∗𝑠) = ℱ(𝑓) ∙ ℱ(𝑠)

The challenge lies in keeping the computational cost for the FT low.

Question 56

What are common simplifications in deep neural networks compared to biological neurons?

Answer 56

• Single biological neurons can compute complex functions that would require several artificial neurons
• The brain processes different streams of information in parallel and in different brain regions. Different data streams a fused into a coherent perception.
• Information processing in the brain is more robust and versatile (data fusion, one-shot learning, memory etc.).
• Deep neural networks suffer from catastrophic forgetting. Performance on a learned task decreases drastically as soon as a new task is trained.

example:
- adversarial patches (techniques to fool ML models)

Question 57

What is a capsule neural network (Hinton, 2017)?

Answer 57

In fully connected neural networks, all neurons process the complete input image, it is therefore not possible to separate the input into different aspects.

A Capsule Neural Network is a ANN that can be used to better model hierarchical relationships. The approach is an attempt to more closely mimic biological neural organization.

New type of neuron: the capsule. Encodes not only a single feature but also adds meta information on its instantiation.

The output of a capsule is a vector that indicates whether the visual feature represented by the capsule is present and how it differs from a prototype instance (pose, transformation, lighting etc.)

Lower-level capsules connect to higher-level capsules with a state that is in agreement with the current output.

➔ difficult to train!

Question 58

What is an advantage of SNNs?

Answer 58

Energy efficiency

Question 59

What are the three basic machine learning paradigms?

Answer 59

• supervised learning
• unsupervised learning
• reinforcement learning