L7

Question 1

What are Parameters in the context of machine learning?

Answer 1

Values learned during training

Question 2

What are Hyperparameters?

Answer 2

Set before training (like learning rate, number of neighbors, or regularization parameter C)

Question 3

What is a Decision function?

Answer 3

Takes a dataset as input and gives a decision as output

Question 4

What is the Loss function?

Answer 4

What you are trying to minimize for a single training example to achieve your objective (e.g. square loss)

Question 5

What is a Cost function?

Answer 5

Average of your loss functions over the entire training set (e.g. mean square error)

Question 6

What is a Training set used for?

Answer 6

Learn model parameters

Question 7

What is a Validation set used for?

Answer 7

Tune hyperparameters

Question 8

What is a Test set used for?

Answer 8

Evaluate final model performance

Question 9

Why do we need SVMs?

Answer 9

To find the best line (or hyperplane) possible with the largest margin between classes

Logistic regression draws a line to separate classes (good for linear problems).
But we want a model that finds the best line possible — not just any line.

Question 10

What does SVM stand for?

Answer 10

Support Vector Machine

• supervised learning algorithm used for both classification and regression.

Question 11

What is the main goal of SVM in classification?

Answer 11

To separate classes with the widest possible gap or margin

Question 12

What is the Margin in SVM?

Answer 12

The distance between the decision boundary and the closest data points (support vectors)

SVM wants to maximize this margin.
The decision boundary is a straight line (or hyperplane in higher dimensions).
Goal : learn a boundary that leads to the largest margin (buffer) from points on both sides

Question 13

What are Support Vectors?

Answer 13

The data points closest to the boundary that define the position of the decision boundary

• They “support” or define the position of the decision boundary.
• Only support vectors matter during prediction

Question 14

What is a Hard Margin SVM?

Answer 14

No errors allowed – aims to find a hyperplane that perfectly separates the classes without any misclassification

Fails when there is overlap or noise
Max margin classification – Focus on observations at the edges of the cluster + Use mid point between them as threshold (Maximal Margin)

Question 15

What is a Soft Margin SVM?

Answer 15

Allows some misclassification or overlap and measures how much an instance is allowed to violate the margin

Question 16

What does hyperparameter C control in Soft Margin SVM?

Answer 16

Trade-off between margin size and classification errors

2 contradicting objectives:
making the slack variables as small as possible to reduce the margin violations
Making wT · w as small as possible to increase the margin

Large margin (simpler model)
Fewer classification errors (fit data better)

Question 17

What does a large C mean?

Answer 17

Large C: less tolerant to errors → narrow margin
Small C: more tolerant to errors → wider margin

Question 18

What is the decision function for a new data point in SVM?

Answer 18

If result ≥ 1 → positive class
If ≤ -1 → negative class
If in between → uncertain zone (margin)

The smaller the weight vector 𝜃, the larger the margin

Question 19

What type of problems do SVMs solve?

Answer 19

Convex quadratic optimization problems with linear constraints

Question 20

What is the kernel trick in SVM?

Answer 20

Projects data into a higher-dimensional space where it becomes linearly separable

• When we transform back this line to original plane, it maps to ellipse boundary. These transformations are called kernels.
• As a function of the original features, the linear SVM model is not actually linear anymore.

Question 21

What is a Linear kernel used for?

Answer 21

Fast, simple, when data is already separable

Key parameters: C (regularization)

Question 22

What does the RBF kernel offer?

Answer 22

Very flexible, works on complex data

Key parameters: C, gamma (width of kernel)

Question 23

What is the purpose of the Gamma parameter in RBF kernels?

Answer 23

Controls how far a single point’s influence reaches

Kernels work best for “small” n_samples → Long runtime for “large” datasets (100k samples)
Real power in infinite-dimensional spaces: rbf! → Rbf (Radial Basis Function / Gaussian kernel) is “universal kernel” - can learn (aka overfit) anything

Question 24

What is One-vs-Rest (OvR) in multiclass classification?

Answer 24

One classifier per class vs all others

One classifier per class vs all others
1vs{2, 3, 4}, 2vs{1, 3, 4}, 3vs{1, 2, 4}, 4vs{1, 2, 3}
Class with highest score

Question 25

What is One-vs-One (OvO) in multiclass classification?

Answer 25

One classifier for every pair of classes One classifier for every pair of classes 1v1, 1v2, 1v3, 1v4, 2v3, 2v4, 3v4

Question 26

What is a One-Class SVM (OC-SVM) used for?

Answer 26

Learns from one class, (e.g., only positives) → Desirable if there are many times of negative examples useful for novelty or anomaly detection Learns the "normal" pattern and flags anything different as outlier.

Question 27

What are the advantages of Linear SVMs?

Answer 27

Accuracy, works well on smaller cleaner datasets, can be more efficient

Question 28

What are the disadvantages of Linear SVMs?

Answer 28

Not suited to larger datasets, less effective on noisier datasets

Question 29

What is a key advantage of Kernel SVMs?

Answer 29

Allow for complex decision boundaries, even if the data has only a few features. Work well on low-dimensional and high-dimensional data (i.e., few and many features)

Question 30

What is a key disadvantage of Kernel SVMs?

Answer 30

Do not scale well with the number of samples 1. Do no scale very well with the number of samples. Running an SVM on data with up to 10,000 samples might work well, but working with datasets of size 100,000 or more can become challenging in terms of runtime and memory usage. 2. Require careful preprocessing of the data and tuning of the parameters. 3. SVM models are hard to inspect; it can be difficult to understand why a particular prediction was made, and it might be tricky to explain the model to a nonexpert.

Question 31

What is the role of preprocessing in SVMs?

Answer 31

Requires careful preprocessing of the data and tuning of the parameters

Question 32

What is the meaning of Support Vectors in SVM?

Answer 32

Closest points to the margin

Question 33

What does Margin refer to in SVM?

Answer 33

Distance between boundary and support vectors

Question 34

What is a Hyperplane in SVM?

Answer 34

Decision boundary

Question 35

What does parameter C control in SVM?

Answer 35

Trade-off between margin size and error

Question 36

What is a Kernel in SVM?

Answer 36

Function that transforms data to higher dimensions

Question 37

What does Gamma control in RBF kernels?

Answer 37

Influence of data points in RBF

Question 38

How do you train a margin-based classifier?

Answer 38

- Decision function of SVMs : maximize the margin between the data points and the hyperplane - search for the optimal parameters (𝜃) by finding a solution that: 1. Correctly classifies the training examples 2. Maximizes the margin 3. Can be found through optimisation via projective gradient descent, etc.

Question 39

What is a polynomial kernel?

Answer 39

Adds interaction terms (e.g., x2x^2, xyxy)

Question 40

What is Soft Margin SVM with 2 features?

Answer 40

With 2 Features, the threshold is a line, blue lines are the margins Handling Overlapping Classifications → Adding a square of the dosages - start with data in a relatively low dimension (in this example one dimension dosage in mg) - add higher dimensions (in this example from one to two dimensions) - find a Support Vector Classifier that separates the higher dimensional data into two groups Key parameters: C, degree (d)

Question 41

What is the similarity function of SVM?

Answer 41

add features computed using a similarity function that measures how much each instance resembles a particular landmark