Evaluation

Question 1

Training dataset

Answer 1

= attributes + labels

Question 2

Input

Answer 2

set of annotated training instances

Question 3

Output

Answer 3

an accurate estimate of the target function underlying the training instance for unseen/testing instance

Question 4

Inductive Learning Hypothesis

Answer 4

any hypothesis found to approximate/generalize the target function well over a sufficient large training data set will also approximate the target function well over held out/unseen test examples

Question 5

Three interest points in evaluating a classifier

Answer 5

• Overfitting
• Consistency
• Generalization

Question 6

Overfitting

Answer 6

fit the training data set too well and did a bad job in generalizing the concept

Question 7

Consistency

Answer 7

how well the model/classifier perform on the training data; does it flawlessly predict all the labels correctly?

Question 8

Generalization

Answer 8

the opposite of overfitting; how well the classifier generalizes from the training instances to predict the target function

Question 9

Classification Evaluation aims to ?

Answer 9

find evidence of consistency and non-overfitting and evidence that support the idea of inductive learning hypothesis (generalization)

Question 10

Generalization

Answer 10

the proportion of the time the class label is correctly labelled for the test instance

Question 11

A good model should?

Answer 11

fit the training data well and generalise well to unseen data

An overfitting model has poorer generalisation than a model with high training error

Question 12

Learning Curves

Answer 12

% of training data versus testing/test Accuracy

Represent the performance of a fixed learning strategy over different sizes of training data, for a fixed evaluation metric; it can also show how much data need to be used in order to achieve certain degree of accuracy

Allow us to visualise the data trade-off

Question 13

Learning Curves Trend Summary

Answer 13

• Too little training instances are used = poor performance on both training and testing data
• the peak of the training accuracy = overfitting because the model fits too well with the training data but perform poorly on the testing data
• training accuracy starts to drop as the model tries to generalize better on the concept and thus the test accuracy starts to rise (the better job the model does in generalization the narrower the gap between training and test accuracy)

Question 14

Relationship between the size of training data and accuracy

Answer 14

Generally, the more training instances are used the better the accuracy for the testing data sets because there are more examples for the computer to learn/generalize the concept and thus predict better

Question 15

Apparent error rate

Answer 15

the error rate that we got from evaluating the training data set

Question 16

True error rate

Answer 16

the error rate that we got from evaluating the testing data set / real instances

With unlimited samples used as training instances, apparent error rate will eventually become the true error rate

True error rate is almost always much higher than training error rate because of overfitting

Question 17

difference between true error rate & error rate of the test dataset?

Answer 17

The true error rate is just an estimation with a risk of overfitting
• too fit for the development data or
• only have good accuracy for the one test dataset or not the others

Question 18

Why we want to know the true error?

Answer 18

this is how we know how well the model do in generalization

Question 19

Possible evidence of overfitting?

Answer 19

• large gap between training and test accuracy in the learning curve
• Complex decision boundary (which is distorted by noise data)
• Lack of coverage of population in the sample data, due to either
o small number of samples or
o non-randomness in the sample dataset (sampling bias)

Question 20

Bias and Variance

Answer 20

model bias
evaluation bias
sampling bias

model variance
evaluation variance

It’s rather hard to tell evaluation and model bias & variance apart

These are informal definitions and they can not be measured quantitatively.

A biased classifier is guaranteed to be making errors; an unbiased classifier might/might not be making errors

Although high bias and high variance are often bad, but it does not mean low bias and low variance are good. They’re generally desirable, all else equal.

Bias is generally binary (black-or-white) whilst variance is generally relative (to other classifiers)

Question 21

model bias

Answer 21

wrong predictions due to the propensity of the classifier; relates to accuracy

o In term of regression problems:
 bias = the average of errors
 a model is biased if the predictions are systematically higher or lower than the true value
 a model is not biased if the predictions are correctly predicted OR some instances are higher and some are lower than the true values

o	In term of classification problems:
	a model is biased if the class distribution of the prediction is the not same with the test dataset
	a model is not biased if the class distribution of the prediction is the same with the test dataset

Question 22

evaluation bias

Answer 22

over or under estimate the effectiveness of the classifier due to the propensity of the evaluation strategy

o the estimate of effectiveness of a model is systematically too low or too high

Question 23

sampling bias

Answer 23

the training dataset does not fully represent the population ( -> inductive learning hypothesis broken)

Question 24

model variance

Answer 24

o In term of regression problems:
 Compare the average of the squared predictions with the square of the average of predictions
 Not sure how to interpret this

o In term of classification problems:
 a model has low variance when different randomly sampled training datasets lead to similar predictions/model (independent from whether the predictions are correct or not)
 a model has high variance when different randomly sampled training datasets lead to very different predictions/model

Question 25

evaluation variance

Answer 25

o the estimate of the effectiveness of a model changes a lot when we alter the training dataset

Question 26

How to control bias and variance?

Answer 26

 Holdout partition size
o More training data: less model variance, more evaluation variance
o Less training data: the other way around

 Repeated random subsampling & M-fold Cross-validation
o Less variance than holdout

 Stratification
o Less variance than holdout

 Leave-one-out cross validation
o No sampling bias at all
o Lowest bias/variance in general

Question 27

Low Bias Classifier

Answer 27

weighted random classifier, polynomial/RBF kernel SVM

Question 28

Low variance classifier

Answer 28

0-R, naïve Bayes

Question 29

True Positive and True Negative

Answer 29

If predicted and actual result match:
• if both True -> TP
• if both False -> TN

Question 30

False Positive and False Negative

Answer 30

If predicted and actual result don’t match:
• if test = False, reality=True -> False Negative
• if test=True, reality= False -> False Positive

Question 31

Classification Accuracy (ACC)

Answer 31

the proportion of instances for which we have correctly predicted the label

Question 32

Error rate (ER)

Answer 32

the proportion of instances for which we have incorrectly predicted the label

Question 33

Error rate reduction (ERR)

Answer 33

compare the ER of a given method with that of an alternative method (ER0)

Question 34

Metrics related to interesting class only

Answer 34

precision and recall

Question 35

Precision/positive predictive value

Answer 35

when we predict an instance is interesting, how often are we correct?

Question 36

Recall/sensitivity

Answer 36

Out of all the instances that are truly interesting, how many did we correctly predict?

Question 37

why there is a trade-off between precision and recall ?

Answer 37

because it is a direct trade-off between FP and FN

Question 38

What is F-score?

Answer 38

the weighted harmonic mean of precision and recall

Question 39

F-1 score interpretation

Answer 39

When F-1 score = 1 (max F-1 score), perfect precision and recall

When F-1 score = 0 (min F-1 score), worst precision and recall

Question 40

Multi-class classification

Answer 40

There is no uninteresting class in multi-class classification.

So we use Confusion Matrix because of the technical definition of accuracy behaves strangely

Question 41

Precision & Recall for multiple categories problem

Answer 41

• They are calculated per-class
• Micro-averaging: combine all test instances into a single pool
• Macro-averaging: calculate P, R per category and then average (by the number of category)
• Weighted averaging: calculate P, R per class and then average, based on the
• proportion of instances in that class

If there is a small category aka a category that has very few instances compared to other categories, using the micro-averaging method is going to make the small category invisible while the macro-averaging method is going to give equal weight to the small category as other big categories

People often calculate both so that they can see from different perspectives

Problems:
 when to do averaging?
 “?” class

Question 42

Clustering Accuracy

Answer 42

Highest accuracy = fairest

Question 43

Holdout

Answer 43

train a classifier over a fixed training dataset and evaluate it over a fixed held-out test dataset

Each instance is randomly assigned to either the test or training dataset; there is no overlapping data between the two datasets (partitioned)

Question 44

Pros and Cons of holdout

Answer 44

Pros:
• Simple to work with
• High reproducibility (same split ratio)

Cons:
• Trade-off between more training and test data (variance vs. bias)
• Representativeness of the training and test data (something might happen only in the test data but not the training data and the model never get to learn it; high bias because there is a mismatch between the training and test data sets); solution: random subsampling

Question 45

Random Subsampling

Answer 45

perform holdout over multiple iterations, randomly selecting the training and test data while maintaining a fixed size for each dataset on each iteration. Evaluate by taking the average across the iterations.

Question 46

Pros and Cons of random subsampling

Answer 46

Pros:
• Reduce variance and bias over holdout method  more reliable result

Cons:
• Reproducibility (because of the randomness)
• Slower than holdout
• Wrong raining set – test set size might lead to misleading result, like holdout

Question 47

M-fold Cross Validation

Answer 47

Split the data into M equal partitions. Take one partition as the test data and the rest as the training data. Train the system M times and the average performance is computed across the M runs.

M typically = 5 or 10.

Evaluation are calculated based on the entire dataset

Better than random sub-sampling & holdout

Number of folds directly impacts runtime and size of datasets:
 Fewer folds: more instances per partition, more variance in performance estimates
 More folds: fewer instances per partition, less variance but slower

Question 48

Pros and Cons of M-fold CV

Answer 48

Pros:
• Training the system M times instead of N times
• Can measure the stability of the system across different training/test combination
• Very reproducible
• Minimise bias and variance on the estimate of the classifier’s performance

Cons:
• The value of M is subjective so might lead to bias (as training data might be different from the test data)
• The result will not be unique unless we always partition the data identically
• Will get slightly lower accuracy compared to leave-one-out method

Question 49

Leave–One–Out Cross–Validation

Answer 49

 M = N (train on all data except one data point)
 Maximize the training data and mimics the actual testing behaviour (every test instance is independent)
 Too computationally expensive

Question 50

Stratification

Answer 50

the process of rearranging the data as to ensure each fold is a good representative of the whole

Question 51

Inductive Learning Hypothesis

Answer 51

Any hypothesis found to approximate the target function well over (a sufficiently large) training data set will also approximate the target function well over unseen test examples

Question 52

Inductive bias (assumptions must be made about the data to build a model and make predictions):

Answer 52

o Different assumptions will lead to different predictions

o In order to optimize performance, we need some sort of priori knowledge as there is no free lunch in ML

Question 53

Stratification

Answer 53

 Assumes that class distribution of unseen instances will be the same as distribution of seen instances

 When constructing holdout/CV partitions, ensure that training data and test data both have the same class distribution as dataset as a whole

Question 54

Result Comparison

Answer 54

Baseline
Benchmark
Random Baseline

Question 55

Baseline

Answer 55

naïve method which we would expect any reasonably well-developed method to better (aka minimum/dumb and simple)

• Sometimes out-perform the complicated methods
• Valuable in getting a feel of how difficult the classification task is
• In formulating a baseline for a medical task, we need to be sensitive to the importance of positives and negatives in the classification tasks

Question 56

Benchmark

Answer 56

established rival technique which we are pitching our method against (aka reasonable), like past performance

Question 57

Random Baseline

Answer 57

 randomly assign a class to each test instance

 randomly assign a class to each test instance, weighting the class assignment according to class distribution of the training class (if we know the prior probability)

 Zero-R/majority class baseline (not suitable for task that discover needles in the haystack); zero attribute is used; only based on the class labels

 One-R: based on one attribute only; for every value of each attribute, use zero-R, pick the one that has the lowest total error rate
o Pros:
 Easy & simple to understand and implement
 Good results
o Cons:
 Unable to capture attribute interactions
 Bias toward attribute that has a lot off values