Section 3 Evaluation of a classifier

Question 1

Name 3 measures to evaluate how a linear regression model performs

Answer 1

Root mean squared error
Mean absolute error
R^2

Question 2

What is the MAP rule

Answer 2

Popular choice along a continuous set of τ (threshold) [0,1] is 0.5, the “maximum a posteriori (MAP) rule” which assigns the most likely class, i.e. corresponding to the largest probability.

Question 3

When might the MAP rule not be a correct choice to make?

Answer 3

This might not be a sensible choice as it assumes the two classes are balanced.
This is not a sensible choice if different misclassification costs are associated with false positives and false negatives. Ex: Medical applications, where category of interest is rare

Question 4

How to check balance of data

Answer 4

First thing you should do with data is have a look at the proportions of the sample.
If data is imbalanced, skewed then using MAP rule will get a skewed distribution.

Question 5

Explain threshold classifier evaluation

Answer 5

If τ = 0, then every observation is predicted as y_hati = 1
If τ = 1, then every observation is predicted as y_hati = 0

Question 6

Explain meaning of TN, TP,FN,FP

Answer 6

T N – True negatives i.e. Number of 0 correctly classified as 0.
T P – True positives i.e.Number of 1 correctly classified as 1.
F N – False negatives i.e. Number of 1 wrongly classified as 0.
F P – False positives i.e. Number of 0 wrongly classified as 1.

Question 7

What is sensitivity

Answer 7

Sensitivity/recall focuses on the positive cases assessing: P(y_hat=1|y=1) Of those truly positive how many are classified as positive

Question 8

What is recall?

Answer 8

Sensitivity/recall focuses on the positive cases assessing: P(y_hat=1|y=1) Also we can get the false positive rate by 1-sensitivity. Of those truly positive how many are classified as positive

Question 9

What is accuracy

Answer 9

Accuracy would be assessing: P(y_hat=y)

Question 10

What is specificity

Answer 10

Specificity focuses on the negative cases assessing: P(y_hat=0|y=0). Also we can get the false positive rate by 1- specificity.
Of those truly negative how many are classified as negative

Question 11

What is precision

Answer 11

Precision would be assessing: P(y=1|y_hat=1)
Of those who are classified as positive how many are truly positive.

Question 12

Describe the relationship of precision vs recall

Answer 12

Recall decreasing will increase the precision metric.

Question 13

If positive class is rare what metrics would we focus on?

Answer 13

Important case is when a positive class (for example) is rare, we want to focus more on the positive class being predicted well. In this scenario we would want to have a good balance between precision and recall.

Question 14

Express recall in terms of precision

Answer 14

Recall = (Precision*prevalence form the model)/(prevalence from the data)

Question 15

What is the ROC curve

Answer 15

The receiver operating characteristic – ROC curve plots sensitivity (true positive rate) versus specificity (false positive rate).The curve illustrates the diagnostic ability of a binary classifier as the discrimination threshold τ is varied.

Question 16

How can we evaluate a classifier by the ROC curve

Answer 16

How often will a randomly chosen 1 outcome have a higher probability of being predicted to be a 1 outcome than a randomly chosen true 0?
A perfect classifier would have AU-ROC = 1,and the ROC curve pushed to the top left Corner. This would imply large sensitivity and large specificity.
A classifier not better than random guessing would have AU-ROC = 0.5
A common way of choosing τ in relation to the ROC curve is to maximise the sum of sensitivity and specificity - balance true positives and negatives

Question 17

When does ROC curve not work

Answer 17

Sensitivity and specificity, in conjunction with the ROC curve, can work in mild imbalanced situations.
However, because sensitivity and specificity are equally weighted, these metrics and ROC curve can be misleading in some very imbalanced applications and provide an optimistic view.

Question 18

What is the PR cruve

Answer 18

The precision/recall – PR curve plots precision versus recall, as a function of the classification threshold τ.
The area under the PR curve - AU-PR is related to the average precision for varying threshold τ.
The larger the area under the curve, the better is the ability of the classifier at correctly identifying the positive(rare) class.

Question 19

How to evaluate a classifer based on the PR curve

Answer 19

The larger the area under the curve, the better is the ability of the classifier at correctly identifying the positive(rare) class.
A good classifier will have both a high precision and high recall, and the PR curve pushed to the top right corner.
The precision is lower bounded by the prevalence of the positive class - which would mean no better than random guessing .

Question 20

What is the F1 score

Answer 20

A quantity to quantify and compare a classifier’s ability to predict positive cases would be the F1 score. This is the harmonic mean of precision and recall
Interpretation: Model’s balanced ability to both detect positive cases (recall) and be accurate with the cases it detects (precision).
The score 0 ≥ F1 ≥ 1, with F1 = 1 denoting perfection

Question 21

Summarise why we use ROC/ sensitivity+specificity

Answer 21

Mild to none imbalance
“Cost” of false positive vs false negative
Interest also in the negative (0) cases

Question 22

Summarise why we use Precision, recall and F1

Answer 22

Positive cases are rare
Want to maintain low false positive rate
Does not consider (True) negative cases

Question 23

Summarise the reasons for using accuracy measure

Answer 23

Easy to interpret and use
Popular
Not always appropriate

Question 24

What measure is used typically for multinomial logistic regression

Answer 24

Typically simple accuracy is used but maybe also:
Class-specific sensitivity, i.e. the proportion of correctly classified observations for class k.
Class-specific false positive rate, i.e. the proportion of instances incorrectly assigned to class k.

Question 25

Why do we not make model very complex to fit perfectly?

Answer 25

In theory, a model can be made arbitrarily complex, as to perfectly fit the data it was estimated on. High complexity will fit model perfectly to the data but will generalise poorly, while low complexity may not fit the data well. Although alot of metrics will reduce as complexity increases the variance of predictions will increase. The variance of predictions is linked to if you can trust your model in reality out of sample.

Question 26

What happens to variance when overfitting occurs

Answer 26

Although alot of metrics will reduce as complexity increases the variance of predictions will increase. The variance of predictions is linked to if you can trust your model in reality out of sample.

Question 27

Explain why we use test data

Answer 27

The data used to learn the model provides an optimistic view of the predictive performance as it can be made arbitrarily complex and the same data is used to learn the model and assess its predictive performance. The data used to fit the model must not be used to evaluate the predictive performance of a model, only can be used for assessing goodness-of-fit and quality of the model.

Question 28

What does test data look like - define it

Answer 28

Data points not used as target cases in the fitting procedure. The test data set is used to estimate the generalisation error of the fitted model. Hence, these data points are used to test the fitted model. Test data should be either external data with similar characteristics to data used for fitting or should be out of sample and split off from the training data.

Question 29

Define training data

Answer 29

Training set: Data points whose target variable values are used in the model fitting procedure, that is are used to learn the parameters of the model. These data points are employed to train the model.

Question 30

What is a key assumption about test and training data

Answer 30

Crucial assumption is that both training and test data are generated by the same data generating process.

Question 31

What is data leakage and how do we avoid it

Answer 31

Data leakage is contaminating the test data with training data information. When we split data into training and test data we standardize the sets individually to avoid data leakage.

Question 32

In reality how do the loss functions for test and training data compare? Similarly for predictive performance measures

Answer 32

The estimated out-of-sample loss is never smaller than the estimated training loss. Similarly, (in expectation) the estimated out-of-sample predictive performance is not greater than the estimated training performance

Question 33

Define the validation set

Answer 33

Validation set: Data points used to evaluate and compare the models to see which is best. Each model in turn is evaluated on these data. Surrogate of the test data

Question 34

What is the summary of steps for evaluating models

Answer 34

Split the available data into: Training data – Validation data – Test data. Model training: Use the training data to estimate the model parameters. Model validation: Use the validation data to perform model selection. Find model which is maximising validation performance. Model testing: Use the test data to assess the predictive performance of the model and the ability to generalise to unseen inputs in real world problems. Estimate generalised predictive performance.

Question 35

What are resampling methods

Answer 35

Resampling methods involve repeatedly drawing random samples from a dataset and refitting and testing a model of interest on each sample in order to obtain additional information about a model. Resampling is replicated a number of times to account for the sampling variability of the process

Question 36

Explain cross validation

Answer 36

Cross validation is a class of resampling methods that estimate the performance by holding out a subset of the training observations from the fitting process. The model is trained on the “kept” observations (in-sample). The model is applied to those held out observations (out-of-sample) to evaluate the predictive performance. The process is replicated a number of times to account for the sampling variability The estimate of the predictive performance is the average predictive performance computed

Question 37

Explain hold out sample cross validation

Answer 37

The data are randomly split into training and hold-out samples. The model is fitted on the training set, and then the predictive performance of the model is evaluated on the hold-out sample. In the case of a single model the hold-out sample corresponds to the test set. In the case of multiple models, the hold-out sample is split into validation and test set.

Question 38

What are common splits of data for hold out sample cross validation

Answer 38

50% training and 25% validation, 25% test

Question 39

Why do we recombine the validation data into training data after model selection?

Answer 39

we combine the training data back with validation data and retrain the model. Always more data is better. Keep test data completely separate.

Question 40

Explain leave-one-out cross validation

Answer 40

The idea is that only one observation in turn is used for validation, while the remaining are employed for training - we are splitting data in deterministic way The cross-validation estimate of the performance is the average performance over the N iterations This method requires training N models

Question 41

Explain k-fold cross validation

Answer 41

Where Leave-one-out can be computationally expensive and time consuming this may be a better approach. K -fold approach can save time and computational resources. In k-fold cross validation, chunks (folds) of data are used to evaluate the performance of a model instead of single observations. The cross-validation estimate of the performance is the average performance over the K folds

Question 42

how does the size of the data determine which cross validation to use

Answer 42

If a large number of observations is available, usually a simple hold-out procedure works fine - smaller samples should use leave-one-out or k-fold.

Question 43

What are the downsides of each cross validation procedure

Answer 43

The simple hold-out procedure can overestimate the generalization error. Leave-one-out overcomes this issue, as prediction is evaluated on a single observation (in turn), so it tends to not overestimate the error. Also the estimated error is less variable. Not screened by random subsets - but computationally intensive. K-fold cross-validation reduces the computational complexity of leave-one-out, but trading for bias in the estimated error.