Evaluate Classifiers

Question 1

Why would you need evaluate classifiers?

Answer 1

To help choose optimal method and parameters

Question 2

What are 3 causes to overfitting?

Answer 2

1. Too many variables
2. Excessive model complexity
3. Data leakage

Question 3

What is the consequence of launching an overfit model?

Answer 3

Deployed model will not generalized

Question 4

What is the formula for accuracy rate?

Answer 4

Accuracy rate = (# of correct classification) / (# of records in datatset)

Question 5

when is accuracy alone is not a sufficient metric?

Answer 5

For imbalance classification problems

Question 6

Along the lines of confusion matrix, what is the formula for accuracy?

Answer 6

Accuracy = (TP+TN) / (P + N)

Question 7

What is the 2 step process for cutoff value for classification?

Answer 7

1. Compute probability of belonging to positive class
2. Compare cutoff value and classify

Question 8

Along the lines of confusion matrix, what is the formula for precision?

Answer 8

Precision =
(TP) / (TP + FP)

Question 9

Along the lines of confusion matrix, what is the formula for False Discovery?

Answer 9

FDR = 1 - Precision

Question 10

Along the lines of confusion matrix, what is the formula for False Omission Rate?

Answer 10

FOR
= (FN) / (TN + FN)

Question 11

Along the lines of confusion matrix, what is the formula for Recall?

Answer 11

Recall = (TP) / (TP + FN)

Question 12

Along the lines of confusion matrix, what is the formula for False Negative Rate?

Answer 12

False Negative Rate = 1 - Recall

Question 13

Along the lines of confusion matrix, what is the formula for false positive rate?

Answer 13

FPR = (FP) / (FP +TN)

Question 14

What does accuracy measure overall?

Answer 14

Correctness

Question 15

What type of data is accuracy good to use with?

Answer 15

Balanced datasets

Question 16

When is accuracy misleading?

Answer 16

When working with imbalance classes

Question 17

What does precision focus on?

Answer 17

Positive prediction

Question 18

When false positives are costly, what is a good metric to use?

Answer 18

Precision

Question 19

What does the ROC Curve & AUC evaluate?

Answer 19

Model performance at different thresholds

Question 20

What does a higher AUC mean?

Answer 20

Better discrimination between classes

Question 21

What does the ROC curve and AUC help find?

Answer 21

Optimal balance between true positive and false positive

Question 22

What does the lift & Gain chart help asses?

Answer 22

How well the model ranks and prioritizes high value cases

Question 23

What does the lift show?

Answer 23

Improvement over random selection

Question 24

What does the gains help visualize?

Answer 24

How well the model capturees true positives early

Question 25

Which evaluation metric is good for fraud detection, spam filter?

Answer 25

Precision

Question 26

Which evaluation is good for medical diagnosis and security alerts?

Answer 26

Recall