# ML Metrics Flashcards

Offline metrics for classification models

Precision, recall, F1 score, accuracy, ROC-AUC, PR-AUC,

confusion matrix

Offline metrics for regression

Mean squared error (MSE)

MAE

RMSE

Offline metrics for ranking system

- MRR
- mAP
- nDCG

Online metric for ad click prediction

- Click through rate
- Ad revenue

Online metric for harmful content detection

- Number of reports
- Actioned reports

Online metric for video recommendations

- Click through rate
- Total watch time
- Number of completed videos

Types of Loss Functions

Mean squared error

Categorical cross-entropy loss

Binary cross-entropy loss

Mean squared error

- Measures the difference between the predicted output and the true output
- Used to optimize the model parameters during training
- what we’re trying to minimize when we train a model

precision

positive predictive value

probability a sample classified as positive is actually positive

TP/(TP+FP)

recall

same as true positive rate

true positives / total positives

TP / (TP+FN)

sensitivity of the classification

What’s the best metric when you have a large number of negative samples

Precision and recall

Precision is not affected by a large number of negative samples because it measures the fraction of true positives out of the number of predicted positives (TP +FP).

Precision measures the probability of correct detection of positive values while FPR, TPR, and ROC measure the ability to distinguish between classes.

Highest value of F1

1.0 indicating perfect precision and recall

Lowest value of F1

0 if either precision or recall are 0

AUC range

0 to 1

ROC

- true positive rate (recall) on the y axis

false positive rate on the x axis - captures the performance of a classification model at all classification thresholds (probability thresholds)
- does not depend on class distribution!
- receiver operating characteristic curve