Accuracy

Percent of all predictions that were correct.

Confusion Matrix

A matrix showing the predicted and actual classifications. A confusion matrix is of size LxL, where L is the number of different label values. Rows for each of the actual values cross-tabbed against columns of the predicted values.

Cross-validation: Overview

A method for estimating the accuracy of an inducer by dividing the data into k mutually exclusive subsets, or folds, of approximately equal size. The inducer is trained and tested k times. Each time it is trained on the data set minus a fold and tested on that fold. The accuracy estimate is the average accuracy for the k folds.

Cross-validation: How

Leave-one-out cross validation

K-fold cross validation

Training and validation data sets have to be drawn from the same population

The step of choosing the kernel parameters of a SVM should be cross-validated as well

Model Comparison

…

Model Evaluation: Adjusted R^2 (R-Square)

The method preferred by statisticians for determining which variables to include in a model. It is a modified version of R^2 which penalizes each new variable on the basis of how many have already been admitted. Due to its construct, R^2 will always increase as you add new variables, which result in models that over-fit the data and have poor predictive ability. Adjusted R^2 results in more parsimonious models that admit new variables only if the improvement in fit is larger than the penalty, which improves the ultimate goal of out-of-sample prediction. (Submitted by Santiago Perez)

Model Evaluation: Decision tables

simplest way of expressing output from machine learning, cells in table represent the resulting decision based on the row and column which represent the conditions

Model Evaluation: Mis-classification error

Define Test error as summed error for classification (false prediction)

Model Evaluation: Negative class

Negative means not having the symptoms

Model Evaluation: Positive class

presence of something we are looking for - 1

Model Evaluation: Precision

Of all predicted positives, how many are positive?

Model Evaluation: Recall

Of all positives how many were predicted as positive?

Model Evaluation: True negative

Hypotheses correctly predicts negative output.

Model Evaluation: True positive

Hypotheses correctly predicts positive output.

Model Selection Algorithm

algorithm that automatically selects a good model function for a dataset.

Precision

Percent of predicted Positives that were correct.

P-values

…

Receiver-Operator Curves

…

Sensitivity

aka Recall or True positive rate. Percent of actual Positives that were correctly predicted.

Specificity

aka True negative rate. Percent of actual Negatives that were correctly predicted.