Accuracy
Percent of all predictions that were correct.
Confusion Matrix
A matrix showing the predicted and actual classifications. A confusion matrix is of size LxL, where L is the number of different label values. Rows for each of the actual values cross-tabbed against columns of the predicted values.
Cross-validation: Overview
A method for estimating the accuracy of an inducer by dividing the data into k mutually exclusive subsets, or folds, of approximately equal size. The inducer is trained and tested k times. Each time it is trained on the data set minus a fold and tested on that fold. The accuracy estimate is the average accuracy for the k folds.
Cross-validation: How
Leave-one-out cross validation
K-fold cross validation
Training and validation data sets have to be drawn from the same population
The step of choosing the kernel parameters of a SVM should be cross-validated as well
Model Comparison
...
Model Evaluation: Adjusted R^2 (R-Square)
The method preferred by statisticians for determining which variables to include in a model. It is a modified version of R^2 which penalizes each new variable on the basis of how many have already been admitted. Due to its construct, R^2 will always increase as you add new variables, which result in models that over-fit the data and have poor predictive ability. Adjusted R^2 results in more parsimonious models that admit new variables only if the improvement in fit is larger than the penalty, which improves the ultimate goal of out-of-sample prediction. (Submitted by Santiago Perez)
Model Evaluation: Decision tables
simplest way of expressing output from machine learning, cells in table represent the resulting decision based on the row and column which represent the conditions
Model Evaluation: Mis-classification error
Define Test error as summed error for classification (false prediction)
Model Evaluation: Negative class
Negative means not having the symptoms
Model Evaluation: Positive class
presence of something we are looking for - 1
Model Evaluation: Precision
Of all predicted positives, how many are positive?
Model Evaluation: Recall
Of all positives how many were predicted as positive?
Model Evaluation: True negative
Hypotheses correctly predicts negative output.
Model Evaluation: True positive
Hypotheses correctly predicts positive output.
Model Selection Algorithm
algorithm that automatically selects a good model function for a dataset.
Precision
Percent of predicted Positives that were correct.
P-values
...
Receiver-Operator Curves
...
Sensitivity
aka Recall or True positive rate. Percent of actual Positives that were correctly predicted.
Specificity
aka True negative rate. Percent of actual Negatives that were correctly predicted.
