Chapter3: Machine Learning Experiments Flashcards
(36 cards)
Describe the accuracy of a classification model
correct / real
describe the error of a classification model
incorrect / real
what is the problem with error and accuracy? what is a better alternative?
it is unreliable for imbalanced data
confusion matrix
how can we compute accuracy from a confusion matrix
total
what does precision and recall measure about a classifier
its ability to classify positive samples
recall =
TP + FN
Precision =
TP + FP
what is the f_1 score
precision + recall
what is specificity
measure for assessing a classifiers ability to classify negative samples
specificity =
TN + FP
1 - specificity is otherwise known as
false positive rate
what is ROC analysis
applies to binary classifiers. we plot sensitivity(true positive rate) and 1 - specificity.
we want the area under the curve to be as close to 1 as possible
what is error in a regression model
difference between predicted and desired output
list types of error for a regression model
root mean square error
mean absolute error
mean absolute percentage error
sum of squares error
what is the coefficient of determination
R^2 score in single output case.
sum of squares error / sum to n (y - 1/n sum to n y) ^2
what is sample error
the error computed using a performance metric from a set of samples
what is true error
the probability a random sample is misclassified
how is true and sample error different in regression
it is the expectation of the error
how do we get bias and variance values
from the expected squared prediction error
what is bias error
(y - E(f))^2
repeat with different sets of training data and measure how true they are
what is variance error
E[(f - E[f])^2]
repeat with different sets of training data and measure how much prediction varies
what is overfitting
model is over complex. Low bias high variance
what is underfitting
model is too simple. high bias, low variance
what is a confidence interval
how good of an estimate of true error is provided by sample error
