Study Guide 12: Criterion Related Validity Flashcards Preview

EPSE 528 > Study Guide 12: Criterion Related Validity > Flashcards

Flashcards in Study Guide 12: Criterion Related Validity Deck (17):
1

Area under the Curve (AUC)

Graph of true positive rate (sensitivity) vs. False positive rate (1-specificity). Interpreted as an estimate of the probability that a randomly chosen depressed person will have a higher depression score than a randomly chosen non-depressed person. AUC of 0.5 = line of no information, recommend: AUC ≥ 0.8.

2

Base rate

Proportion of current employees who are successful, but have not been selected using any test

3

Binomial effect size display (BESD)

a tool for reporting the magnitude of effect size
Concurrent evidence: validity evidence demonstrating the degree to which criterion and test that are measured at the same time correlate.

4

Criterion (or criterion variable):

a measure of some attribute or outcome that is of primary interest.

5

Criterion-related evidence

validity evidence demonstrating the correlation between performance on the test with performance on relevant criterion.

6

Negative predictive value (NPV):

TN (TN + TP) - % of people who are truly not depressed (according to the criterion) out of those that the scale identified as non-depressed

7

Positive predictive value (PPV)

TP/(FP + TP) - % of individuals who are truly depressed (according to the criterion) of those that the scale identified as depressed
- influenced by prevalence of X in population
- As prevalence , PPV
- small differences in  specificity levels can strongly influence  PPVs
- changes in sensitivity have little impact on PPV

8

Predictive evidence:

validity evidence demonstrating the degree to which criterion and test that are measured at different times correlate.

9

Prevalence:

the proportion of the population or sample found to have the condition.

10

Receiver operating characteristic (ROC) curve

graph sensitivity and specificity fro all of the possible scores of a scale by plotting true positives (sensitivity) versus false positives (1 – specificity).

11

Selection ratio:

# of applicants hired/# of applicants selected. All else being equal, the smaller the selection ratio, the more useful the test.

12

Sensitivity:

TP/(FN + TP). The proportion of actual positives which are correctly identified as such. How sensitive a measure is at detecting the condition

13

Specificity:

TN/(TN + FP). The proportion of actual negatives which are correctly identified. How specific a measure is at ruling out those without the condition.

14

Standard error of estima

an estimate of the amount of error to be expected in the predicted criterion score, speaks to the validity of the measure. Whereas, SEM speaks to the reliability of the measure.

15

Taylor-Russell tables

table demonstrating the connection between the validity of a test and the likelihood of the test resulting in a successful selection of a candidate.

16

Utility analysis

frames validity in terms of a cost versus benefit analysis of test use. Is a test worth using? Does it outweigh the costs?

17

Validity generalization

Type of meta-analysis that seeks to evaluate validity coefficients across a group of studies. Can reveal a) average level of validity coefficient found across a number of studies, b) degree of variability among smaller studies, and c) source of variability across smaller studies.