CHAPTER 7:UTILITY Flashcards
Factors that affect a test’s utility:
- Psychometric Soundness
-validity sets ceiling on utility
-a test must be valid to be useful - Cost
-refers to disadvantages, losses or expenses in both economic and noneconomic
terms
-economic, financial or budget-related in nature must certainly be taken into
account - Benefits
-refers to profits, gains or advantages
family of techniques that entail a cost-benefit analysis designed to yield information
relevant to a decision about the usefulness and/or practical value of a tool of assessment
Utility Analysis
General Approaches in Utility Analysis:
- Expectancy Data
- Brodgen-Cronbach-Gleser Formula
- Decision Theory
Expectancy Data
provide an indication of likelihood that a testtaker
will score within some interval of scores on a criterion measure – an
interval may be categorized as “passing”, “acceptable” or “failing”
EXPECTANCY TABLE/CHART
EXPECTANCY DATA
Tables-estimate of the percentage of employees hired by a
particular test who will be successful to their jobs
Taylor-Russell Tables-estimate
EXPECTANCY DATA
estimate of the percentage of employees hired by a
particular test who will be successful to their jobs
TAYLOR-RUSSELL TABLES
EXPECTANCY DATA
used for obtaining the difference between the means
of the selected and unselected groups to derive an index of what the test is
adding to already established procedure
NAYLOR-SHINE
formula used to calculate the dollar amount of a utility gain resulting from the
use of a particular selection instrument under specified conditions
BRODGEN-CRONBACH-GLESER FORMULA
a body of methods used to quantitatively evaluate selection procedures,
diagnostic classifications, therapeutic interventions or other assessment or
intervention-related procedures in terms of how optimal they are (most typically
from a cost-benefit perspective)
DECISION THEORY
A correct classification
EX. a qualified driver is hired; an unqualified driver is not hired
HIT
An incorrect classification; a mistake
EX. a qualified driver is not hired; an unqualified driver is hired
MISS
the proportion of people that an assessment tool accurately identified
as possessing a particular variable
EX. the proportion of qualified drivers with a passing score who actually
gain permanent employee status; the proportion of unqualified drivers with a
failing score who did not gain permanent status
HIT RATE
proportion of people that an assessment tool inaccurately identified
as possessing a particular variable
EX. the proportion of drivers whom inaccurately predicted to be qualified;
the proportion of drivers whom inaccurately predicted to be unqualified
MISS RATE
falsely indicates that the testtaker possesses a particular variable
EX. a driver who is hired is not qualified
FALSE POSITIVE
falsely indicates that the testtaker does not possess a particular variable
EX. the assessment tool says to not hire but driver would have been rated as
qualified
FALSE NEGATIVE
Some Practical Considerations:
-the issue of how many people would actually accept the employment position
offer to them even if they were found to be qualified candidate
The Pool of Job Applicants
-many of the top performers on the test are people who are also being offered
positions by one or more other potential employers
The Pool of Job Applicants
the more complex the job, the more people differ in how well or poorly they do
that job (Hunter et. al.)
The Complexity of the Job
-Cut Score/Cutoff Score
-a (usually numerical) reference point derived as a result of a judgment and used to
divide a set of data into two or more classifications, with some action to be taken
or some inference to be made on the basis of these classifications
The Cut Score in Use
reference point – in a distribution of test scores used to divide
a set of data into two or more classifications – that is a set based on norm-related
considerations rather than on the relationship of test scores to a criterion
-aka norm-referenced cut score
-normative
-relative cut score-a
in a distribution of test scored used to divide a
set of data into two or more classifications – that is typically set with reference to a
judgment concerning a minimum level of proficiency required to be included in a
particular classification
-aka absolute cut score
-criterion
-fixed cut score
the use of two or more cut scores with reference to one predictor for the purpose
of categorizing testtakers
Multiple Cut Scores
-the achievement of a particular cut score on one test is necessary in order to
advance to the next stage of evaluation in the selection process
Multiple-stage or Multi Hurdle
-a model of applicant selection based on the assumption that high scores on one
attribute can balance out low scores on another attribute
-Compensatory Model of Selection