Chapter 7: Utility

Question 1

this term refers to how useful a test
is. More specifically, it refers to the practical value of using a test to aid in decision making.

In addition, in the context of testing and assessment as the usefulness or practical value of testing to improve efficiency.

Answer 1

Utility (test utility)

Question 2

what are the indexes related to psychometric properties?

Answer 2

1. Index of Reliability
2. Index of Validity
3. Index of Utility

Question 2

this term refers to the reliability and validity of a test, indicating its overall quality for a specific purpose.

Answer 2

Psychometric Soundness

Question 3

this index reflects how consistently a
test measures what it is intended to measure.

Answer 3

this index indicates the degree to which
a test measures what it claims to measure.

Question 3

this index measures the practical value or usefulness of the information derived from test scores, especially in aiding decision-making

Answer 3

Index of Utility

Question 4

what are the costs associated with test utility?

Answer 4

1. Economic Costs
2. Personnel and Facility Costs
3. Additional Business Costs

Question 5

this term refers to the profits, gains, or advantages obtained from administering a test.

are weighed against costs to assess the overall value and effectiveness of using a particular test.

Answer 5

benefits

Question 6

what is benefit in the context of test utility?

Answer 6

benefit refers to profits, gains, or
advantages in both economic and noneconomic terms.

Question 6

what is cost in the context of test utility?

Answer 6

cost refers to disadvantages, losses, or expenses in both economic and noneconomic terms.

Question 7

this term may be be broadly defined as a family of techniques that entail a cost–benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment

is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding various kinds of output

Answer 7

utility analysis

Question 7

this type of utility analsis instrument can provide an indication of the likelihood that a testtaker will score within some interval of scores on a criterion measure—an interval that may be categorized as “passing,” “acceptable,” or “failing.”

Answer 7

expectancy table/chart (scatterplot and analysis regression)

Question 8

what does a hit (true positive) mean?

Answer 8

a test score correctly identifies the condition of interest and implies that a qualified employee was hired

Question 9

what does a correct rejection (true negative) mean?

Answer 9

the test score correctly identifies the absence of the condition of interest and implies that an unqualified employee was not hired

Question 10

what does a miss (false negative) mean?

Answer 10

the test score failed to identify a trait or condition of interest and implies that a qualified employee was not hired

Question 11

what does a false alarm (false positive) mean?

Answer 11

the test score incorrectly identifies the condition of interest and implies that a unqualified employee was hired

Question 11

what is base rate (prevalence)?

Answer 11

base rate is the proportion of individuals with the condition of interest

Question 12

this term refers to an estimate of the benefit (monetary or otherwise) of using a particular test or selection method.

Answer 12

utility gain

Question 12

this type of utility analysis instrument provides an estimate of the extent to which inclusion of a particular test in the selection system will improve selection.

provides the personnel officer with
an estimate of how much using the test would improve selection over existing methods.

determines the increase over current procedures

Answer 12

Taylor-Russell Tables

Question 13

what is selection ratio?

Answer 13

selection ratio is the proportion of individuals indicating the presence of the condition of interest

Question 13

this term is a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired

For instance, if there are 50 positions
and 100 applicants, then the ________ _____ is 50/100, or .50

Answer 13

selection ratio

Question 14

this utility analysis instrument provided an indication of the difference in average criterion scores for the selected group as compared with the original group in response to the potential problems of the Taylor-Russell tables

the use of this entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test (or some other tool of assessment) is adding to already established procedures.

determines the increase in average score on some criterion measure

Answer 14

Naylor-Shine tables

Question 14

this term refers to the percentage of people hired under the existing system for a particular position

For example, a firm employs 25 computer programmers and 20 are considered successful, the ____ ____ would be .80

Answer 14

base rate

Question 15

what are the limitations of the Taylor-Russell tables?

Answer 15

1. if job performance levels off at some point in time, then the Taylor-Russell tables would be inappropriate no matter how high the score is
2. the potential difficulty of identifying a criterion score that separates “successful” from “unsuccessful” employees

Question 16

this type of utility analysis instrument is used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions

Answer 16

Brogden-Cronbach-Gleser

Question 17

this utility analysis instrument is also used to predict performance and is under the expectancy tables or charts category

Answer 17

regression analysis

Question 18

this term refers to a set of cost-benefit analysis techniques designed to assess the usefulness or practical value of an assessment tool, training program, or intervention.

Answer 18

utility analysis

Question 19

this utility analysis provides estimates of how adding a particular test to a selection system may enhance selection quality.

Answer 19

Taylor-Russel Tables

Question 19

what is the purpose of utility analysis in test evaluation?

Answer 19

Utility analysis can guide decisions on whether: ○ One test is preferable over another for a specific purpose. ○ One type of assessment tool (e.g., a test) is more useful than another (e.g., behavioral observation) for the intended purpose ○ Adding new assessments to an existing process is beneficial

Question 20

this term refers to a value indicating the proportion of positions available relative to the number of applicants.

Answer 20

Selection Ratio

Question 20

this utility analysis is used to estimate the likelihood that a test-taker will achieve scores within specific intervals (e.g., “passing,” “acceptable,” “failing”) on a criterion measure.

Answer 20

Expectancy Tables

Question 20

what is a limitation of the Taylor-Russel Tables

Answer 20

Requires a linear relationship between the test (predictor) and performance (criterion) and a clear distinction between “successful” and “unsuccessful” outcomes.

Question 20

this utility analysis tool calculates utility gain, often as a monetary estimate, from implementing a specific selection tool, helping determine the financial benefit of using a test.

Answer 20

Brogden-Cronbach-Gleser Formula

Question 21

this utility analysis tool helps calculate the added value of a test by measuring the difference between the performance means of selected and unselected groups

Answer 21

Naylor-Shine Tables

Question 22

this term refers to an estimated benefit (monetary or otherwise) from using a test or selection method, often in terms of productivity gain, or the increased

Answer 22

Utility Gain

Question 23

this theory offers a framework for setting optimal cutoff scores, helping maximize test utility by determining the most effective thresholds for selection or classification decisions

Answer 23

Decision Theory

Question 24

this term refers to a numerical reference point used to divide test scores into classifications, guiding actions or inferences based on these divisions.

Answer 24

Cut Scores

Question 25

what are the five types of cut scores?

Answer 25

1. Relative Cut Score (Norm-referenced) 2. Fixed Cut Score (Absolute) 3. Multiple Cut Scores 4. Multiple Hurdles

Question 25

this type of cut score is set based on the performance of a group rather than on an absolute standard. It divides scores in relation to the norm group's performance

Answer 25

relative cut score

Question 26

this type of cut score involves using two or more cut scores within one predictor to categorize test-takers into distinct groups or performance levels

Answer 26

Multiple Cut Scores

Question 27

this selection model allows for high scores in one area to compensate for lower scores in another, supporting a more flexible assessment approach.

Answer 27

Compensatory Model of Selection

Question 27

this type of cut score is set based on a minimum level of required proficiency, regardless of group performance. This standard is fixed and criterion-referenced.

Answer 27

Fixed Cut Score

Question 28

this type of cut score is a multistage approach where test-takers must meet a cut score at each stage to advance, often used in sequential selection processes.

Answer 28

Multiple Hurdles

Question 29

what are some factors that influence cut scores?

Answer 29

● The size and characteristics of the job applicant pool. ● The complexity of the job, which may dictate higher or more specific cut scores. ● The cut score in use, whether relative, fixed, or part of a multiple hurdle model.

Question 30

what are the main methods for setting cut scores?

Answer 30

1. Angoff Method 2. Known Groups Method 3. IRT-Based Methods 4. Item-Mapping Method 5. Bookmark Method

Question 31

this method for setting fixed cut scores can be applied in personnel selection or for identifying the presence or absence of a particular trait or ability

Answer 31

Angoff Method

Question 32

this method for setting cut scores collects data on a predictor from groups known to possess or lack a particular trait, attribute, or ability. A cut score is established based on which score best distinguishes between these two groups.

Answer 32

Known Groups Method

Question 33

this method for setting cut scores involves creating a histogram of test items grouped by equivalent value in terms of difficulty or importance.

Answer 33

Item-Mapping Method

Question 33

this method for setting cut scores has each test item tied to a specific difficulty level. Test-takers must correctly answer items above a set minimum difficulty to "pass," with experts determining this level as the cut score.

Answer 33

IRT-Based Methods

Question 34

this method of setting cut scores considers the number of positions available, likely acceptance rates, and score distributions in setting cut scores.

Answer 34

Method of Predictive Yield

Question 34

this method for setting cut scores has experts receive training on the minimum required knowledge or skills needed to pass the test. They review a "book" of items arranged in ascending order of difficulty and place a bookmark between items separating those who meet the minimal requirements from those who do not. Multiple rounds of bookmmarking may occur, with experts allowed to discuss their decisions. The final difficulty level set as the cut score is agreed upon by test developers

Answer 34

Bookmark Method

Question 35

this method of setting cut scores uses decision theory principles to guide cut score placement.

Answer 35

Decision-Theoretic Approach

Question 35

this method of setting cut scores analyzes relationships between variables (e.g., test scores) and groups (e.g., successful vs. unsuccessful job candidates)to help determine an optimal cut score.

Answer 35

Discriminant Analysis