Test Bias & Test Utility

Question 1

If we discover that one group of people score higher on a test than another group of people, what are the possible underlying reasons for this?

Answer 1

The test may be biased or the two groups may actually be different

Question 2

If our test is biased then what are our options for dealing with this?

Answer 2

Have an accommodation within existing tests (e.g. extra time, point correction); redevelop the test; develop an alternative special test for the group; avoid testing altogether (but the alternatives may not be less biased)

Question 3

What are some issues with alternative tests like the BITCH (black intelligence test for cultural homogeneity) and Chitling tests?

Answer 3

They may not exhibit psychometric validity; tend to yield much higher scores for groups at which they’re directed, but haven’t demonstrated predictive validity (e.g. not predictive of real world outcomes, like job performance or academic achievement)

Question 4

Describe how we can use regression lines to model test bias

Answer 4

We can look at regression lines for criterion validity scatterplots separately for different groups; e.g. look at the relationship between how well people are predicted to do at a job (test score) and how well they actually do (job performance)

Question 5

If all of group A are scoring higher on both test score and job performance than group B, what does this mean?

Answer 5

Either the test isn’t biased, or both the job performance measure and the test score are equivalently biased (discriminating against group B)

Question 6

When does Intercept Bias occur, and what does it mean?

Answer 6

When the slope of the regression lines of the groups are the same, but they intercept the vertical axis at different places; test is biased between one group but the test is still equally predictive of job performance for both groups (although they can’t be compared)

Question 7

When does Slope Bias occur, and what does it mean?

Answer 7

When the slope of the regression lines of the groups are different; test is biased against one group and is also less predictive of job performance for them; test is differentially valid for the 2 groups

Question 8

What does “differentially valid” mean?

Answer 8

When the scatter of points is greater for one of the groups, so there’s a lower correlation between job performance and test score

Question 9

Some people have argued that the construct of race is primarily social and has no biological meaning (especially in societies like the US). What argument has been made for the idea that race differences in intelligence test scores are due to genetics?

Answer 9

Black children raised in white families with white education tend to still do worse at school and score lower on IQ tests (shows innate race differences)

Question 10

What arguments have been made against the idea that race differences in intelligence scores are due to genetics?

Answer 10

School achievement and even IQ is partly a function of teacher expectation (e.g. Pygmalion effect); A minority group member may still face many disadvantages in such a situation, especially where group membership may be superficially obvious (e.g. race)

Question 11

Describe the Pygmalion Effect found in Rosenthal and Jacobson’s experiment

Answer 11

School children were given a non-verbal IQ test; teachers were given a list of children who performed in the top 20% and were identified as bloomers (actually chosen at random); for earliest grades, bloomers scored significantly higher at the end of the year

Question 12

Steele and Aronson had students complete the Graduate Record Examination and divided them into 2 groups. How did they demonstrate the effect of self-stereotyping?

Answer 12

Group 1 were told the test measured intellectual ability and group 2 were told it was about problem-solving; African-Americans did worse than white Americans when in group 1 but didn’t differ in group 2

Question 13

Describe the experiment that Shih et al. carried out to demonstrate the effect of self-stereotyping with Asian American women

Answer 13

They gave them a maths test; when previously given a questionnaire relating to racial identity they did better on the maths test than controls, but performed worse when previously completing a questionnaire relating to gender

Question 14

List the Commonwealth of Australia Acts that impact the use of psychological tests for employment purposes

Answer 14

Racial discrimination act; age discrimination act; human rights and equal opportunity commission act; sex discrimination act; disability discrimination act; fair work act

Question 15

According to the Disability Discrimination Act, to overcome a claim of discrimination, what must the deficit be directly tied to?

Answer 15

An inherent requirement of the job

Question 16

What disabilities does the Disability Discrimination Act include?

Answer 16

Physical, intellectual, psychiatric, sensory, neurological, learning disabilities, physical disfigurement, and the presence of disease-causing organisms in the body

Question 17

Describe the requirements for psychological testing in relation to employment under Australian law

Answer 17

All tests must measure the person for the inherent requirements of the job, not the person in the abstract (content and criterion validity needed); aptitude and personality tests must relate directly to the genuine requirements of the job; should only be used to assess an applicant’s suitability for the position based on selection criteria; other information (e.g. personality/private life) shouldn’t be used when deciding about their suitability

Question 18

The only Australian case that discussed personality tests in the context of employment law, was between Australian Industrial Relations Commission vs. Coms21 (1999). What occurred?

Answer 18

Employees from a consulting firm were unfairly dismissed based on a personality test, and an inaccurate consultant’s report without regard to objective, reasonable and justifiable selection criteria (no associated skill or competency tests were used)

Question 19

Discuss the issues that have arisen when the use of psychological testing has been taken to court on the grounds of bias

Answer 19

Overrepresentation of mentally-retarded children with Spanish surnames; IQ tests used to place children in educable mentally retarded classes (disproportionate effect on black children); When WISC-R and Stanford-Binet were argued to be racially biased, judge ruled evidence as unconvincing; Inconsistencies in court decisions are commonplace; validity/reliability of many psychological tests currently being decided by the courts

Question 20

What does Test Utility refer to, and why is it important?

Answer 20

The practical usefulness of a test; it’s not enough for a test to be good (i.e. reliability/validity), it has to yield a worthwhile benefit that outweighs its costs (e.g. money, time, inconvenience, etc)

Question 21

Is a test with poor reliability and validity likely to have good utility?

Answer 21

Not if the utility includes interpreting the test score, but there may be situations when the test score is less important, such as if its being used for some other purpose (e.g. liar detector tests can be useful even if they don’t work)

Question 22

What is utility analysis?

Answer 22

A family of different techniques that can be used to decide the usefulness of a test; can also be applied to interventions (e.g. to decide most preferable training/therapy program)

Question 23

What is an expectancy table, and how do you calculate one?

Answer 23

It’s a utility analysis strategy generated from a criterion validity scatterplot; the scatterplot is divided into categories, based on test performance and job performance criteria, a cut-off (pass mark) is determined, and the number of correct hits, false positives, false negatives and correct negatives are calculated

Question 24

What are the selection ratios in the context of expectancy tables?

Answer 24

The ratio between available job positions and number of applicants (e.g. 2 positions: 4 applicants = selection ratio of .5)

Question 25

What are false positives and false negatives in the context of expectancy tables?

Answer 25

False positives: test incorrectly identifies person as being good when they’re not;
False negatives: test incorrectly identifies person as being no good when they’re good

Question 26

What is a cut off in the context of expectancy tables?

Answer 26

The minimum test score needed in the test to be hired (pass mark)

Question 27

Imagine you are an HR manager. Describe how you could use expectancy table techniques to deal with (1) a high selection ratio and (2) a low selection ratio

Answer 27

1. High selection ratio: if we had lots of job positions, we could lower the pass mark of the test, so no good people missed out
2. Low selection ratio: if we only had a few job positions to fill, we could raise the pass mark of the test, so no bad people would be hired

Question 28

What risks would you be running with either a high or low selection scenario?

Answer 28

High selection: lowering the pass mark could lead to an increase in false positives (more bad people hired);
Low selection: raising the pass mark could lead to an increase in false negatives (good people not hired)

Question 29

How many categories can an expectancy table have?

Answer 29

Multiple categories rather than just two (good/bad or pass/fail)

Question 30

What are the disadvantages of using the expectancy table technique?

Answer 30

It assumes a linear relationship between job performance and test score; doesn’t take into account other factors (e.g. minority status, physical health of applicant, etc)

Question 31

What influence does the criterion validity of a particular selection test have on its usefulness in recruiting people?

Answer 31

It can tell us if we’re choosing the right people; a test with low criterion validity is no better than selecting applicants at random (more scatter); a test with high criterion validity is very good at choosing the right people (positive correlation)

Question 32

How did Schmidt et al. use utility analysis to save employers millions of dollars?

Answer 32

Through analysis of the Programmer Aptitude Test (PAT) to select computer programmers; when compared with previous selection procedures, it yielded a much higher validity coefficient, saving the employer $6 million p/yr