Test Bias and Fairness

Question 1

3 aspects/facets of the definition of bias

Answer 1

1. The presence, in scores, of construct-irrelevant variance
   ○ Systematic variation in the scores that is due to construct-irrelevant variables
   ○ Depends on the definition of the target construct
   ○ Not error variance / random variation / measurement error; it’s systematic, and due to construct-irrelevant variances
   
   2. Varies as a function (f) of group membership
      ○ The amount/extent of construct-irrelevant variance in the scores changes as a function of group membership (it will be higher in some groups and lower in some others)
      § What could be those groups?
      □ Gender, ethnicity, age, other
   3. Scores wind up systematically over- or under-estimating the target construct for a particular group
      The estimation could be high or low - it’s not always under-estimation (it can be over too)

Question 2

Definition of bias (1 sentence)

Answer 2

Bias is the presence of construct-irrelevant variance in scores, based on group membership, such that scores wind up over or under estimating the target construct for that particular group

Question 3

How can we determine that a certain factor constitutes bias in a test?

Answer 3

If there is one facet of the group membership (age, gender, etc) that influences the test scores BEYOND the true construct of interest, in a systematic fashion

Question 4

2 myths about bias

Answer 4

It’s only bias if it places a group at a disadvantage (not true - can also put groups at an advantage)
It’s always against minority members (not true - it can be against anyone)

Question 5

Definition of fairness

Answer 5

Whether the outcomes of administering a test in a particular context are described as being equitable and socially just

Question 6

Relation between bias and fairness

Answer 6

If a test based on the data is judged to be biased, then a biased test would probably be considered unfair (almost always)
BUT, a test without bias could also be judged as being unfair

Question 7

Caveat about interpreting/noticing bias

Answer 7

Simply observing that members from different groups obtain different average scores on a particular test is not evidence for bias by itself, especially if those groups differ on variables that are relevant for target constructs

Question 8

Item/content bias

Answer 8

refers to if the wording of the items (instructions, items themselves, choices in MC, etc) contains terms that everyone will understand - if not then there is item/content bias
• There will always be people who will not understand, but there are ways to maximize the understanding in our selected population
○ Avoid idioms, slang
○ Try not to use the same slang as a client, since it will not be perceived well (failed attempt to ingratiate yourself with the client and appropriate a culture that is not yours)

Question 9

Predictive bias

Answer 9

does prediction using the test scores vary due to construct irrelevant variables for different groups

Question 10

Construct bias

Answer 10

concerns if the test measures the same theoretical domains for members of different groups through the test

Question 11

Method bias

Answer 11

Special because the test is involved, but it’s more of an issue in individual testing
• The impact of the examiner on people from different groups may not be the same for many reasons unrelated to the construct measured (ex: reaction to a mental health professional, a doctor, a professor, etc)
• The impact that an authority figure has on the examinees during testing (the stimulus value of the authority figure)
• Some culture might be more reluctant to let strangers handle sensitive matters like health or mental health, therefore might react more strongly to the tester
Some might also be worried about what will happen to their results (if they will be shared with others, etc)

Question 12

2 main types of educational assessment

Answer 12

• Classroom assessment - teachers use various assessment strategies to support ongoing teaching and learning, and to report on the achievement of learning
  
  • External assessment - includes standardized tests and large-scale assessments developed commercially which are used to determine individual levels of achievement in reference to a norm group

Question 13

Comparison of fairness with reliability and validity

Answer 13

• Fairness is similar to validity/reliability in that it’s not dichotomous, it’s determined by degree

Unlike validity and reliability, fairness is not a technical quality, but it’s affected by technical quality

Question 14

Historical events that impacted the view of fairness

Answer 14

Emerged in the 20th century as a result of 2 earlier events
• Interest in the mind
• Compulsory public education made the number of students rise dramatically

Assessment methods at the time allowed for a lot of subjectivity (essays and oral examinations) - which influenced fairness
The first concerns relating to fairness were about reliability
• Standardization was an interesting solution
• Intelligence tests after WWI were highly racist - faced a lot of criticism
• “Culture-free” test failed
• Fair and unbiased were synonyms at the time, but the definition of bias was refined with the advancement of statistics, so they became two separate concepts

End of 20th century
• There was a shift of focus towards validity issues
• Ownership of the consequences of testing was emphasized
Still a debate whether validity is about ethics or measurement

Question 15

3 conditions for fairer educational assessment

Answer 15

• Opportunity to learn: can simply mean exposure to test content or the alignment between curriculum and assessment, can also relate to the availability of learning resources (teachers, tutoring, etc)
  
  • Constructive environment: one that respectfully encourages students to fully participate through the assessment process - requires the assessment to be perceived as useful and the teachers to be perceived as trustful and competent
  • Evaluative thinking: involved asking questions, identifying assumptions, seeking evidence and considering different explanations (AKA critically evaluating assessment practices) - also includes self-reflection in teachers

Question 16

Strategies to be used for increased fairness in educational testing

Answer 16

• Transparency: students should know how their work will be judged before an assessment begins (clear instructions)
• Opportunity to demonstrate learning: students should have many opportunities to demonstrate their learning (increases reliability) - varied assessment methods can prevent any type of student from being advantaged over another
Balance between care and respect: making sure that learning opportunities are engaging without being superficial, and challenging without being impossible

Question 17

Construct bias in the context of cultural testing

Answer 17

overlap in definitions of the construct across cultures

Question 18

Item bias in the context of inter-cultural testing

Answer 18

due to poor translation, ambiguities, low familiarity for certain groups, influence of culture-specific factors on the understanding of an item

Question 19

What is a possible origin to the test bias controversy?

Answer 19

The American society strongly believes in equal opportunity for every person
Leads to a resistance to labelling and alternative placement (esp. In education): we want to believe that everyone has the same opportunities and potential for great achievement

Question 20

Intervening variable in a test

Answer 20

a psychological process, if it is treated only as a component of a system and has no properties beyond the ones that operationally define it

Question 21

Hypothetical construct

Answer 21

if it is thought to exist and have properties beyond its defining ones
Intelligence and personality are hypothetical constructs

Measurements of hypothetical constructs require lengthy processes of test development, and the scores obtained through testing can be interpreted in different ways; it’s not the same thing as absolute measurement (like height or weight) that contain very little uncertainty

Question 22

How can we find bias in a test?

Answer 22

Finding bias: is done through observing the mean difference in scores between groups
• But those differences can be due to many other factors than bias

Question 23

Steelean effects

Answer 23

the fear of confirming a stereotype impedes minorities’ performance

Question 24

7 possible sources of bias

Answer 24

1. Inappropriate content
   ○ Tests are constructed according to majority values
   ○ Correct responses depend on material tha is unfamiliar to minority individuals
   2. Inappropriate standardization samples
      ○ Minorities representation in norming samples is proportionate but insufficient to allow them any influence over test development
   3. Examiners’ and language bias
      ○ Minorities can be intimidated by White examiners who speak standard English
   4. Inequitable social consequences
      ○ Ethnic minority individuals are already disadvantages because of stereotyping and past discrimination, labelling effects of testing adds to their discrimination
   5. Measurement of different constructs
      ○ Tests based on majority culture measure different characteristics for minority groups
   6. Differential Predictive Validity
      ○ Tests for majority don’t predict any relevant behaviours for minority members
   7. Qualitatively distinct aptitude and personality
      Suggests that minority and majority ethnic groups possess characteristics of different types, therefore test development must begin with different definitions for majority/minority groups

Question 25

Difference between bias and fairness

Answer 25

Fairness: moral, philosophical or legal issue on which reasonable people can disagree
Bias: empirical property of a test (statistically estimated)
• To calculate bias, we need scores
Test reviewers therefore can’t estimate bias

Question 26

Cultural loading

Answer 26

(often associated with culture fairness) the degree to which a test or item is specific to a particular culture (greater potential for bias when administered to another group)
Does not render tests biased or offensive by themselves - but creates a potential for them

Question 27

Cultural test bias hypothesis (CTBH)

Answer 27

differences in mean performance for members of different ethnic groups do not reflect real differences among groups but are artifacts of tests or of the measurement process that depend on the test content and do not reflect accurately internal abilities
For ex: intelligence tests containing material more familiar to high-SES children than to low-SES children will indicate higher scores for high-SES children

Most studies have refuted this hypothesis - item biases account for <1% to 5% of variation in test scores and is usually counterbalanced across groups

Question 28

Influence of the presence of a certain group in the norming samples on that group’s capacity to score higher on the test

Answer 28

He observed that the low presence of minorities in norming samples made it impossible for those groups to have any influence on the results of a test
• Experiment with rats (from different strains, aka ethnicities): he hypothesized that mazes (intelligence tests) that were normed on populations dominated by a certain strain would yield higher scores for that strain - hypothesis was confirmed
○ BUT: other authors have observed that some groups (like Asians), despite being underrepresented in norming samples, score consistently higher than other groups - therefore there must be a cultural component to the difference observed in scores (argument backed up by empirical studies)
○ Harrington’s results were an overgeneralization - rats and humans are not the same in terms of intelligence

Question 29

3 fallacious assumptions that impede the scientific study of test bias

Answer 29

1. The egalitarian fallacy
   If all groups are equal in the characteristics measured by a test, assuming that any difference in score must result from bias is a fallacy
   
   2. Culture-bound fallacy
      Reviewers can assess the culture loadings of items through casual inspection
   3. Standardization fallacy
      A test is necessarily biased when used with any group not included in large numbers in the norming sample

Question 30

Why is assuming no difference just as detrimental to research than assuming a difference

Answer 30

Assuming no difference is just as detrimental to research than assuming a difference
Ex: Black children have been found to have higher creativity than White children in some contexts, but assuming that those differences are due to test bias is detrimental to the access of Black children to special education classes

Question 31

4 limitations of mean differences

Answer 31

• Mean differences do not always tell us about the typical score
○ After re-evaluating mean differences in scores, a researcher found that the groups differed more in terms of variation than in their typical score
• Mean differences provide no information as to why the two groups differ - it’s a starting point to asking that question
• The mean is an inaccurate measure of center (if the distribution is skewed, the mean can be pulled away from the center)
○ Symmetry should never be assumed, even with big norming samples
A mean is a point estimate: the variation in scores for one group can be so big that the mean is no longer representative

Question 32

Bias in predictive validity

Answer 32

constant error in an inference to an error in a prediction that exceeds the smallest feasible random error, as a function of membership in a particular group
Results in an erroneous regression equation for the group in which there is bias, therefore in a wrong prediction

Question 33

Bias in construct validity

how to detect it

Answer 33

• How to detect it:
	○ Factor analysis (most often exploratory) - when it shows evidence that there is more than 1 factor underlying the scores, then it's an indication of construct bias
		§ Researchers calculate a correlation (Pearson) called factor congruence/invariance

Question 34

Situational bias

Answer 34

influences in the test situation, but independent of the test itself, that may bias test scores
Ex: characteristics of test setting, instructions, examiners, etc
They are not test bias because they’re not part of the test themselves, but they should still be considered

Question 35

Content validity

how to detect content validity bias

Answer 35

the extent to which the content of a test is a representative sample of the behaviour to be measured
• Items with content bias will behave differently from group to group for people with the same level of abiltiy
• Ex: the item has information that is unfamiliar to minority individuals
• There has been shown to be differences in some items of the SATs and GREs for African American and White examinees
• How to detect item bias:
○ DIF
○ CT (contingency table)
○ Partial correlation procedure to estimate DIF
○ Chi-square technique
○ Correlations between P decrements
○ Partial correlation between an item score and a nominal variable (ex: ethnic group)

Question 36

Internal consistency reliability

Answer 36

the extent to which all items of a test are measuring the same construct
If the reliabilities are similar from group to group, the test is said to be unbiased

Question 37

Helm’s 7 forms of cultural equivalence

Answer 37

1. Functional equivalence: the extent to which test scores have the same meaning for different cultural groups
   
   2. Conceptual equivalence: whether test items have the same meaning and familiarity in different groups
   3. Linguistic equivalence: whether tests have the same linguistic meaning to different groups
   4. Psychometric equivalence: the extent to which tests measure the same thing for different groups
   5. Testing condition equivalence: whether groups are equally familiar with testing procedures and view testing as a means of accessing abiltiy
   6. Contextual equivalence: the extent to which a cognitive ability is assessed similarly in different contexts in which people behave
      Sampling equivalence: whether comparable samples of each cultural group are available at the test development, validation, and interpretation stages

Question 38

3 reasons to translate/adapt tests

Answer 38

• Facilitate comparative ethnic studies
  
  • Allow individuals to be tested in their own language
  • Reduce the time and cost of developing new tests

Question 39

When are tests from different languages/cultures considered equivalent?

Answer 39

• When members of each linguistic/cultural group who have the same standing on the construct measured by the tests have the same probability of selecting the correct item response

Question 40

Judgmental designs used to establish item equivalence

Answer 40

• Judgmental design: rely on a person’s / group’s decision regarding the degree of translation equivalence of an item
○ Forward translation: translators adapt or translate a test to the target culture/language, other translators then assess the equivalency between the 2 versions
○ Back translation: translators adapt or translate a test to the target culture/language, other translators readapt the items back to the original culture or language - both versions must fit

Question 41

Statistical designs used to establish item equivalence

Answer 41

• Statistical designs: depend on the characteristics of the sample
○ Bilingual examinees: bilingual individuals take both the original and translated test
○ Source and target language monolinguals design: monolinguals in the original language take the original or the back-translated version
○ Monolinguals design: monolinguals in the original language take the original and the back-translated version
Afterwards, statistical procedures are done to assess DIF (factor analysis, item response theory, logistic regression, Mantel-Haenszel technique)

Question 42

4 guidelines to ensure equitable assessment

Answer 42

1. Investigate possible referral source bias, because evidence suggests that people are not always referred for services on impartial, objective grounds
   
   2. Inspect test developer’s data for evidence that sound statistical analyses for bias have been completed
   3. Conduct assessments with the most reliable measure available
   4. Assess multiple abilities and use multiple methods