Reliability and Validity

Question 1

What is a psychometric test?

Answer 1

A standardised test that uses psychological measurements to quantify a person’s ability, strengths or characteristics

Question 2

What does a psychometric test consist of?

Answer 2

One or more stimuli that people respond to. Responses can be overt (e.g., key press) or covert (e.g., skin conductance response)

Question 3

How are psychometric tests standardised?

Answer 3

Data is collected from from a large number of people to establish norms, including cut-offs. This allows identification of individuals outside or inside the population norms.

Question 4

What are two key properties of a good psychometric test?

Answer 4

Reliability (internal properties of the scale) and validity (external properties of the scale)

Question 5

What is reliability?

Answer 5

Consistency/stability of a measure across:
- Time
- Setting
- Individuals
‘Quality’ of measurement (the extent to which you can measure a person’s ‘true’ score each time)

Question 6

What is True Score Theory?

Answer 6

A person’s ‘true’ score reflects their genuine ability, characteristics, or potential. Every observed score is made up o the true score plus some measurement error.

Question 7

What are the two sources of variability (error) in observed scores?

Answer 7

1. Variability in ‘true’ scores within a population (individual differences)
2. Fluctuations in measurement error (systematic and random)

Question 8

What are the two types of measurement error of reliability?

Answer 8

Systematic error
Random error

Question 9

How does systematic error affect reliability?

Answer 9

Consistently affects measurement (bias)
- Fire alarm during exam
- Race/gender bias on the test items
- Driving errors equipment has faulty controller

Systematic errors alter the mean of the data: shift
(everyone shifts in the same direction)

Question 10

How does random error affect reliability?

Answer 10

• Random variations are not consistent across the sample (noise/chance)
  
  • Test takers might be hungry, tired, nervous, etc., but not everyone taking the test might be in the same state
• Random variations increase the variability of the data
• (+ and - to the true score)

Question 11

What is reliability determined by?

Answer 11

how much error variance is in the measurement

Question 12

How do we calculate reliability?

Answer 12

variability of true score / variability of observed score

*greater denominator = less reliable

Question 13

What are the types of reliability?

Answer 13

Test-retest
Alternate/parallel forms
Internal consistency (including co-efficient alpha, split-half/inter-item reliability, and item-total reliability)
Inter-rater agreement

Question 14

What are the types of validity?

Answer 14

Construct validity (including content and face validity)
Criterion validity (including predictive, known groups, convergent and discriminant validity)

Question 15

What is predictive validity?

Answer 15

The extent to which data from a measure can predict something it should theoretically be able to predict. For example, does a high extraversion score predict the number of friends someone has?

Question 16

What is convergent validity?

Answer 16

The degree to which multiple measures of the same construct show similar results. For example, do self-report and clinician-rated measures of depression show similar scores?

Question 17

What is discriminant validity?

Answer 17

The extent to which a measure does NOT relate to constructs it should NOT relate to. For example, a measure of verbal validity should not be strongly correlated with measures of athletic ability.

Question 18

What are the two types of errors in measurement?

Answer 18

Systematic error (bias)
Random error (noise/chance)

Question 19

What is systematic error?

Answer 19

Error that consistently affects measurement. Examples include a fire alarm during an exam or race/gender bias in test items. Systematic errors shift the mean of the data in the same direction fr everyone

Question 20

What is random error?

Answer 20

Random variations in scores that don’t have a consistent effect across the sample. Examples include test-takers being hungry, tired or nervous. Random error increases the variability of the data.

Question 21

What is test-retest reliability?

Answer 21

A type of reliability assessed by administering the same test to the same people on multiple occasions and calculating the correlation between scores. A higher correlation indicates higher reliability.

Question 22

What considerations are important for test-retest reliability?

Answer 22

The time interval between tests is important. Longer intervals allow more room for natural change, while shorter intervals can lead to carryover effects. Test-retest reliability is only useful for stable characteristics.

Question 23

What is alternate/parallel forms of reliability?

Answer 23

A type of reliability assessed by developing multiple versions of a test, administering each version to the same participants at multiple times, and correlating the results. A higher correlation indicates better reliability.

Question 24

What is split-half reliability?

Answer 24

A type of reliability assessed by administering a test to participants, splitting the test in half, and computing the correlation between the two halves.

Question 25

What are the problems with split-half reliability?

Answer 25

The way that the test is split is important, as different splits can yield different reliability values. Reliability is also reduced because the number of items in each half is smaller.

Question 26

What is coefficient alpha?

Answer 26

A measure of internal consistency reliability that assesses the correlation of each test item with other items. The most common method is Cronbach's alpha.

Question 27

What is a problem with coefficient alpha?

Answer 27

It can be sensitive to test size, as more items can lead to higher reliability estimates even if the items are poorly inter-correlated.

Question 28

What are the two main types of validity?

Answer 28

Construct Validity (how well a measure reflects the intended construct) and criterion validity (how well a measure relates to concrete, observable criteria)

Question 29

What are the subtypes of construct validity?

Answer 29

Content validity Face validity

Question 30

What is content validity?

Answer 30

The extent to which the content of each test item measures the intended construct

Question 31

What is face validity?

Answer 31

A superficial measure of whether the test appears to measure the intended construct. It can be important because it can affect how respondents approach a test.

Question 32

What is known-groups validity?

Answer 32

The extent to which a measure differentiates between groups who should theoretically perform differently on it

Question 33

What are some good practices for constructing valid test items?

Answer 33

Use ambiguous language Avoid items that might cause response bias Use a blend of positively and negatively keyed items Ensure items assess all aspects of a construct

Question 34

What is social desirability and why is it important to consider in test reconstruction?

Answer 34

The tendency for people to want to present themselves in a positive light. Items that no one wants to rate themselves high r low on tend to be poor items because they fail to capture the range of variability in the construct. These items can be high in reliability but low in validity.