Reliability

Question 1

What is reliability?

Answer 1

The extent to which a measurement tool gives you consistent measures; also refers to the degree to which test scores are free from errors of measurement

Question 2

How can we measure reliability?

Answer 2

By seeing if someone gets close to the same score if they complete the same questionnaire a number of times; or if people give similar responses to a series of questions supposed to be measuring the same thing

Question 3

What is classical test theory?

Answer 3

The traditional conceptual basis of psychometrics; it’s the idea that every actual/observed score we measure can be decomposed into two parts: the true score and measurement error

Question 4

What is the true score, according to classical test theory?

Answer 4

The aspect of what we strive to measure; this is constant for an individual

Question 5

What is measurement error, according to classical test theory?

Answer 5

What we don’t want to measure; it’s random, and unrelated to the true score

Question 6

What is the equation for classical test theory?

Answer 6

X (observed score) = T (true score) + E (errors of measurement)

Question 7

What is a true score theory?

Answer 7

Another term for classical test theory

Question 8

What is reliability in terms of the relationship between true and total variance?

Answer 8

Reliability (r) is the proportion of true variance (variation of test scores in a sample without measurement error) to the total variance (actual variation in data – including error); true variance/total

Question 9

What does a lower measurement error tell us about the reliability

Answer 9

It will be higher

Question 10

Why do we describe classical test theory in terms of variance rather than standard deviations?

Answer 10

Because it is additive and can be broken up into components

Question 11

If a person took the same test multiple times and we ended up with a lower reliability, what would we expect in regards to the spread of their scores?

Answer 11

We’d expect them to be more spread out due to measurement error (and less spread out if higher reliability)

Question 12

Describe the various sources of measurement error

Answer 12

Test construction (item sampling/content sampling); Test administration (e.g. distractions during the test, fatigue, etc); Test scoring (e.g. biased examiners, ambiguous scoring guidelines, technical errors); Other influences (e.g. self efficacy, motivational factors, etc)

Question 13

What is item sampling/content sampling?

Answer 13

Only certain items or content are included in the test so scores may vary according to this; e.g. in an exam, not everything is included so people may be advantaged or disadvantaged depending on what they focused on when revising

Question 14

Why can we only estimate the reliability of a test and not measure it directly?

Answer 14

Because true variance is hypothetical/theoretical, so instead we estimate reliability via different methods

Question 15

What are four methods available to us to help estimate the reliability of a test?

Answer 15

Internal consistency; Test-retest reliability; Alternate-forms reliability; Inter-rater reliability

Question 16

What is internal consistency (aka inter-item consistency or internal coherence)?

Answer 16

It’s how much the item scores in a test correlate with one another on average; are responses consistent across items? (e.g. Cronbach’s alpha, KR-20)

Question 17

What is test-retest reliability?

Answer 17

The correlation between scores on the same test by the same people done at two different times

Question 18

Why might test-retest reliability not always be appropriate?

Answer 18

People might remember from the first attempt (but counterbalancing alternate forms can get around this)

Question 19

What is Cronbach’s alpha, and when should it be used?

Answer 19

A measure of internal consistency; when there’s more than 2 possible outcomes to a question

Question 20

Describe the steps involved in calculating Cronbach’s alpha by hand.

Answer 20

1. Split the questionnaire in half
2. Calculate the total score for each half
3. Work out the correlation between the total scores for each half
4. Repeat steps 1-3 for all possible two way splits of the questionnaire
5. Work out the average of all possible split-half correlations
6. Adjust the correlation to account for shortening the test by applying a special version of the Spearman-Brown formula

Question 21

What is KR-20 and what is it used for?

Answer 21

Kuder-Richardson 20 formula, used for estimating internal consistency for questions with 2 outcomes (e.g. true/false); like Cronbach’s alpha, it gives an estimate of the mean of correlations between all possible halves of your questionnaire (then corrected for halving)

Question 22

What’s the difference between parallel forms and alternate forms?

Answer 22

Parallel forms are 2 versions of a test where their mean, SD and correlations with other tests must be the same; Alternate forms are 2 versions that only need equivalent content and difficulty level

Question 23

What is the coefficient of equivalence in the context of a test with parallel forms?

Answer 23

It’s the correlation between two versions of the same test

Question 24

List five issues/situations that might affect which reliability estimate you use

Answer 24

1. Homogeneity/heterogeneity of the test
2. Static vs. dynamic characteristics
3. Restriction of range/variance
4. Speed tests
5. Criterion-referenced tests

Question 25

If a test is heterogeneous, what might be an inappropriate estimate of reliability, and what could you do about this?

Answer 25

Internal consistency; could look at each subscale separately (e.g. the big 5 personality test)

Question 26

What’s the difference between measuring something static and something dynamic?

Answer 26

Static remains the same over time (e.g. intelligence); dynamic is expected to change over time (e.g. fatigue/state anxiety)

Question 27

Which reliability measure could have issues if measuring something dynamic?

Answer 27

Test-retest reliability, as it assumes the thing being measured stays the same

Question 28

What’s the difference between a homogeneous test and a heterogeneous test?

Answer 28

Homogeneous – the test items measure the same thing; Heterogeneous – more than one independent thing is being measured (i.e. there are subscales that don’t intercorrelate highly)

Question 29

What special considerations do we need to take into account when analysing the psychometric properties of a speed test compared with a power test?

Answer 29

Measuring internal consistency may not be appropriate for a speed test as people may get all attempted questions correct but run out of time (spurious correlation between items)

Question 30

Which reliability measures would we use for a speed test?

Answer 30

Alternate-forms or test-retest reliability

Question 31

How does having more items on a test affect it’s reliability?

Answer 31

It tends to increase

Question 32

How can we estimate the change in reliability if the test is shortened or lengthened?

Answer 32

Using the Spearman-Brown formula

Question 33

What does n stand for in the Spearman-Brown adjusted reliability formula?

Answer 33

Number of items in new test divided by number of items in old test

Question 34

Does doubling the length of a test mean doubling the reliability?

Answer 34

No, increasing the numbers of items in a test has diminishing returns

Question 35

What are the performance measures used in the Neale Analysis of reading?

Answer 35

Reading accuracy, reading rate, and comprehension

Question 36

The Neale Analysis of reading tests oral reading comprehension and fluency aimed at what age group?

Answer 36

Students aged 6 – 12 years; also used to diagnose reading difficulties in older readers

Question 37

How is a Neale Analysis of reading test carried out?

Answer 37

The child reads a selection of stories out loud then completes a comprehension test on the story; test administrator notes any errors and length of reading time

Question 38

In clinical practice, we usually only test a client on a particular test once or twice. No test is perfectly reliable so their scores will be an inaccurate measure of the underlying trait. What does this mean?

Answer 38

It’s critical to know HOW INACCURATE the scores we obtain from the test are likely to be in order to make sensible judgements

Question 39

Describe in detail what the standard of error of measurement is supposed to represent

Answer 39

If an individual takes a test a number of times, we assume the distribution will be approximately normal; the average distance of each attempt from their average obtained score (the SD) is the SEM, which tells you the likely margin of error (confidence interval) in the test score

Question 40

Why do we have to add and subtract double the SEM from an individual score in order to get the 95% confidence interval?

Answer 40

Because we assume that the client’s true score (their score if perfectly reliable) is at the middle of the distribution, where we’d also assume the actual score would be if they only took the test once

Question 41

What is the margin of error we usually report?

Answer 41

That 95% of an individual’s scores will be within 2 SEMs of the true score (+/- 2 standard deviations)

Question 42

If we assume a normal distribution, what percentage of an individual’s scores will be within 1 SEM of the true score?; 3 SEMS?

Answer 42

68%;

99.7%

Question 43

Since we can’t actually work out the SEM by testing every client multiple times on a test, how do we work out what the SEM is?

Answer 43

We estimate it: SEM = Sx (SD of lots of test-taker’s scores, not one individual’s) * square root of (1 – reliability of test); or can look up in a table

Question 44

What can we assume about the SEM?

Answer 44

That it will be the same for everyone who takes the test

Question 45

When estimating the SEM, if 95% of scores fall within 2 SD (1.96) of the mean (which is the person’s actual score), then the 95% confidence interval is what?

Answer 45

The actual score +/- (2 x the SEM)

Question 46

If the reliability for the WAIS IQ test is .98 with a SD of 15, how would we work out the SEM?

Answer 46

SEM = 15 * (square root of) 1-.98 =2.12

Question 47

If someone gets an IQ score of 105, what would their 95% confidence interval be?

Answer 47

105 +/- (2 * 2.12); so from 101 to 109

Question 48

The standard error of the difference is used to work out whether someone’s score is significantly different from what?

Answer 48

Their own score on the same test (e.g. after intervention); Their score on another test of the same thing; Someone else’s score on the same test; Someone else’s score on another test

Question 49

If the 2 scores differ by more than 2 SEdiff, then what can we say?

Answer 49

That they’re significantly different at a 95% level of confidence

Question 50

The reliable change index is often used in clinical contexts in place of the standard error of the difference; how is it calculated?

Answer 50

Work out the difference between 2 scores (e.g. changes after an intervention) and divide by the SEdiff; if the reliable index is greater than 1.96 (i.e. 2 SEdiff) then the change is statistically significant

Question 51

What does alternate forms reliability measure?

Answer 51

How much a person’s scores correlate if they do two equivalent versions of the same test (sometimes called coefficient of equivalence)

Question 52

What is inter-rater reliability?

Answer 52

The correlation between scores on the same test by the same people provided by two different examiners

Question 53

What will be affected if scores are inappropriately restricted in the amount they can vary?

Answer 53

The correlation, and ALL reliability estimates

Question 54

How might criterion-referenced tests affect reliability estimates?

Answer 54

There may be little variation in people’s responses (e.g. first-aid tests where most people pass); there’s a restriction of range, leading to problems with ANY reliability estimates (as they’re all based on variance)