Study Guide 7: CTT

Question 1

True score

Answer 1

• A person’s average score on a test over an infinite number of repeated tests (error would be eliminated over an infinite #). It describes the theoretical performance on a test.
• in CTT, ‘T’ incorporates systematic error

Question 2

Systematic error

Answer 2

• Error that affects the individual the same way each time he or she takes the test (e.g. reading ability, test wiseness).
• In CTT, systematic error is incorporated in the true score ‘T’
• Affects validity of scores, NOT reliability

Question 3

Unsystematic error

Answer 3

• Random error that affects individuals differently each time a test is taken (eg noise, anxiety).
• In CTT, ‘E’ refers to unsystematic error.

Question 4

Classical Test Theory (CTT)

Answer 4

X = T + E

where X is observed score, T is true score and E is error (unsystematic)

Question 5

Index of Reliability

Answer 5

• The proportion of true score variance reflected in the observed score variance.
• quantifies the closeness of the relationship btw X and T for a set of examinees

Question 6

Reliability Estimates

Answer 6

1) alternate forms reliability
2) test-retest reliability
3) split-half reliability
4) cronbach’s coefficient alpha
5) kuder-Richardson formula 20

Question 7

Sources of error

Answer 7

1) content sampling error: error d/t items selected/heterogeneity
2) time sampling error: error d/t daily fluctuations that affect test performance
3) scorer error: error d/t scorer variability in test-retest

Question 8

Variables that affect reliability

Answer 8

1) test length: longer tests increase reliability
2) group heterogeneity: greater diversity increases reliability
3) item difficulty: items of medium difficulty increase reliability

Question 9

Sources of error

Answer 9

1) content sampling error: error d/t items selected/heterogeneity
2) time sampling error: error d/t daily fluctuations that affect test performance
3) scorer error: error d/t scorer variability in test-retest

Question 10

Variables that affect reliability

Answer 10

1) test length: longer tests increase reliability
2) group heterogeneity: greater diversity increases reliability
3) item difficulty: items of medium difficulty increase reliability

Question 11

observed score

Answer 11

: True score + error (X = T + E); result obtained from a single sampling

Question 12

reliabiliaty

Answer 12

• The extent to which test scores remain consistent over repeated administrations of the same or parallel test
• The degree to which test scores are free from measurement error
• Increases with 1) greater test length, 2) greater group heterogeneity, and 3) item difficulty closer to medium.

Question 13

Reliability coefficient:

Answer 13

: Indicates % of variability in observed scores due to individual (true score) differences (or variability); implies the remainder is due to random measurement error.

Question 14

standard error of measurement

Answer 14

• average size of error scores
• helps to interpret accuracy of test scores
• sem = so √1-Rxx, where Rxx= reliability
• if Rxx = 1, sem = 0

Question 15

tau equivalence

Answer 15

tests are “parallel” if:
a) the tests measure the same psychological construct – the true scores on one test are equal to the true scores on the other test
b) the tests have the same level of error variance
if the items meet tau equivalence, then alpha, KR-20, and the split-half reliability
will all give identical and accurate estimates of reliability
if the items meet tau equivalence, then alpha, KR-20, and the split-half reliability
will all give identical and accurate estimates of reliability

Question 16

essential tau equivalence

Answer 16

this is less strict than tau equivalence
 theoretically, this means that the true scores for two tests (or two versions of a
test) are the same
 this is estimated, practically-speaking, by seeing if the observed scores on the
two tests (or test versions) have the same (or nearly the same) mean
 the requirement for equal error variances (as seen for tau equivalence) is not
made if the items only meet essential tau equivalence, then the alpha and KR-20 will
give identical and accurate estimates, but the split-half reliability estimate will not

Question 17

what happens to Alpha, KR-20 and tau equivalence if neither tau or essnetial tau equivalence is met?

Answer 17

if the items meet neither tau equivalence or essential tau equivalence, then alpha
and KR-20 will underestimate the reliability (although it is not known by how
much or how little) and the split-half reliability estimate will be inaccurate

Question 18

split-half reliability

Answer 18

reliability as an estimate of interal consistency – when you divide your test into two (odd vs. even numbered items, or 1-10, 11-20) and then correlate performance on the two halves.

Question 19

alternate forms reliability

Answer 19

an estimate of reliability as equivalence

Question 20

cronbach’s alpha coefficient

Answer 20

this is THE most common estimate of reliability. It takes every possible way of splitting a test into two halves - and then takes the average of those split-half reliabilities. KR-20 is incorporated in SPSS calculations of cronbach’s alpha.