reliability

Question 1

What is Classical Test Theory (CTT)?

Answer 1

A theoretical framework for reliability defined by assumptions describing how measurement errors influence observed test scores.

Question 2

What is the fundamental equation of reliability theory? (Assumption 1 of CTT)

Answer 2

X = T + E, where X is the observed test score, T is the true score, and E is the random error score.

Question 3

What does ε(E) = 0 signify in CTT?

Answer 3

The average error score across repeated testing is zero, meaning positive and negative errors cancel each other out.

Question 4

What are the implications of ε(X) = T in CTT?

Answer 4

It allows us to derive that ε(E) must be 0, confirming that measurement errors are random.

Question 5

What does ‘E’ represent in the context of reliability?

Answer 5

Unsystematic, or random, measurement error that deviates an examinee’s observed score from the true score.

Question 6

What are the assumptions of CTT regarding error scores? (Assumption 3)

Answer 6

Errors are independent and do not correlate with true scores. P(ET) = 0

Question 7

What does it mean if two tests are parallel according to CTT?

Answer 7

They are tau-equivalent:
They satisfy Assumptions 1 through 5, measure the construct equally well (T = T’), and have the same level of error variance.

Question 8

How is the reliability coefficient defined?

Answer 8

It is the proportion of observed score variance attributable to true-score variance.

Question 9

What does a reliability coefficient of 1 indicate?

Answer 9

Observed-score variance reflects entirely true-score variance, indicating perfect reliability.

Question 10

What does a reliability coefficient of 0 indicate?

Answer 10

Observed-score variance reflects entirely error-score variance, indicating zero reliability.

Question 11

What is the significance of the equation σ²_X = σ²_T + σ²_E?

Answer 11

It means observed-score variance is equal to the sum of true-score variance and error-score variance.

Question 12

What is the implication of having a heterogeneous sample for reliability estimation?

Answer 12

Greater variability among people increases true-score variance, which enhances reliability.

Question 13

Fill in the blank: According to CTT, if two tests are essentially tau-equivalent, they have true scores that are the same except for an _______.

Answer 13

additive constant.

Question 14

True or False: Congeneric measures have perfectly correlated true scores.

Answer 14

True.

Question 15

What does a higher reliability indicate about estimating true scores from observed scores?

Answer 15

The higher the reliability, the more confident we can estimate true scores from observed scores.

Question 16

What does it mean when reliability falls between 0 and 1?

Answer 16

Observed-score variance includes some true-score variance and some error-score variance.

Question 17

What is the relevance of error variance in relation to the reliability coefficient?

Answer 17

Reliability reflects the degree to which error variance is minimal compared to the variance of observed scores.

Question 18

What is a primary challenge in estimating reliability based on CTT?

Answer 18

There is no way of knowing the true scores or the error associated with test responses.

Question 19

What is the difference between parallel tests and essentially tau-equivalent tests?

Answer 19

Parallel tests have equal error variance, while essentially tau-equivalent tests do not.

Question 20

What are the assumptions that must be satisfied for two tests to be considered parallel?

Answer 20

They must meet Assumptions 1 through 5 and measure the construct equally well.

Question 21

How does the reliability coefficient relate to observed and true scores?

Answer 21

It represents the correlation between observed and true scores.

Question 22

What does the assumption ε(X) = T imply in terms of test scores?

Answer 22

It implies that the observed scores reflect the true scores without systematic error.

Question 23

What is the relationship between true-score variance and observed score variance in parallel tests?

Answer 23

The correlation between scores on two parallel forms of a test is equal to the ratio of true-score variance to observed score variance.

Question 24

how to prove reliability in the context of parallel tests?

Answer 24

Reliability is proven by equal observed score variance for parallel tests, assuming errors are random and uncorrelated.

Question 25

Fill in the blank: The correlation between scores on two parallel forms of a test is equal to the ratio of _______ to observed score variance.

Answer 25

[true-score variance]

Question 26

What is cov(X, X) in relation to the covariance of scores?

Answer 26

cov(X, X) is equal to the variance of X.

Question 27

What does the notation ρ^2 represent?

Answer 27

ρ^2 represents the ratio of true-score variance to observed score variance.

Question 28

True or False: Errors in parallel tests are correlated.

Answer 28

False.

Question 29

What assumptions are made about errors in the calculation of reliability?

Answer 29

Errors are assumed to be random and uncorrelated.

Question 30

What does the formula cov(T, T') represent?

Answer 30

It represents the covariance between true scores T and T'.

Question 31

List the components of the covariance formula for true and error scores.

Answer 31

* cov(T, T) * cov(T, E) * cov(E, T) * cov(E, E)

Question 32

What does the notation σ represent in this context?

Answer 32

σ represents the standard deviation of the scores.

Question 33

What is the significance of Assumption 6 in this context?

Answer 33

Assumption 6 is used to derive relationships between the variances and covariances of scores.

Question 34

True or False: The observed score variance is equal to the sum of true score variance and error variance.

Answer 34

True.

Question 35

Fill in the blank: The formula for the correlation coefficient is _______.

Answer 35

[cov(T, T) / (σ_T^2)]

Question 36

assumption 1-5 of ctt

Answer 36

X = T + E (1) E(X) = T (2) ρ_ET=0 (3) ρ_E1T2=0 (4) ρ_E1E2=0 (5)

Question 37

assumption 6 of ctt

Answer 37

Observed scores X and X’ satisfy Assumptions (1) to (5) Tau-equivalent condition: The two tests measure the construct equally well, thus T = T’  T1 = T2 The tests have the same level of error variance (〖ϑ_E〗^2= 〖ϑ_E'〗^2)

Question 38

Assumption (7): Two tests are considered essentially tau-equivalent if (when tau-equivalent cannot be met):

Answer 38

* They have observed scores X and X' that satisfy Assumptions (1) to (5) * They have true scores that are the same except for an additive constant (i.e., T1 = T2 + c)

Question 39

What is Test-retest reliability

Answer 39

* Correlate the scores obtained from the same test administered on two different occasions, based on the assumptions: o (1) that the true scores are stable across the two occasions o (2) the error variance of the first testing equals to that of the second testing. * This provides an estimation of the stability of the test scores.

Question 40

Problems due to assumptions of test-retest reliability

Answer 40

Assumptions o (1) that the true scores are stable across the two occasions o (2) the error variance of the first testing equals to that of the second testing. Problems: o Length of test-retest interval is critical.  Too short: Possibility of carryover effects e.g. memory/practice effects  Too long: if true scores are not stable across time, and true score changed over time o Estimate inappropriate if the construct measured is not stable over time (e.g., a state-like construct).  Need to try to maintain the error variance of measurement o Hence, only use test-retest reliability if the construct is known to be stable across time/less susceptible to carryover effects e.g. visual acuity

Question 41

Assumptions of test-retest reliability

Answer 41

o (1) that the true scores are stable across the two occasions o (2) the error variance of the first testing equals to that of the second testing.

Question 42

Advantages of using Internal consistency reliability rather than test-retest reliability

Answer 42

* A useful practical alternative because it requires respondents to complete only one test at only one point in time, thus avoiding the problems associated with repeated testings. o premise is that different parts of a test can be treated as different forms of the test* and correlating the scores for these different parts provide a reliability estimate. --> Mitigates the issue that the construct needs to be stable across time

Question 43

How is internal consistency reliability obtained?

Answer 43

* This reliability provides an estimate of how different parts of the test are consistent with each other. * The premise is that different parts of a test can be treated as different forms of the test* and correlating the scores for these different parts provide a reliability estimate. * Three different approaches to the internal consistency method of estimating reliability: o (1)split-half o (2) Cronbach’s alpha o (3) standardized alpha

Question 44

assumptions of using internal consistency reliability (and its problem)

Answer 44

o Two halves have are as good as each (same validity)  Problem: two forms may not be parallel if the validity of the two halves are not equal

Question 45

assumptions of split-half approach

Answer 45

o Items added/removed are parallel (Test items are good)

Question 45

Split-half approach (how it is done)

Answer 45

The test is split into two halves (e.g., odd/even or first/second half), and scores for the two halves (Y and Y') are correlated. As the correlation is only a measure of the reliability of one half of the test, the reliability of the entire test (X = Y + Y‘) would be greater and is estimated using the Spearman-Brown formula: Reliability is a function of test length (Longer test  Higher reliability) If we just correlate the two halves  underestimation of reliability as the actual test is longer  need for estimation of the reliability of the entire test using the Spearman-Brown formula ρ_XX'=(2ρ_YY')/(1+ρ_YY' ) where ρ_YY' is the correlation of the spilt halves of the tests Y and Y’ are parallel and X and Y and Y’ are parallel; X is the real test Hence, should adhere to assumption 6 of CTT

Question 46

disadvantage of split half approach

Answer 46

* Different ways to split – odd-even; first-second half - reliability estimates are not the same

Question 47

spearman brown formula

Answer 47

refer to notes ρ_XX'=(Nρ_YY')/[1+(N-1)ρ_YY' ] where N is the factor by which the test is increased if split into half then N =2, ρ_YY' is the correlation based on split halves

Question 48

what is cronbach's alpha and how does it split tests

Answer 48

* Alpha is a measure of internal consistency, which refers to the interrelatedness of a set of items. * Splits test to the item-level o Such that we get a best estimate of reliability