Chapter 4: Reliability Flashcards by Genevieve Benjamin

Reliability

Consistency or stability of test scores

How well did you know this?

Not at all

Perfectly

Factors that impact reliability

When the test is administered
Items selected to be included
External distractions (ex- noise)
Internal distractions (ex- fatigue)
Person administering test
Person scoring test

How well did you know this?

Not at all

Perfectly

Two components of score

True score (representative of true knowledge or ability)
Error score

How well did you know this?

Not at all

Perfectly

Systematic error

Error resulting from receiving a different set of instructions for test

How well did you know this?

Not at all

Perfectly

Classical test theory equation

Xi=T+E
Xi- obtained score
T- true score
E- error

How well did you know this?

Not at all

Perfectly

What measurement error reduces

Usefulness of measurement
Generalizability of test results
Confidence in test results

How well did you know this?

Not at all

Perfectly

Content sampling error

Difference between sample of items on test and total domain of items

How well did you know this?

Not at all

Perfectly

How good sampling affects error

Reduces it

How well did you know this?

Not at all

Perfectly

Largest source of measurement error

Content sampling error

How well did you know this?

Not at all

Perfectly

Time sampling error

Random fluctuations in performance over time

Can be due to examinee (fatigue, illness, anxiety, maturation) or due to environment (distractions, temperature)

How well did you know this?

Not at all

Perfectly

Inter-rater differences

When scoring is subjective, different scorers may score answers differently

How well did you know this?

Not at all

Perfectly

Clerical errors

Adding up points incorrectly

How well did you know this?

Not at all

Perfectly

Reliability (mathematic definition)

Symbol: rxx

Ratio of true score variance to total score variance (number from 0 to 1, where 0 is total error and 1 is no error)

How well did you know this?

Not at all

Perfectly

Reliability equation

rxx= (sigma^2T)/(sigma^2X)

How well did you know this?

Not at all

Perfectly

Reliability’s relation to error

Greater the reliability, the less the error

How well did you know this?

Not at all

Perfectly

What reliability coefficients mean

rxx of 0.9: 90% of score variance is due to true score variance

Test-retest reliability

Administer the same test on 2 occasions
Correlate the scores from both administrations
Sensitive to sampling error

Things to consider surrounding test-retest reliability

Length of interval between testing
Activities during interval (distraction or not)
Carry-over effects from one test to next

Alternate-form reliability

Develop two parallel forms of test
Administer both forms (simultaneously or delayed)
Correlate the scores of the different forms
Sensitive to content sampling error (simultaneous and delayed) and time sampling error (delayed only)

Things to consider surrounding alternate-form reliability

Few tests have alternate forms

Reduction of carry-over effects

Split-half reliability

Administer the test
Divide it into 2 equivalent halves
Correlate the scores for the half tests
Sensitive to content sampling error

Things to consider surrounding split-half reliability

Only 1 administration (no time sampling error)
How to split test up
Short tests have worse reliability

Kuder-Richardson and coefficient (Cronbach’s) alpha

Administer test
Compare each item to all other items
Use KR-20 for dichotomous answers and Cronbach’s alpha for any type of variable
Sensitive to content sampling error and item heterogeneity
Measures internal consistency

Inter-rater reliability

Administer test
2 individuals score test
Calculate agreement between scores
Sensitive to differences between raters

Composite scores

Scores that are combined to form a combined score | Reliability of these is usually better than their individual parts

Difference scores

Calculated difference between 2 scores Reliability of these is usually lower than their individual parts (information is lost: only can see change, not initial baseline)

Choosing a reliability test to use

Multiple administrations: test-retest reliability | One administration: homogeneous content uses coefficient alpha and heterogeneous content uses split-half coefficient

Factors to consider when evaluating reliability coefficients

Construct being measured Time available for testing How the scores will be used Method of estimating reliability

High-stake decision tests: reliability coefficient used

Greater than 0.9 or 0.95

General clinical use: reliability coefficient used

Greater than 0.8

Class tests and screening tests: reliability coefficient used

Greater than 0.7

How to improve reliability

Increase number of test items Use composite scores Develop better items Standardize administration

Standard error of measurement (SEM)

Standard deviation of test administered to the same individual an infinite number of times Useful when interpreting test scores When reliability increases, this decreases

How to calculate confidence intervals

Use SEM and SD

Relationship between reliability and confidence interval

Reliability increases, confidence interval decreases

Test manuals/researchers report: information included

Internal consistency Test-retest Standard error of measurement (SEM) Information on confidence intervals

Generalizability theory

Shows how much variance is associated with different sources of error