Ch. 5 - Reliability

Question 1

reliability

Answer 1

consistency in measurement (not good or bad, right or wrong, just consistent); the proportion of the total variance attributed to true variance

Question 2

reliability coefficient

Answer 2

a proportion that indicates the ratio between the true score variance on a test and the total variance

Question 3

concept of reliability - equation

Answer 3

Observed Score = True Score + Error

Question 4

we use X to describe test score variability / reliability

Answer 4

variance

Question 5

the proportion of the total variance attributed to true variance is

Answer 5

reliability

Question 6

the greater the reliability…

Answer 6

indicates that you are capturing more true variance than “noise”

Question 7

measurement error

Answer 7

all of the factors associated with the process of measuring some variable, other than the variable being measured

Question 8

error variance

Answer 8

variance from irrelevant, random sources

Question 9

sources of error variance

Answer 9

test construction (content sampled, way items are worded
test administration (environment: lighting, temperature; testtaker variables: sick, bad mood; examiner-related variables: "giving away" answers with tone of voice)

Question 10

more sources of error variance

Answer 10

computer glitches or errors in hand-scoring; testtakers may over or under report
sampling error - only contacting voters with landlines

Question 11

test-retest reliability

Answer 11

a method of reliability. obtained by correlating pairs of scores from the same people on two different administrations of the same test. use when measuring something that’s stable over time (trait)

Question 12

as the time between test administrations increases, the correlation usually…

Answer 12

decreases

Question 13

coefficient of stability

Answer 13

the estimate of test-retest reliability, when the interval between testing is greater than six months

Question 14

coefficient of equivalence

Answer 14

the degree of the relationship between various forms of a test

Question 15

parallel forms (reliability)

Answer 15

for each form of the test, the means and variances of observed test scores are equal

Question 16

alternate forms (reliability)

Answer 16

these don’t necessarily met the requirements of parallel forms (same means and variances) but are equivalent in terms of content, level of difficulty, etc

Question 17

parallel or alternate forms relaibility

Answer 17

the extent to which item sampling and other errors have affected test scores on versions of the same test

Question 18

how do you obtain parallel or alternate forms reliability estimates?

Answer 18

administer test two times with same group (like test-retest but don’t have to wait)
same problems: scores affected by item sampling, testtaker variables, etc
time consuming and expensive

Question 19

estimate of inter-item consistency

Answer 19

degree of correlation among all items on a scale

Question 20

how do you do a split-half reliability estimate?

Answer 20

(1) divide test into equivalent halves
(2) find Pearson r between the scores on each half
(3) adjust the half-test reliability with Spearman-Brown formula

Question 21

what is a split-half reliability estimate?

Answer 21

obtaining reliability estimate evaluating the internal consistency of the test (no need for two firms or time elapsing).

Question 22

how should split the test for a split-half reliability estimate?

Answer 22

not down the middle
randomly assign items
split odd-even
divide by content and difficulty

i.e. make mini parallel forms!

Question 23

Spearman-Brown Adjustment

Answer 23

determines the reliability of a whole test from a shortened version. (half)

Question 24

don’t use split-half reliability with what kind of test?

Answer 24

heterogeneous (measures more than one trait)

Question 25

reliability usually increases as...

Answer 25

test length increases

Question 26

alternatives to the Spearman-Brown reliability estimate (for split-half)

Answer 26

Kuder-Richardson (for tests with dichotemous items) Average Proportional Distance Cronbach's alpha - "mean of all possible split-half correlations"

Question 27

reliability coefficients range from

Answer 27

0 to 1. possible to get negative, but usually a mistake in data entry

Question 28

measures of reliability are subject to

Answer 28

error. they are estimates

Question 29

a reliability coefficient may not be acceptable if

Answer 29

it is done with the same test on a very different set of testtakers

Question 30

what's a good reliability?

Answer 30

like grades! .90 is an A, .80 is a B

Question 31

if reliability is really high on a split-half estimate, what is likely the cause?

Answer 31

redundancy in test items

Question 32

the more homogeneous a test is...

Answer 32

the more inter-item consistency it can be expected to have (duh)

Question 33

split-half reliability, odd-even, Spearman-Brown formula, Kuder-Richardson (KR-20), alpha, and Average Proportional Distance are all methods of evaluating...

Answer 33

the internal consistency of a test

Question 34

inter-scorer reliability

Answer 34

the degree of agreement or consistency between two or more scorers/judges/raters

Question 35

if inter-scorer reliability is high,...

Answer 35

test scores can be derived in a systematic, consistent way by trained scorers

Question 36

what are the three approaches for estimating reliability?

Answer 36

test-retest, alternate or parallel forms, internal or inter-item consistency

Question 37

what about the nature of a test might influence reliability? (5)

Answer 37

``` homogeneous vs heterogeneous test dynamic vs static characteristics restriction or inflation of range speed vs power test criterion-referenced vs norm-referenced tests ```

Question 38

heterogeneous vs homogeneous test

Answer 38

measures different factors; measures one factor/trait

Question 39

traditional ways of estimating reliability are often not appropriate for what kind of test?

Answer 39

criterion-referenced

Question 40

what kind of reliability estimate is best for a heterogeneous test?

Answer 40

test-retest (not inter item consistency because that will be low)

Question 41

what kind of reliability estimate is best for a measurement of dynamic characteristics?

Answer 41

inter-item consistency (not test-retest)

Question 42

power test

Answer 42

has a long time limit, but some items are so hard that no testtaker will get a perfect score

Question 43

speed test

Answer 43

must be done in a certain amount of time. easy items but tough to get them all done (typing)

Question 44

classical test theory believes that...

Answer 44

everyone has a "true score" on a test. very test-dependent, though

Question 45

what are alternatives to classical test theory?

Answer 45

domain sampling theory generalizability theory Item Response Theory (IRT)

Question 46

domain sampling theory

Answer 46

test's reliability is an objective measure of how precisely the test measures the "domain" of the test (ex: behavior). takes issue with the true score + error = score

Question 47

generalizability theory

Answer 47

a person's test scores vary from testing to testing because of the variables in the testing sitaution. takes issue with the true score + error = score

Question 48

Item Response Theory (IRT)

Answer 48

hundreds of varietys; items vary in many different ways including: Difficulty and Discrimination

Question 49

what tells us how much error could be in single test score?

Answer 49

Standard Error of Measurement (SEM)

Question 50

Standard Error of Measurement

Answer 50

estimates the extent to which an observed score deviates from a "true" score

Question 51

the higher reliability of a test, the ____ the SEM

Answer 51

lower

Question 52

if a person were to take a bunch of equivalent tests, scores would be...

Answer 52

normally distributed with their true score at the mean

Question 53

confidence interval

Answer 53

the range or band of scores that is likely to contain the true score

Question 54

95% confidence interval - what does it mean?

Answer 54

we are 95% confident that the true score is within +- 2 standard errors of measurement. 95% of this testtaker's scores are expected to fall within this range on the distribution

Question 55

true differences in a characteristic being measured might be from another source besides error or change from one testing to another. what might that be?

Answer 55

an actual difference. might be what you're looking for in psychotherapy outcome reasearch

Question 56

standard error of the difference helps you determine

Answer 56

if your research showed statistically significant results of something weird!

Question 57

the standard error of the difference will always be ___ compared to the standard error of measurement for a score.

Answer 57

larger, because both include error.