Ch 3 - Reliability

Question 1

reliability =

Answer 1

the ability of test scores to be interpreted in a consistent and dependable manner across multiple test administrations

Question 2

the property of reliability resides with what?

Answer 2

the test scores, not the test itself

Question 3

if test reliability is high, how much weight do you give that test?

Answer 3

more weight

Question 4

Classical test theory =

Answer 4

describes a set of psychometric procedures that can be used to test the reliability, difficult, and discriminatory properties of test items and scale

Question 5

Classical Test Theory (CTT) equation:

Answer 5

X = T + E
raw score = true score + random error
*can try to reduce random error of tests

Question 6

When error is minimized, a test produces ___?

Answer 6

more reliable scores

Question 7

two sources of measurement error

Answer 7

systematic error - when a test is being used and consistently measures something other than the trait it was designed to assess (imperfect construct validity)

unsystematic error (random error) - collection of factors that contribute to the variation in scores across administrations
 - can be related to test, its construct, administration or scoring or to the individual

Question 8

4 sources of measurement error =

Answer 8

1. time sampling error
2. content sampling error
3. test administration error
4. test taker variables

Question 9

Time sampling errors result from ___?

Answer 9

repeated administrations of a test to the same individual

Question 10

3 effects that can create a time sampling error =

Answer 10

1. carryover effect - when an experimental treatment continues to affect a participant long after the treatment is administered
2. practice effect - when individuals improve their scores across test administrations as a result of increased familiarity and comfort with a test and the content that is being assessed
3. fatigue - when clients tire from multiple administrations of a test

Question 11

why do counselors care about reliability?

Answer 11

if you know how reliable a test is, you are better prepared to determine what weight to assign the results of a test when making decisions

Question 12

Correlation =

Answer 12

statistical technique used to measure and describe a relationship between two variables
*doesn’t speak to causation

Question 13

positive relationship for correlation =

Answer 13

the scores on each of the two variables move in the same direction

Question 14

negative relationship for correlation =

Answer 14

the scores move in opposite directions

Question 15

correlation coefficient =

what is the range

Answer 15

numeric value that indicates the strength of the relationship between two variables

range between -1.00 and +1.00

Question 16

what is a perfect correlation?

Answer 16

coefficient of -1.00 and +1.00 are perfect correlations
- means that each and every change in one variable is met with a constant change in the other variable

• the closer a number is to 1 in either direction, the stronger the correlation is

Question 17

what does coefficient of 0 mean?

Answer 17

indicates no discernible relationship between the variables

Question 18

what does positive/negative describe for a coefficient?

Answer 18

positive relationship means scores of the two variables move in the same direction (more study, better score)

negative relationship means the scores move in opposite directions (more depressive symptoms, lower the mood score)

Question 19

What are the four methods of estimating reliability?

Answer 19

Test-retest reliability
Alternate forms reliability
Internal consistency
Inter-rater reliability

Question 20

test-retest reliability =

Answer 20

assessing how reliable or stable scores on an instrument are over time - participants tested on same test on two separate times

Question 21

3 challenges of test-retest reliability

Answer 21

carryover effect, maturation, memory loss

Question 22

alternate forms reliability =

Answer 22

assessing stability using different but equivalent versions of a test

Question 23

internal consistency reliability =

Answer 23

used to determine whether errors associated with content sampling are present (are the items consistent and assessing the same concept)

Question 24

Inter-rater reliability =

Answer 24

used to assess level of agreement between two or more raters in their evaluation of a particular outcome (to what extent are the raters the same in how they rate a test)

Question 25

Evaluating reliability coefficients table =

5 levels

Answer 25

very high = greater than 0.90
high = between 0.80 and 0.89
acceptable = between 0.70 and 0.79
Questionable = between 0.60 and 0.69
Unacceptable = less than 0.59

Question 26

4 factors affecting reliability (significant for counseling)

Answer 26

1. increase length of the test
2. make sure test used is designed for population being assessed
3. increase heterogeneity of group used to norm the test (was there diversity)
4. Use optimal time interval between test administrations