Study Guide 9: Reliability: Estimation, Interpretation, & Impact Flashcards

1
Q

Adjusted true score estimate

A

it takes measurement error into account, and adjust the point estimate to the mean. Xest= X+Rxx (Xo – X)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Alternate forms reliability

A

method for estimating reliability of test scores by obtaining scores from two different forms of a test and computing the correlation between them.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

(Cronbach’s) coefficient alpha

A

most widely used method for estimating reliability. useful for determining the extent to which the ratings from raters are internally consistent

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Cohen’s Kappa

A
measures the agreement multiple raters, participants or measurement categories-  who each classify items into mutually exclusive categories. It’s a more robust measure of agreement than a simple percentage aggrement because it corrects for agreement that would be expected by chance alone. Thus Kappa =0 is agreement by chance only, and 1.0=perfect chance-corrected agreement. 
0-0.2=slight
0.21-0.40=fair
0.41-0.6=moderate
0.61-0.8=substantial
0.81-1.0=almost perfect
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Composite score

A

if a test includes multiple items, and if the overall score for the test is computed from the responses to those items on a test, the overall score is the composite score.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Confidence interval or error band

A

reflect the accuracy or precision of the point estimate as reflective of an individual’s true score. The greater the sem the greater the average difference between observed scores and true scores.
95% confidence interval=Xo+/-(1.96)(sem)
68% (+ 1 SEM), 95% (+ 1.96 SEM), 99% (+ 2.58 SEM)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Correction for attenuation

A

is a statistical procedure, due to Spearman (1904), to “rid a correlation coefficient from the weakening effect of measurement error”

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Essential tau equivalence

A

when two tests measure the same psychological construct. rests on more liberal assumptions than that of parallel tests (tau equivalence) – ie. the assumption of equal error variance is not required. Thus, estimates from alpha are likely to be accurate more often than those from methods like the split-half approach.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Internal consistency reliability ·

A

practical alternative to alternative forms or test-retest procedure.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Inter-rater reliability

A
  • how repeatable the scores are when two or more different people are scoring or observing the same behavior
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Kuder-Richardson formula 20 or KR-20

A

measure of internal consistency reliability for measures with binary items. It is analogous to Cronbach’s alpha, but for dichotomous choices.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Point estimate

A

single best estimate of the quantity of an underlying psychological attribute at the moment the individual took the test

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Random (unsystematic) error

A
  • is caused by any factors that randomly affect measurement of the variable across the sample. It does not have any consistent effects across the entire sample. Instead, it pushes observed scores up or down randomly. This means that if we could see all of the random errors in a distribution they would have to sum to 0 – there would be as many negative errors as positive ones. The important property of random error is that it adds variability to the data but does not affect average performance for the group.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Regression to the mean

A

likelihood that, upon a second testing, an individual’s score is likely to be closer to the group mean than was his or her first score.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Spearman-Brown correction

A

formula that allows you to calculate the reliability of a revised test (ie., a test that has been lengthened or shortened)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Split-half estimate of reliability

A

– splitting a test into two parallel halves of equal size, and correlating the performance on those halves

17
Q

Standardized coefficient alpha

A

relies only on correlations (pair-wise correlations) – then take the average of all the correlation- this reflects the degree to which responses to all of the items are generally consistent with each other. Then estimate reliability by using average interitem correlation within the Spearman and Brown formula.

18
Q

Describe the relationship between (a) reliability and SEM, and (b) reliability and confidence intervals

A

Larger sem means less reliability

More reliable tests will produce narrower confidence intervals.

19
Q

Describe the implications of reliable measures for research

A

1) observed correlations (ie. between measures) will always be weaker than true correlations (ie. between psychological constructs)
2) the degree of attenuation is determined by the reliabilities of the measures -the poorer the measure, the greater the attenuation
3) error constrains the maximum correlation that could be found between two measures.
4) It is possible to estimate the true correlation between a pair of constructs. By knowing the observed correlation between measures, and their estimated reliabilities, they can solve for true correlation.
a. The equation used for this is “correction for attenuation” because it allows researchers to estimate the correlation that would be obtained if it were not affected by attenuation.

20
Q

Understand what elements contribute to the correlation between two (observed) scores

A

a) the correlation between the true scores of the two psychological constructs being assessed
b) the reliabilities of the two measures -

21
Q

change scores

A

the change in a score on one test from one point to another – difference scores. Concern variability.

22
Q

discrepancy scores

A

the difference scores are computed by subtracting scores from one type of test (eg. An achievement test) from a different type of test (eg. IQ test).
In order to create discrepancy scores, the test scores used in the calculation should be on similar metric scales. Thus, if scores on two tests are in different metrics, standardization of the scores is necessary in order to calculate difference scores and a discrepancy.

23
Q

Understand and describe the difference and relationship between internal consistency and dimensionality

A

Internal consistency doesn’t necessarily mean a test is unidimensional, although this is a tempting conclusion. An internal consistency estimate could be high (eg. Alpha=.75) even if a test is multidimensional because a composite test might have items within each composite test that correlate highly with each other, but the items from different sets correlate weakly.
Factor analysis is a more appropriate method for evaluating dimensionality.