Reliability and Validity Statistics

Question 1

Extent to which a measurement is consistent and free from error
Can be expected to repeat the same score on two different occasions provided that the characteristic of interest does not change

Answer 1

Reliability

Question 2

Ratio of total variability of scores (measurements) compared to individual variability within scores (measurements)
Measured as a unitless coefficient

Answer 2

Relative Reliability

Question 3

Indicates how much of a measured value, expressed in the original units, is likely due to error

Answer 3

Absolute Reliability

Question 4

Absolute measure of reliability
In same units of measurement as variable
SD of the distribution of theoretical multiple measurements

Answer 4

Standard Error of Measurement

Question 5

Often used to evaluate scales and questionnaires
How well do these questions reflect the SAME construct, not do they actually measure the construct

Answer 5

Internal Consistency

Question 6

Each subject assessed by the same set of raters
Used for test-retest and inter-rater reliability
We can generalize findings to other raters

Answer 6

ICC Model 2

Question 7

Each subject is assessed by the sames set of raters, but the raters represent the only raters of interest
Used for inter-rater reliability or when you do not wish to generalize the scores to other raters

Answer 7

ICC Model 3

Question 8

Represents correlation among items and correlation of each individual item with the total score

Answer 8

Cronbach’s Alpha

Question 9

Proportion of agreement between raters after chance agreement has been removed
Can be used on both nominal and ordinal data
Can be interpreted like ICC

Answer 9

Kappa Coefficient