Module 6: Reliability and Validity Flashcards
Reliability
Reliability refers to the consistency or stability of a measuring instrument. In other words, the measuring instrument must measure exactly the same way every time it is used.
Systematic errors
Problems that stem from the experimenter and the testing situation.
Trait errors
Problems that stem from the participants. Were they truthfull, did they feel well?
True score
The true score is what the score on the measuring instrument would be if there were no error.
Error score
The error score is any measurement error (systematic or trait).
Observed score
The score recorded for a participant on the measuring instrument used.
Conceptual formula for observed score
Observed score = True score + Error score
Random errors
Errors in measurement that lead to measurable values being inconsistent when repeated measurements of a constant attribute or quantity are taken
Conceptual formula for reliability
Reliability = True score / (true score + error score)
Correlation coefficients
A correlation coefficient measures the degree of relationship between two sets of scores and can vary between -1.00 and +1.00. The stronger the relationship between the variables, the closer the coefficient is to either 1.00 or +1.00.
Positive correlation
A positive correlation indicates a direct relationship between variables: When we see high scores on one variable, we tend to see high scores on the other
Graph going from left bottom to right top
Negative correlation
A negative correlation indicates an inverse, or negative, relationship: High scores on one variable go with low scores on the other and vice versa.
Graph going from left top to right bottom
Rules-of-thumb correlation coefficient
.29 = none to weak .30-.69 = moderate .70-1.00 = strong
Types of reliability
- Test/retest
- Alternate forms
- Split-half
- Interrater reliability
Test/retest reliability
One of the most often used and obvious ways of establishing reliability is to repeat the same test on a second occasion. The correlation coefficient needs to be high on both tests for the reliability to increase as well.
Practice effects
Some people get better at the second testing, and this practice lowers the observed correlation
Alternate form reliability
using alternate forms of the testing instrument and correlating the performance of individuals on the two different forms
Split-half reliability
where you give one group one half of the test and the other group the other half.
Interrater reliability
Here you test how consistent the assessment of two or more raters or judges are
Con: you need to test the reliability of the interraters between them.
Conceptual formula interrater reliability
Interrater reliability = Number of agreements / number of possible agreements x 100
Cronbach’s alpha
A measure of the internal consistency as a kind of average correlation between the items
(I.e. measuring one participant in group 1 and comparing that to another participant from group 1 to see if they match up)
Rules-of-thumb Cronbach’s alpha
- > .80: reliability = good.
- .60 - .80: reliability = sufficient.
- Less than .60 = insufficient
Validity
Validity refers to whether a measuring instrument measures what it claims to measure. The extent to which the observations reflect what we want to measure, i.e. the extent to which the observation reflects the concept or construct under investigation.
Differences validity and reliability
- Reliability refers to observations (scores)
- Validity refers to conclusions based on observations.
- Reliability concerns random measurement error.
- Validity issues have to do with systematic error. E.g. our policemen are only 90 meters apart instead of 100, So, the observation does not reflect what we want to measure.
E.g. John scores higher on the IQ-test than Peter. Reliability: are we sure their true scores are different? Validity: Is John more intelligent than Peter?
