Module 6: Reliability and Validity

Question 1

Reliability

Answer 1

Reliability refers to the consistency or stability of a measuring instrument. In other words, the measuring instrument must measure exactly the same way every time it is used.

Question 2

Systematic errors

Answer 2

Problems that stem from the experimenter and the testing situation.

Question 3

Trait errors

Answer 3

Problems that stem from the participants. Were they truthfull, did they feel well?

Question 4

True score

Answer 4

The true score is what the score on the measuring instrument would be if there were no error.

Question 5

Error score

Answer 5

The error score is any measurement error (systematic or trait).

Question 6

Observed score

Answer 6

The score recorded for a participant on the measuring instrument used.

Question 7

Conceptual formula for observed score

Answer 7

Observed score = True score + Error score

Question 8

Random errors

Answer 8

Errors in measurement that lead to measurable values being inconsistent when repeated measurements of a constant attribute or quantity are taken

Question 9

Conceptual formula for reliability

Answer 9

Reliability = True score / (true score + error score)

Question 10

Correlation coefficients

Answer 10

A correlation coefficient measures the degree of relationship between two sets of scores and can vary between -1.00 and +1.00. The stronger the relationship between the variables, the closer the coefficient is to either 1.00 or +1.00.

Question 11

Positive correlation

Answer 11

A positive correlation indicates a direct relationship between variables: When we see high scores on one variable, we tend to see high scores on the other

Graph going from left bottom to right top

Question 12

Negative correlation

Answer 12

A negative correlation indicates an inverse, or negative, relationship: High scores on one variable go with low scores on the other and vice versa.

Graph going from left top to right bottom

Question 13

Rules-of-thumb correlation coefficient

Answer 13

.29 = none to weak
.30-.69 = moderate
.70-1.00 = strong

Question 14

Types of reliability

Answer 14

• Test/retest
• Alternate forms
• Split-half
• Interrater reliability

Question 15

Test/retest reliability

Answer 15

One of the most often used and obvious ways of establishing reliability is to repeat the same test on a second occasion. The correlation coefficient needs to be high on both tests for the reliability to increase as well.

Question 16

Practice effects

Answer 16

Some people get better at the second testing, and this practice lowers the observed correlation

Question 17

Alternate form reliability

Answer 17

using alternate forms of the testing instrument and correlating the performance of individuals on the two different forms

Question 18

Split-half reliability

Answer 18

where you give one group one half of the test and the other group the other half.

Question 19

Interrater reliability

Answer 19

Here you test how consistent the assessment of two or more raters or judges are

Con: you need to test the reliability of the interraters between them.

Question 20

Conceptual formula interrater reliability

Answer 20

Interrater reliability = Number of agreements / number of possible agreements x 100

Question 21

Cronbach’s alpha

Answer 21

A measure of the internal consistency as a kind of average correlation between the items

(I.e. measuring one participant in group 1 and comparing that to another participant from group 1 to see if they match up)

Question 22

Rules-of-thumb Cronbach’s alpha

Answer 22

• > .80: reliability = good.
• .60 - .80: reliability = sufficient.
• Less than .60 = insufficient

Question 23

Validity

Answer 23

Validity refers to whether a measuring instrument measures what it claims to measure. The extent to which the observations reflect what we want to measure, i.e. the extent to which the observation reflects the concept or construct under investigation.

Question 24

Differences validity and reliability

Answer 24

• Reliability refers to observations (scores)
• Validity refers to conclusions based on observations.
• Reliability concerns random measurement error.
• Validity issues have to do with systematic error. E.g. our policemen are only 90 meters apart instead of 100, So, the observation does not reflect what we want to measure.

E.g. John scores higher on the IQ-test than Peter. Reliability: are we sure their true scores are different? Validity: Is John more intelligent than Peter?

Question 25

Statistically significant

Answer 25

What is important for validity coefficients is that they are statistically significant at the .05 or .01 level (with p-value)

Question 26

7 types of validity

Answer 26

• Content validity
• Face validity
• Criterion validity
  
  • Concurrent validity
  • Predictive validity
• Construct validity
• Statistical Conclusion validity
• Internal validity
• External validity
  
  • Population validity
  • Ecological validity

Question 27

Content validity

Answer 27

Looks at the content of tests. Does it cover a representative sample of the domain you are researching?

Question 28

Face validity

Answer 28

Face validity is whether or not a test looks valid on its surface (not the content!). Does the operationalization appear to be valid on it’s surface?

Question 29

Criterion validity

Answer 29

Criterion validity measures how accurately an instrument the behavior or ability predicts. There are two types of criterion validity.
• Concurrent validity is used to estimate present performance. Is the test for bipolar disorder good at distinguishing people with and without depression?
• Predictive validity is used to estimate future performance. Is the personality test a good predictor for study success?

Question 30

Construct validity

Answer 30

Assesses the extent to which a measuring instrument accurately measures a theoretical construct or trait that it is designed to measure.

Some examples of theoretical constructs or traits are verbal fluency, neuroticism, depression, anxiety, intelligence, and scholastic aptitude. The conclusion that were made, can they be concluded from the research they did?

Question 31

(Statistical) Conclusion validity

Answer 31

Do the observations allow for the conclusion that variables are related?

Question 32

Internal validity

Answer 32

Does the operationalization allow for the conclusion that variables are causally related?

Question 33

External validity

Answer 33

Extent of generalizability of the conclusions
• Population validity: does the sample allow for conclusions about the target population?
• Ecological validity: does the procedure followed in the study allow for conclusions about more natural circumstances?

Question 34

Cons of test/retest-reliability

Answer 34

Con:

• Practice effects
• Individuals may remember how they answered previously, both correctly and incorrectly. In this case we may be testing their memories and not the reliability of the testing instrument

Question 35

Cons alternate form reliability

Answer 35

Con:

• Difficult to make them parallel, same number of items, difficulty, etc.
• Practice effects (not as much as test/retest)

Question 36

Con split-half reliability

Answer 36

Con:

• Helps with reliability of itself, but not over time (the test is not being done twice in its entirety).
• Difficult to divide the items equally