Reliability and Validity Flashcards Preview

PSYC1111 > Reliability and Validity > Flashcards

Flashcards in Reliability and Validity Deck (39):
1

What is reliability

Reliability refers to the consistency/repeatability of results of a measurement. “How” reliable something is relative and depends on the situation.

2

Types of reliability

• Observers: Inter-Observer reliability
• Observations: Internal (Split-half) reliability
• Occasions: Test-retest reliability

3

What is inter-observer reliability?

Inter-observer reliability is the degree to which observers agree upon an observation or judgement
• Can be frequency or categorical judgement

4

How is inter-observer reliability tested?

Measure by looking at shared relationships between observers (i.e. correlation). Cohen’s kappa, Pearson’s correlation coefficient (r), etc. depending on whether continuous or discrete

5

What is internal reliability?

Internal reliability is the degree to which specific items/observations in a multiple item measure behave the same way. I.e. are they measuring the same thing?

6

How is internal reliability tested?

Tested by dividing test into 2 halves, then looking at correlation between them. If it has high internal reliability, an individual's performance on the first half should correlated with the second half.

7

What is test-retest reliability?

Test-retest reliability is the extent to which scores on a test/measure remain stable over time.

8

What is validity?

Validity refers to how well a measure or an
operationalised variable corresponds to what it is supposed to measure/represent.

9

Types of validity

• Internal validity
• External validity
• Population validity
• Ecological validity
• Construct validity
• Content validity
• Criterion validity

10

Internal validity

• How convincing is the evidence for causality in a study/series of experiments?
• i.e. how strong is the inference that the independent
variable and the dependent variable are causally
related?

11

J.S. Mill: 3 requirements to establish causality

• Covariation
• Temporal Sequence
• Eliminating confounds (rival explanations/hypotheses)
• Third-variable problem

12

Equifinality

Another threat to internal validity. Most things have multiple causes .

13

External validity

How well does a causal relationship
hold across different people, settings,
treatment variables, measurements
and time.
Two types: Ecological and population validity.

14

Population validity

• Making cross-cultural inferences from Western, Educated, Industrialized, Rich, Democratic samples?
• Differences in tasks ranging from motivation, reasoning and even visual perception
• Muller-Lyer Illusion: Americans vs. the San people of the Kalahari

15

Ecological Validity

How well do results of laboratory experiments generalise to real-life settings?
• E.g. aggression studies in the lab vs. in real life
• Bandura (1961, 1963): Bobo doll experiment

16

Construct validity

How well do your operationalized variables
(independent and/or dependent) represent the
hypothetical or abstract variables of interest

17

Content validity

Degree to which the items or tasks adequately sample the
target domain
• i.e. how well does a measure/task represent all the facets of a construct

18

Criterion validity

To what extent can a procedure be used to infer or predict some criterion (outcome)
Two types: concurrent and predictive

19

Concurrent validity

A type of criterion validity - to what extent can a procedure be used to infer some criterion

20

Predictive validity

A type of criterion validity - to what extent can a procedure be used to predict some criterion

21

When do person confounds occur?

Person confounds occur when a variable seems to cause something because people who are high or low on this variable also happen to be high or low on some individual difference variable (e.g. demographics characteristic) that is associated with the outcome variable of interest.

22

When do operational confounds occur?

Operational confounds occur when a measure designed to assess a specific construct such as depression, memory, or foot size inadvertently measures something else as well.

23

What is a confound?

A threat to internal validity, that undermines a causal explanation

24

What is an artifact?

A threat to external validity, a by-product of testing procedure or sample that biases all results.
Unlike confounds, artifacts stay constant and are present in all groups

25

What are some artifacts?

Hawthorne effect, history effect, and selection bias/non-response bias

26

What is the Hawthorne effect?

That mere act of measurement changes the nature of something
- a form of participant reaction bias

27

Advantages to Surveys?

Quick and efficient
Large samples
Obtain public opinion almost immediately
Easy to use

28

Limitations of surveys?

Non-random sampling
Haphazard samples (availability and volunteer bias)
Question wording effects

29

Advantages of naturalistic observation?

Allows the study of issues not amenable to experimentation
Useful in the initial stages of investigation

30

Limitations of naturalistic observation?

Low internal validity
Time consuming
Hawthorne effect - threat to external validity

31

Advantages of participant observation?

It can be used in situations that otherwise might
be closed to scientific investigation

32

Limitations of participant observation?

The dual role of the researcher maximizes the chances for the observer to lose objectivity and allow personal biases to enter into the description

33

Advantages of longitudinal studies?

• Genuine changes and stability of some
characteristics observed
• Major points of change observed
• Temporal sequence
• Minimise age-cohort effects

34

Disadvantages of longitudinal studies?

• Time consuming and expensive
• Participant attrition – threat to validity

35

Advantages of cross-sectional studies?

• Relatively inexpensive and less time consuming • Low attrition rate

36

Disadvantages of cross-sectional studies?

• Cannot observe changes in individuals
• Insensitive to abrupt changes
• Age-Cohort effects

37

What are age-cohort effects in cross-sectional studies?

Cohort differences are confounded with age differences.

For example, measuring computer skills at different age points - Different exposure to computers

38

What is experimental mortality also known as? What's an example of a situation in which it is a problem?

Experimental mortality is also known as heterogenous attrition.

For example, in study on antipsychotics and schizophrenia, experimental group receive antipsychotics which produce side effects and also reduce potentially pleasurable symptoms such as schizophrenia. Hence, more drop out of the experimental group.
This becomes a threat to internal validity.

39

Individual difference confounds include...

Experimental mortality
Participant Reaction Bias (participant expectancies, participant reactance, evaluation apprehension)