C4. Internal and External Validity of RCTs

Question 1

Define internal validity

Answer 1

A study is said to have internal validity if its findings for the sample are credible.

Question 2

Define external validity

Answer 2

A study is said to have external validity if its findings can be credibly extrapolated to the population or Real World policy of interest.

Question 3

4 consideration for internal validity

Answer 3

Contamination

Non-compliance

Hawthorne Effect

Placebo Effect

Question 4

What is contamination?

Answer 4

• People in the control group access the treatment anyway
• Some of the untreated turn out to be treated.

Question 5

What in non-compliance?

Answer 5

• Individuals who are offered a treatment refuse to take it.
• Some of the treated turn out to be untreated.

Question 6

What is the Hawthorne Effect?

Answer 6

A phenomenon in which participants alter their behaviour as a result of being part of an experiment or study.

Question 7

What is the Placebo Effect?

Answer 7

The placebo effect impacts final outcomes because of perceived changes (different from the Hawthorne effect where the outcome changes due to imperceptible changes).

Question 8

What can we expect about the distribution of outcomes from an RCT with external validity?

Answer 8

The distribution of outcomes that would occur in the population under the policy of interest would be the same as the distribution of outcomes realised by the experimental treatment group.

Question 9

Problem of sample vs. population

Answer 9

Sample used in RCT may differ from the population of policy interest due to the small scale/local nature of the RCT.

Then establishing external validity requires more work.

Question 10

Solution to differences in sample and population

Answer 10

Doing lots of RCTs and varying the circumstances under which the RCT takes place and seeing which aspects of the environment matter/don’t matter to the outcome.

Question 11

Problem of the assumption of individualistic treatment

Answer 11

The assumption of individualistic treatment response may be valid within the design of the RCT but may not hold in the population where there is the potential for spillover effects.

This affects external validity.

Question 12

Problems with measurement

Answer 12

serious measurement problem often occurs when studies have short durations. We often want to learn long-term outcomes of treatments, but short studies reveal only immediate, surrogate outcomes.

This affects external validity.

Question 13

What does credible policy require?

Answer 13

Internal AND external validity