Lecture_4 Flashcards
(25 cards)
Experiment
X:
* Cause
* Precedes Y
* Treatment
* Independent variable (always categorical)
Y:
* Effect
* Follows X
* Outcome
* Dependent variable (categorical or metric)
Experiment: An experiment is formed when the researcher manipulates one or more independent variables (“X”) and
measures their effect on one or more dependent variables (“Y”), while controlling for the effect of extraneous variables.
Extraneous
“fremd”, “überflüssig”, “unerheblich” oder “nicht zugehörig”
Experimental Research Designs
Manipulation of independent variable in controlled setting,
Goal: measure causal effect of independent on dependent variable
Example: independent variable: advertising spot A (vs. spot B) dependent variable: sales
Can We Prove Causality?
Key Criteria for Inferring Causality
* Co-variation: A change in one variable consistently results in a change in another
* Time Order: The cause must precede the effect in time
* Elimination of Alternatives: All other possible causal explanations are ruled out
➢ Despite rigorous evidence, causality can’t be definitively proven, only inferred with strong certainty
Absolute proof of causality is elusive = “schwer fassbar”. Strong evidence only allows us to infer causal relationships.
Central Concepts in Experimental Design
Independent Variable (IV)
* Variable manipulated in an experiment (cause)
Dependent Variable (DV)
* Affected by the variation of the IV (effect)
Experimental Treatment
* Action or manipulation of the IV
Extraneous Variables
* All variables, besides IV, that might affect the DV
Test Units / Units of Analysis
* Entities responding to the IV or treatments (e.g., consumers, organizations)
Test Group vs. Control Group
* Test Group: Receives the treatment
* Control Group: Does not receive the treatment
Types of Hypotheses in Experiments
Difference Hypothesis
* Example: “Ad campaign 1 yields lower sales compared to ad campaign 2.”
Coherence Hypothesis
* Example: “Increased ad spending leads to higher sales.”
One-Tailed Hypothesis (specific direction is presumed)
* Example: “Ad campaign 1 has lower sales than ad campaign 2.”
Two-Tailed Hypothesis (no specific direction is assumed)
* Example:“Ad campaign affects sales volume.”
Independent vs. Dependent Variables
- Questions to answer before running an experiment:
- What is my dependent variable (“Y”)? How should it be measured? What analysis does this type of
measurement require? Is this measure valid and reliable? - What is my independent variable (“X”)? How should it be manipulated? What materials are needed to
manipulate and measure the independent variable? - What are the dependent and independent variables of the following research statements?
- Smoking causes cancer.
- Fat-reduced food makes people take-in more calories.
- Taking this class improves students’ knowledge about empirical methods.
? Students learn more about empirical methods by attending this class than by reading a book.
Internal vs. External Validity
Internal validity
* Extent to which changes in the dependent variables(s) can be explained by the experimental
manipulation and not by external factors
➢ Degree to which a causal conclusion can be drawn
External validity
* Extent to which the results of the experiment can be generalized– from sample to population
➢ Degree to which findings are representative
➢ Achieving both internal and external validity can be challenging. While a study may be internally valid, it might not be generalizable to broader contexts (external validity). However, internal validity is foundational; without it, external validity is compromised.
The Hawthorne Effect
- Context: 1920s research at Western Electric’s Hawthorne Works
- Objective: Assess how lighting levels influenced worker productivity
- Unexpected Finding: Productivity rose regardless of lighting changes
- Real Cause: Workers’ awareness of being observed — coined as the
“Hawthorne Effect”
Validity Insights
➢ Internal Validity: Seemingly high due to controlled lighting
➢ External Validity: Challenged, as the true productivity driver was observation, not lighting
Threats to Internal Validity: Extraneous Variables
📆 Temporal Factors
* History: External events coinciding with the experiment
* Maturation: Test units evolving over time
🔬 Experimental Factors
* Instrumentation: Variability in measurement tools or observers
* Experimenter Effect: Biases due to experimenter characteristics like age, race, or gender
❌ Loss Factors
* Mortality (Survivorship Bias): Loss of test units during the experiment
🧠 Participant Behavior
* Social Desirability & Demand Effects: Modifying behavior due to perceived expectations
📝 Assignment Factors
* Selection Bias: Improper assignment of test units to treatment conditions
📏 Measurement Factors
* Testing effects (reactivity): Changes caused by the process of experimentation
⚖️Score Variability Factors
* Regression to the mean: Individual extreme scores move closer to average over time
➢ Randomization helps control for the effects of extraneous variables!
Regression to the Mean
- Positive Reinforcement: Israeli flight force instructors tried it, but it was unsuccessful
- Punishment: Instructors experienced it to be effective for addressing bad performance
➢ Kahneman’s Observation: He suggested the opposite of the instructors’ experiences
True vs. Quasi-Experiment
Randomization (random assignment) is the process of assigning participants to
one of the experimental conditions – test group(s) or control group – by chance.
True experiment:
Randomization
Quasi-experiment:
No randomization
Quasi-Experiments vs. „True“ Experiments
What is randomization?
* When we talk about randomization in experimental research, we usually mean random assignment
* Random assignment is the process of assigning participants to one of the experimental conditions by chance
* Can be achieved by flipping a coin, or by using the random number capability of the survey software, etc.
Why is randomization so important?
* Helps ensure that participants in different experimental conditions are similar to each other prior to the treatment
* Helps establish internal validity (→ confidence in causal relationship)
Lab vs. Field Experiments
Factor Laboratory Field
Internal Validity High Low
External Validity Low High
Controlling Extraneous Variables
Extraneous variables are also called confounding variables to illustrate that these variables can confound the results of the experiment by influencing the dependent variable.
Four ways to control for confounding variables:
* Randomization: randomly assigning participants to experimental groups.
* Statistical control: measuring extraneous variables and adjusting for their effects through statistical methods (e.g., demographics).
* Matching: matching participants on a set of key variables before assigning them to conditions.
* Design control: using specific experimental designs to control for confounding effects (treating extraneous variables as additional IV’s)
The NASA Twins Study: A Unique Matched Experiment
Mark and Scott Kelly
* In Space: Scott embarked on a year-long journey in space
* On Earth: Mark, his twin, served as the control, remaining Earth-bound
Study Focus
* Exploring the physiological impacts of space travel on the human body
Key Findings
* Telomeres Dynamics: Scott’s telomeres (DNA end caps) unexpectedly lengthened in space, but shortened rapidly
upon his return
* The ‘Space Gene’ Mystery: 93% of Scott’s altered genes reverted to their pre-flight state. The remaining 7%
suggest long-term changes in areas such as DNA repair and bone formation
Means to Increase Internal Validity
Manipulation Check
* Purpose: Assess if the treatment impacts variables other than the intended dependent
variable
* Attention: Confirm participants comprehend the treatment
* Implementation: Often conducted as a pre-test
Plausibility Check
* Aim: Determine if experimental conditions mirror real-world situations
* Context: Crucial for lab-based or scenario-driven experiments
Abbreviations:
- X = exposure of a group to an IV (treatment)
- O = DV is observed/measured
- R = Random assignment of participants (test units) to groups (treatments)
- EG = Experimental Group
- CG = Control Group
- TE = Treatment Effect
Truly Experimental Designs
Pretest-posttest control group:
EG: R O1 X O2
CG: R O3 x O4
TE = (O2– O1) – (O4– O3)
Possible problem: The interactive
testing effect (O1 may influence X) is
not controlled.
Posttest-only control group:
EG: R X O1
CG: R x O2
TE = O2– O1
Possible problem: Despite
randomization, it is possible that
EG and CG have differed in the
DV prior to the experiment (i.e.
independent of the IV)
NOT Truly Experimental Designs (Quasi-Experiments)
One-group pretest-posttest design:
O1 X O2
no randomization,
no control group…
→ extraneous variables largely uncontrolled
Static group design:
EG: X O1
CG: X O2
no randomization (→ selection bias)
One-shot case study:
X O1
no randomization,
no control group…
Nonequivalent-groups design:
EG: O1 X O2
CG: O3 X O4
no randomization → CG may differ from
EG in more than the treatment effect (X)
Factorial Designs
Simple experiment
2x2 experiment
2x3 experiment
3x2x2 experiment
- Factorial Designs: Manipulate more than one independent variable (factor).
Allows testing for interactions between different factors, i.e., when differences on one factor depend on the level you are on on another factor
Between-Groups vs. Within-Subjects Designs
Between-groups designs (all we’ve seen so far)
* Only one exposure to treatment (one none in CG)
* Only one testing
EG: R O1 X O2
CG: R O3 x O4
Within-subject designs
* Exposure to several / all treatments (stimuli)
* Repeated testing
X1O1 X2O2 X3O3 …
Between-Groups vs. Within-Subjects Designs
Between-Groups:
Advantages
* Simplicity
* Lower fatigue and practice effects
* Useful if it’s impossible to switch to other
experimental conditions (e.g., male vs. female)
Disadvantages
* High number of participants necessary,
particularly for complex designs (many IV’s with
many levels)
* Weaker effects, manipulation needs to be strong
Within-Subject:
Advantages
* Requires fewer participants
* Higher sensitivity (higher chance that
manipulation has an effect)
Disadvantages
* Carry-over effects from one condition to the next
* Fatigue and practice effects
* Remedy: Counterbalancing (random sequence)
Limitations of Experimentation
- Experiments can be time consuming, particularly if the researcher is interested in measuring the long-term effects
- Experiments can be costly due to requirements such as having experimental and control groups and multiple measurements
- Experiments can be difficult to administer. It may be impossible to control for the effects of the extraneous variables, particularly in a field environment.
- Results may not be generalizable when testing occurs in an ‘artificial’ environment
