Validity Flashcards
(24 cards)
What does validity refer to?
Validity refers to the degree with which correct influences can be made from the results of a research study
NOT THE SAME AS THE VALIDITY OF A MEASURE
What is external validity?
Extent to which the results of a research study can be generalized outside of the study
What are threats to validity?
Any factor that could potentially undermine the accuracy or trustworthiness of the results of a study is considered a threat to validity
When you work hard to design and carry out a study, you want to be able to draw strong conclusions
Threats to validity prevent you from doing this but only if you don’t plan for them
What is generalization from a sample to the general population?
To what extent can results from our study be generalized to individuals who differ from our actual participants?
What is generalization from one research study to another?
To what extent can results from our study generalize to another study of the same issue?
What is generalization from a research study to a real-world situation?
To what extent can the results of our study generalize to the “real” world?
What is considered a threat to external validity?
Any characteristic of a study that limits our ability to generalize our results
What are the threats to generalizing across participants or subjects?
- Selection bias: a threat to external validity that occurs when the selection process produces a sample with characteristics that are different from those in the population. Also known as sampling bias
- Over-utilizing college students: WEIRD participants (Western, Educated, Industrialized, Rich, and Democratic)
- Volunteer bias: People who volunteer may not be like those who don’t
- More approval motivated
- More Sociable
- Participant characteristics: our accessible population may be inherently biased
What are threats to generalizing across features of a study?
- Novelty effects: In the context of a study, subjects may perceive and respond differently than they would in the “real” world
- Multiple treatment interference: If you are testing a series of treatments or conditions, participations in one might affect the behavior in another
- fatigue
- practice
- Experimenter characteristics: the way subjects behave may be influenced by who the experimenter is
What are the threats to generalizing across features of the measures?
- Sensitization: when you’re being tested on something, you may perform differently than if we weren’t testing you
- We are testing a new program to develop self-esteem in students where we are actually measuring self-esteem and demonstrating a change
- If we employ this new program without actually measuring self-esteem, the outcomes might be very
different - Generality across response measures: The operational definition you choose may lead to results that would have not been obtained if you used a different one
- A study may find that relaxation training works to treat phobias when the fear response is measured
as heart rate but this training may not be effective if the operational definition was self-reported fear - Time of measurement: When you are measuring your outcome matters
-Smoking cessation programs generally are effective when you measure outcomes immediately after completing the program but maybe not so much if you measure 6 months after the program
What is internal validity?
Extent to which a study produces a single, unambiguous explanation of the relationship between variables
What is good internal validity?
- Allows us to conclude with confidence that the relationship observed in the study are due to the study’s independent variable and NOT other variables
- Allows us to conclude with confidence that the results of the study are not due to methodological errors
- Allows us to conclude with confidence that the results of the study have not been influenced by biases
What are extraneous variables?
Any variable in a research study that is NOT being directly investigated
Every study has hundreds of these extraneous variables
Our jobs as researchers is to consider these extraneous variables BEFORE we carry out the study and determine the extent to which they might affect what we are studying
When do extraneous variables become a pain?
Occasionally, an extraneous variable sneaks past us and ultimately ends up
influencing and/or distorting the results of the study
- When this happens, the extraneous variable is no longer just a nuisance, it’s a
CONFOUND
What is a confounding variable?
A Confounding Variable is an uncontrolled extraneous variable that provides an alternative explanation for the observed relationship between the two variables
What does confounding variables result in?
Low internal validity
What is the third variable problem? Example?
A confounding variable (also confounding factor, a confound or a confounder) is
also referred to as a “Third Variable”
This “Third Variable” explains away some or all of the relation between the
two variables of interest
Examples:
- I.e.: Although there is no REAL association between X and Y, the fact that Z is related to both X and Y will make them APPEAR to be related.
- If you measured Z because you planned ahead and thought that Z might be an issue, then Z would just be something you controlled in your analysis. If you did not measure Z, it would be a confounding variable!
What categories do confounding variables fall into?
Environmental variables
- refers to the general environment of the study
Individual differences
- refers to personal characteristics that differ from one individual to another
Time-related variables
- Refers to any variable that can change over time and influences participants behavior
How can you control environmental variables?
Obviously you cannot control EVERY aspect of the environment
- Your goal as researchers is to ensure that the environment in which your study is taking place is to avoid systematic differences in environmental conditions
These systematic differences might be related to:
- The investigator ( different investigators working with different group of participants)
- The location of data collection ( different sites of recruitment)
- The immediate environment (I.e. maintenance schedule)
- The timing of data collection (I.e. beginning vs. end of semester on SONA)
How can you control individual differences?
Obviously you cannot control EVERY individual difference
- Controlling the ones that are KNOWN to impact what you are
studying is possible though!! - I.e., hypothyroidism, prenatal care, etc…
- In group-based studies you also need to avoid Assignment Bias
- A type of research bias that occurs when the process of assigning participants to groups skews the results of a study
- Easy to avoid with the use of Random Assignment
How can you control time-related variables?
Any variable that differs based on the timing of the study can confound your results. These may include:
- Maturation: physiological processes occurring within the participants that could account for any changes in their behavior
Mortality or Attrition: Differential dropping out of some subjects from the comparison groups before the study is finished, resulting in differences between the groups that may be unrelated to the treatment effects
- History: Extraneous events occurring during the course of the experiment that may affect the participants’ responses on the dependent measure. Could be major events occurring in society (e.g., social upheaval) or minor events occurring within the experimental situation (e.g., equipment malfunction)
How are internal and external validity related reciprocally?
- Controlling internal validity decreases external validity
- Controlling external validity decreases internal validity
What is one way to effectively deal with the relation between internal and external validity?
- First demonstrate an effect in a highly controlled environment or a highly
selective population (i.e., prioritize internal validity)
-Second replicate the study in a more realistic, natural setting or in a more
inclusive population (i.e., prioritize external validity)
