Module 7, Evaluating Quantitative Research Flashcards
(29 cards)
Reliability - What is a Reliable Instrument?
an instrument that gives the same measurements when you repeatedly measure the same unchanged objects or events
- a test must be reliable for it to be valid!
my notes:
- if you were to step on a scale 5 times within 30 seconds of each other and you get the same weight from the scale, we would say those scores are reliable and consistent (getting same measurement)
◦ it is important we would not
expect the weight to change
within 30 seconds but if we
do it 5 times throughout the
day it is possible the weight
could change
- must produce consistent scores when we do not expect anything to change in order for it to be accurate (if the weight was different each time, the scale would be unreliable and not accurate
Internal Validity
the extent to which the change in the dependent variable can be attributed to the manipulation of the independent variable
- what is really the manipulation of the independent variable that caused the change in the DV or was there other factors?
- can we establish cause and effect
- extraneous variables are alternative explanations (do not want to see them show up)
External Validity
extent to which researchers can generalize their findings to other people, situations and times
- people is population validity (is the extent to which we can generalize our findings to other people)
◦ example: do the findings
generalize from adolescents
to young adults?
- situations is ecological validity (whether we can generalize the findings to other situations or contexts)
◦ example: does a lab based
study or intervention, what
that study generalize to be
able to say more real world
situations where exercises
are actually working out in a
gym
- times (different time periods) - do not consider much but we need to
◦ different time periods bring
about different attitudes and
beliefs and so research done
earlier on may not actually
be applicable now
◦ example: study interested in
woman participation in
sports done in 1980 (may not
be applicable in today’s
social context)
Internal Vs. External Validity
- want to maximize both but very challenging to do both in one study
trade-off between internal and external validity - the more you maximize one, you have to give up the other just as much
- when researchers try to maximize internal validity, they try to control for extraneous variables or alternative explanations for findings - in doing so what happens is the study starts to look less like the real world or they might start to restrict the inclusion criteria
- if a researcher is trying to maximize external validity and the study to look more like real world where it is generalizable then the researcher has to give up tight control; where they move from lab based to real world context like a gym
series of experiments (studies) - each study has a specific goals and limitations
- real world intervention vs. lab based intervention (more of a continuum
- you would apply your lab based results to the real world
Threats to Internal Validity
what is going to threaten our ability to establish cause and effect
- experimental procedures, treatments, or experiences of the participants that threaten the researcher’s ability to draw correct inferences from the data about the population in an experiment
10 Threats to Internal Validity
history, maturation, testing (reactivity, warm-up effect and loss of naivety), instrumentation, selection bias, experimental mortality, selection-maturation interaction and expectancy
Threats to Internal Validity - History
events occurring in the course of experiment that cause changes in the DV and are not the intervention (IV)
- an unplanned event that coincides with the IV could be responsible for the observed changes in the DV (true cause)
- not history of participant rather coinciding
Threats to Internal Validity - Maturation
processes within the participants that operate as a result of time passing
- eg. DV is a physical fitness test (beep test)
my notes:
- may have nothing to do with PA rather they are just physically maturing
- looking at expected or anticipated physiological changes or maturation over time but it does not always mean improvement could also mean decline
- natural development is key
Threats to Internal Validity - Testing (Reactivity)
the effects of one test on subsequent administrations of another test
- the pre-test is affecting post-test not the intervention
reactivity: exposure to pretest changes behaviour
- eg. self-monitoring - athletes increase their effort simply because they were asked to record it (apple watch is given and all of a sudden you start moving around more) - impact of pre-test
Threats to Internal Validity - Testing (Warm-Up Effect)
do better on the subsequent measure/test because they are more familiar with it
- they are better at the test not that their aerobic capacity has increased for yo-yo test
- based on a performance measure, it is not filling out a questionnaire on emotion regulation, or wellbeing (this is an assessment) - only an issue for performance measure like yo-yo
Threats to Internal Validity -Testing (Loss of Naivety)
people start catching on to what you are testing/measuring
- example: we do not tell them we are measuring self compassion because that will impact how they are answering the questions themselves
Threats to Internal Validity - Instrumentation
quantitative research relies on measurement:
- eg. leadership style, MRI, motivation etc.
instrumentation:
- changes in instrument calibration, including lack of agreement within and between observers
- it is important that the data acquired from instruments be accurate or else the interpretation of results will be inaccurate
Threats to Internal Validity - Selection Bias
choosing comparison groups in a non-random manner (not randomly assigned)
pre-existing differences between groups - functions as confounding or extraneous
- control groups and experimental group - identified extraneous variables - you allow people to pick which group (people who are more motivated about PA will sign up for experimental group)
Threats to Internal Validity - Experimental Mortality
loss of participants from comparison groups for non-random reasons (participants decide the study is not for them and they drop out)
- participants drop out and those who remain tend to be more motivated (results in higher scores on post-test)
example:
- control and experimental group both have 50 participants and at the pre test they are scoring same amount of moderate to vigorous activity but at the end of the study you find that the experimental group has more moderate to vigorous activity than control group
- since these are based on averages it could be that in the experimental group who were doing less activity in pre-test are less motivated in first place so they drop out of the study (did not like PA intervention) and you are left with people who were already scoring high, thus it will inflate your post test scores
Threats to Internal Validity - Selection-Maturation Interaction
the passage of time affects one group but not the other in nonequivalent group designs (groups have not been formed randomly)
- could be the case if we are studying teams or children (can be hard to put people randomly into groups)
- may randomly assign teams, classes, groups as compared to people individually
- randomly assign coaches - does not have to do with the intervention (leadership) - maybe the field they plan on or maturation
Threats to Internal Validity - Expectancy
researchers anticipating that certain participants will perform better
- people expectations will impact peoples motivation, effort, behaviour
- if people act people to perform better they may interact with them differently (encouraging) ~ could be sub-conconscious
3 Ways to Control Threats to Internal Validity
randomization, placebos, blind & double-blind studies and standardize experiments and instruments as much as possible
Controlling Threats to Internal Validity - Randomization
- random assignment – controls for history
- matched pairs – e.g, age, gender
◦ if we cannot do random
assignment - good
secondary option - randomizing treatments or counterbalancing
◦ ABC v. ACB v. BAC v. BCA v.
CAB v. CBA - interaction of timing - example: randomized other of clips because they wanted to figure out if features for penalties were valid not the order the clips came in (just getting at information)
my notes: - random assignment is equal probability of an individual being placed into any level of the IV
- on average, the groups are the same on a IV (everyone in the groups is not same but on average-
- random assignment is equal probability of an individual being placed into any level of the IV
- on average, the groups are the same on a IV (everyone in the groups is not same but on average the groups are the same) (trying to even out any pre-existing differences) which controls for a lot of extraneous variables - internal validity is impacted
◦ any changes in the dependent variable or differences between group and dependent variable can be attributed back to the IV - random assignment is best tool to control for threats to internal validity
Controlling Threats to Internal Validity - Placebos, Blind and Double-Blind Studies
helps reduce experimenter bias/participant expectancies
Placebo
often times people think they are receiving the treatment and will report improvements even though they received placebo (its psychological)
- do not want the participant to know what group they are in - whether they are getting treatment or not
- intervention group is getting aerobic capacity and the other group is not getting it but in order to blind it they could do PA that is less aerobic, like stretching, yoga etc. - this is called an attention control group
- attention control group: self-compassion study - she had another group, an attention control group (not impact IV but are doing something) - they did writing activities (around sport) whether the writing was impacting the outcomes as imposed to the self-compassion writing activities (intervention)
Single Blind
single blind study either the participant does not know what group they are in (typical) or rare scenario whether people who are collecting data do not know what group they are in
- single blind is typically when research participant is unaware of an aspect of the study, so for example a participant would not be aware when they are taking the placebo medication and when they are taking the actual medication
Double Blind
- in double blind study both the participants and the researcher who is collecting data or administrating test do not know what groups they are in (both)
in a double blind study neither the participant or investigator know that information and double blind can sometimes be more difficult to do but would be more robust from a research point of view
- this way participants cannot use any preconceptions or biases while in the study as they are unaware and investigators need to be blinded so they do not ask questions or treat the participants differently based on whether they are getting air pollution or filtered air
Standardize experiments and instruments as much as possible
eg. script every researcher uses, everything in the same order, make sure instruments are calibrated after every test
Threats to External Validity
arise when experimenters draw incorrect generalization from the sample data to other persons, other settings, and past or future situations
