Evaluation design II Flashcards Preview

ESS3705 Sport, Physical Activity and Health > Evaluation design II > Flashcards

Flashcards in Evaluation design II Deck (52)
Loading flashcards...


• Randomised cross-over trials
○ Exposed to diff treatments - randomise order of treatments - indvs act as own controls
• Parallel randomised control trials
- Randomly assigned to diff groups


the basic experimental principle

• Intervention only diff between 2 groups
- Achieved by random assignment


Pretest-posttest experimental design

• Essential components
○ Identify study popn and determine appropriate sample size for exp and control groups
○ Randomly assign to exp and control groups
○ Pre-test everyone with standardised instrument
○ Introduce IV (intervention) to exp group while withholding from control
○ Post-test both groups with same instrument and under same conditions as pretest
- Compare amount of change in DV for both exp and control groups
- Exp designs attempt to provide max control for threats to internal val


Pretest-posttest experimental design research

Robbins et al. (2012)


Robbins et al. (2012)

The primary purpose of the study was to determine whether girls in one school receiving nurse counseling plus an after-school physical activity club showed greater improvement in physical activity, cardiovascular fitness, and body composition than girls assigned to an attention control condition in another school (N = 69). Linear regressions controlling for baseline measures showed no statistically significant group differences, but the directionality of differences was consistent with greater intervention group improvement for minutes of moderate to vigorous physical activity/hour (t = 0.95, p = .35), cardiovascular fitness (t = 1.26, p = .22), body mass index (BMI; t = -1.47, p = .15), BMI z score (t = -1.19, p = .24), BMI percentile (t = -0.59, p = .56), percentage body fat (t = -0.86, p = .39), and waist circumference (t = -0.19, p = .85). Findings support testing with a larger sample



see notes


why randomise people to groups?

Ensure that any factors that may influence outcome balanced (evenly distributed) between intervention and control groups


Threats to internal validity

• Bias
○ Selection
○ Perf
○ Detection
○ Attrition
- Random error


Threats to internal validity research

Halperin et al. (2015)


Halperin et al. (2015)

Internal validity refers to the degree of control exerted over potential confounding variables to reduce alternative explanations for the effects of various treatments. In exercise and sports-science research and routine testing, internal validity is commonly achieved by controlling variables such as exercise and warm-up protocols, prior training, nutritional intake before testing, ambient temperature, time of testing, hours of sleep, age, and gender. However, a number of other potential confounding variables often do not receive adequate attention in sports physiology and performance research. These confounding variables include instructions on how to perform the test, volume and frequency of verbal encouragement, knowledge of exercise endpoint, number and gender of observers in the room, influence of music played before and during testing, and the effects of mental fatigue on performance. In this review the authors discuss these variables in relation to common testing environments in exercise and sports science and present some recommendations with the goal of reducing possible threats to internal validity.


Selection bias

• When indvs in intervention and control groups systematically diff on factor that may effect outcome thereby leading to systematic error in outcome
• E.g. older people in one group, more females - related to prevalence of activity
• Randomisation reduces selection bias so diffs in outcomes between 2 groups can be attributed to diff treatment of groups (intervention/control) and not to
1. Confounding factor
2. Effect modifier
○ Similar to confound
○ Affects outcome measure
○ E.g. older people in intervention - intervention related to age - reduced likelihood of change - reduced effectiveness of intervention - being modified by age
○ Systematically be biased between groups and modifies outcome
3. Chance


Selection bias research

Bolzern et al. (2018)


Bolzern et al. (2018)

○ Background
○ Cluster-randomised controlled trials require different methodology from individually randomised controlled trials and have unique vulnerabilities to bias. We aimed to compare selection bias between samples of these two types of trials published in four high-impact journals.
○ Methods
○ The JAMA archives and OVID Medline were searched by single selection by one of us (JB) for cluster-randomised and individually randomised controlled trials published in four journals ( BMJ, The Lancet, JAMA, and The New England Journal of Medicine). Two of us (JB and NM) independently double-extracted data from the 20 most recently published trials of each type up to July 3, 2017. Fixed-effects forest plots were generated to show any imbalances in baseline mean participant age between trial arms for each trial. Pooled imbalance was calculated for each trial type. The characteristic of age was chosen because it is reported universally, in standard units (years).
○ Findings
○ For individually randomised controlled trials, age imbalance between trial arms was not statistically significant (0·005 years, 95% CI −0·026 to 0·035). For cluster-randomised controlled trials, age imbalance was ten times greater and was statistically significant (−0·050, −0·057 to −0·043).
○ Interpretation
- Randomisation distributes participant characteristics equally between trial arms except when baseline imbalances occur at random. When studies are pooled, such random imbalances cancel out to become negligible: if imbalance is not negligible across pooled trials, it indicates compromised randomisation or that selection bias has acted on the sample. The significant age imbalance in the cluster-randomised but not the individually randomised trials suggests that cluster-randomised trials are more vulnerable to selection bias. This imbalance might not affect trial outcomes since age in the control group was not substantially greater than in the intervention group. However, it indicates that selection bias can enter though the design of cluster-randomised controlled trials—possibly during the widespread practice of post-randomisation recruitment—and is therefore concerning. One limitation is that selection bias levels may be different in a more general sample of trials than one from four high-impact journals. Our method of examining selection bias has indicated that cluster-randomised controlled trials may require a more robust approach. We recommend taking steps to minimise selection bias in this type of trial.



• When a 3rd/'other' variable influences the association between 2 other variables (intervention and outcome) and 'confounds' the results of the study
- Has to be associated with exposure variable (intervention/control) and the outcome/change in the outcome

see notes


Confounding research

Fuller (2019)


Fuller (2019)

It is sometimes thought that randomized study group allocation is uniquely proficient at producing comparison groups that are evenly balanced for all confounding causes. Philosophers have argued that in real randomized controlled trials this balance assumption typically fails. But is the balance assumption an important ideal? I run a thought experiment, the CONFOUND study, to answer this question. I then suggest a new account of causal inference in ideal and real comparative group studies that helps clarify the roles of confounding variables and randomization.


Performance bias

• Insufficient adherence to study protocol by researchers/Ps
• Researchers may not deliver intervention consistently to all Ps and Ps may differ in how they adhere
- Over/underestimate intervention - researchers may treat people diff depending on which group they are in - deliver intervention in same way to all people


Performance bias research

Gold et al. (2012)


Gold et al. (2012)

○ Objective: Randomised controlled trials (RCTs) aim to provide unbiased estimates of treatment effects. However, the process of implementing trial procedures may have an impact on the performance of complex interventions that rely strongly on the intuition and confidence of therapists. We aimed to examine whether shifting effects over the recruitment period can be observed that might indicate such impact.
○ Method: Three RCTs investigating music therapy vs. standard care were included. The intervention was performed by experienced therapists and based on established methods. We examined outcomes of participants graphically, analysed cumulative effects and tested for differences between first vs. later participants. We tested for potential confounding population shifts through multiple regression models.
○ Results: Cumulative differences suggested trends over the recruitment period. Effect sizes tended to be less favourable among the first participants than later participants. In one study, effects even changed direction. Age, gender and baseline severity did not account for these shifting effects.
- Conclusion: Some trials of complex interventions have shifting effects over the recruitment period that cannot be explained by therapist experience or shifting demographics. Replication and further research should aim to find out which interventions and trial designs are most vulnerable to this new kind of performance bias.


Detection bias

• Systematically diff outcome measures between groups
• Researchers can administer outcome measures diff between groups
• Ps receiving intervention they like may over report changes in behav
- Researchers that want intervention to work - encourage people more at follow up test rather than baseline to help prove the effectiveness - overexaggerate


Detection bias research

Torgerson and Torgerson (2003)


Torgerson and Torgerson (2003)

Reporting or detection bias occurs when researchers are more assiduous in their reporting of events in one group compared with the alternative. For example, if there were an evaluation of an intervention to reduce school exclusions, schools in the intervention group might under- or over-report exclusions compared with the control schools, when in fact there were no differences between the groups. This difference might occur because the ‘active’ group may be more assiduous in their data collection. For example, in the experimental schools minor exclusions such as for a day or for a lunch time, may have been reported when data on similar events in the control schools may not have been collected. Alternatively, the control schools could be more assiduous in reporting exclusions because of the possible demoralisation felt due to being randomised to act as a control. Reporting bias can be minimised if reporting is undertaken blind to treatment allocation, that is, the person who is measuring the outcomes does not know which group is the intervention. We accept, however, this is not always possible. Alternatively, event reporting can be ascertained from differential sources (e.g. participant and teacher and local authority), which should minimise the risk of detection bias.


perf and detection bias

• Perf and detection bias can be avoided by blinding researchers and/or Ps to group allocation - double blind trial
- Often difficult in behavioural studies to blind Ps


perf and detection bias research

Agouropoulos et al. (2014)


Agouropoulos et al. (2014)

○ Objectives: To evaluate the effect of biannual fluoride varnish applications in preschool children as an adjunct to school-based oral health promotion and supervised tooth brushing with 1000 ppm fluoride toothpaste.
○ Methods: 424 preschool children, 2-5 year of age, from 10 different pre schools in Athens were invited to this double-blind randomized controlled trial and 328 children completed the 2-year programme. All children received oral health education with hygiene instructions twice yearly and attended supervised tooth brushing once daily. The test group was treated with fluoride varnish (0.9% diflurosilane) biannually while the control group had placebo applications. The primary endpoints were caries prevalence and increment; secondary outcomes were gingival health, mutans streptococci growth and salivary buffer capacity.
○ Results: The groups were balanced at baseline and no significant differences in caries prevalence or increment were displayed between the groups after 1 and 2 years, respectively. There was a reduced number of new pre-cavitated enamel lesions during the second year of the study (p = 0.05) but the decrease was not statistically significant. The secondary endpoints were unaffected by the varnish treatments.
○ Conclusions: Under the present conditions, biannual fluoride varnish applications in preschool children did not show significant caries-preventive benefits when provided as an adjunct to school-based supervised tooth brushing with 1000 ppm fluoride toothpaste.
- Clinical significance: In community based, caries prevention programmes, for high caries risk preschool children, a fluoride varnish may add little to caries prevention, when 1000 ppm fluoride toothpaste is used daily. (C) 2014 Elsevier Ltd. All rights reserved.


Attrition bias

• Systematic diffs in no. drop outs from study between intervention and control groups
○ Often more in control if not getting anything
• Leads to systematic diffs in groups at follow up - not balanced
• Can be managed by intention to treat analysis methods
○ Analyse data on people still in study and keep everyone in who were lost at follow up
○ Just baseline measure = carry forward - assume the same
○ Midline test - 2 tests before outcome - did some but not outcome - last observation carried forward
- No. people randomised = no. people in analysis


Attrition bias research

Hindmarch et al. (2015)


Hindmarch et al. (2015)

○ Background: Failure to retain participants in randomised controlled trials and longitudinal studies can cause significant methodological problems. We report the recruitment and retention strategies of a randomised controlled trial to promote fire-related injury prevention in families with pre-school children attending children's centres in disadvantaged areas in England.
○ Methods: Thirty-six children's centres were cluster randomised into one of three arms of a 12-month fire-related injury prevention trial. Two arms delivered safety interventions and there was one control arm. Retention rates compared the numbers of participants responding to the 12-month questionnaire to the number recruited to the trial. Multivariable random effects logistic regression was used to explore factors independently associated with participant retention.
○ Results: The trial exceeded its required sample size through the use of multiple recruitment strategies. All children's centres remained in the study, despite increased reorganisation. Parent retention was 68% at 12 months, ranging from 65% to 70% across trial arms and from 62% to 74% across trial sites. There was no significant difference in the rates of retention between trial arms (p = 0.58) or between trial sites (p = 0.16). Retention was significantly lower amongst mothers aged 16-25 years than older mothers [ adjusted odds ratio (AOR) 0.57, 95% CI 0.41, 0.78], those living in non-owner occupied accommodation than in owner occupied accommodation (AOR 0.53, 95% CI 0.38, 0.73) and those living in more disadvantaged areas (most versus least disadvantaged quintiles AOR 0.50, 95% CI 0.30, 0.82).
- Conclusions: Studies recruiting disadvantaged populations should measure and report attrition by socioeconomic factors to enable determination of the extent of attrition bias and estimation of its potential impact on findings. Where differential attrition is anticipated, consideration should be given to over-sampling during recruitment and targeted and more intensive strategies of participant retention in these sub-groups. In transient populations collection of multiple sources of contact information at recruitment and throughout the study may aid retention.


Reporting of RCTs

see notes


Reporting of RCTs research

Schuller et al. (2009)