Evaluation design III Flashcards Preview

ESS3705 Sport, Physical Activity and Health > Evaluation design III > Flashcards

Flashcards in Evaluation design III Deck (52)
Loading flashcards...

Quasi-experimental designs

Can be used when random assignment is not possible

Less internal val than "true" exps

Still provide moderate amount of support for causal inferences


Quasi-experimental designs research

Sharma et al. (2021)


Sharma et al. (2021)

BackgroundAccredited Social Health Activists (ASHA) are community health workers responsible for improving the health status of people by facilitating their access to healthcare services. The life skills of ASHA are known to be effective in negotiating behaviour change in the community; however, there has been a meagre focus towards improving them. Considering this gap, we adopted a comprehensive training program, known as Personal Advancement and Career Enhancement (P.A.C.E.), to empower ASHAs on life skills and financial literacy. The present study intends to assess the training program in two districts of Uttar Pradesh, India, by examining changes in knowledge, perceptions, and practices of ASHAs about life skills and financial literacy.

MethodsWe conducted a quasi-experimental, non-randomized, controlled study with pre-and post-test assessments. Data were collected on socio-demographic characteristics, knowledge, and practices related to life skills (communication skills, self-confidence, problem-solving and decision-making skills, time and stress management skills) and financial literacy. Additionally, change perceptions on gender-, life skills-, and savings-related practices at the personal, community, and workplace levels were assessed in the intervention group. Factor analysis was performed to obtain the change patterns by assessing the degree to which the four life skills, financial literacy, and change perceptions on practices were correlated. A general linear regression model was performed to assess associations among change pattern scores and socio-demographic variables.

ResultsWe analyzed the data of 171 ASHAs (intervention group:86 and control group:85). There was a significant improvement in the average post-test scores of all the life skills and financial literacy in the intervention group (p<0.001). Three distinct change patterns were found post-training in the intervention group. Factor 1 (high loadings for change perceptions on practices) was positively associated with ASHAs aged 38 and above and with experience of 12years. On the contrary, the change in financial literacy and self-confidence scores was common among ASHAs with more than 12years of experience.

ConclusionsThe P.A.C.E training program was found effective in improving the life skills and financial literacy of ASHAs in India


Controlled before and after study design

see notes


Controlled before and after study design research

Goga et al. (2020)


Goga et al. (2020)

Objectives We report the effectiveness of a mentoring approach to improve health workers' (HWs') knowledge, attitudes and confidence with counselling on HIV and infant feeding.

Design Quasi-experimental controlled before-after study. Setting Randomly selected primary healthcare clinics (n=24 intervention, n=12 comparison); two districts, South Africa.

Participants All HWs providing infant feeding counselling in selected facilities were invited.

Interventions Three 1-2 hours, on-site workshops over 3-6 weeks. Primary outcome measures Knowledge (22 binary questions), attitude (21 questions-5-point Likert Scale) and confidence (19 questions-3-point Likert Scale). Individual item responses were added within each of the attitude and confidence domains. The respective sums were taken to be the domain composite index and used as a dependent variable to evaluate intervention effect. Linear regression models were used to estimate the mean score difference between intervention and comparison groups postintervention, adjusting for the mean score difference between them at baseline. Analyses were adjusted for participant baseline characteristics and clustering at health facility level.

Results In intervention and comparison sites, respectively: 289 and 131 baseline and 253 and 114 follow-up interviews were conducted (August-December 2017). At baseline there was no difference in mean number of correctly answered knowledge questions; this differed significantly at follow-up (15.2 in comparison; 17.2 in intervention sites (p<0.001)). At follow-up, the mean attitude and confidence scores towards breast feeding were better in intervention versus comparison sites (p<0.001 and p=0.05, respectively). Controlling for confounders, interactions between time and intervention group and preintervention values, the attitude score was 5.1 points significantly higher in intervention versus comparison groups.

Conclusion A participatory, low-intensity on-site mentoring approach to disseminating updated infant feeding guidelines improved HWs' knowledge, attitudes and confidence more than standard dissemination via a circular. Further research is required to evaluate the effectiveness, feasibility and sustainability of this approach at scale.


types of controlled conditions in quasi-exp design

Control group in quasi-exp study may receive diff intervention, selected components of intervention being tested/something that mimics time and attention paid to Ps (i.e. placebo)

Or use wait-list control, which means control Ps receive nothing during study period but will eventually receive intervention some time after study period


the cycling demo towns programme

1st phase: Oct 2005-Oct 2008

All towns funded at approx. £6 per head per year, matched by local authority

All towns 'medium-sized'; larger ones focused effort on part of popn
○ Exeter one of towns

Infrastructure changes - painting cycling lanes


Ride as groups

Media coverage




Secondary analysis of Sport England's Active People Survey (2005/5 and 2007/8)

Sport and active recreation
○ APS1 (2005/6) n = 1000 per LA (local authority)
○ APS2 (2007/8) n = 500 per LA
○ Phone; random digit dialling; representative sample
○ Didn’t necessary take part in campaign
- Intervention designed to affect whole popn level not just those P in intervention - should have sufficient reach to affect all of local authority

see notes

Left: at baseline 10-12% cycled at least once per month - equality at baseline - more variation in demonstration compared to other authorities - due to larger sample size - increase in cycling prevalence in 2008 with demonstration group - smaller change in non intervention - error bars don’t overlap - there is effect of intervention

Right: higher standard of cycling - lower prevalence - 2.5% - increases more in demo towns than non demo towns - bigger error bars -overlap - diffs not sig - less of change than if cycled less freq


Matched analysis

Possibility of confounders, e.g. age, gender

CDTs matched with comparison areas using National Statistics 2001 Area Classification

Closest stat neighbour on demographics

see notes

Closer r'ship in before and after with just once a month - closer r'ship with matched groups

Error bars overlap - just by chance


Matched analysis research

Rose and van der Laan (2009)


Rose and van der Laan (2009)

Matched case-control study designs are commonly implemented in the field of public health. While matching is intended to eliminate confounding, the main potential benefit of matching in case-control studies is a gain in efficiency. Methods for analyzing matched case-control studies have focused on utilizing conditional logistic regression models that provide conditional and not causal estimates of the odds ratio. This article investigates the use of case-control weighted targeted maximum likelihood estimation to obtain marginal causal effects in matched case-control study designs. We compare the use of case-control weighted targeted maximum likelihood estimation in matched and unmatched designs in an effort to explore which design yields the most information about the marginal causal effect. The procedures require knowledge of certain prevalence probabilities and were previously described by van der Laan (2008). In many practical situations where a causal effect is the parameter of interest, researchers may be better served using an unmatched design.


Changes in cycling between 2006-2009: ICM matched analysis (n=c.9,000)

Mean time spent cycling 1.23h-1.25h

% cycling in past year: 24.3%-27.7%

New to cycling: 1.8%-2.8%
○ Only people who respond may already cycle already

Cycled in last 7 days: 41.7%-49.4%

No sig change: mean days in last week; mean time in last week; % ride to work; days cycled to work


Changes in cycling between 2006-2009: ICM matched analysis (n=c.9,000) research

Sustrans and Davis (2019)


Sustrans and Davis (2019)

intervention typologies:
§ City and town wide interventions
§ Building or improving routes or networks
§ Social marketing including marketing of infrastructure
§ Workplace and other institution based interventions
§ Interpersonal interventions
§ School based interventions

We distinguish city and town wide interventions from the other intervention typologies by virtue of the fact of the approach applied being usually a combination of measures. These combinations typically include measures included in the other typological clusters. The other groups of identified typologies tend, in the research literature and in the practicality of delivery, to be relatively more localised.

Overall, the review concludes that there is strong evidence for the positive impact of interventions to increase active travel. This in turn increases levels of physical activity 13 15 . Of the different intervention typologies the evidence was strongest (in terms of volume and robustness) for city or town-wide interventions. Each of the other intervention types reported some increases in walking and or cycling.


Quasi-experimental statistical methods

Diff-in-Diff analysis: method involves comparing changes before and after the programme for indvs in programme and control

Regression analysis: attempts to address problem of confounding by controlling for diff at baseline


Quasi-experimental statistical methods research

Nygaard et al. (2020)


Nygaard et al. (2020)

Often an intervention is applied in an area (e.g. community, municipality) without it being an experiment and without a control group, this can be categorized as a natural experiment. Such a situation offers the opportunity to exploit exposure contrasts between areas regarding the specific intervention for evaluation. In the present study, we will employ the difference-in-difference approach to evaluate the natural experiment (the structural intervention) comparing measures of health and social factors retrieved from registers in the two social housing areas before and after the intervention. A 'natural experiment' study comparing individual and aggregated level differences in register-based information on health and social variables across time including the entire study period is included in the research project. The population includes all residents with an address in the study area and the control area at any point during the years 2015-2025 (∼3,000 residents in each area). All residents are linked to the Danish social and health registers by the unique personal identification number, which makes it possible to follow all permanent and former residents over time. Hereby we plan to study if the structural changes (the structural intervention, the 'natural experiment') give rise to differences in health (such as use of general practitioner, hospitalizations, use of selected medications) and social factors (e.g. divorces, income levels, unemployment) compared to the control area. The control area is representing a similar social housing area in the same municipality, which will not undergo structural changes until 2023. Findings will be evaluated drawing upon knowledge gained from the entire study from surveys and qualitative interviews as well experiences from the interventions. In this presentation, we wish to discuss how best to include the knowledge based on other methodologies in the register-based analyses


advantages of controlled before and after design

Provides some assurance that outcomes actually results of programme

Most practical option for conducting outcome evals in community interventions

By using preexisting/self-selecting groups, such as indvs already enrolled in programme, it avoids additional steps required with random assignment to study conditions

Overcomes ethical concerns involved in withholding/delaying treatment/substituting less effective treatment for one group of study Ps


disadvantages of controlled before and after design

Can demand more time and resources

Requires access to at least 2 similar groups

Without randomisation, study groups may differ in imp ways that account for some of groups diffs in outcomes after intervention

Selection bias


Interrupted time series design

Helpful for overcoming secular trends - what is going on in background with prevalence of outcome interested in

see notes

May be other factors that explain deviation


Interrupted time series design research

Turner et al. (2020)


Turner et al. (2020)

Objectives: Interrupted time series (ITS) designs are frequently used in public health to examine whether an intervention or exposure has influenced health outcomes. Few reviews have been undertaken to examine the design characteristics, statistical methods, and completeness of reporting of published ITS studies.

Study Design and Setting: We used stratified random sampling to identify 200 ITS studies that evaluated public health interventions or exposures from PubMed (2013-2017). Study characteristics, details of statistical models and estimation methods used, effect metrics, and parameter estimates were extracted. From the 200 studies, 230 time series were examined.

Results: Common statistical methods used were linear regression (31%, 72/230) and autoregressive integrated moving average (19%, 43/230). In 17% (40/230) of the series, we could not determine the statistical method used. Autocorrelation was acknowledged in 63% (145/ 230) of the series. An estimate of the autocorrelation coefficient was given for only 1% of the series (3/230). Measures of precision were reported for 63% of effect measures (541/852).

Conclusion: Many aspects of the design, methods, analysis, and reporting of ITS studies can be improved, particularly description of the statistical methods and approaches to adjust for and estimate autocorrelation. More guidance on the conduct and reporting of ITS studies is needed to improve this study design.


Time series design with control

see notes


Time series design with control research

Zhang et al. (2020)


Zhang et al. (2020)

Objective To discuss the study design and data analysis for three-phase interrupted time series (ITS) studies to evaluate the impact of health policy, systems, or environmental interventions. Simulation methods are used to conduct power and sample size calculation for these studies.

Methods We consider the design and analysis of three-phase ITS studies using a study funded by National Institutes of Health as an exemplar. The design and analysis of both one-arm and two-arm three-phase ITS studies are introduced.

Results A simulation-based approach, with ready-to-use computer programs, was developed to determine the power for two types of three-phase ITS studies. Simulations were conducted to estimate the power of segmented autoregressive (AR) error models when autocorrelation ranged from -0.9 to 0.9 with various effect sizes. The power increased as the sample size or the effect size increased. The power to detect the same effect sizes varied largely, depending on testing level change, trend changes, or both.

Conclusion This article provides a convenient tool for investigators to generate sample sizes to ensure sufficient statistical power when three-phase ITS study design is implemented.


Impact of free access to leisure facilities and community outreach on inequalities in physical activity: a quasi-experimental study

Re:fresh programme introduced free access to leisure facilities in Blackburn and Darwen in July (50+), September 2008 (16-24) and then April 2009 (25-49)

5 full time health trainers offered around 700h 1-1 coaching per year

see notes

Offered in more deprived areas to reduce inequality

see notes

On average upward trajectory

Don’t know whether it represents new people or existing members


advantages of time series design

Enables detection of whether programme effects LT or ST

Series of tests before intervention can eliminate need of control

Series of tests before can be used to project results which would be expected

Can be used if only one site to conduct eval

Can detect secular trends

Good external val

Kontopantelis et al. (2015)
- Interrupted time series analysis is arguably the “next best” approach for dealing with interventions when randomisation is not possible or clinical trial data are not available
- Although several assumptions need to be satisfied first, this quasi-experimental design can be useful in providing answers about population level interventions and effects


disadvantages of time series design

Problem of confounding

Changes in instruments during series of measurements

Loss/change of cases

Changes in group composition

Kontopantelis et al. (2015)
- However, their implementation can be challenging, particularly for non-statisticians


The effectiveness and cost-effectiveness of a complex community sport intervention to increase PA: an interrupted time-series design (Anokye et al., 2018)

○ Effectiveness and cost-effectiveness analyses of 2-staged community sports interventions; taster sports sessions compared w/ portfolio of community sport sessions

○ Quasi-exp using interrupted time series design

○ Community sports projects delivered by 8 lead partners in London

○ Inactive people aged 14+ (n=246)

○ Community sports interventions delivered in 2 stages, 6-week programme of taster sport sessions (stage 1) and 6-week programme of portfolio of community sporting sessions delivered by trained coaches (stage 2)

Outcome measures
a. Change in days with >=30min self-reported vigorous intensity PA, moderate intensity PA, walking and sport
b. Change in subjective well-being and EQ5D5L quality-adjusted life-years (QALYs)

○ Interrupted time series analysis evaluated effectiveness of 2-staged sports programmes
○ Cost-effectiveness analysis compared stage 2 with stage 1 from providers perspective, reporting outcomes of incremental cost per QALY (2015/16 price year)
○ Uncertainty assessed using deterministic and probabilistic sensitivity analyses

○ Counterfactual change at 21 days lower for vigorous, moderate PA and sport
○ Stage 2 increased walking
○ Counterfactual change as 21 days in well-being pos particularly for 'happiness'
○ Stage 2 more expensive but increased QALYs
○ Cost per QALY for stage 2 £50000 and has 29% chance of being cost-effective (£30000 threshold)

○ Community-based sport interventions could increase PA among inactive people
- Less intensive sport sessions may be more effective and cost-effective