L11 - Reviewing the evidence

Question 1

What is a meta-analysis?

Answer 1

A quantitative synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way

Question 2

Describe the limitations of using evidence from single studies

Answer 2

a

Question 3

What is the purpose of a systematic review?

Answer 3

A systematic review summarises the results of available carefully designed healthcare studies (controlled trials) and provides a high level of evidence on the effectiveness of healthcare interventions. Judgments may be made about the evidence and inform recommendations for healthcare.

Question 4

Under what circumstances should a systematic review be undertaken?

Answer 4

It should have:

1. Clearly focused question
2. Explicit statements about:
   (i) types of study
   (ii) types of participants
   (iii) types of interventions
   (iv) types of outcome measures
3. Systematic literature search
4. Selection of the material
5. Appraisal
6. Synthesis (possibly including a meta-analysis)

Question 5

What is a systematic review?

Answer 5

An overview of primary studies that used explicit and reproducible methods.

Question 6

What is a ‘forest-plot’?

Answer 6

A graphical representation of a meta-analysis which plots a measure of study size against a measure of effect (e.g. Odds Ratio)

Question 7

What are the common difficulties in systematic reviews?

Answer 7

a

Question 8

What are the common difficulties in meta-analyses?

Answer 8

1. Heterogeneity between studies
   (i) modelling for variation: fixed vs random effects model
   (ii) analysing the variation: sub-group analysis
2. Variable quality of the studies
3. Publication bias in the studies

Question 9

What is publication bias?

Answer 9

When studies with statistically significant or ‘favourable’ results are more likely to be published than those studies with non-statistically significant or ‘unfavourable’ results - this particularly applies to smaller studies.

Question 10

What effect does publication bias have?

Answer 10

Publication bias leads to a biased selection of studies towards the demonstration of effect

Question 11

List three types of observational study: one descriptive and two association studies

Answer 11

Descriptive (proportion) - prevalence study

Association (ratio) - cohort and case-control studies

Question 12

Name an experimental study type

Answer 12

Randomised controlled trials (RCTs) (comparison - ratio)

Question 13

What is the difference in the analysed ratios of cohort/case-control studies and randomised controlled trials?

Answer 13

Cohort/case-control studies is a ratio of association

RCTs is a ratio of comparison

Question 14

Name three types of broad classification or studies

Answer 14

1. Primary research studies
2. Literature reviews: narrative, systematic
3. Decision analyses

Question 15

What is the purpose of a decision analysis study?

Answer 15

Harms and benefits

Cost-effectiveness

Question 16

Describe the features of a narrative review

Answer 16

Implicit assumptions, opaque methodology, not reproducible -> biased and subjective

Question 17

Describe the feature of a systematic review

Answer 17

Explicit assumptions, transparent methodology, reproducible -> unbiased and objective

Question 18

List some type of primary research studies

Answer 18

RCTs
Cohort studies
Case-control studies

Question 19

What are the purposes of a meta-analysis?

Answer 19

1. To facilitate the synthesis of a large number of study results
2. To systematically collate study results
3. To reduce problems in interpretation due to variations in sampling
4. TO QUANTIFY EFFECT SIZES AND THEIR UNCERTAINTY AS A POOLED ESTIMATE

Question 20

What are the quality criteria for a meta-analysis?

Answer 20

They should have a formal protocol specifiying:

1. Compilation of complete set of studies
2. Identification of common variable or category definition
3. Standardised data extraction
4. Analysis for sources of variation

Question 21

Why would a systematic review not necessarily include a meta-analysis?

Answer 21

If e.g. clinical heterogeneity was too great.

Question 22

How does a meta-analysis analyse all the studies?

Answer 22

1. ORs and % CIs calculated for all studies
2. Pooled estimate OR
3. Studies weighted due to their size and the uncertainty of their OR (smaller EF, greater weight to result)

Question 23

How do you interpret a forest plot?

Answer 23

ORs are represented by squares
95% CIs by libes
Size of square proportional to weight given to the study
Diamond=pooled estimate
Diamond centre (dotted line) = pooled OR
Diamond width= pooled 95% CI
Solid line = null hypothesis OR

Question 24

What are the assumptions of the fixed effect model for calculating the pooled estimate odds ratio and 95% CI?

Answer 24

That the studies are estimating exactly the same effect size

Question 25

What are the assumptions of the random effects model for calculating the pooled estimate odds ratio and 95% CI?

Answer 25

Assumes that studies are estimating similar, not the same, effect sizes

Question 26

Random effects modelling can only be used for modelling variation (accounting for it), what can be used to explain variation?

Answer 26

Sub-group analysis can help explain heterogeneity which may provide further insight into the effect of a treatment or exposure

Question 27

What are the two types of sub-group analysis?

Answer 27

1. Stratification by study characteristics e.g. length of follow-up 2. Stratification by participant profile e.g. age group, sex (however data is often unavailable)

Question 28

Studies are of variable quality. List from easier to assess to harder to assess the following causes of poor quality studies: Poor protocol implementation poor study design poor design protocol

Answer 28

poor study design poor design protocol poor protocol implementation

Question 29

``` List the following studies from least prone to bias and confounding to most prone: randomised controlled trials cohort studies non-randomised controlled trials case-control studies ```

Answer 29

randomised controlled trials non-randomised controlled trials cohort studies case-control studies

Question 30

What are the two approaches for assessing the quality of studies that are included in meta-analyses

Answer 30

1. Define a basic standard that studies must meet to be included e.g. Cochrane studies used to only include RCTs 2. Score each study for its quality and then: (i) incoportate the quality store into the weighting allocated to each study (ii) use sub-group analyses to explore differences (iii) meta-regression analyses - i.e. weighted linear regression of effect size against quality

Question 31

What components of e.g. RCTs need to be considered when assesing their quality?

Answer 31

allocation methods e.g. randomisation blinding and outcome assessment patient attrition - intention-to-treat appropiate statistical analysis

Question 32

How can you check for publication bias in the selection of studies?

Answer 32

1. Check meta-analysis protocol for how they searched for and identified unpublished studies 2. Plot identified studies against a measure of their size - A FUNNEL PLOT 3. Use a statistical test for publication bias (tend to be weak statistical tests)

Question 33

What does a 'funnel plot' plot?

Answer 33

A measure of study size (y-axis) against a measure of effect (x-axis) (e.g. Odds ratio)

Question 34

How would you analyse a funnel plot for publication bias?

Answer 34

Smaller studies can be expected to vary further from the central effect size. Publication bias is likely to exist if there are few small studies with results indication small or 'negative' measure of effect

Question 35

What is the p-value of tests for (statisical) heterogeneity if the differences in individual trial results are not consistent with heterogenity (e.g. systematic heterogeneity between trial results is likely)?

Answer 35

p=

Question 36

If heterogeneity test results for a meta-analysis were below 0.05 why would that raise concerns about the analysis?

Answer 36

It would raise concerns as to whether the individual trial results should be pooled. A statistical solution would be to use a different statistical analysis based on a 'random effects' model rather than a 'fixed effects' model, giving wider confidence intervals

Question 37

Give an example of a critical appraisal tool that can be used for systematic reviews

Answer 37

CASP