Week 11- Evidence Level in Medical Research

Question 1

What is the aim of evidence based medicine/practice?

Answer 1

• Aims to optimize medical practice and decisionmaking by emphasizing the use of evidence from well
  designed and conducted research
• Medical evidence is classified by its scientific strength,
  requiring that only the strongest types of research can
  yield strong recommendations

Question 2

What is the hiearchy of evidence?

Answer 2

Question 3

Why is evidence-based medicine important?

Answer 3

• High quality research is capable of changing or
  improving current clinical practice e.g., encourage
  healthcare professionals and systems to use
  treatments that have been proven to be both
  clinically and cost-effective, while disinvesting from
  practice that does not meet these objectives

Question 4

DATA SYNTHESIS AND EXPERT OPINION DIAGRAM

Answer 4

Question 5

What is a systematic review?

Answer 5

• A review of a clearly formulated question with
• systematic and explicit methods to identify, select,
  and critically appraise relevant research, and
• collect and analyse data from the studies that are
  included in the review.
• Statistical methods (meta-analysis) may or may not
  be used to analyse and summarise the results of the
  included studies.

Question 6

What are the steps for undertaking a systematic review?

Answer 6

1. Formulate a review question
2. Find studies
3. Select and Appraise relevant studies
4. Summarise and synthesise results
5. Interpret and apply results

Question 7

What should be considered when determining the research question?

Answer 7

1. PICO/PECO framework
   -> types of Participants
   -> types of Intervention (exposure)
   -> Comparators (e.g. placebo/other drugs)
   -> types of Outcomes
2. Types of studies (observational/intervention studies)

Question 8

How does meta-analysis fit into systematic reviews?

Answer 8

It is an optional part of systematic review

Question 9

What is meta-analysis?

Answer 9

• The statistical component of a systematic review
• It may not always be possible or appropriate to
  conduct meta-analysis as part of a systematic
  review
• Pooling together of results and measure
  overall/pooled association/estimates
• This will show as if the there is true difference
  /association and that the results are not due to
  chance

Question 10

What caused the use of meta-analysis in medical research?

Answer 10

• Formation of Cochrane Collaboration (1993)
  spurred growth in meta-analyses in medical
  research

Question 11

Why do we conduct meta-analysis as part of systematic review?

Answer 11

• A single small study may not give very precise
  results about the effect of a treatment
• Combining results from several studies gives
  more precise estimates
• May also increase generalizability of results to
  different countries/settings

Question 12

Why is meta-analysis important?

Answer 12

• Researchers put little faith in a single study of an effect, no
  matter how good the study and how statistically significant
• When many studies were done, someone would write a
  narrative (= qualitative) review trying to explain why the
  effect was/wasn’t real in the studies
• Each study produces a different estimate of the magnitude
• Enlightened researchers now realize that all effects are real
• The aim of research is therefore to get the magnitude of an
  effect with adequate precision
• Meta-analysis combines the effects from all studies to give
  an overall mean effect and other important statistics

Question 13

When is meta-analysis not performed?

Answer 13

• Do not have data in suitable format
• Studies are poor quality or at risk of bias
• HETEROGENEITY Studies not sufficiently
  similar
  – differences in:
• Populations (eligibility criteria)
• Interventions
• Outcome definitions
• Effect estimates

Question 14

What is quality assessment (critical appraisal)

Answer 14

• Good quality and poor quality studies
• Many validated quality assessment tools
• Some used more than the others
• Choose a reliable one
• Conclusion we made depends on the quality
  of studies

Question 15

What are examples of quality assessment tools for non-intervention and intervention studies?

Answer 15

• Risk of bias tools for use in systematic reviews of non intervention
  studies:
• The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised
  studies in meta-analyses (cohort and case control studies
• NICE Quality appraisal checklists
  – Appendix D Methodology checklist: cohort studies
  – Appendix E Methodology checklist: case–control studies
• Risk of bias tools for use in systematic reviews of intervention
  studies:
• RoB 2.0 tool (RCTs)
• ROBINS-I tool (Risk Of Bias in Observational Studies focused on Interventions)

Question 16

What is publication bias?

Answer 16

• A meta-analysis reflects only what’s published.
  – But statistically significant effects are more likely
  to get published.
  – Hence published effects are biased high.
  Publication bias = selective publication of studies
• investigators fail to report completed studies
• typically those that show ‘no effect’
• suspect when a limited number of small trials all
  suggest benefit
• big ‘bad pharma’
  suppresses evidence

Question 17

What are the different organisations that provide levels of evidence?

Answer 17

-Oxford Centre for Evidence Based Medicine (2009) (WHAT WE USE)
– National Institute for Health and Care Excellence (NICE) levels of
evidence (2001)
– Australian National Health and Medical Research Council
(NHMRC) levels of evidence (2009)
– United States Preventive Services Task Force (USPSTF) levels of
evidence (1988)
– Grading of Recommendations Assessment, Development and
Evaluation (GRADE) levels of evidence (2004)

Question 18

What are the levels of evidence (specific)

Answer 18

Question 19

Explain evidence level 1A

Answer 19

• Types of studies:
  – Systematic reviews/meta-analyses of randomized controlled
  trials
• Strengths:
  – Interventional study (exposure assigned not assessed)
  – High internal validity (very low random error, bias,
  confounding)
  – High external validity (assuming representative samples and
  low drop out in studies)
  – Can prove temporality in associations
  – Strongest evidence for inferring causality
• Limitations:
  – Depend on the quality of included studies; i.e. if original
  studies poorly conducted, this cannot be fixed

Question 20

Explain Evidence Level 1B

Answer 20

• Types of studies:
  – High quality individual randomized controlled trials (narrow
  confidence intervals, low drop out, random allocation
  maintained)
• Strengths:
  – Interventional study (exposure assigned not assessed)
  – High internal validity (low random error, bias, confounding)
  – Can prove temporality in associations
  – Can infer causality (but with caution - single study)
• Limitations:
  – Single study, so external validity could be low if the study
  sample is too specific (non-representative)

Question 21

Explain Evidence Level 1C

Answer 21

• Types of studies:
  – High quality individual non-randomized/non-controlled trials
  (narrow confidence intervals, low drop out)
• Strengths:
  – Interventional study (exposure assigned not assessed)
  – Moderate internal validity (lower bias than in observational
  studies)
  – Can prove temporality in associations
• Limitations:
  – Prone to confounding due to lack of randomisation (see
  session 11a)
  – Single study so external validity could be low if the study
  sample is too specific (non-representative)

Question 22

Explain Evidence Level 2A

Answer 22

• Types of studies:
  – Systematic reviews/meta-analyses of cohort studies
• Strengths:
  – Acceptable internal validity (usually very large
  sample, low
  random error)
  – High external validity (assuming representative samples in
  studies)
  – Can prove temporality in associations
• Limitations:
  – Prone to bias and confounding due to methodological
  limitations of included studies (observational) (see session
  10a)
  – Causality cannot be inferred

Question 23

Explain Evidence Level 2B

Answer 23

• Types of studies:
  – High quality individual cohort studies (narrow confidence
  intervals, low drop out, accurate assessments, adjustment for
  confounding)
  – Low quality individual RCTs (wide confidence intervals, high
  drop out, random allocation not maintained)
• Strengths:
  – Acceptable internal validity (large sample, low random error) –
  applies only for cohort studies in this case
  – Can prove temporality in associations
• Limitations:
  – Prone to bias and confounding due to methodological
  limitations of included studies (see session 10a)
  – Causality cannot be inferred

Question 24

Explain Evidence Level 3A

Answer 24

• Types of studies:
  – Systematic reviews/meta-analyses of case-control studies
• Strengths:
  – Usually very large sample, low random error
  – High external validity (assuming representative samples in
  studies)
• Limitations:
  – Low internal validity (bias, confounding) (see session 10a)
  – Cannot prove temporality in associations
  – Causality cannot be inferred

Question 25

Explain Evidence Level 3B

Answer 25

• Types of studies:
  – High quality individual case-control studies (narrow
  confidence intervals, accurate assessments, no serious recall
  bias, adjustment for confounding)
• Strengths:
  – Large sample, low random error
• Limitations:
  – Low internal validity (bias, confounding) (see session 10a)
  – Cannot prove temporality in associations
  – Causality cannot be inferred

Question 26

Explain Evidence Level 4

Answer 26

• Types of studies:
  – Case-series (following-up a group of patients who receive a
  specific treatment and observe their prognosis)
  – Cross-sectional studies (session 10a)
  – Low quality cohort and case-control studies (wide confidence
  intervals, high bias, affected by confounding)
• Strengths:
  – Easy to conduct and cheap
  – Can generate hypotheses for further higher quality research
• Limitations:
  – Very low internal validity (random error, bias, confounding)
  – Cannot prove temporality in associations (apart from cohort)
  – Causality cannot be inferred
  – External validity low

Question 27

Explain Evidence Level 5

Answer 27

• Types of studies:
  – Individual expert opinion/report
  – Expert committee opinion/report
  – Clinical experiences of respected authorities
• Strengths:
  – Very easy to conduct and usually does not involve a cost
  – Could generate hypotheses for further higher quality research
• Limitations:
  – Do not qualify as scientific research!
  – Not based on empirical research findings, just clinical
  observations
  – Both internal and external validity extremely low
  – Definitely cannot infer causality

Question 28

What are the 4 key factors that affect strength of recommendation?

Answer 28

• Quality of evidence
• Balance desirable/undesirable effects
• Values and preferences
  (patients/professionals/paymasters/public)
• Cost (utilization of available resources)

Question 29

Explain the relationship between quality of evidence and strength of recommendation

Answer 29

You can have a weak recommendation, but high quality evidence or strong recommendation, but low quality evidence

Question 30

What does grade (quality of evidence) include?

Answer 30

• Imprecision 95% CI
• Limitations in study design (bias) quality
• Inconsistency of results heterogeneity
• Publication bias publication bias
• Indirectness of evidence generalisability

Question 31

What is ‘GRADE’?

Answer 31

• A common, sensible and transparent approach to grading
  quality (or certainty) of evidence and strength of
  recommendations

Question 32

Why rate the certainty in the evidence and strength of
recommendations?

Answer 32

– judgments about evidence and recommendations in
healthcare are complex
– clinical actions are likely to differ depending on whether
one concludes that the evidence is low or high
– also need to decide whether any incremental health
benefits are worth the additional costs
– can help to improve communication of practice guidelines

Question 33

What are the levels of quality acc to GRADE approach?

Answer 33

Question 34

What factors may decrease quality level of evidence?

Answer 34

Question 35

What factors may increase quality level of evidence?

Answer 35