January 20th

Question 1

CRITICAL EVALUATION OF PUBLISHED RESEARCH-

Answer 1

despite peer review, validity of design or conclusions not guaranteed
ultimately health professional responsible for judging validity and relevance of published material
proper attitude is resolute skepticism, due to:
probabilistic and provisional nature of science
investigation of complex phenomena
ethical or economic constraints to desired levels of control

Question 2

aim of critical evaluation:

Answer 2

identify strengths and weaknesses of research publication

ensure patients receive assessment and treatment based on best available evidence

Question 3

Examine:

Answer 3

criteria for critical evaluation of quantitative research paper
implications of identifying problems in design, measurement and analysis in a publication
strategies for summarizing and analyzing evidence from a set of papers
implications of critical evaluation of research for health care practice

Question 4

CRITICAL EVALUATION OF PUBLISHED RESEARCH

Answer 4

despite peer review, validity of design or conclusions not guaranteed
ultimately health professional responsible for judging validity and relevance of published material
proper attitude is resolute skepticism, due to:
probabilistic and provisional nature of science
investigation of complex phenomena
ethical or economic constraints to desired levels of control

Question 5

CRITICAL EVALUATION OF THE INTRODUCTION

Answer 5

inadequacies may signal research erroneously conceived or poorly planned
Adequacy of the literature review
sufficiently complete to reflect current state of knowledge in the area
no significant omissions relevant to the presented research

unbiased in presenting unfavorable points of view

Clearly defined aims or hypotheses
clearly and operationally stated
if lacking, conceptual advances ambiguous
Selection of appropriate research strategy
strategy appropriate to aims
eg. Survery inappropriate for demonstrating causal effects correlation, not causeexperimental instead

Selection of appropriate variables
operational definition of variable should be suitable to phenomenon studied

Question 6

Internal Validity

Answer 6

ability to attribute differences or changes observed to the independent variable

Question 7

External Validity –

Answer 7

extent to which results can be generalized to other samples or situations
Subjects
– shows if sample representative
Sampling model – unbiased, optimizes representation?
Sample size – appropriate to heterogeneity of population?
ensures representative
important for statistical power
Description of the sample – clear description of characteristics/variables (eg. demographics)?

Validity and reliability
used standardized apparatus or measurement tool?
established reliability and validity of new apparatus?—compare to another instrument or method
Description of measurement tool
full description of novel instrumentation?
allows to be replicated

Question 8

methods- procedure

Answer 8

full description necessary for replication and evaluation of internal/external validity
Adequacy of the design
controls for extraneous influences?
ie. minimizes threats to internal validity
should control for alternative explanations of the data
Control groups
used (eg. placebo, no treatment, conventional treatment)?
controls for extraneous effects
if not used, internal validity questionable, size of effect difficult to estimate

Question 9

mwethods Procedure (cont.)
Subject assignment

Answer 9

avoids initial differences between subject groups?

also important in quasi-experimental or natural comparison studies

Question 10

Treatment parameters

Answer 10

describes all treatments to different groups?

treatments have same intensity and applied equally by different personnel?

Question 11

Rosenthal (bias) and Hawthorne (expectancy) effects

Answer 11

ie. if experimenters/observers or subjects aware of aims and predicted outcomes
uses single or double blind design?

Question 12

methods- procedure

Settings

Answer 12

describes sufficiently to evaluate generalizability?

has implications for external validity (ecological)

Question 13

Procedure (cont.)

Times of treatments and observations

Answer 13

clearly indicates sequence of treatments and observations?
variability in treatment and observation times can affect internal validity
controls for or considers series effects? (eg. that pre-tests produce effect)

Question 14

CRITICAL EVALUATION OF THE RESULTS

Answer 14

statistically correct summary and analysis of the data?
inadequacies could have produced erroneous inferences
complete summaries of all relevant findings?
Tables and graphs
correctly tabulated or drawn?
adequately labeled for interpretation?

Question 15

results: Selection of statistics

Answer 15

appropriate descriptive and inferential statistics used?
if not appropriate could distort findings/inferences
Calculation of statistics
no errors?
computers generally ensure

Question 16

CRITICAL EVALUATION OF THE DISCUSSION

Answer 16

draws inferences from data in relation to aims or hypotheses
if inferences incorrectly made, conclusions may lead to useless or dangerous treatments being offered

Question 17

Drawing correct inferences from the data

Answer 17

considers limitations of descriptive and inferential statistics?
Eg. correlations don’t necessarily imply causation
Eg. lack of significance could be type II error (miss of effect)

Question 18

Logically correct interpretations of the findings

Answer 18

interpretations follow from statistical inferences, without introducing extraneous evidence?
Eg. n = 1 design – shouldn’t claim procedure generally useful

Question 19

Protocol deviations

Answer 19

indicates and takes into account unexpected deviations from intended design?
Eg. placebo/treatment code broken
Eg. “contamination” between control and treatment groups discovered
researchers obligated to report so that implications for results can be considered

Question 20

Generalization from the findings

Answer 20

data from sample only generalized to that population?

frequent tendency to generalize to subjects or situations not considered in original sampling

Question 21

Statistical and clinical significance

Answer 21

appropriate conclusion of clinical applicability/significance?
should consider size of effect, side-effects, cost effectiveness, value of outcome

Question 22

Theoretical significance

Answer 22

relates results to previous relevant findings?

unless logically related to the literature, theoretical significance of findings unclear

Question 23

OVERALL EVALUATION

Answer 23

even if investigation flawed, may be useful information to be drawn
negative results are also useful
Journal of Negative Results in Biomedicine (est. 2002)
Journal of Negative Results in Ecology and Evolutionary Biology (est. 2004)
Text – Table 23.1 Checklist

Question 24

CRITICAL EVALUATION OF THE LITERATURE: META-ANALYSIS

Answer 24

unusual that results of a single research publication sufficient for clinical decision
need to consider multiplicity of papers
Review – critical summary of literature and implications
Meta-analysis – systemic procedures to summarize overall implications of set of papers

Question 25

To review/evaluate literature:

Answer 25

1. identify relevant literature (Section 2 – Research Planning)
2. evaluate critically the key papers (as discussed in this Section)
   might reject some if have irrepairable errors or don’t fit selection criteria
3. identify patterns of findings in the literature
   tabulate (table listing the previous findings in the literature)
4. identify crucial disagreements and controversies
5. propose valid explanations for disagreements
   provide theoretical framework for resolving controversies, proposing future research

Question 26

Two main strategies for summarizing findings from multiple papers:

Answer 26

a. Quantitative – condense results from several papers into a single statistic
represents average effects size

b. Qualitative – tabulate key features of related publications
Eg. designs, subject characteristics, measures used
allows relating differences in features to outcomes

Question 27

Qualitative comparison (cont.)

Answer 27

tabulates key info and outcomes
enables emergence/demonstration of pattern
clear pattern doesn’t always emerge
conflict might emerge about nature and causes of findings
eg. Table 22.2 – large difference between results of Smith and Jones versus those of Brown and Miller
explanation not necessarily true, but hypothesis to guide future investigations