Week 10 - Assessing Research Flashcards
(16 cards)
Faulty conclusions in research result from:
Internal validity threats - whether DV change is actually due to IV
External validity threats - whether findings are generalisable
Vaccines and autism example
1998 - described vaccines as ‘precipitating event’ for autism
Paper retracted in 2010 - biased sample, data manipulation, undisclosed COIs
Burden of misconduct
Most retractions involve scientific fraud (fabrication, falsification, plagiarism) or other misconduct (fake peer review)
Critical appraisal
Assessment of strengths of research against limitations (process + results)
Must consider all aspects - RQ clarity, design quality, stat analyses, result interpretation
Inherently retrospective (unless registered report)
Very important - whether to ‘believe’ effect, build future research off it, endorse particular therapy or intervention
Formal assessment tools
Assessment of quality
- CASP (Critical Appraisal Skills Program) - for specific study types
- Cochrane’s Risk of Bias - for RCTs specifically
Article reporting guidelines
- CONSORT checklist (Consolidated Standards of Reporting Trials)
- APA JARS
Assessment of strength of evidence for specific interventions
- NHMRC evidence hierarchy
NHMRC evidence hierarchy
Only thing better than RCTs or prospective cohort studies are systematic reviews of those studies
NHMRC body of evidence & grades of recommendation
Evidence base, consistency, clinical impact, generalisability, applicability
(all ranked from A to D)
Gives overall A to D grade
CASP checklist (RCP version)
A. Are results valid?
- 1. clearly focused issue, 2. random assignment, 3. all participants accounted for at conclusion, 4. blind study, 5. similar groups at start, 6. groups treated equally
B. What are the results?
- 7. size of effect, 8. precision of effect
C. Will results help locally?
- 9. can results be applied to local population, 10. were all clinically important DVs considered, 11. are benefits worth harms/costs
Appraisal guidelines
Geared to intervention studies, looking for scientific evidence of potential effects
RCTs considered gold standard (trumped by syntheses)
Need to consider why someone is reading article - research direction, clinical practice, personal interest (this will affect which appraisals get most weight)
Gamification study example - A. are results valid?
- Clear focus
- Random allocation
- Details about attrition rates given
- Not fully blind (but tricky in psychology)
- Baseline groups similar on demographics but differed on key outcomes (floor/ceiling effects)
- Waitlist group treated differently (received less emails)
How to deal with attrition?
Check for differential attrition (could lead to bias)
Simple imputation - last observation carried forward or mean imputation
Multiple imputation (best) - sophisticated missing data imputed
Intention-to–treat protocol is the ideal - includes all participants randomized in a trial, regardless of whether they completed it, in the analysis
Gamification study example - B. what are the results?
- Active group differed from waitlist but not control on most things (not clearly reported), decent adherence effect (especially in active group)
- No error bars or CIs included (poor precision
Gamification study example - C. will the results help locally?
- Fairly narrow and homogenous sample
- Reliability of measures seems good, knowing if all important outcomes were considered requires discipline knowledge
- Study is robust but room for improvement, still benefits do not seem worth it as intervention is no better than existing treatment
Quality assessment
Global assessment of paper quality (assign scores to appraisal criteria, then sum scores)
- Pros - quantifies quality, enables comparisons, regardless of journal IF
- Cons - assumes same weight of items, score could fail key areas and pass overall
Journal quality
Peer-review should reduce bias (but different standards of rigour, predatory journals)
Journals have impact factor (JIF) - citations over citable items in 2 year period
- Does this measure quality? - not necessarily (disciplines with few researchers, squeaky wheel)
- Different types of IF exist with different data weighting
- Can IF mislead? - some article types more likely to be cited than others, journals can act to increase their IF (excessive self-citation)
How to evaluate and do good science
Use the assessment tools to better design, report and evaluate studies
Consider pre-registration or data sharing
Open Science Framework
