Critical Appraisal Flashcards
(24 cards)
What is critical appraisal?
Process of assessing and interpreting evidence from a research paper, by systematically considering its validity, results and relevance.
To determine whether evidence is ‘good’ must consider.
Who, When, Where, How and What
Who has carried out the research?
Is it someone who has reputation in the filed? do they look as though they are qualified to be carrying out this research? One person or team of researchers. Where are they based? Is location likely to have a bearing on your practice? Are you certain no biases and no vested interest e.g. funded by a company who would benefit from positive research results?
When and where was it published?
Should be published in a peer-reviewed journal. Reviewer should be blind to who the author and institution is, and their role to ensure the research is not flawed. Look at citation indices.
Be careful of any papers over 10 years old which are not continuing to be cited.
How is the research designed?
You need to think about participants and the sample size. Are these relevant to larger population you are supposed to be studying? The methodology should be transparent so it can be replicated. Is there a control group or comparison with another intervention? Is there any inter- and intra-rater reliability? Have they controlled bias as much as possible?
What did they find/conclude?
Were the aims stated clearly enough that you can see whether they were met or whether the results have been written up to fit an agenda (confirmation bias).
Do the authors compare their work to other studies and critically evaluate if they have found similar results/ something new and why?
Do the authors reflect on the limitations of the research and discuss how it might be improved?
How to do critical appraisal (4 main areas)
- Aims/method
- Participants
- Intervention
- Analysis/outcomes
Aims/methods- questions to asks
Are the aims of the study clearly stated
Is there a theoretical rationale for intervention
Is the review of the literature adequate
Are the methods used to address research questions appropriate?
Are those assessing the cases blind to their status/blind to the intervention that is being given (double blind)?
Was there rationale for the sample size?
Participants
- Who were the participants (Age, SES, severity, motivation etc.) is this the group you are studying?
Background factors e.g. bilingualism, hearing, cognitive levels- these could be confounding factors that impact the outcomes.
- Participants in comparison group comparable to the intervention group? Same number in each group? Same age-range? Communication disorder in each group (not just SSD- lots of types!) must clearly describe the type of communication disorder and time since onset of disorder.
- How were the participants allocated to groups?
- Were all subjects seen throughout the study- was the issue of attrition addressed?
Intervention- questions to ask self
Is it clear what the intervention consisted of? What happened, how long for, by whom? Who provided the intervention? Normal practitioners or “special provision”.
Is it clear what happened to the comparison group- nothing, other specified intervention?
Were interventions common practice or a new intervention type?
Any ethical implications of interventions?
Would it be possible to repeat interventions given a comparable group of participants?
Analysis/outcomes- questions to ask
Was the study analysed appropriately?
Were the most appropriate outcomes used? e.g. if about SSD, was PCC used?
Valid/reliable outcome measures? (standardised assessment?)
How real were the changes- were the differences statistically significant? - look at P value
Are the results meaningful in the current SLT context?
Easy Pros/Cons of a study to learn for discussion:
Were groups comparable?
Were outcome measures objective (quantitative data) or subjective measures (patient-reported outcome measure)
Were biases reported?
Lack of follow-up
Small sample sizes
Reliability on transcription (qualitative study) reliability of interview transcription.
Should I use this intervention/approach? 3 KEY QUESTIONS
- What are the results?
- Are the results of the study valid?
- Will the results help me in caring for my own client/patient?
What are threats to validity?
Any features of the study which make you doubt integrity of the results
SELECTION BIAS
participants allocated disproportionately not randomly.
Need parity across groups in terms of age, SES, severity, condition.
SAMPLING BIAS
When the sample is not representative of the target population.
e.g. aim is to research pre-schoolers. in methodology had to recruit higher age group for numbers.
EXCLUSION BIAS
when particular groups from the sample are excluded.
Particularly look at follow-up, who has been unable to follow up? Low SES, cannot complete whole trial over the time.
PERFORMANCE BIAS
When researcher/participant know which group they are in/what treatment they are getting. Sometimes not possible to get over in SLT.
DETECTION BIAS
Outcomes are influenced by the knowledge of the researcher/participant.
e.g. SSD research. Pre and post-therapy measures. Someone transcribing speech but they know which group pre and post, may be subject to detection bias.
CONFIRMATION BIAS
when the researcher shapes the methodology/ analysis to fit hypothesis.
QUESTION-ORDER BIAS (qualitative)
Order of questions asked in influence answers given
RECALL BIAS (qualitative)
Studies where people are questioned about something in the past, their recollections may be effected by current realities.
ACQUIESENCE BIAS (common in service evaluation/audit)
Occurs when people want to be agreeable; they are more likely to answer affirmatively.
PUBLICATION BIAS
When studies are carried out but not published.
