Give 2 definitions for evaluation
Evaluation is the assessment of whether a service achieves its objectives
OR
Evaluation is a process that attempts to determine as systematically and objectively as possible the relevance, effectiveness and impact of activities in the light of their objectives
Give some examples of types of evaluation in healthcare
Evaluation of: health services, single intervention, public health intervention
Give 2 frameworks for health service evaluation?
Donabedian’s:
- Structure
- process + output
- outcome
Maxwell’s 3A’s and 3 E’s
What is evaluated in terms of ‘structure’ of a health service?
What there is
Buildings, staff, equipment
e.g. number of ICU beds per 1000, number of vascular surgeons per 1000…
What is evaluated in terms of ‘process’ of a health service?
What is done
e.g. number of patients seen in A+E, HOW IS PRIORITY assessed in a/e, number of operations performed
What is evaluated in terms of ‘outcome’ of a health service?
Classification of health outcomes: mortality (30 day mortality rate), morbidity (complication rates), QoL/PROMs, patient safety
Or another classification (5 Ds): death, disease, disability, discomfort, dissatisfaction
What are the 5 Ds when evaluating the outcome of a health service?
death, disease, disability, discomfort, dissatisfaction
What is an example of a PROMs?
Oxford hip score questionnaire
How can Quality of healthcare be evaluated?
Donebedian’s: structure, process, outcome
OR
Maxwell’s dimensions of quality (3Es + 3As):
Effectiveness, efficiency, equity
Acceptability, accessibility, appropriateness (relevance)
Describe the 3 Es of maxwell’s dimensions of quality
Effectiveness: Does the intervention / service produce the desired effect?
Efficiency: Is the output maximised for a given input (or is the input minimised for a
given level of output)?
Equity: Are patients being treated fairly?
Describe the 3 As of maxwell’s dimensions of quality
Acceptability: How acceptable is the service offered to the people needing it?
Accessibility: Is the service provided? Geographical access; Costs for patients; Information available; Waiting times
Appropriateness: Is the right treatment being given to the right people at the right time? [Overuse? Underuse? Misuse?]
What are the two methods of evaluation of health services?
Qualitative
Quantitative
What is qualitative evaluation? Give some examples
Consult relevant stakeholders as appropriate (e.g. staff, patients, relatives and carers, policy makers,
commissioners as appropriate)
Methods: observation, interviews, focus groups, review of documents / questionnaires
Give examples of quantitative evaluation?
- Routineley collected data (e.g. hospital admissions + mortality)
- Review of records (medical, administrative)
- Surveys
- Other special studies
Exam Q (will come up): Donabedian’s “structure, process, outcome” is a useful framework to use when carrying out evaluation of health services. Explain what is meant by “structure” and “process”
Structure: What there is (e.g. number of ICU beds per 1000, number of vascular surgeons per 1000)
Process: What is done (e.g. the number of patients seen in A+E)
Exam Q (will come up): When assessing the quality of health services, Maxwell’s classification lists six dimensions. List the six dimensions?
Maxwell’s dimensions of quality (3Es + 3As):
Effectiveness, efficiency, equity
Acceptability, accessibility, appropriateness (relevance)
Exam Q (will come up): Although using measures of health outcomes is desirable in evaluation of health services, there are potential limitations. Explain why it may be difficult to attribute a health outcome to the service provided.
(Issues with health outcomes)
- Link (cause and effect) between health service provided and health outcome may be difficult to establish as many other factors may be involved e.g. case-mix, severity, other confounding factors
- Time lag between service provided and outcome may be long e.g. between healthy eating intervention in childhood and incidence of Type 2 diabetes in middle age
- Large sample sizes may be needed to detect statistically significant effects
- Data may not be available
- There may be issues with data quality –> consider CART – Completeness, Accuracy, Relevance, Timeliness
What is the basic framework for evaluating health services?
structure, process, outcome
Exam Q: how can “outcome” be classified?
EITHER: mortality, morbidity, QoL/PROMs, Pt satisfaction
OR 5 D’s: death, disease, disability, discomfort, dissatisfaction
For hip replacement for OA, use structure/process/outcome to evaluate the service?
Structure: Number of orthopaedic surgeons / 1000 population
Process: number of operations in 1 year for OA
Outcome: Oxford hip score