Clinical Audit, Research or Service Evaluation

Question 1

What is an audit ?

Answer 1

a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change

AKA Does it reach a pre- determined standard?

Question 2

What is research ?

Answer 2

the attempt to derive generalisable new knowledge by addressing clearly defined questions with systematic and rigorous methods

AKA Are we doing it right according to clinical standards? If not, how can we improve?

Question 3

What is service evaluation ?

Answer 3

Review process undertaken solely to define or judge current service with the intention of benefiting those who use it

• Used to inform local practice
• May result in service redesign

AKA How well does a service achieve its intended aims (patient experience and outcomes)? (without reference to a standard)

Question 4

What is the purpose of audits, research, and service evaluation ?

Answer 4

RESEARCH
-Generalizable new knowledge

AUDIT
-Inform delivery of best care

SERVICE EVALUATION
-Define or judge current care

Question 5

Which kinds of interventions does research, audit, and service evaluation make use of ?

Answer 5

RESEARCH
-Novel use or application

AUDIT and SERVICE EVALUATION
-Already in use

Question 6

To what extent does research, audits, and service evaluation include data additional to usual care ?

Answer 6

RESEARCH
-Yes, including invasive tests

AUDITS and SERVICE EVALUATION
-Can include questionnaire / interview

Question 7

Are patients allocated to an intervention in research, audits, and service evaluation ?

Answer 7

RESEARCH
-Yes, usually

AUDITS and SERVICE EVALUATION
-No

Question 8

To what extent is there randomisation in research, audit, and service evaluation ?

Answer 8

RESEARCH
-Maybe

AUDITS and SERVICE EVALUATION
-No

Question 9

Identify the type of questions which may be asked as part of a service evaluation.

Answer 9

Does the service achieve its objectives? (including structure, process, and outcome)

• How ?
• What are the costs?

Does the service meet the patient’s needs ?

Is this service equitable ?

EXAMPLES
• What are the patient experiences of attending a community pulmonary rehabilitation service?
• What are the waiting times for pts attending the A&E department in XXX hospital?

Question 10

Identify the type of questions which may be asked as part of an audit.

Answer 10

Does this service reach a predetermined standard?

EXAMPLES
What proportion of pts presenting to AandE are seen within 4 hours?

What proportion of pts referred for urgent investigation of colorectal cancer are seen within 2 weeks?

Question 11

Identify the type of questions which may be asked as part of a resratch.

Answer 11

How do patient outcomes compare between this service (or treatment) and an alternative?

EXAMPLE
1) Quantitative research
What is the impact of a nurse-led dermatology clinic in primary care on the quality of life of children with eczema?

2) Qualitative reseatch
What are the concerns of women being recalled for a repeat cervical smear?

Question 12

Identify the main parts of a quantitative research question.

Answer 12

Parts to a (quantitative) research question: PICO

Patient / Intervention / Comparison / Outcome

Question 13

What is the link between audit and research ?

Answer 13

• Research findings can identify areas for audit
• Audit can be the final stage of a research project
• Audit can help with dissemination of research findings
• Audit can identify gaps in research evidence

Question 14

Identify an example of audit which has taken in Scotland.

Answer 14

Audit of implementation of NICE recommendations to measure CO in the breath of all pregnant women at booking.

Measured proportion of women offered CO breath test between two hospitals over span of two weeks. This led to a question: why the difference ? Which led to a Scotland-wide research study of opinions of midwives on the new policy

Question 15

What are the common elements between audits and research ?

Answer 15

➡Question-driven
➡Professionally led
➡Influence on clinical practice
➡Formal data collection – use of a proforma 
➡Can involve service users
➡Methodological rigour
➡Data analysis / interpretation
➡Publishable ??
➡Ethics: research yes, audit no but may be needed if it judged to put patients at risk

Question 16

Describe the main methods of audits.

Answer 16

Retrospective (notes review, BUT beware missing data)

Prospective (ongoing data collection BUT beware Hawthorne effect)

Question 17

Define Hawthorne effect.

Answer 17

“type of reactivity in which individuals modify an aspect of their behavior in response to their awareness of being observed”

Question 18

Identify the main steps involved in an audit.

Answer 18

1) Identify a topic / problem
2) Identify local resources (local audit dept)
3) Choose the standard, create the audit proforma
4) Define the sample
5) Collect data (? do a pilot)
6) Compare data with the
standard
7) Develop and implement
change
8) Re-audit

Question 19

What are the criteria for choosing an audit topic ?

Answer 19

Is the topic of high cost, high volume, or risk to staff or users?

Is there evidence of a serious quality problem, for example patient complaints or high complication rates?

Is evidence available to inform standards, for example systematic reviews or national clinical guidelines?

Is the problem amenable to change? Is the topic a priority

• for the organisation?
• for a national policy initiative?

Is there potential for involvement in a national audit project?

Question 20

What are the different kinds of standards which can be chosen for an audit ? What are the criteria for choosing the standard in an audit ?

Answer 20

Minimal: lowest acceptable level of performance

Ideal: the care possible under ideal conditions (e.g. 100% survival ? – elusive!)

Optimal: realistic under normal conditions of practice (somewhere between minimal and ideal)

Is it evidence based (e.g. from NICE, SIGN, Cochrane review) and related to aspects of care ?

Question 21

Identify the main steps in selecting the sample.

Answer 21

1) Identify inclusion (e.g. age, comorbidites) and exclusion criteria
2) Choose size of sample (with small sample size, large standard error and vice versa)

Question 22

What is standard error, in the context of audits ? What is the use of it ?

Answer 22

SE is a measure of the error of your estimate

Standard error (SE) of percentage = √ [ P (100 – P) / n] where ‘n’ = sample size and P is estimated percentage

SE used to derive a 95% confidence interval (interval in which we are 95% confident that the TRUE percentage lies)

Question 23

Interpret the following:

• ‘Standard’ = referral criteria
• 4 weeks, N of admissions = 93
• N of inappropriate admissions = 18
• Proportion = 18 / 93 = 0.193 or ~19%
• Standard error = 4% (measure of imprecision)
• ~2 x SE → 95% confidence interval
• 95% CI = 11 to 27%

Answer 23

• Interpretation: based on a sample of 93, we can be 95% confident that the true proportion of inappropriate admissions is between 11% and 27%

Question 24

In the following:
n=93, P=19%, SE (P) = 4%, 95% CI = 11% to 27%

Suppose we want a more accurate estimate, with a standard error of 2%, how many admissions would be needed ?

Answer 24

Use the formula

P = √ [ P (100 – P) / n ]

n = 385

To increase precision, increase sample size

Question 25

Graph relationship between sample size and standard error.

Answer 25

Refer to slide 24

Question 26

State who to contact in the NHS for an audit or service evaluation. For research.

Answer 26

Audit or Service Evaluation:
Clinical Effectiveness Department

Research:
Research and Development Office

Question 27

Read through this scenarios to decide whether it constitutes audit, research or service evaluation.

BACKGROUND: Oral chemotherapy agents provide patients with choice and home- based therapy but demand greater emphasis on patient support and education to minimise toxicities. To meet this demand, a nurse-/pharmacy-led clinic is to be established for the provision of oral capecitabine to metastatic colorectal cancer patients to provide a controlled and supportive environment.
QUESTION: Is a nurse-/pharmacy-led clinic for the provision of home-based oral capecitabine safe, effective and acceptable to patients?
METHODOLOGY: We propose to conduct a prospective review of notes of patients attending the clinic and a retrospective survey of patient experiences to assess clinic effectiveness. Information will be collected from notes on pt demographics, dosing schedules, number of treatment modifications, response to treatment, adverse effects and the need for consultations with healthcare practitioners. The survey questionnaire will seek information about the provision of information, attendance at the clinic, problems encountered during chemotherapy, and the support they received.

Answer 27

Service evaluation

Question 28

Read through this scenarios to decide whether it constitutes audit, research or service evaluation.

OBJECTIVE: To assess the knowledge of current guidelines and tetanus prevention practices of various staff members in accident and emergency (A&E) departments.
METHODS: A structured questionnaire on tetanus guidance, anti-tetanus practice, vaccination preparations and information on the population served by the department will be sent to medical and nursing staff in A&E departments across Scotland.
QUESTIONS: What proportion of A&E staff follow the national policy?
How does adherence to guidelines differ amongst staff grades and with years of experience?

Answer 28

Audit

Question 29

Read through this scenarios to decide whether it constitutes audit, research or service evaluation.

BACKGROUND: There is increasing evidence that clinical guidelines can lead to improvements in clinical care. However, they are not self-implementing.
QUESTION: Does an outreach visit by a pharmacist improve prescribing behaviour of GPs?
METHODS: We will distribute the recently published guidelines for management of Helicobacter pylori to all general practices in one CHP. Practices will be randomised to receive just the guidelines or the guidelines together with the offer of a visit and the conduct of an audit by a pharmacist trained in the techniques of outreach visiting.
MEASUREMENTS: The intervention will be evaluated using level three Prescribing Analysis and Cost (PACT) data for metronidazole and omeprazole for the 12 month period before and after the introduction of the guidelines.

Answer 29

Research

Question 30

Read through this scenarios to decide whether it constitutes audit, research or service evaluation.

BACKGROUND: Previous studies have identified medication errors in preparing and administering intravenous medicines of 13-84% in hospitals in individual countries.
OBJECTIVE: To compare the effect of the design and implementation of systems for the preparation and administration of intravenous therapy in hospitals in three European countries on the number of observed medication errors.
SETTING: Six hospital departments in the UK, Germany and France.
METHODS: Direct observation of the preparation and the administration of intravenous drugs made by a single observer in each country.
MAIN OUTCOME MEASURES: Medication process errors.

Answer 30

Audit + (Service Evaluation)

Question 31

Read through this scenarios to decide whether it constitutes audit, research or service evaluation.

BACKGROUND: Drug utilization review (DUR) programs are being conducted in hospitals with the aim of improving the appropriateness of prescriptions. However, there is little evidence of their effectiveness.
OBJECTIVE: To assess the impact of both a retrospective and a concurrent DUR program on the quality of in-hospital prescribing.
METHODS: We will conduct an interrupted time series quasi-experimental study. Using explicit criteria for quality of prescribing, the natural history of cisapride prescription will be established retrospectively in three hospitals. A retrospective DUR will be implemented in one of the hospitals, a concurrent DUR in another, whereas the third hospital will serve as a control. A pharmacist will abstract records of all patients who receive cisapride during the observation period to determine the appropriateness of prescriptions for the indication for use.

Answer 31

Audit + Research

Question 32

Read through this scenarios to decide whether it constitutes audit, research or service evaluation.

OBJECTIVE: To describe the drugs and types of medicine management problems most frequently associated with preventable drug related admissions to an acute medical admissions unit.
DESIGN: Observation study.
SETTING: Medical admissions unit in a teaching hospital in the UK.
PARTICIPANTS: Patients seen by pharmacists on the medical admissions unit.
MAIN OUTCOME MEASURES: Proportion of admissions that are drug related and preventable, classification of the underlying causes of preventable drug related admissions, and identification of drugs most commonly associated with preventable drug related admissions.

Answer 32

Research + (Audit?)