W5-T3 Implementation Outcomes

Question 1

define implementation outcomes

Answer 1

outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda

1. serve as indicators of implementation success
2. proximal indicators of implementation processes
3. key intermediate outcomes

Question 2

recognize Proctor et al.’s (2011) taxonomy of implementation outcomes

Answer 2

acceptability – perception amongst stakeholders that new intervention is agreeable

adoption – the intention to apply or the application of a new intervention

appropriateness – perceived relevance of intervention of setting, audience or problem

feasibility – the extent to which an intervention can be applied

fidelity – the extent to which an intervention gets applied as originally designed/intended

implementation costs – costs of the delivery strategy, including the costs of the intervention itself

coverage/reach – the extent to which eligible patients/population actually receive intervention

sustainability – the extent to which a new intervention becomes routinely available/is maintained post-introduction

Question 3

suitability of different methods for assessing implementation outcomes

Question 4

understand the importance of validated and pragmatic quantitative measures

Question 5

know where to identify validated implementation outcome instruments

Question 6

define three types of outcomes

Answer 6

1. implementation outcomes
   (acceptability, adoption, appropriateness, costs, feasibility, fidelity, reach, sustainability)
2. service outcomes (i.e. A&E visits)
   (efficiency, safety, equity, and timeliness)
3. patient/client outcomes (i.e. alcohol consumptions)
   (Functions, symptoms)

Question 7

what are other frameworks (beside Proctor) that can be used to identify the implementation outcomes

Answer 7

Consolidated Framework for Implementation Research (CFIR)

RE-AIM

Question 8

how CFIR works in identifying implementation outcomes

Answer 8

assesses 39 constructs over 5 domain
1. intervention characteristics
2. outer setting
3. inner setting
4. characteristics of individuals
5. process of implementation

Question 9

how RE-AIM works in identifying implementation outcomes

Answer 9

assesses 5 dimensions across individual, organisation and community levels:
1. reach
2. effectiveness
3. adoption
4. implementation (i.e. fidelity)
5. maintenance

Question 10

define how implementation outcomes are measured

Answer 10

1. qualitative interviews or focus group
2. observation
3. survey or questionnaires
4. routinely collected data

Question 11

how is the level of analysis and implementation stage considered and way to measure

Answer 11

Outcomes: adaptability
Level of analysis: individual provider: consumer provider
Implement stage: early/mid/late
measurement: survey/qualitative interviews/admin data

outcomes: adoption
level of analysis: individual provider: organisation or setting
Implementation stage: early to mid
measurement: admin data, observation, qualitative interview, survey

outcomes: sustainability
level of analysis: administrators: organisation or setting
implementation stage: late
measurement: case audit, qualitative interviews, questionnaires, checklists

Question 12

describe how implementation outcome is used in implementation study

Answer 12

implementation study: improving the uptake of apps that support people quitting smoking or
reducing their alcohol intake

Implementation strategy: a text message sent to hospital patients identified as smokers or risky drinkers via the electronic health record

implementation outcomes:

acceptability: explored before sending the text messages, via focus groups with patients and staff to help develop the message content and refine the process

feasibility: explored using the hospital’s electronic health – data collection: assess how many patients
were recorded as smokers and drinkers at risky levels and whether they had mobile phone numbers.

adoption: recording the proportion of patients who access the apps: patients that had received the link to the apps via text message.

Question 13

why it is important to validate implementation outcome instruments

Answer 13

lack of consensus on which instruments should be used for measuring the same outcome

inconsistencies in the outcomes reported and difficulties in comparing these outcomes in systematic reviews and meta-analyses.

There is variability in the quality in terms of reliability and validity of instruments, and it’s not clear if the best instrument is being used for a given outcome

Question 14

define the concepts of reliability and validity

Answer 14

a measure has to be reliable before it can be valid, but reliability does not infer validity.

Question 15

define measurement properties

Answer 15

reliability (reliability, measurement error, internal consistency)

validity (content validity, criterion validity, construct validity)

responsiveness (the ability of a measure to detect the change in an individual over time. )

Question 16

define pragmatic measure and why we need them

Answer 16

to usefully inform the assessment of implementation determinants, mechanisms, processes, strategies, and outcomes. measures must be both psychometrically sound and pragmatic

Question 17

define pragmatic measure construct

Answer 17

toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping

Question 18

what is concept mapping exercise

Answer 18

1. asked the stakeholders to conceptualize the domains that comprise the pragmatic measure construct
2. a systematic review found 47 criteria identified to be grouped into four categories: acceptable, compatible, easy and useful

Question 19

define pragmatic rating scale

Answer 19

pragmatic measures for implementation research: development of the psychometric and pragmatic evidence scale

– 11 items or questions belonging to these four categories

– 6 point rating system assigned to each question

Question 20

define three brief validated instruments

Answer 20

Outcome: Acceptability of Intervention Measure (AIM)
Items: … meet my approval, … is appealing to me, I like…, I welcome…

Intervention Appropriateness Measure (IAM)
Items: … seems fitting, …seems suitable, …seems applicable,…seems like a good match

Feasibility of Intervention Measure (FIM)
Items: …seems implementable, …seems possible, …seems doable, …seems easy to use

Question 21

define three options for online repositories of implementation outcome instruments

Answer 21

Society of Implementation Research Collaboration (SIRC)
— They use the findings of their systematic reviews of implementation outcome instruments used in mental
health settings, to populate the repository
— They identify implementation outcome instruments that assess all of the 39 consolidated frameworks for implementation research constructs
– require membership

Grid Enabled Measures (GEM)
— online repository of implementation outcome instruments
— relies on crowd-sourcing, where instrument developers proactively add their publication to the repository
— Any measure can be added to this repository without any validation dataset

KCL and UEA (University of East Anglia)
— based on the findings of the systematic review of
implementation of outcome instruments in physical health settings
— allows you to search for instruments to assess the implementation outcome used in Procter’s taxonomy
— To view a summary of the instrument; the
number of items; the country of application; and the level of analysis (i.e. patient, provider, organisation)
— to consider the methodological quality of the
psychometric studies included in the repository
— to consider the instrument quality
— to view the usability rating of the instrument
— Where permission is granted, the repository provides access to both the psychometric study and the published instrument.