Week 9 - Research in the Real World Flashcards
(15 cards)
Evaluation
“Does it work?”
- Consider timeframe
- What is ‘it’? (aspect or outcome)
- What does ‘work’ mean (what is the aim?)
What is program evaluation?
Systematic and objective assessment of design, implementation or results of program/activity
Purpose - continuous improvement, accountability, inform decisions
Evaluation principles
Fit-for-purpose - choice to evaluate and scale should be proportional to program
Useful - inform program delivery, define achievable outcomes
Robust and ethical - identify potential bias, provide performance insights, culturally appropriate
Credible - conducted by capable people, impartial, systematic, integrity
Transparent - unless good reason not to, findings given to stakeholders
Pure research vs evaluation
Pure research - more general, theoretical, led by researchers
Evaluation - specific and applied, led by those funding evaluation
Why evaluate?
Benefits government, policymakers, community
Commonwealth entities and companies required to by law
Aus government spends 50 million on evaluations from private sector
Whether to evaluate
Consider priorities
Evaluation should be proportional to program (1-5% of total budget)
Time/approach must be fit for purpose (ongoing? risk of not evaluating?)
Factors in evaluation
Government/entity priority
Stakeholder priority
Monitoring and feedback
High profile/sensitivity
Evaluation commitment
Previous evaluations
Internal/external
Timeframes
Resources/funding
Risk
When to evaluate
Evaluation should be built into program planning
Before implementation - are current needs being met? what can be done to address need? is the intervention likely to work?
During implementation - is program going as planned? what can be learned?
After implementation - did it work? what is the impact? are objectives being met?
Who evaluates?
In-house - small scale, low cost, low stakes
Independent - large scale, high cost, high stakes, must avoid COIs
Flow chart
- Is evaluation required? > Does it require independent > Are funds available internally? Are funds available for an independent?
Step 1 - establish advisory group
Panel of specialists including SMEs, evaluation specialists, community reps., stakeholders
Role - guide design, ensure objectivity, interpret findings
Step 2 - develop program logic (model)
Visual rep. of how program should work
Key components - inputs (resources used), activities (actions/processes involved), outputs (products/services resulting from program), outcomes (changes or benefits from activities and outputs)
Example
- Inputs - new policy, 2 years funding, staff
- Activities - staff training package made, training sessions for staff, client resources made
- Outputs - smoking cessation intervention delivered, clients given resources
- Outcomes - increased awareness, increased use of services, quit attempts, improved health
Step 3 - develop evaluation plan
Overview of program - history, program activities, key stakeholders, SMART goals, program logic
Purpose of evaluation - what is being figured out?
Evaluation audience
Evaluation questions - one indicator per question
Evaluation design - between or within-subjects? instruments? analysis?
Potential risks - internal validity? poor sample/response rates
- Risk rating = likelihood x consequence
Resources and role - include timeline
Governance - describe advisory group
Reporting - target audience
Step 4 - engage independent evaluator
Prepare request for proposal, issue invitations, assess applications, select one
Step 5 - project manage
RFP should include draft work plan, refine in consultation with experts, regular meetings and updates, feedback on final report
Step 6 - disseminate findings
Feedback session to stakeholders, reported to relevant communities and management structures, sent to papers or journals
