Lecture 6 Evaluation

Question 1

What is evaluation?

Answer 1

A process that critically examines a program

It involves systematically collecting and analyzing information about the activities, characteristics, and outcomes

Question 2

Why do we evaluate programs?

Answer 2

To determine the effect(s) of the program
For accountability: is the money spent wisely?
For development: to improve the effectiveness, implementation on a larger scale and/or inform programming decisions.
For ethical aspects (unwanted side effects)

Question 3

What are reasons why evaluation is often omitted?

Answer 3

Evaluation typically is the last consideration in planning and implementing an intervention
Further reasons are: Money, threat to current practices, time constraints, already proven effective, intervention is still developing.

Question 4

What is included in an evaluation plan?

Answer 4

Relevant evaluation question(s)
Appropriate design & methods
A program / intervention
Outcomes 
Involvement of stakeholders

Question 5

What are the 3 types of evaluation?

Answer 5

Cost, process, effect

Question 6

What is effect evaluation?

Answer 6

It evaluates how successful a program is in preventing/reducing the health problem.
It examines the effect of the program.
It also provides an answer to the primary question.

Question 7

What are primary and secondary outcomes? And which of the two is the same as the dependent variable?

Answer 7

The primary outcome is the same as the dependent variable.
Primary outcome: outcome of greatest importance
Secondary outcome: used to evaluate additional effects of the intervention

Question 8

For what four objectives can effect evaluation/program outcomes be described?

Answer 8

Health, Quality of life, Behaviour, Environment

Question 9

What is process evaluation?

Answer 9

It is about understanding processes in order to strengthen or improve the program being evaluated
What is the reason the program is (un)succesful in preventing/reducing the health problem.
It’s about quality of the implementation, examining delivery and content of the program, barriers and facilitators of the implementation, organisational context

Question 10

What is economic evaluation

Answer 10

It is about if the intervention is cost-effective in preventing/reducing the health problem.
It can give a comparison in different costs: intervention, treatment, and societal costs.

Question 11

What are the seven basic evaluation question topics?

Answer 11

1. research, integrity, acceptability
2. observed change
3. internal validity
4. effect explanation
5. cost-benefit assessment
6. applicability
7. generalisability

Question 12

What are the four designs in which an evaluation can be done?

Answer 12

Study protocol (design paper)
Effect study (effect paper)
Process evaluation (process paper)
Economic evaluation (cost effectiveness paper)

Question 13

What does the acronym RE-AIM stand for?

Answer 13

Reach
Effectiveness
Adoption
Implementation
Maintenance

Question 14

What does the R in the RE-AIM framework stand for? Explain what it is.

Answer 14

Reach.
The percent and representativeness of individuals willing to participate in an initiative program.
Explores characteristics of study participants

Question 15

What does the E in the RE-AIM framework stand for? Explain what it is.

Answer 15

Effectiveness
The impact of the intervention on targeted outcomes
The health outcomes can be: health promotion, intermediate, health.

Question 16

What does the A in the RE-AIM stand for? Explain what it means.

Answer 16

Adoption.
The percent and representativeness of settings and intervention staff that agree to deliver a program
What are facilitators and barriers for adoption?

Question 17

What does the I in the RE-AIM framework stand for? Explain what it means.

Answer 17

Implementation.

The consistency and skill with which various program elements are delivered by various staff.

Question 18

What does the M in the RE-AIM stand for? Explain what it is.

Answer 18

Maintenance.
The extent to which individual participants maintain behaviour change long term and, at the setting level.
The degree to which the program is sustained over time within the organizations delivering it

Question 19

What does the acronym GRADE stand for?

Answer 19

Grading of Recommendations Assessment, Development and Evaluation

Question 20

What factors decrease the quality of evidence?

Answer 20

Study limitations (risk of bias)
Inconsistency of results
Indirectness of evidence
Publication bias
Imprecision

Question 21

What factors increase the quality of evidence?

Answer 21

Large magnitude of effect

Dose-response gradient