Program Evaluation

Question 1

Program

Answer 1

• More of less coherent system of activities that have been organized to address specific needs and resolve particular problems of a given population
• A bunch of activities that are interrelated so we put them together because they make sense to be together, they’ve been organized to meet specific needs

Question 2

Program Evaluation

Answer 2

• It’s in many fields, but it’s especially important in community psych
• Is the program doing what it says it’s doing?
• We need to make sure programs are doing the same thing as others and we need to make sure they are productive
• We don’t want to have programs that are wasting us money or have redundant movies
• Kind of like research but like not really

Question 3

Components of a program and of a program logic model

Answer 3

• Program mission
• Program processes
• Program outcomes

Question 4

Program processes - implementation/program evaluations

Answer 4

• Inputs: what you need, staff/supplies, how much money…
• Activities: What are you actually doing/outreach, intake, actual activities, program exits, etc.
• Outputs: Numbers that prove we’re doing something, how many people are coming or learning, how are the following the curriculum
• This step answers what our program is actually doing

Question 4

Theory of change

Answer 4

• A statement about how a program’s activities are intended to produce particular outcomes
• How do you think you will make a difference? Or how do think the program will make a difference

Question 4

Program logic models

Answer 4

• A depiction of the relationships among program resources, program activities, outputs, and outcomes
• Can be in a table/text/graphic forms
• Important for understanding a program’s theory of change and the program’s feasibility
• How does this actually run/ how is it running or organized/ how do all the activities work together and do all these things make sense
• Theory of change

Question 4

Program mission/needs assessment

Answer 4

• What is the program claiming to due
• Ex: Ending hunger in Ottawa
• What are the needs that drive the program
• What values and principles drives the way we go about things
• Ex: homeless people may not like many systems so you have to find ways to get them off the streets and into the shelters

Question 4

Program outcomes - Impact/Outcome evaluations

Answer 4

• Initial outcomes
• Intermediate outcomes
• Long term outcomes

Question 4

Possible constraints to programs

Answer 4

• Could be things like taxes, or finding good people to work, etc.

Question 5

Main types of program evaluation

Answer 5

• Needs assessment
• Process (implementation) evaluation
• Outcome (impact) evaluation

Question 5

Potential users of program evaluation findings

Answer 5

• Program clients
• Program staff
• Program management
• Program funders
• Policy makers and decision makers
• We don’t just want to grade the program we want to give them helpful feedback

Question 5

What is a needs assessment

Answer 5

• Assessment of scope and nature of problem
• Evaluation of services in place to address problem
• Examination of service alternatives
• Development of plan for new program
• Usually do this when starting a program but can also be done with existing programs
• Ex: homelessness in Ottawa has changed a lot in the last 20 years, but the people running the programs haven’t, so we do these evaluations to almost revamp the program

Question 6

Needs assessment sample questions

Answer 6

• What is the prevalence of persons who are homeless in Ottawa?
• What services are available in Ottawa for persons who are homeless?
• How well are current services serving persons who are homeless?

Question 7

Needs assessments methods primary data sources

Answer 7

• Primary data sources (you collect it as the evaluator)
• Key informant surveys
• Focus groups
• Community forums
• Community surveys

Question 8

Needs assessment methods secondary data sources

Answer 8

• Secondary data source (data that already exists from someone else)
• Social indicators
• Service utilization data
• Published studies

Question 9

Process or implementation evaluation

Answer 9

• How is the program actually running and what is it actually doing
• If the activities are being delivered to the intended recipients in the intended amount at the intended level of quality
• What we want to do comes from the program logic model

Question 10

Why must we do a program logic evaluation before a process evaluation

Answer 10

• We don’t know why people are benefiting from the program
• We don’t know if it is poorly implemented theory

Question 11

Types of program failure

Answer 11

• Implementation failure: Can be corrected if we improve how we operate
• Theory failure: Your theory/assumptions is just wrong. No evidence saying what you want to do is going to change

Question 12

Key questions in a process/implementation evaluation

Answer 12

1. Is the program reaching the intended population?
2. Is the program being implemented as planned?
3. How or why has be the actual program differed from the intended program?

Question 13

Common methods of implementation evaluation

Answer 13

• Chart reviews (secondary)
• Observation of service delivery (primary)
• Staff interviews (primary)
• Client interviews (primary)
• Daily logs of services delivered (secondary)

Question 14

Outcome/Impact evaluation

Answer 14

• Assessment of whether interventions have actually produced their intended effects
• Ex: produced planned client benefits

Question 15

How do we get evidence that people have actually benefited from programs

Answer 15

• Self reports
• Comparison groups (is it ethical?)
• Gets more complicated and more expensive

Question 16

Outcome evaluations sample questions

Answer 16

• Are clients of a community support program actually experiencing the intended improvement in functioning as a result of their participation in the program?
• Is service intensity associated with client outcomes?

Question 17

Rigor and practical considerations in program evaluations

Answer 17

• We want it to be the best it can be but a lot of our research methods aren’t going to work for practical, cost, ethical, etc. reasons
• We don’t always have the best evidence, because we don’t have the best design, and that makes us ask more questions

Question 18

Type of designs for an outcome evaluation

Answer 18

- Experimental approach - Quasi experimental - Non experimental approach - A lot of qualitative because that's all we can really do