exam 1 Flashcards
(86 cards)
1
Q
functions of government (5)
A
- protect their sovereignty
- preserve order
- provide services for their citizens
- socialize their citizens (especially their younger citizens) to be supportive of the system
- collect taxes from their citizens
2
Q
3 reasons for studying public policy
A
- scientific understanding: better understand how the world works and the impacts public policy can have on people/society
- professional advice: have a practical application of the knowledge in public policy
- policy recommendations: help inform the people who are actually making the policy choices
3
Q
Policy Process Model (6)
A
- problem identification: defining issues
- agenda setting: getting problems seriously considered by policymakers
- policy formulation: proposed policy actions (inactions) to address problems
- policy legitimation: providing legal force to decisions
- policy implementation: putting the policy into action
- policy evaluation: assessment of policy or program
4
Q
Weiss’ definition of policy evaluation
A
the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy
5
Q
outcome (summative) evaluation
A
concerned with the end results of the program
6
Q
process (formative) evaluation
A
focused not on the end results but the program in practice and procedure
7
Q
Covert Purposes of Evaluations (4)
A
- postponement
- ducking responsibility
- window dressing
- public relations
8
Q
postponement
A
the initiator or client may be trying to delay a decision on a program
9
Q
ducking responsibility
A
the client may be trying to have the evaluation make their decision
10
Q
window dressing
A
the client may be trying to disguise their decision with the evaluation
11
Q
public relations
A
the client may be trying to gain support for the program through the evaluation
12
Q
4 Unfavorable Conditions
A
- program is unclear and unstable
- participants are unsure about the purpose of the program
- initiators are trying to eyewash or whitewash the program
- evaluation has a lack of resources
13
Q
program is unclear and unstable (2)
A
- there doesn’t seem to be much adherence to the goals
- since it’s unclear what the program actually is, it might be unclear what the evaluation is or what it means
14
Q
participants are unsure about the purpose of the program
A
a process evaluation might be warranted to try and figure out what’s going on with the program
15
Q
initiators are trying to eyewash or whitewash the program (3)
A
- eyewash: attempting to justify a program by selecting certain aspects of an evaluation to look good
- whitewash: trying to cover up by avoiding any objective appraisal
- might not have the necessary information to complete an evaluation properly
16
Q
evaluation has a lack of resources
A
not just talking about money; time and people are also necessary
17
Q
Whorely’s 3 Criteria of Evaluability Assessment
A
- the program should operate as intended
- it should be relatively stable
- it should seem to achieve positive results
18
Q
ethics of evaluators (5 guiding principles)
A
- systematic inquiry
- competence
- integrity/honesty
- respect for people (treatment of people)
- responsibilities for general and public welfare
19
Q
systematic inquiry
A
evaluators conduct systematic, data-based inquiries
20
Q
competence
A
evaluators provide a competent performance for stakeholders
21
Q
integrity/honesty
A
evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process
22
Q
respect for people
A
evaluators respect the security, dignity, and self-worth of respondents, program participants, clients, and other evaluation stakeholders
23
Q
responsibilities for general and public welfare
A
evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation
24
Q
barriers to ethical analysis (4)
A
- technocratic ethos: relies on the things that can be measured
- too many designs: frames the policy research at the expense of looking at other areas
- advocacy vs. analysis: your analysis and advocacy become blurred within the evaluation
- disturbing: don’t want to examine the results of the program
25
Weiss' 4 I's of Stakeholders
1. ideology:
2. interests: each stakeholder has their own self-interest in the course of action
3. information: stakeholders have different knowledge from their own experience/understanding of different reports
4. institution: decisions are made in an organizational context
26
importance of program knowledge (5)
1. to develop a good sense of the issues
2. to formulate questions
3. to understand and interpret the data
4. to make sound recommendations
5. for reporting
27
3 main steps of planning an evaluation (Posavac)
1. identify the program and its stakeholders: who are the program personnel, sponsors, and who is being served by the program?
2. become familiar with information needs: who wants the evaluation?
3. plan the evaluation: examine the literature, plan the methodology, and present a written proposal
28
Whorley's 5 categories of evaluation questions
1. program process
2. program outcome
3. attributing outcomes to the program
4. links between processes and outcomes
5. explanations
29
program process questions (3)
1. questions that are trying to understand what is going on in the program
2. aligned to the program's design
3. might be more open
30
program outcome questions (3)
1. questions focused on the impact of the program
2. designed on the client's situation
3. these questions normally come from the program's goals or various stakeholders
31
attributing outcomes to the program questions (2)
1. questions that clearly show that changes observed are due to the program
2. trying to understand the extent to which the program was responsible for changes
32
links between processes and outcomes questions
questions focused on what processes or features of a program are related to different outcomes
33
explanations questions
questions designed to help understand why the program achieved its results
34
sources of information of data collection (7)
1. informal interviews
2. observations
3. formal interviews
4. written questionnaires
5. program records
6. data from other institutions
7. other sources (focus groups, testing, more documents)
35
4 general levels of measurement
1. nominal (special case dichotomous)
2. ordinal
3. interval
4. ratio
36
program outcome measures in terms of effects on (4)
1. people served: what are the changes in attitudes, values, behavior, and skills
2. agencies: is there a change in the agency or institution
3. a larger system: is there a change in the network of agencies or community
4. the public: changes in the public's views, attitudes, and perceptions
37
2 components of public interest
1. there is the instrumental achievement of a particular set of objectives that represent widely shared interests
2. there is the adherence to a set of procedures that, if followed in selecting and pursuing objectives by society, yields an acceptable outcome for the group
38
instrumental rationality
program evaluation steadfastly acts on the basis that rationally connecting objectives, means, and outcomes can improve the outcomes and, consequently, the public interest
39
Emison's key points for the Client (5)
1. know what your client's interests are
2. know what success it
3. have a principal who can do something
4. put the evaluation in a management context
5. do the right thing
40
Emison's 4 C's
1. client
2. content
3. control
4. communication
41
client (3)
1. the client provides the purpose for the conduct of the evaluation
2. the key to success is understanding our client
3. understanding the client's interests is essential to having a program evaluation that goes beyond study to action
42
content (2)
1. governs the substantive nature of the evaluation
2. understand how the important features of the program work
43
control (2)
1. successful program evaluations are managed well
2. controlling the analysis from the outset can ensure that a product will emerge for the client to consider
44
communication
we must communicate the evaluation in a manner that makes it easy for the client to understand the content, agree with the conclusions, and direct that actions be taken
45
Emison's key points for Content (5)
1. build the analysis on facts
2. align your evidence and your conclusions
3. simplicity always trumps elegance
4. don't let the illusion of the perfect drive out the reality of the good
5. never underestimate the power of accurate description
46
validity
how well does the indicator measure the concept; allows the evaluator to have more confidence in what is being measured
47
reliability
if you were to repeatedly measure this concept, would you get the same results?
48
characteristics of measurement (7)
1. validity
2. reliability
3. direction
4. variance/sensitivity to differences
5. currency/salience
6. access
7. bias of the data collection
49
possible stakeholders and their motivations in an evaluation report
1. legislature - influence program direction, gain knowledge of program, lay basis for long-term change
2. political executive - influence program direction, justify prior decision, draw attention to program
3. program manager - improve operational program, justify prior decision, draw attention to program
4. program client - influence program direction, lay basis for long-term change
50
program theory (3)
1. can be seen as preceding and then evolving and expanding into the Theory of Change, which is more relational and holistic
2. it emerged from the need to better understand programs' rationale and, more importantly, the chain of causality that lead to its outcome(s)
3. the assumption is that there is a logic that leads to the achievement(s) and that understanding this logic is paramount to understanding the success and failure of the program
51
main characteristics of theories of change (4)
1. logical thinking and critical reflection
2. flexibility and openness
3. innovation and potential improvement in programs
4. performance management
52
advantages of a theory of change (5)
1. ownership
2. relevancy
3. focus
4. value for money
5. measurement
53
TOC advantage: ownership (2)
1. the TOC provides a unique moment of stakeholder participation where all those who have a stake in the program can meaningfully contribute to it from the design and conceptualization stage
2. increasing ownership increases commitment and collective synergies and hence the program's overall chances of success
54
TOC advantage: relevancy
planning with others and with an ear firmly to the ground is hugely helpful in ensuring that the program meets the needs of its targets in context and will, therefore, be relevant
55
TOC advantage: focus (2)
1. a TOC planning process begins with a definition of the desired change which means that everything thereof will be defined and decided per reference to this change
2. the desired change is the focus, and the focus determines the means
56
TOC advantage: value for money (2)
1. programs that do not think through the elements that a TOC often 'forces' a planning process to go through can have less value for money
2. the lack of focus on the ultimate change may lead to the implementation of multiple activities that in the end are a distraction and do not contribute to generate impact
57
TOC advantage: measurement (2)
1. it helps evaluators ensure that they are measuring the right activities and that they have developed appropriate research tools
2. articulating a theory of change at the outset and gaining agreement on it by all stakeholders reduces, but does not eliminate, the problems associated with causal attribution of impact
58
key elements of an evaluation plan (10)
1. introduction and background to the program
2. a summary of relevant, previous evaluations: their findings and the methodologies they employed
3. evaluation questions
4. overall evaluation design
5. methods
6. ethics
7. timescales
8. main outputs
9. project management
10. the evaluator(s)
59
good evaluation questions must be: (3)
1. reasonable and appropriate
2. answerable
3. contain the criteria for program performance
60
principles related to participants' rights (3)
1. voluntary participation
2. do no harm
3. confidentiality and anonymity
61
participants' rights: voluntary participation (2)
1. participants must willingly participate in the evaluations, namely in the workshops, interviews, focus groups, and all the other situations by which data and information are to be collected
2. the option not to participate should be made clear and available to them as an equally valid and respected option
62
participants' rights: do no harm (2)
1. participants, contributors, and evaluation stakeholders more broadly will incur no harm if and when they decide to participate
2. this principle needs to be mainstreamed throughout and influence the overall evaluation rationale and process - from the selection of the methods to the actual implementation of these
63
participants' rights: confidentiality and anonymity (2)
1. regardless of the information they provide, they will not be identified as the source
2. no statements or other type of information that may identify participants should be shared with others or publicly displayed by the evaluator
64
types of research design (4)
1. informal study design (self-evaluation & expert judgement)
2. formal study design
3. quasi-experimental designs (one-group design, extensions of the one-group design, comparison group studies)
4. experimental design
65
8 threats to internal validity
1. history: events could happen between measurements
2. maturation: participants change and age
3. testing: Hawthorne effect
4. instrumentation: changes in measurement (different observers may observe in different ways)
5. regression: may be measuring at an extreme position (wouldn't know with only one measurement)
6. selection: introduce bias by selecting of groups
7. mortality: loss of participants along the way
8. interaction of these factors
66
history of evaluation research: pre WWII (3)
1. 1912: comparison between autopsies and medical diagnoses
2. 1933: 8 year study - outcomes of students in traditional vs. progressive schools
3. studies done by academics and interest groups
67
history of evaluation research: post WWII
a slew of social scientists hired by the government under FDR after the Great Depression; government began to participate in evaluation research
68
history of evaluation research: war on poverty (3)
1. LBJ - avalanche of social programs
2. large-scale government-funded evaluations
3. federal government funding of social programs, requiring systematic evaluation to know how money is being spent (ex: Elementary Secondary Education Act (ESEA))
69
history of evaluation research: development of the field (3)
1. late 60s-early 70s: evaluation as its own field, government officials, use of cost benefit analysis
2. by end of 70s: at federal level, evaluation is pretty commonplace, basically every cabinet office has its own evaluation office
3. institutionalization: university research centers devoted to evaluation; for-profit enterprises
70
history of evaluation research: Reagan administration (2)
1. cuts funding to social services --> fewer evaluations
2. evaluations sponsored by the federal government focused on cost reduction and eliminating services
71
history of evaluation research: Clinton administration (2)
1. renewed emphasis on evaluation --> focus on effectiveness returned
2. conservatives emphasize efficiency and ability to reduce size of government/services; liberals emphasize effectiveness and improving situations
72
significance of the Government Performance and Results Act (1993) to evaluation (4)
1. Clinton administration - requires that federal agencies have performance measurements/targets (has now trickled down to state agencies); accountability of program/using resources
2. many are able to make a career out of being an evaluator
3. internationalization: more international organizations established to formulate ethics of evaluation research
4. diversifying: more diverse evaluators, non-profit organizations, lack of training
73
Emison's purposes of evaluation (2)
1. to advance the public interest
2. advance achievement of public objectives while observing appropriate procedures within a democratic society
74
traditional role of evaluators (6)
1. objective outsider
2. fact-seeking
3. valid report on program
4. carefully avoid bias
5. methodology focus (absolute neutrality is not possible)
6. recommendations based on data analysis, not from a stakeholder perspective
75
different roles of an evaluator
objective outsider (detached, traditional) vs. co-investigator (participative)
76
empowerment evaluator (2)
1. late 80s-90s, stakeholders are in charge of the evaluation
2. evaluators offering help from the sidelines
77
collaborative evaluator (4)
1. most common
2. critical friend (joint venture between practitioner and evaluator)
3. evaluator brings research skills to the table while program personnel bring knowledge of the program
4. evaluator doesn't make recommendations but encourages practitioners to reflect on the data
78
stakeholder evaluator (2)
1. convenes various stakeholders, structured engagement with stakeholders
2. evaluator is in charge of the study, but is seeking stakeholder input for recommendations
79
exogenous factors and ethics per Fox, Grimm, & Caldeira (3)
1. literacy level
2. power relations
3. intercultural communication
80
considerations in developing program process measures (8)
areas to consider:
1. types of programs offered
2. characteristics of the staff and clients
3. frequency of service
4. duration of service
5. intensity
6. size of group receiving service
7. stability of the service offered
8. quality of the service
81
considerations in developing program input measures (7)
raw materials of the program include:
1. budget
2. nature of the staff
3. location
4. plan of activities
5. methods of service
6. purposes
7. client eligibility standards
82
reasons experimental design is considered the hallmark of scientific research (2)
1. avoids many issues of internal validity
2. allows the researcher to establish a strong causal relationship
83
problems with randomization of assignment of groups (4)
1. refusal to participate
2. nonattendance
3. attrition: may leave program, death
4. outside interference
84
external validity (2)
1. whether results from the evaluation can be generalized to other situations
2. more formally, it is the validity with which we can infer that a causal relationship which we observe during the evaluation can be generalized across different types of persons, settings, and times
85
internal validity (2)
1. whether the evaluation can demonstrate plausibly a causal relationship between the treatment and the outcome
2. in other words, is the relationship between an independent and dependent variable a causal relationship?
86
importance of formality and rigor in evaluative information (4)
1. evaluation is a formal and rigorous process (separates it from informal evaluation)
2. informal evaluations by the people running the program tend to be overly optimistic
3. bringing rationality to policymaking
4. market situations: performance is evaluated by the market
