Midterm Flashcards
(111 cards)
1
Q
The person, program, idea, policy, product, object, performance, or any other entity that you are evaluating
A
Evaluand
2
Q
The absolute or relative quality of the evaluand, either intrinsically or in regard to a particular criterion
A
Merit
3
Q
The evaluand’s value in a particular context (extrinsic value)
A
Worth
4
Q
An evaluation that continually informs the management and main stakeholders of an ongoing intervention about early indications of progress (or lack of progress) in achieving results of a project or program, or other kind of support to an outcome.
A
Process
5
Q
An evaluation done during the planning or implementation of a program or product that is designed to provide feedback to improve the evaluand.
A
Formative evaluation
6
Q
An evaluation done at the end of or on completion of a program.
A
Summative Evaluation
7
Q
Broad metaphysical constructs that include sets of logically related philosophical assumptions.
A
Paradigms
8
Q
A set of beliefs about how a program works or why a problem occurs.
A
Program theory
9
Q
True or false: Pragmatists such as Tashakkori and Teddlie believe that you (the researcher) are free to study what interests you and is of value to you, it’s okay to study it in different ways that you deem important, and results can and should be used in ways that bring about positive consequences in your value system.
A
True
10
Q
A model that displays the sequence of actions in a program, describes what the program is and will do, and describes how investments will be linked to results.
A
Logic model
11
Q
What is of key importance in David Fetterman’s model of evaluation known as “empowerment evaluation?”
A. Empowerment evaluation is only focused on summative evaluation.
B. The key purpose is to empower the external evaluator.
C. Participation in the evaluation process will bias the results and should be avoided.
D. People who are coached by evaluators and conduct their own evaluations will develop capacity for conducting evaluations and are likely to use the results of the evaluation.
A
D
12
Q
Which philosophical assumption would you be dealing with if you felt compelled to turn in a fellow student because you knew that she bought her final research paper off of the internet?
Epistemological
Ontological
Axiological
Constructivist
A
Axiological
13
Q
What are the components of Stufflebeam’s CIPP model?
A
Context, Input, Process, Product
14
Q
Postpositivists believe that there is/are _______ reality and that reality/those realities ___________.
A
one, can be known within a certain level of probability
15
Q
In an input evaluation (in the CIPP model), which of the following is a key type of decision?
A. Gauge the success of the program in addressing needs.
B. Judge performance and interpret outcomes.
C. Determine feasibility and potential cost-effectiveness, and choose among competing action plans.
D. Define goals and priorities and desired outcomes
A
C
16
Q
The constructivist paradigm focuses mainly on _____________ and the postpositivist paradigm focuses on __________.
A
Values, Methods
17
Q
What evaluator developed the CIPP evaluation model?
A
David Stufflebeam
18
Q
According to Shadish (1991), evaluation theories should meet the following criteria
A. Knowledge, use, valuing, practice, and social programming
B. Planning, design, use, and explaining
C. Measurement, utility, value, and social programming
D. Both A and B
A
D
19
Q
True or false: In a context evaluation, you evaluate needs, problems, assets, and opportunities.
A
True
20
Q
True or false: Stufflebeam and Shinkfield (2007) include the following six components in program evaluation theory: overall coherence, core concepts, tested hypotheses on how evaluation procedures produce desired outcomes, workable procedures, ethical requirements, and a general framework for guiding program evaluation practice and conducting research on program evaluation.
A
True
21
Q
True or False: Context evaluation is designed to provide the “big picture” into which a program and its evaluation fit.
A
True
22
Q
Evaluators typically take on more than one role throughout an evaluation.
A
True
23
Q
True or False: Evaluators do not have to worry about unique cultural factors limiting their ability to generalize from theories primarily developed in the US.
A
False
24
Q
If you were interested in fostering improvement and self-determination in program participants, what model of evaluation would serve that purpose?
A. UFE
B. Training evaluation
C. Empowerment Evaluation
D. CIPP
A
C
25
The Transformative paradigm primarily focuses on what branch in evaluation?
Methods
Social Justice
Values
Use
Social Justice
26
The first step in a UFE is to identify the primary users who will need to act on the evaluation findings.
A. True
B. False
True
27
An evaluator should only focus on the evaluation process, data, and program outcomes, not on limited resources or power imbalances.
True
False
False
28
The evaluator’s primary role for the method branch of evaluation is which of the following?
A. evaluator as objective neutral party
B. communicator who engages in meaningful dialogue
C. social relations manager to facilitate use
D. relationship builder based on trust and cultural respect, and investigates structural inequalities
A
29
In a constructivist evaluation, researchers must be aware of their own values.
A. True
B. False
True
30
______is an approach that focuses on the theories people have about what it takes to have a successful program.
An approach that focuses on the theories people have about what it takes to have a successful program. In simple terms, you could think about how people learn or how they change their behavior. What conditions need to be in place for that to happen.
Who developed this approach?
Theory-based evaluation
Chen and Rossi
31
```
Connoisseurship evaluation takes a _____ stance
A. Phenomenological
B. Experimental
C. Positivist
D. Transformative
```
A
32
_______ is the assessment of the environment in which an innovation or program will be used, to determine the need and objectives for the innovation and to identify the factors in the environment that will impact the success of its use. This analysis is frequently called a needs assessment, and it is used in making program planning decisions
Context evaluation
33
Which statement best fits “naturalistic” or fourth-generation evaluation?
A. The evaluator focuses on testing preconceived hypotheses
B. The evaluator observes the program with objectivity.
C. The evaluator uses an experimental approach
D. None of the above.
D
34
Constructivists believe that it is possible for researchers to eliminate the influence of values in their work.
A. True
B. False
False
35
Here, evaluation questions are raised about the resources that will be used to develop and conduct the innovation/program.
Input evaluation
36
```
Many theorists in the values branch of evaluation advocate for the use of _________ methods in evaluation.
A. statistical
B. improved
C. qualitative
D. quantitative
```
C
37
Who developed the responsive evaluation approach?
Robert Stake
38
used to examine the ways in which an innovation/program is being developed, the way it is implemented, and the initial effectiveness, and effectiveness after revisions.
Process Evaluation
39
What is the hermeneutic process?
A. The process of constructing reality by the research participants through conversational interaction with the researcher, and multiple meanings are often constructed and analyzed.
B. The process of determining which evaluation approach to use based on question analysis and discussion with your mentor.
C. The process of determining the mental constructs of program participants.
D. A and C
A
40
How is a constructivist evaluator most likely to uncover hidden meetings and construct reality during an evaluation?
A. Random sampling techniques
B. Using structured interviews in the data collection process
C. Prolonged involvement with program participants
D. Random assignment
C
41
focuses on the success of the innovation/program in producing the desired outcomes. Includes measuring the outcome variables specified in the program objectives, identifying unintended outcomes, assessing program merit, and conducting cost analyses.
Production evaluation
42
According to Robert Stake, evaluators have a responsibility to look at what is happening in the program first, and then choose value questions and criteria for judging merit and worth.
A. True
B. False
True
43
Which evaluation type begins the evaluation with preliminary data collection to determine evaluation questions?
A. All evaluation approaches use this approach
B. Responsive evaluation
C. Postpositivist
D. Objectives-based
B
44
Rossi's Five Domains Model
```
needs assessment
program theory assessment
Implementation assessment
impact assessment
Efficiency assessment
```
45
What are some of the primary principles that make up the basis of the transformative paradigm’s axiological assumption?
A. Being culturally respectful
B. Furthering human rights and addressing inequities
C. Promoting social justice
D. All of the above
D
46
```
If you were conducting an evaluation that examined how social injustice stemmed from gender inequities, you would be interested in which theory/theories of evaluation?
A. Critical race theory of evaluation
B. Feminist evaluation theories
C. Culturally responsive evaluation
D. Responsive evaluation theory
```
B
47
```
What does CRT stand for?
A. Critical Rational Theory
B. Critical Race Theory
C. Criterion Referenced Evaluation Theory
D. Critical Race Transformation
```
B
48
```
Evaluators that are interested in examining how indigenous people have suffered due to colonization would be advocates of which theory?
A. Feminist theory
B. Transformation theory
C. Critical race theory
D. Participatory theory
```
C
49
According to Lee (2009), an evaluator is not “culturally competent,” but with appropriate understandings and experiences, the evaluator can conduct a culturally competent evaluation.
A. True
B. False
True
50
Which of the following is NOT true regarding the transformative paradigm (TP)?
A. TP includes a belief that the lives and experiences of diverse groups of people are of central importance in the evaluation to address issues of power and justice.
B. TP focuses on viewpoints of marginalized groups and interrogating systemic power structures.
C. TP relies heavily on mixed methods to further social justice and human rights.
D. TP does ALL of the above
D
51
A key difference between the constructivist and transformative paradigms is that constructivists emphasize understanding the influence of values in the process of inquiry, but transformatists believe that their values should and will influence the process and outcomes of their work.
A. True
B. False
True
52
```
Who are some important philosophers that influenced the transformative paradigm?
A. Kant, Hegel, Habermas, and Marx
B. James, Dewey, Mead, and Peirce
C. Campbell, Tyler, Cook, Cronbach
D. Guba, Denzin, and Lincoln
```
A
53
In regard to Kirkhart’s five justifications for considering validity from a multicultural perspective, the interpersonal justification is:
A. Congruence with the lived experience of participants in the program and in the evaluation process
B. Cultural appropriateness of measurement tools and design configurations
C. Quality of interactions between and among participants in the evaluation process.
D. Cultural congruence of theoretical perspectives underlying the program, evaluation, and assumptions of validity
C
54
Which of the following is important in deliberative democratic evaluation?
A. Cultural diversity
B. Authentic processes, constraints on self-interest, and focus on issues
C. Collaboration and rules and principles
D. All of the above are used
D
55
Uses the commonsense idea that an effective way to determine “what works” is to examine successful cases and compare them to other cases.
Who developed this?
Success case method (SCM)
Brinkerkoff
56
What is an example of a group data collection strategy?
A. In-depth interviews
B. Focus group
C. Concept mapping
D. B and C are both examples of group strategies
D
57
```
To which kind of evaluation approach is a logic model most closely tied?
A. Theory-based evaluation
B. Participatory evaluation
C. Responsive evaluation
D. Utilization-Focused Evaluation
```
A
58
According to your text, many indigenous people prefer to speak of “relationships” rather than “partnerships.”
A. True
B. False
True
59
How can a theoretical framework provide guidance to inform the identification of the evaluand and context?
A. Provide an understanding of the evaluator’s point of view
B. Provide a basis for decision about the appropriate components of a program and variables to examine
C. Tell the evaluator the correct evaluation model that must be used
D. Tell the evaluator the correct advisory board to use
B
60
What kind of relationship does a Values Branch evaluator tend to have with stakeholders?
A. Distant
B. Involved
C. Value neutral
D. Universal
B
61
A very basic template for a logic model would have which three components?
A. inputs, models and resources
B. methodology, values, and models
C. resources/inputs, activities, and outputs/outcomes/impacts
D. evaluator, evaluand, approach
C
62
What are some stakeholder groups in evaluation?
A. customers, employees, business owners
B. administrators, teachers, students
C. city residents, elected officials, voters
D. All of the above could be stakeholders in an evaluation.
D
63
According to Mertens and Wilson, identification of stakeholders is context-specific.
A. True
B. False
True
64
Which of the variables shown below would be considered contextual variable(s)?
A. Local setting and time
B. Cultural characteristics of the stakeholders
C. Characteristics of program participants
D. All of the above could be contextual variables
D
65
A logic model is helpful in depicting the evaluand, its process, and
its outcomes?
True
False
True
66
You can use Kirkpatrick’s evaluation model for any kind of program.
A. True
B. False
False
67
Type of evaluation done for and with specific intended users for specific, intended uses." The cardinal is that the utility of an evaluation is to be judged by the degree to which it is used. The evaluator focuses on use from the beginning until the end of the evaluation, and during that time he or she continually facilitates use, and organizational learning or any other process that helps ensure that the evaluation results will continue to be used once the evaluator leaves the organization.
Who developed this
UFE
Patton
68
What might be a disadvantage of using an experimental approach in evaluation?
A. The variables and interactions between variables are complex to be worked out in advance.
B. Inability to control exposure to the intervention.
C. Ensuring that treatment and control groups are separate during the study
D. All of the above
D
69
```
Which of the following names is NOT associated with theory-based evaluation?
A. Huey-Tsyh Chen
B. David Fetterman
C. Mark Lipsey
D. Stewart Donaldson
```
B
70
How can you identify learning in Kirkpatrick’s model?
A. Examine pretest to posttest change on learning scores
B.Interview participants
C. Measure how much participants enjoyed the training
D. Attempt to capture the effect of the training program on the organization’s performance.
A
71
What are the four levels (in order) of the Kirkpatrick model of evaluation?
A. reaction, learning, results, behavior
B. behavior, reaction, results, learning
C. reaction, learning, behavior, results
D. results, behavior, reaction and learning
B
72
```
Positivists and postpositivists believe that ___________ the object of study reduces bias.
A. distance from
B. limiting contact with
C. introspection concerning
D. A and B
```
D
73
One of Donald Campbell’s major contributions to the field of evaluation is his work on experimental and quasi-experimental designs.
True
False
True
74
Which of the following adds some justification of the methodological choice of a randomized design?
A. When you want to establish a cause-and-effect-relationship
B. When random assignment is not problematic.
C. When there is uncertainty about the effect of an intervention.
D. All of the above
D
75
If a randomized experimental approach is not feasible, what are some other alternatives?
A. In-depth case studies
B. Statistical analysis of observational data
C. Quasi-experimental comparison group studies
D. All of the above might be an alternative
D
76
The methodological assumption(s) in the postpositivist-and-methods-branch of evaluation includes which of the following?
A. Scientific method, quantitative methods, and hypothesis
B. Distant, objective
C. Deductive reasoning, one reality knowable within a certain level of probability
D. Respect, justice, beneficence
A
77
Empowerment evaluation
David Fetterman
78
Practical participatory evaluation
Brad Cousins
79
designed to provide feedback on an ongoing basis for continuous improvement.
Who?
Developmental evaluation
Patton
80
A model of evaluation that works with businesses and schools to establish these organizations cultures with regard to evaluation. They identified three inquiry phases in the learning organization evaluation model:
focusing the inquiry
carrying out the inquiry
applying learning
Learning organization evaluation
Preskill and Torres
81
The basic principle is that the program staff is in charge of the direction and execution of the evaluation, while an evaluator serves as a facilitator, coach, advisor, or guide.
This is an approach that aims to increase the likelihood that the programs will achieve results by increasing the capacity of the program stakeholders. The role of the evaluator is not as an expert or director but rather a critical friend
Who did this?
Empowerment evaluation
Wandersman and Fetterman
82
According to Lincoln, constructivists must use only qualitative data collection methods.
True
False
False
83
Many constructivists would consider themselves relativists.
True
False
True
84
```
__________ are commonly used by values branch evaluators because they serve as a good approach to gain understanding about the day-to-day activities of a program and uncover hidden meanings.
Closed-ended questionnaires
Experiments
Case studies
None of the above
```
Case Studies
85
Which of the following statements fits constructivism?
A. There is one agreed-upon socially-constructed reality.
B. In research and evaluation, reality is constructed by the researcher and research participants through interactive dialog.
C. Constructivism is a branch of postpositivism.
A & B
B
86
study of the power structures that perpetuate social inequities
Transformative participatory research
87
vehicle for organizing concerns about pluralism and diversity in evaluation, and as a way to reflect upon the cultural boundaries of our work.
Multicultural validity
Kirkhart
88
The study of the quality of the evaluation.
Meta-evaluation
89
Kirkhart's Five Justifications for Multicultural Validity
Interpersonal—The quality of the interactions between and among participants in the evaluation process.
Consequential—The social consequences of understandings and judgments and the actions taken based upon them.
Experiential—Congruence with the lived experience of participants in the program and in the evaluation process.
Theoretical—The cultural congruence of theoretical perspectives underlying the program, the evaluation, and the assumptions of validity.
Methodological—The cultural appropriateness of measurement tools and cultural congruence of design configurations.
90
The belief that the social world can be studied in the same way as the natural world, that there is a value-free method for studying the social world, and that explanations of a causal nature can be provided. Focuses primarily on quantitative designs and data.
Postpositivist
91
The belief that knowledge is socially constructed by people active in the research process, and that researchers should attempt to understand the complex world of lived experience from the point of view of those who live it. Focuses primarily on identifying multiple values and perspectives through qualitative methods.
Constructivist
92
The belief that the lives and experiences of diverse groups of people are of central importance in the evaluation to address issues of power and justice. Focuses primarily on eliciting and understanding viewpoints of marginalized groups and interrogating systemic power structures through mixed methods to further social justice and human rights.
Transformative
93
The belief that reality is individually interpreted and that the methodological choices will be determined by the evaluation questions. Focuses primarily on data that are found to be useful by stakeholders, and advocates for the use of mixed methods.
Pragmatic
94
Resources that go into a program including staff time, materials, money, equipment, facilities, volunteer time.
Inputs
95
Shadish's Criteria for Developing Evaluation Theories
Knowledge: What do we need to do to produce credible knowledge?
Use: How can we use the knowledge we gain from an evaluation?
Valuing: How do we construct our value judgments?
Practice: What do we evaluators actually do in practice?
Social programming: What is the nature of social programs and their roles in solving societal problems?
96
The activities, products, and participation generated through the investment of resources. Goods and services delivered.
Outputs
97
Results or changes from the program such as changes in knowledge, awareness, skills, attitudes, opinions, aspirations, motivation, behavior, practice, decision-making, policies, social action, condition, or status.
Outcomes
98
the variable that is measured by the researcher to see if there is a change in knowledge, behavior, attitude, amplitude, or some other relevant construct.
Dependent variable
99
The program that is implemented in hopes of seeing a change in knowledge, behavior, attitude, amplitude, or some other relevant construct. This variable is manipulated by the researcher
Independent variable
100
more immediate or direct results
Proximal outcomes
101
an approach that focuses on the theories people have about what it takes to have a successful program. In simple terms, you could think about how people learn or how they change their behavior.
Who?
Theory-based evaluation
Chen and Rossi
102
Who defined program theory driven evaluation science as the systematic use of substantive knowledge about the phenomena under investigation and a scientific methods to improve, to produce knowledge and feedback about, and determine the merit, worth, and significance of the evaluands such as social, educational, health, community, and organizational programs?
Donaldsen
103
more ultimate or longer term outcomes
Distal outcomes
104
Specifies the expected distal or longer term outcomes.
Conceptual theory
105
Specifies the expected immediate or proximal outcomes of the program actions
Action theory
106
The program is implemented, but the theory is faulty; hence, the expected results do not occur
Theory failure
107
The planned program services are not delivered; hence, the expected results do not occur
Implementation failure
108
Uses the commonsense idea that an effective way to determine “what works” is to examine successful cases and compare them to other cases.
Who did this?
Success case method
Bricnkerkoff
109
Service utilization and organizational plan
Program process theory
110
Causal validity
Internal validity
111
has to do with the degree to which the results of a study can be generalized to and across populations of persons, settings, times, outcomes, and treatment variations.
external validity
