Evaluation Flashcards
(26 cards)
Johnson, Onwuegbuzie, & Turner (2007)
They argue that MM is the third research paradigm.
- MM is based on a philosophy of pragmatism.
- MM is based on a desire for method triangulation and multiple operationalization of constructs
- Definition of MM: Using elements of both QUANT and QUAL research techniques for purpose of breadth and depth of understanding and corroboration
- Can be either equal status or Qual/quant dominant
Rossi, Lipsey, & Henry (2019)
Definition of Evaluation
1. The systematic assessment of intervention programs designed to improve social conditions
Goal: informing decision makers
2. Program sponsors can influence the evaluation
3. Formative/Summative
Five domains of Evaluation:
- Needs
- Theory and Design
- Process
- Impact
- Efficiency
The Five Domains of Evaluation
Rossi, Lipsey, & Henry (2019)
- Needs assessment
- Theory and design
- Process
- Impact
- Efficiency
Mertens (2018)
Argues that MM research is best suited for addressing complex social problems/wicked problems
Purpose of Evaluation: informing stakeholders about the value of an evaluand
Four branches of Evaluation:
METHODS (postpositive): value QUANT and RCT as the gold standard
USE(pragmatism): focus on applying results to program decision making
VALUES (constructivist): focuses on the participant and understanding his perspective
SOCIAL JUSTICE (transformative): focuses on giving voice to marginalized groups and addressing inequality.
Complexity theory
Mertens (2018) describes these aspects of complexity theory:
- change is non-linear
- changes can emerge and be different than intended, different than sum total of parts
- people and systems are adaptive
- systems are dynamic
- participants co-evolve
The global conveyor belt exchanges ideas from each branch
Phillips (2018)
Evaluation emerged as a field in the 1960s due to new govt. programs and need for increased accountability.
Two roles fo evaluation:
1. Chronbach: to provide information for stakeholders to use to make conclusions
2. Scriven: to make a value judgment about the worth of the program including unintended consequences (the scientific merit).
Johnston, Midgett, Doumas, & Moody (2018)
GOAL:The goal of the study was to evaluate the effectiveness of an intervention to increase knowledge, awareness, and strategy use for decreasing bullying
DESIGN: MM Convergent design with pre-post of surveys and post interviews
PARTICIPANTS: A stratified random sample of 200 high school students
MEASURES: surveys for awareness, knowledge, and confidence for dealing with bullying
ANALYSIS: ANOVA, t-test, descriptives
FINDINGS: The authors found significant increase in knowledge and confidence (ANOVA, T-test), They found increase frequency of awareness (descriptive), and they found which strategies Ss used more and less. QUAL: corroborated quant findings
NEW INSIGHTS: identified the unintended consequence of tension with peers when using strategies
DEPTH: The lowest frequency was for reporting bullying to teachers. The qual interviews revealed that Ss didn’t trust teachers to improve the problem.
IMPLICATIONS: Counselors should use the program, need an anonymous reporting mechanism
Bryson (2004)
The author argues that identifying all relevant stakeholders is important for the success of a program.
1. The author argues for a broad definition of stakeholder. Using a broad definition and consulting with all relevant stakeholders will help ensure that the intervention creates public value.
2. In order to build consensus for the program, you need to identify the “supra interests” that can unify diverse stakeholders under a common good.
3. The author provides strategies such as a power matrix and participation matrix to help identify stakeholders.
Five things to consider about stakeholders:
1. Organizing participation
2. Creating ideas for interventions
3. Finding supra-interests
4. Identifying conflicts
5. Proposal development
Bryk et al. (2015)
The authors argue that to make improvements you have to first see the system that produces the current outcomes.
Park et al. (2015) said that all systems are perfectly designed to achieve the outcomes they produce.
1. Complex systems produce results that no one designed for.
2. You can perform a causal system analysis to understand the system better. This involves continuously asking “why” things occur. You can make a fishbone diagram to show the current system.
3. Next, use prior research to design a solution system. All the parts of the solution system need to work together
4. You also need adaptive integration in order to adapt the intervention and the system to achieve desired results for specific contexts.
Lipsey (2007)
The author argues that effective program evaluation requires establishing an understanding of the “black box” of the treatment effect.
1. We need to use a Theory of Treatment to help explain the causal chain that allows the treatment to produce the desired effect.
GOAL: the goal of the theory of treatment is to enable the researcher to claim that the treatment produced the effect and rule out rival confounding factors
PARTS of TOT:
1. Specific problem definition
2. Critical inputs and interrelationships
3. The causal mechanism of the treatment
4. The expected output, including the magnitude of effect
Rossi, Lipsey, and Henry (2019)
Treatment Theory
They define the Theory of Treatment as a conceptualization of what needs to occur to create the intended outcomes.
- makes explicit the cause and effect sequence to produce outcomes (implicit=black box)
- The ToT helps define the program boundaries.
Cook et al (2010)
The authors describe strategies for participant recruitment for evaluation studies.
- Recruitment refers to obtaining the desired number of participants who meet certain eligibility requirements
- Retention refers to maximizing the number of participants who complete the study and are available for follow up activities
- In designing a study it’s important to balance the need for data collection with the potential negative effects on participants.
- Involve participants in the study planning to ensure cultural responsiveness
Mclaughlin et al (2010)
The authors argue that a logic model is a kind of hypothesis or convincing argument about how the program will transform resources into activities for certain people which will lead to the result the program is expected to achieve in a specific context.
It includes: Resources, activities, outputs, and short, medium, and long-term outcomes.
Rossi, Lipsey, Henry, (2019)
Process evaluation
Process evaluation involves assessing how well a program is operating and performing the intended functions.
Process evaluation: Evaluating whether a program is implemented consistent with the program design and delivered with sufficient quality, frequency, and intensity
Program failure can be due to problems with implementation.
A process evaluation can help identify and improve problems
Fidelity of implementation
The extent to which the program adheres to program theory and design (Rossi, Lipsey, and Henry, 2019)
Coverage
The extent to which participation by the target population achieves the levels specified in the program design.
Bias
The degree to which some subgroups participate in greater proportions than others
Zhang et al (2011)
The authors conducted an evaluation study to evaluate the benefit of a service learning intervention that paired pre-service teachers with at-risk grade school readers.
- The authors used the Stufflebeam’s (2003) CIPP model to evaluate the effectiveness of the program. This model helped ensure that all participants were involved in the different evaluation stages (e.g. schools involved in needs assessment)
- Results: the program resulted in increases in outcomes for teachers, such as improving their knowledge of teaching reading. The program resulted in increases in student outcomes, such as reading ability and reading confidence.
Stufflebeam (2003)
The author presents the CIPP model for program evaluation. The goal is to assess the merit of programs to assist decisions about whether to continue programs.
CONTEXT: The context aspect focuses on identifying the needs of a target group of participants
INPUT: Focuses on Identifying the resources and strategy to meed the identified needs
PROCESS: This is an ongoing check on a plan’s implementation and documentation of the check.
PRODUCT: Involves measuring, interpreting, and judging a program’s merits including the intended and unintended outcomes.
Dusenbury et al. (2003)
The author’s define fidelity of implementation as the extent to which program service providers implement the program design as intended by the developers.
Five components of fidelity: Adherence, dose, quality of delivery, participant responsiveness, program differentiation
Shadish, Cook, & Campbell
Experiments and Causal Inferences
- Experimentation involves direct manipulation of something to observe the effects.
- Most causes are actually INUS conditions (insufficient, non-redundant, unnecessary, part of a sufficient condition
- Counterfactual: What would have happened to a treatment group if they simultaneously did not receive the treatment
- The goal of an experiment is to approximate the counterfactual in order to test the causal chain to identify
whether certain variables lead to expected effects. - Causal description=a description of the causes attributable to an effect; Causal explanation=a explanation of the mechanisms and conditions under which a causal mechanism holds.
- All science involves a level of trust. There is no “naive realism” in which experiments provide a perfect description of reality. Science does not provide a “clear window” onto nature. CONCLUSION: we need to accept the ambiguity of science.
Shaddish, Cook, & Campbell (2002)
Statistical Conclusion Validity
Internal Validity
Validity=The approximate truth of an inference. The evidence supports the inference as being true. Validity is a property of inferences.
Statistical conclusion validity=The appropriate use of statistics to infer whether the IV and DV covary.
Validity threats: Reasons that we can be wrong about inferences we made about the experiment.
Threats to statistical conclusion validity: Reasons researcher may be wrong about inferences drawn about the existence and size of covariation
Low power, unreliable measures, unreliable implementation, violated assumptions
Internal Validity=The ability to infer that a causal relationship exists between IV and DV. The ability to infer that the treatment caused the observed effects and not some other exogenous factor. The focus is usually on Molar causes.
Internal validity is the sine qua non of experimentation.
Threats to Internal Validity: Other possible causes for the effect that would have produced the effect even without treatment. Selection bias, history, maturation, attrition, regression to mean, instrumentation
We can’t control for all threats, we just need to control the most plausible ones.
Shadish, Cook, & Campbell (2002)
Construct Validity
It involves making valid inferences from the sample to the higher order constructs they represent.
Constructs involve social creation of meaning, so agreement is never complete. There is never a 1 to 1 relationship between the construct and sampling particular.
The goal is to do pattern matching between the prototypical case and the particular case.
To ensure construct validity:
- Think through definition of constructs
- Differentiate from similar constructs
- Find multiple measures for the constructs
Shadish, Cook, & Campbell (2002)
External Validity
Inferences about the extent to which causal inferences hold across different people, settings, treatments, and outcomes.
The goal of experimentation is to be able to extend to untested contexts where the results will be beneficial.
Inferences can be narrow to broad, broad to narrow, or the same level, similar or different units
Threats: different units produce different effects, treatment variation, different settings affect results,
