Chapter 10 Flashcards
When is an experiment considered a true experiment?
if all aspects of a basic experimental design are present, and absent any threats to internal validity
Are all topics amenable to the use of a true experiment?
no
When are single case experimental designs used?
when only one appropriate participant is available.
When are quasi experimental designs used?
used when random assignment for a variable is not possible
Can anything that departs from a true experiment e sued to support claims about causality?
no.
Can applied research and evaluation research present difficulties that prevent researchers from using true experiments?
yes.
What are some reasons that may prevent applied researchers and evaluation researchers from conducting true experiments?
Sometimes researchers are invited to participate to late in the process and are not able to provide input on the best measurement techinques.
Budget limitations can rule out some form of data collection
Program evaluation research raises many of the issues found in what contexts?
applied contexts
What is program evaluation? What are they usually looking to achieve? Where are the programs often implemented?
Research designed to evaluate programs (e.g., social reforms, innovations) designed to produce changes or certain outcomes in a target population.
Usually looking to achieve some positive effect on a group of people.
Such programs can be implemented in schools, work settings, or entire communities.
Who urged a culture of evaluation in which all programs are thoroughly evaluated to determine whether they are effective?
Donald Campbell
How has the field of program evaluation progressed?
the initial focus of evaluation research was outcome evaluation (did the program result in the positive outcome for which it was designed?)
Evaluation research then broaden the scope to many other questions which can be categorized into 5 types.
What are the 5 types of questions that can guide program evaluations? Is this a sequential process? What type of questions are emphasized depend on?
This is a sequential process and the type of questions that are emphasized depend on the purpose of evaluation.
The categories/5 types of program evaluation research
- needs assessment
- are there problems that need to be addressed in a target population? - Program Theory Assessment
- How will the problems be addressed? Will the proposed program actually address the needs appropriately? - Process Evaluation
- Is the program addressing the needs appropriately? Is it being implemented properly? - Outcome Evaluation
- Are the intended outcomes of the program being realized? - Efficiency Assessment
- Is the cost of the program worth the outcomes?
What are examples of questions researchers could ask relating to needs assessments (evaluation research?)
- what services do homeless people need most?
- Do repeat juvenile offenders have particular personal and family problems that could be addressed by an intervention program?
- What are some of the most overall important needs among Portuguese Canadians?
How do researchers usually collect data for a needs assessment (evaluation research)?
often come form surveys, interviews, and existing archival data maintained by public health, criminal justice, and other agencies.
Explain we need to think about when assessing program theory?
Who does assessing theory usually involve?
What does assessing program theory include?
After identifying needs, a program can be designed to address them. The program must be based on valid assumptions about the causes of problems and a cogent rationale for the best way to address these problems.
usually involves researchers, service providers, a prospective client program clients collaborating to ensure that the program assesses the target populations’ needs.
includes articulating a rationale for how members will benefit from the program, including how they will access the program services.
What is another name for process evaluation?
When the program is underway, what do the researchers monitor?
What is this research very important for?
How is data for this research often collected?
program monitoring.
monitor whether it reaches the target pop., whether its attracting enough clients, and whether staff is providing he planned services.
This research is really important to avoid concluding that a program is ineffective, when really it is not being implemented properly.
Often collected through questionnaires, interviews, observational studies, and analyses of records kept by program staff.
What is another name for outcome evaluation?
How do researchers determine this?
What do we need to know when addressing this aspect of program evaluation?
What methods do we use to conduct outcome evaluation?
Impact assessment.
the researchers must devise a way of measuring the outcome and then study and then study the impact of the program on the outcome measure.
We need to know what participants of the program are like and what they would be like if they had not completed the program.
Ideally, a true experiment with random assignment to conditions would be carried out to answer questions about outcomes. However, other approaches like the quasi-experimental and single case designs, can be useful ways of assessing the impact of the intervention when random assignment is not possible or is not ethical.
What are researchers trying ot figure out hen doing efficeny assessment?
once it is shown that the progrma does have the intended affect, researchers can determien whtehr the benefits are worth the cost, or if the resources used to implement this program coudl be better used.
What is a shoestring evaluation? Who came up with it?
full program evaluations can be extensive, longterm undertakings, particularly if every question is being addressed. Researchers liek Bamberger and colleagues, are developing systematic approaches to respond to the specific challenges that arise while doing evaluation research. Shoestring evaluation is when there are constraints related to time, budget, and data collection options.
What are quasi experimental designs?
A study design that has many features of an experiment, but due to necessity lacks some aspects of a true experimental design (and so cannot support causal inferences).
What is quasi a latin term for?
As if
What do Quasi-experiments lack from a true experimental design out of necessity?
What is the result of this?
What do quasi experimental designs provide a good example of?
control conditions, random assignment
As a result, they cannot be used to make causal inferences, but rather indicate how variables are related. (i.e associations, similar to correlational studies with measured variables.)
Provide a good example of how studies that claim to b using true experimental designs might fall short of that goal, and inadvertently include elements that prevent them from being true experimental designs, and therefore do not permit causal inferences.
When using quasi experimental designs, what is it important to distinguish between?
(1) quasi-experimental designs used intentionally and out of necessity
(2) flawed experiments that purport to be true experiments but are actually quasi experimental in nature.
Are Quasi-experiments bad or poor experiments?
no. in many circumstances they are used out of necessity.
