Weeks 9 & 10: Mixed Method Designs & Program Evaluation

Question 1

What is a mixed-method research design?

Answer 1

A form of triangulation in which two sets of findings from different strategies leads to a third, more meaningful interpretation; the combining of more than one method of research design (quantitative and/or qualitative)

Question 2

Mixed method research can be either “within” or “between” methods. How is “within” different from “between” in mixed-method research, and which is more common?

Answer 2

-“Within” means that the mixed methods come from within the same type of design (either quantitative or qualitative)

-“Between” means that methods from two different types of research design are used (ex. one quantitative and one qualitative); this is the most common type

Question 3

Why might someone choose to use mixed-method research design? (4)

Answer 3

-To counter-balance the limitations of the other method(s)
-To answer variations of the research question, either in order to generalize or to gain depth
-To add more breadth and depth to the overall results
-To use one method as a preliminary to the main method (ex. assessing the appropriateness of a questionnaire using focus groups before administering the questionnaire)11

Question 4

What are the types of mixed-method design? (4)

Answer 4

DEEEm+:
-Descriptive
-Exploratory
-Explanatory
-Exposing marginalization
-Sequential
-Divergent

Question 5

What does the “Creswell two-phase” mixed method research design entail?

Answer 5

The quantitative and qualitative designs occur in two separate phases, one after the other

Question 6

What does “Creswell’s dominant/less dominant” mixed method research design entail?

Answer 6

One methodology is the primary design and then a second methodology simply complements the first; both occur with the same timeframe
-Equivalent to “Engel & Schutt’s embedded”

Question 7

What does “Creswell’s mixed methodology” research design entail?

Answer 7

Two designs are integrated throughout within the same timeframe; not often used, as it would compromise the strength of each design alone
-Equivalent to “Engel & Schutt’s convergent/parallel”

Question 8

What does “Creswell’s convergent design” entail?

Answer 8

The quantitative and qualitative designs occur within the same study timeframe and convergences and divergences inform the results

Question 9

What does “Creswell’s explanatory sequential” mixed-method research design entail?

Answer 9

Two methods in sequence, with the second one (usually qualitative) informing the first (quantitative)
-Equivalent to “Engel & Schutt’s explanatory sequential”

Question 10

What does “Creswell’s exploratory sequential” mixed-method research design entail?

Answer 10

Two methods in sequence where the qualitative occurs first to create measures and then the quantitative method uses those measures

Question 11

What are some examples of “Creswell’s complex options” of mixed-method research design? (4)

Answer 11

-Experimental (intervention) designs: experiments with a qualitative component
-Participatory action research designs: more than one method as part of PAR, decided on by participants
-Multiple case study designs: mixed methods used to examine case studies
-Evaluation designs: mixed methods used in program evaluation

Question 12

What does “Engel & Schutt’s convergent/parallel” mixed-method research design entail?

Answer 12

Quantitative and qualitative methods each have equal weight in contributing to the answer to the research question and are implemented at the same time; findings are integrated and interpreted together
-Equivalent to “Creswell’s mixed methodologies”
-Also referred to as “integrated” designs

Question 13

What does “Engel & Schutt’s explanatory sequential” mixed-method research design entail?

Answer 13

The quantitative method is implemented first, then a qualitative method is used afterwards to provide more depth
-Equivalent to “Creswell’s explanatory sequential”

Question 14

What does “Engel & Schutt’s embedded” mixed-method research design entail?

Answer 14

The main method (qualitative or quantitative) is only complemented by the secondary design, which is a small addition during the study’s timeframe used to obtain further depth (not sequential)
-Equivalent to “Creswell’s dominant/non-dominant”

Question 15

What does “Engel & Schutt’s transformative” mixed-method research design entail?

Answer 15

A social justice focus, in the same kind of way as participatory action research

Question 16

What does “Engel & Schutt’s multiphase” mixed-method research design entail?

Answer 16

A long-term design that likely occurs over a period of several years; each design’s results informs the next design
-Also referred to as “staged design”

Question 17

“Two-eyed seeing” can use any mixed-method research design, but what sets it apart from purely eurowestern research design? (2)

Answer 17

-Combines the strength of Indigenous and eurowestern worldviews
-Incorporated reciprocity, Indigenous rituals, and accountability

Question 18

What are the four types of possible difference between mixed-method research designs?

Answer 18

-Timing (concurrent vs. sequential)
-Weight (dominant vs. less dominant)
-Purpose (descriptive, explanatory, exploratory, o exposure of marginalization, or a combination)
-Degree of integration (fully integrated vs. presented separately)

Question 19

What are some disadvantages to mixed-method research designs? (4)

Answer 19

-Time consuming
-Require expertise in both paradigms
-Scope of study becomes less focused/more complicated
-Lengthy reporting requirements

Question 20

What would “purists” say about mixed-method research designs?

Answer 20

They cannot be done/are impossible

Question 21

What would “situationalists” say about mixed-method research designs?

Answer 21

Their feasability/potential depends on the situation they are used in

Question 22

What would “pragmatists” say about mixed-method research designs?

Answer 22

The method that is used must depend on the research question being asked

Question 23

What would “pluralists” say about mixed-method research designs?

Answer 23

That both quantitative and qualitative methods must be incorporated in order to gain knowledge about both subjective and objective realities

Question 24

What is the most common type of research conducted by social work practitioners?

Answer 24

Program evaluation

Question 25

When conducting program evaluation from a structural/anti-oppressive perspective, what must be included?

Answer 25

An advisory committee: the perspectives/participation of service users in the development and implementation of the design

Question 26

What is program evaluation?

Answer 26

Research, usually for a specific program/initiative, used to reach the following goals:
-assessing the strengths and needs for change/additional programming
-showing client satisfaction
-reporting on output (services provided, completion rates)
-demonstrating outcomes, including effectiveness

Question 27

Why are program evaluations conducted? (5)

Answer 27

-Evidence of effectiveness for service users
-Evidence of cost-effectiveness for funders
-Evidence for accreditors that interventions are not harmful
-To help decision-makers (board to directors, managers, etc.) to make decisions about change
-Information for frontline workers about what is working/not working/harmful

Question 28

What are the three types of program evaluation?

Answer 28

-Needs/strengths assessment
-Formative/process
-Outcome

Question 29

What type of evaluation is most appropriate for the “determining need” stage of a program?

Answer 29

Needs/strength assessment

Question 30

What type of evaluation is most appropriate for the “planning” stage of a program?

Answer 30

Formative/process

Question 31

What type of evaluation is most appropriate for the “implementation” stage of a program?

Answer 31

Formative/process

Question 32

What type of evaluation is most appropriate for the “completed” stage of a program?

Answer 32

Outcome

Question 33

What is meant by “descriptive” research when it comes to mixed-methods, and what kind of statistics does it use?

Answer 33

Describes the population/group in question; univariate statistics

Question 34

What is meant by “exploratory” research when it comes to mixed-methods, and what kind of statistics does it use?

Answer 34

Obtaining depth or investigating a new topic area; in-depth qualitative data rather than statistics

Question 35

What is meant by “explanatory” research when it comes to mixed-methods, and what kind of statistics does it use?

Answer 35

Prediction of cause/effect relationship between variables and determining whether interventions are effective; requires inferential statistical analysis

Question 36

What is the second most common type of research for social workers, and what is it used for?

Answer 36

Needs/strengths assessment; used to examine need for/availability of resources, identify barriers to accessing resources, develop a program plan

Question 37

What are the most common designs for a needs/strengths assessment? (2)

Answer 37

-Survey
-Focus groups, sharing circles, interviews

Question 38

Do needs/strengths assessments tend to be exploratory, explanatory, descriptive, or exposing of marginalization?

Answer 38

-Usually descriptive
-Increasingly exposing of marginalization
-May be exploratory or explanatory, but not as common

Question 39

What is a program logic model, and what is it used for?

Answer 39

A flow chart representing the connection of various components of a program

Used in program planning to develop ideas and decide on vision, mission, goals, objectives, activities, and intended outcomes

Also used to guide program evaluation questions about inputs, process, outputs, and outcomes

Question 40

What is an environmental scan?

Answer 40

A method of monitoring a program/service’s external environment for relevant present factors (demographics, waitlists, etc.) and future trends (drawn from past evaluation results, census data, literature, etc.)

Question 41

What is a formative evaluation?

Answer 41

AKA “process evaluation”; a process conducted through expert consultation which is focused on the planning stage of program development; investigates…
-Organizational structure
-Supports available to workers
-How practice decisions are made
-Whether a program is being implemented the way it was designed

Question 42

What is the difference between inputs, program processes, outputs, and outcomes in program evaluation?

Answer 42

Inputs: client characteristics and number entering the service

Program processes: activities, services offered

Outputs: number of “graduates”, amount of training

Outcomes: impacts on clients (ex. feelings of depression)

Question 43

What is meant by “target population” in program evaluation?

Answer 43

Includes not just those who attend a program, but all people who ought to be able to attend

Question 44

What is meant by “stakeholders” in program evaluation?

Answer 44

Everyone who has a stake in the program: service users, funders, accreditors, staff, and the public

Question 45

What is a summative evaluation?

Answer 45

AKA “outcome evaluation”; a process that focuses on the effectiveness of the program through explanatory research designs; requires consideration of internal and external validity to determine the strength of the design

Question 46

What is the best way to objectively show effectiveness of an intervention?

Answer 46

Establishing cause and effect through the use of a pre-test/post-test control group

Question 47

What is a pre-test/post-test comparison group?

Answer 47

If a control group is not possible but it is possible to have two groups of participants, they are given similar (but non-equivalent) conditions; cannot establish cause and effect, but does show how much service users change in comparison to another condition (better than no measurement at all)

Question 48

What is meant by “one group pre-test/post-test”?

Answer 48

There is no option to compare groups, as all participants must be part of the same group; limited external and internal validity, but does show whether service users changed between the start and end of the program

Question 49

What is a “one group post-test” in program evaluation?

Answer 49

A measurement of how service users functioned (or their satisfaction) at the of a program when we cannot compare groups or have a pre-test (often a survey); limited validity and low response rate, but better than no measurement at all

Question 50

What makes satisfaction surveys less favourable than randomized controlled trials in program evaluation? (3)

Answer 50

-Low response rate
-Mostly positively skewed
-Satisfaction does not mean effectiveness

Question 51

What is a “single subject/system design” in program evaluation?

Answer 51

Measuring one unit of analysis (a single subject, a single system/group of people, etc.) rather than comparing groups like you would do in an experimental design
-You still measure an outcome/dependent variable and an independent/intervention variable, but measurements occur repeatedly throughout all phases of the research
-Analysis is done through graphs/trends rather than through statistics

Question 52

What notation is used in single-subject/system design program evaluation?

Answer 52

A = baseline
B = intervention

So B = measurement during intervention only, AB = measurement at baseline and intervention, ABA = measurement at baseline, intervention, and withdrawal, etc.

Question 53

What are some strengths of single-subject/system designs in program evaluation? (4)

Answer 53

-Easy to comprehend
-Focuses on the individual, respecting culture and dignity
-Almost instant feedback
-Can be incorporated directly into therapeutic sessions

Question 54

What are some weaknesses of single-subject/system designs in program evaluation? (4)

Answer 54

-Inability to generalize
-Questionable accuracy (self-reporting; reactive effects, testing effects)
-Issues of worker fidelity, incompetence, and/or dual relationship
-Requires cooperation during administration

Question 55

What are the two types of efficiency evaluation/analysis in program evaluation?

Answer 55

-Cost/benefit analysis: determining if a program is worth its cost

-Cost effectiveness: determining which program is most cost effective through comparing several against one another