Final Flashcards
(176 cards)
1
Q
An evaluation that uses the data collected before a program begins. Examples include pilot projects, baseline surveys, and feasibility studies.
A
Design evaluation
2
Q
Coefficient of stability: Evaluator administers the same instrument twice, separated by a short period of time. Results are compared, using a statistic such as a correlation coefficient
A
Reliability coefficient/ test-retest reliability
3
Q
Evaluator administers two equivalent versions of the same instrument (parallel forms) to the same group of people. Results are compared, using a coefficient of stability.
A
Alternate-form coefficient
4
Q
A nonexperimental research method relying on questionnaires or interview protocols
A
Survey
5
Q
The extent to which researchers can make this statement with confidence
A
Internal validity
6
Q
Study conducted at a single time period, and data are collected from multiple groups
A
Cross-sectional study
7
Q
Data are collected at two or more points in time
A
Longitudinal study
8
Q
Design that combines cross-sectional and longitudinal elements by following two or more age groups over time
A
Cohort-sequential design
9
Q
Two observers’ data are compared to see whether they are consistently recording the same behaviors when they view the same events. Statistical procedures such as correlation or percentage of agreement can be used to establish this type of reliability.
A
Interrater reliability
10
Q
Interpretive research approach relying on multiple types of subjective data and investigation of people in particular situations in their natural environment
A
Qualitative research
11
Q
Use of multiple data sources, research methods, investigators, and/or theories/ perspectives to cross-check and corroborate research data and conclusions
A
Triangulation
12
Q
used to determine whether a single observer is consistently recording data over a period of time.
A
Intrarater reliability
13
Q
To what degree does all accumulated evidence support the intended interpretation of scores for the proposed purpose?
A
construct validity
14
Q
commonly used data collection instruments or procedures designed to measure personality, aptitude, achievement, and performance.
A
Tests
15
Q
Items on the test represent content covered in the program (e.g., did the teacher teach the children that blue on the map means water?). Evaluators can work with content specialists to list the content that is part of the program, and can compare the test items to see whether they correspond.
A
Content-related evidence
16
Q
a self-report data collection instrument that is filled out by research participants.
A
Questionnaire
17
Q
The researcher takes independent samples from a general population over time and the same questions are asked.
A
Trend study
18
Q
a situation where the interviewer asks the interviewee a series of questions
A
Interview
19
Q
indicates that the measure actually reflects current or future behaviors or dispositions.
A
Criterion-related evidence
20
Q
a situation where a focus group moderator keeps a small and homogeneous group (of 6–12 people) focused on the discussion of a research topic or issue.
A
Focus group
21
Q
Evaluators need to be aware of the consequences of using data, especially with regard to the potential to worsen inequities.
A
Consequential evidence
22
Q
Researcher watches and records events or
behavioral patterns of people
A
Observation
23
Q
Observation conducted in real-world situations
A
Naturalistic observation
24
Q
Observation conducted in lab setting set up by the researcher
A
Laboratory observation
25
Evaluators need to stay on site for sufficient time to “get the story right.” If a study is conducted over too short a time or interviews are conducted with too few people, it is possible that an evaluator will reach “premature closure”
Prolonged and substantial engagement
26
the ability to generalize the results of a study to the population from which the research population was drawn.
External validity
27
A word that produces an emotionally charged reaction
Loaded term
28
A question that suggests how the participants should
| answer
leading question
29
Asking about two or more issues in a single question
Double-barreled question
30
Nonoverlapping response categories
Mutually exclusive categories
31
Response categories that cover the full range of possible responses
Exhaustive categories
32
An ordered set of response choices, such as a 5-point rating scale, measuring the direction and strength of an attitude
Rating scale
33
Descriptors placed on points on a rating scale
Anchor
34
Participant must select from the two response choices provided with an item
Binary forced-choice approach
35
A scaling technique that is used to measure the meaning that participants give to attitudinal objects or concepts and to produce semantic profiles
Semantic differential - a scaling technique that is used to measure the meaning that participants give to attitudinal objects or concepts and to produce semantic profiles
36
An item directing the participant to different follow-up questions depending on the initial response
Contingency question
37
Tendency for a participant to respond in a
| particular way to a set of items
response
38
Evaluators need to be aware of their assumptions, hypotheses, and understandings, and of how these change over the period of the study.
Progressive subjectivity
39
those incurred to provide a treatment or a service
direct costs
40
resources lost due to the disorder.
indirect costs
41
Events that occur during the study other than the experimental treatment effect that can influence the results
History threat
42
describing design that is intended to simplify life for everyone by making products, communications, and the built environment more usable by as many people as possible at little or no extra cost
Universal design
43
naturally occurring physical or psychological changes
Maturation threat
44
A monitoring system that would involve community members in determining what indicators of change are important to them
Most Significant Change Method
Christian Commission for Development in Bangladesh and Rick Davies
45
Administration of a pretest that affects participants’ performance on a posttest (i.e., participants become “testwise”).
testing threat
46
Having pretests and posttests that differ in terms of difficulty; this can lead to seeming changes that are really due to the difference in the tests.
Instrumentation
47
Having extreme groups in a study (i.e., people very high or very low on a particular characteristic); it is possible that changes will be seen on the dependent variable because of this threat.
statistical regression
48
Differences between the experimental and control groups on important characteristics, other than receipt of the intervention.
differential selection
49
Differential dropouts of participants from either the experimental or control groups.
experimental mortality
50
treatment was not implemented as intended
treatment fidelity
51
monetary (i.e., dollar) values are estimated for the resources used (i.e., costs) and for the program effects (i.e., benefits), and these two components (costs and benefits) are then compared to determine the worth of a program.
cost-benefit analysis
52
based on cost and effect data (rather than cost and benefit data as in benefit-cost analysis).
cost-effectiveness analysis
53
What a concept means in abstract or theoretical terms
Conceptual definition of a sample
54
What you will observe and/or measure; links the concept you want to sample to the real world
Operational definition of a sample
55
Group to which you wish to generalize your results
target
56
List of people who match the conceptual definition
Experimentally accessible population
57
List of people in the experimentally accessible population
Sampling frame
58
Match between accessible population and target population
population validity
59
voluntary consent without threat or undue inducement; it includes knowing what a reasonable person would want to know before giving consent (informed), and explicitly agreeing to participate (consent).
informed consent
60
collecting, analyzing, storing, and reporting data in such a way that the data cannot be traced back to the individual who provides them.
confidentiality
61
involves the selection of a sample from a population in a way that allows for an estimation of the amount of possible bias and sampling error.
Probability-based sampling
62
Those in which every member of a population has a known nonzero probability of being included in the sample.
Random samples
63
Every member of the population has an equal and independent chance of being selected.
Simple random sampling
64
A robust description of the context in which qualitative research is conducted
Thick description
65
Selection of easily obtainable participants for sample group and usually the cheapest and fastest way of obtaining a sample group; interviewers happen to be available at a program site.
Convenience sampling
66
When will you have a poor strength of treatment?
A. When the participants are sensitized to the posttest.
B. When the evaluator uses different types of measurement for dependent variables.
C. When the treatment “dose” was not strong enough to produce the expected change.
D. When the evaluator systematically conducts an intervention at different times of the day.
C
67
In evaluation, a limitation of traditional definitions of needs assessment (according to your textbook) is that they focus only on needs and lack any attention to assets?
A. True
B. False
True
68
The assumption that all people within a particular subgroup are similar to each other in terms of their other background characteristics, or similar enough where differences are not identified.
Myth of homogeneity
69
Which of the following is(are) needed to infer causality?
A. X precedes Y
B. X is related to Y
C. Confounding variables can be ruled out as causes
D. All of the above are required.
D
70
```
When an extraneous variable systematically varies with the independent variable and influences the dependent variable, it is called:
A. Another dependent variable
B. A confounding variable
C. A moderating variable
D. An unreliable variable
```
B
71
Data analysis tends to be an ongoing and iterative (nonlinear) process in qualitative research.
The cyclical process of collecting and analyzing data during a single research study).
Interim analysis
72
the difference between the sample and the population.
Sampling error
73
A major characteristic of experimental and quasi-experimental designs is an independent variable that can be manipulated.
A. True
B. False
True
74
Used in telephone surveys; computer generates a random list of phone numbers.
Random - digit dialing
75
```
If an evaluation is focused on addressing human rights, with what paradigm is it most closely aligned?
A. Postpositivist
B. Transformative
C. Constructivist
D. Pragmatic
```
B
76
Take every nth name off a list. Suppose you have 1,000 names on the list and you need a 10% sample. You pick a random number between 1 and 10, start there, and then take every 10th name on the list.
Systematic sampling
77
Which of the following is NOT a purpose for an evaluation?
A. To identify needed inputs, barriers, and facilitators to program development or implementation.
B. To determine inequities on the basis of gender, race, ethnicity, disability, and other relevant dimensions of diversity.
C. To determine how a program can be made to look positive despite its lack of impact/effect.
D. To demonstrate that accountability requirements are fulfilled.
C
78
If there are different groups (strata) that you want to be sure to include, then you can divide the population into subgroups first and then randomly sample from the subgroups.
Stratified sampling
79
```
What are some designs used by the methods branch to determine the effectiveness of an intervention?
A. Experimental, quasi-experimental, single-group, and survey
B. Narrative, phenomenological, ethnographic, and case study
C. Dialectical, transformative, and case study
D. Gender, race, and class analysis
```
A
80
Use with naturally occurring groups (e.g., classrooms, school districts, city blocks). Units are randomly selected from full list of possible sites. Then you can collect data from the members in the randomly selected unit.
Cluster sampling
81
Which of the following is legitimately considered (according to your textbook authors) a type of evaluation?
A. Organizational assessment
B. Participatory evaluation
C. Cost analysis/assessment
D. All of the above are considered legitimate
D
82
Use a combination of sampling strategies over the course of the study (e.g., start with cluster sampling, then use simple random sampling within clusters).
Multistage sampling
83
Cost Analysis is defined as an evaluation’s determination of whether a program’s effect was worth its cost.
A. True
B. False
A
84
Pilot testing data collection instruments is not very important.
A. True
B. False
A
85
Choose unusual or special individuals (e.g., highly successful or unsuccessful school principals).
Extreme or deviant sampling
86
Identify instances where the phenomenon of interest is strongly represented. Look for rich cases that are not necessarily extreme.
Intensity sampling
87
What are some examples of qualitative data collection options as listed by your textbook authors?
A. Observations, interviews, review of artifacts, and focus groups
B. Performance assessments and structured observations
C. Interviews, surveys with random samples, and norm-referenced tests
D. Structured observations, criterion-referenced tests, and portfolios
A
88
```
What are some forms of evidence used to support validity/credibility in qualitative data collection?
A. Peer debriefing
B. Member checks
C. Persistent observations
D. All of the above
```
D
89
Choose individuals that represent maximum variation of the phenomenon (e.g., teachers in isolated rural, suburban, and inner-city areas).
Maximum variation sampling
90
Identify strongly homogeneous cases; find individuals who share relevant characteristics and experiences.
Homogeneous sampling
91
What is universal design as discussed by the textbook authors?
A. A way to simplify life for everyone by making products, communications, and the built environment more useable by as many people as possible.
B. A technique for making tests more accessible to disabled people.
C. Use of multiple languages in a data collection instrument
D. All of the above.
A
92
In most cases, use homogeneous groups (e.g., if service providers and participants are included in the same focus group, this might yield biased results).
Focus group sampling
93
This is the opposite of the extreme or deviant sampling strategy; you want to identify the typical or average
Typical case sampling
94
What is INTRArater reliability?
A. It is used to determine whether a single rater or observer is consistent over time.
B. It compares the data of two raters or observers to see whether they are rating the same behavior consistently.
C. It is used to compare two kinds of data collection to see whether they are describing the same event.
D. It is used to compare when different raters are administering similar instruments
A
95
Strategy combines the identification of strata of relevant subgroups with purposeful selection from those subgroups.
Stratified purposeful sampling
96
Use cases that can make a point dramatically or are important for other reasons. Patton (2002b) says that the key to identifying a critical case is “If it’s true of this one case, it’s likely to be true of all other cases” (p. 243).
Critical case sampling
97
```
An evaluator created a test. In order to test reliability, he had the participants take the test and he analyzed the results to examine the consistency of their responses. What is this an example of?
A. Repeated measures reliability
B. Intraparticipant reliability
C. Internal-consistency reliability
D. Multi-dimensional reliability
```
C
98
Start with key informants who are then asked to recommend others you should talk with—some who agree with them and some who disagree with them.
Snowball or chain sampling
99
Set up criteria to specify what characteristics people in the study need to have.
Criterion sampling
100
Reliability and validity are commonly used terms to describe the quality of quantitative data collection. What does validity mean in this situation?
A. Does the instrument measure cultural competence?
B. Does the instrument (as used with the participants) really measure what it is supposed to measure?
C. Does the instrument measure what is it is supposed to measure consistently?
D. Does the instrument reliably measure what it is supposed to measure over time?
B
101
If the evaluation is focused on a theoretical construct such as creativity, you need to describe the meaning of that construct, and then identify individuals who theoretically exemplify that construct.
Theory-based sampling
102
In multiple regression, when we say that we control for the effects of some variable(s) we are:
A. statistically adjusting or subtracting the effects of a variable to see what a relationship would have been without it
B. actually removing a variable from a model so that it does not interact with the effects of other variables
C. changing the mediating capabilities of an endogenous variable
D. changing the mediating capabilities of an exogenous variable
A
103
Look for cases that both confirm and disconfirm emerging hypotheses.
Confirming and disconfirming case sampling
104
```
What kind of data collection is useful when you want people to discuss a particular topic?
A. Case studies
B. Interviews
C. Focus groups
D. Open-ended questionnaires
```
C
105
Selection of individuals emerges as the study progresses; you do not know a priori who will need to be included.
Opportunistic sampling
106
What are some critical issues related to data collection?
A. Language of participants
B. Literacy level of participants
C. Use of a dominant or colonizing language
D. All of the above
D
107
Randomly choose individuals from a purposefully defined group.
Purposeful random sampling
108
```
The ______ sampling strategy chooses unusual or special individuals (e.g., highly successful or unsuccessful school principles)?
A. Maximum variation sampling
B. Stratified purposeful sampling
C. Extreme or deviant sampling
D. Snowball or chain sampling
```
C
109
Determine whether there is a political reason for including particular areas and individuals for the credibility and perceived usefulness of the study.
Politically important case sampling
110
What is the definition of a sampling frame in your textbook (Mertens and Wilson)?
A. The target population of your study
B. List of all the people in the experimentally accessible population.
C. The people you plan to observe
D. None of the above
B
111
```
Which of the following is NOT one of the 13 categories of disabilities in the Individuals with Disabilities Education Improvement Act of 2004?
A. Emotional disturbance
B. Speech of language impairment
C. Visual impairment
D. Majority-minority group membership
```
D
112
The set of cases selected from the population
Sample
113
```
Which of the following is a type purposeful/theoretical sampling?
A. Simple random sampling
B. Critical case sampling
C. Systematic sampling
D. Cluster sampling
```
B
114
The full group to which one wants to generalize
Population
115
```
Which major sampling option is more commonly used in the Values Branch?
A. Probability-based sampling
B. Multistage sampling
C. Theoretical/Purposeful sampling
D. Simple Random Sampling
```
C
116
A numerical index based on sample data
Statistic
117
```
What is the opposite of deviant case or extreme sampling?
A. Typical case sampling
B. Homogeneous sampling
C. Critical case sampling
D. Snowball sampling
```
A
118
A numerical characteristic of a population
Parameter
119
```
What kind of sampling strategy begins with a random start, includes that element, and then includes every nth name off a list?
A. Interval sampling
B. Systematic sampling
C. Random digit sampling
D. Multistage sampling
```
B
120
The type of statistical analysis focused on describing, summarizing, or explaining a set of data
Descriptive statistics
121
According to your text, the “myth of homogeneity” means assuming that all people within a particular subgroup are similar to each other in terms of their other background characteristics, or at least sufficiently similar that you do not have to focus on those differences.
A. True
B. False
True
122
The type of statistical analysis focused on making inferences about populations based on sample data
Inferential statistics
123
What is nested sampling?
A. You gather samples using a variety of methods.
B. You use different people from different populations
C. You have identical samples for both the quantitative and qualitative parts of the study.
D. Data are collected from a group using one method; then a subset of that group is selected to provide data using another method.
D
124
The theoretical probability distribution of the values of a statistic that would result if you selected all possible samples of a particular size from a population
Sampling distribution
125
```
Which sampling strategy uses cases that can make a point dramatically or are important for other reasons?
A. Stratified random sampling
B. Critical case sampling
C. Homogeneous sampling
D. Politically important case sampling
```
B
126
The theoretical probability distribution of the means of all possible samples of a particular size selected from a population
Sampling distribution of the mean
127
The standard deviation of a sampling distribution
Standard error
128
A statistic that follows a known sampling distribution and is used in significance testing
Test statistic
129
An evaluation that is allowed to evolve throughout the course of the project. Examples include participatory, qualitative, critical, hermeneutical, bottom-up, collaborative, and transdisciplinary approaches.
Emergent evaluation
130
The branch of inferential statistics focused on obtaining estimates of the values of population parameters
estimation
131
Use of the value of a sample statistic as one’s estimate of the value of a population parameter
Point estimation
132
Placement of a range of numbers around a point estimate
Interval estimation
133
Use a word or short phrase to summarize the topic found in a passage of the data.
Descriptive codes
134
Use the exact language of the participants as a code.
In vivo codes
135
Captures actions in the data and usually ends in “-ing.”
Process coding
136
Labels emotions that are expressed by the participants.
Emotion coding
137
Can reflect the values, attitudes, or beliefs expressed by participants.
Values coding
138
A hypothesis states that there is no difference between the scores of the experimental group and the control group
Null hypothesis
139
The hypothesis states that there will be a difference between means in the population.
Alternative hypothesis
140
An interval estimate inferred from sample data that has a certain probability of including the true population parameter.
Confidence interval
141
A set of data, where the rows are “cases” and the columns are “variables”
Data set
142
The analysis guides the design of subsequent stages of a study and leads to further analysis that integrates the data from these stages.
Sequential integration
143
Data arrangement in which the frequencies of each unique data value is shown
Frequency distribution
144
The branch of inferential statistics focused on determining when the null hypothesis can or cannot be rejected in favor of the alternative hypothesis
Hypothesis testing
145
Depicting frequencies and distribution of a quantitative variable
Histogram Graph
146
The point at which one would reject the null hypothesis and accept the alternative hypothesis
Alpha level
147
The average deviation of data values from their mean in squared units
Variance
148
The area on a null hypothesis sampling distribution where the observed value of the statistic, if it fell in this area, would be considered a rare event
Critical region
149
The square root of the variance
Standard deviation
150
The likelihood of the observed value (or a more extreme value) of a statistic, if the null hypothesis were true
Probability value (p value)
151
Conclusion that an observed finding would be very unlikely if the null hypothesis were true
statistically significant
152
Used to determine if the difference between the means of two groups is statistically significant
Independent samples t test:
153
Claim made when a statistically significant finding seems large enough to be important
Practical significance
154
Methods used to gather data in an emergency response situation in order to share information in real time.
Rapid evaluation and assessment methods (REAM)
155
An index of magnitude or strength of relationship
Effect size indicators
156
Rejection of a true null hypothesis
Type 1 error
157
Failure to reject a false null hypothesis
Type II error
158
Which of the following is used for group differences evaluation questions?
A. t test for independent samples
B. Pearson product-moment coefficient of correlation
C. Mean and variance
D. Range
A
159
What are some strategies for analyzing qualitative data?
A. Engage in continuous and ongoing data analysis
B. Reflectively reading interview transcripts and field notes to get a holistic picture of the research question.
C. Determining codes for the data that suggest emergent concepts
D. All of the above
D
160
```
Who are the researchers who initiated grounded theory as a systematic method?
A. Glaser and Strauss
B. Campbell and Shadish
C. Pope and Wallace
D. Patton and Stake
```
A
161
One reason that statistical analysis is useful is:
A. It helps you in coding interview transcripts
B. It helps you reduce a large amount of data into more meaningful terms such as an average
C. It is a systematic system for organizing data into relevant categories or themes.
D. It is useful for obtaining a thick description of the data
B
162
According to your textbook, generalizability is only a concern in the interpretation of quantitative data
A. True
B. False
B
163
According to the authors of your text, it is sometimes appropriate to involve stakeholders in the analysis phase of the evaluation.
A. True
B. False
A
164
What are some theoretical frameworks commonly used in qualitative data analysis?
A. Postpositivism
B. Postpragmatism
C. Postmodernism
D. Feminist theory and indigenous theory
D
165
According to your text, it is rarely if ever important for evaluators to use a particular theoretical framework lens in analyzing data.
A. True
B. False
B
166
What factor(s) might influence whether you decide to analyze your data using software or manually?
A. The amount of time you have available to analyze your data.
B. The amount of data you have collected
C. The training and support available at your institution.
D. All of the above are factors are important to consider.
D
167
```
The ______ sampling strategy chooses unusual or special individuals (e.g., highly successful or unsuccessful school principles)?
A. Maximum variation sampling
B. Stratified purposeful sampling
C. Extreme or deviant sampling
D. Snowball or chain sampling
```
C
168
What is the definition of a sampling frame in your textbook (Mertens and Wilson)?
A. The target population of your study
B. List of all the people in the experimentally accessible population.
C. The people you plan to observe
D. None of the above
B
169
```
Which of the following is NOT one of the 13 categories of disabilities in the Individuals with Disabilities Education Improvement Act of 2004?
A. Emotional disturbance
B. Speech of language impairment
C. Visual impairment
D. Majority-minority group membership
```
D
170
```
Which of the following is a type purposeful/theoretical sampling?
A. Simple random sampling
B. Critical case sampling
C. Systematic sampling
D. Cluster sampling
```
B
171
```
Which major sampling option is more commonly used in the Values Branch?
A. Probability-based sampling
B. Multistage sampling
C. Theoretical/Purposeful sampling
D. Simple Random Sampling
```
C
172
```
What is the opposite of deviant case or extreme sampling?
A. Typical case sampling
B. Homogeneous sampling
C. Critical case sampling
D. Snowball sampling
```
A
173
```
What kind of sampling strategy begins with a random start, includes that element, and then includes every nth name off a list?
A. Interval sampling
B. Systematic sampling
C. Random digit sampling
D. Multistage sampling
```
B
174
According to your text, the “myth of homogeneity” means assuming that all people within a particular subgroup are similar to each other in terms of their other background characteristics, or at least sufficiently similar that you do not have to focus on those differences.
A. True
B. False
A
175
What is nested sampling?
A. You gather samples using a variety of methods.
B. You use different people from different populations
C. You have identical samples for both the quantitative and qualitative parts of the study.
D. Data are collected from a group using one method; then a subset of that group is selected to provide data using another method.
D
176
```
Which sampling strategy uses cases that can make a point dramatically or are important for other reasons?
A. Stratified random sampling
B. Critical case sampling
C. Homogeneous sampling
D. Politically important case sampling
```
D
