Research and Program Evaluation II Flashcards
1
Q
Positivism
A
An objective truth exists and can only be understood if directly observable. Truth must be measurable
2
Q
Post-positivism
A
Truth can only be approximated bc of inherent errors present when measuring reality
3
Q
Constructivisim
A
There are multiple realities or perspectives for any given phenomenon
4
Q
Critical/ideological paradigm
A
Researchers taking a proactive role and confronting the social structure and conditions facing oppressed or underprivileged groups
5
Q
Nuremberg Code
A
Stemmed from Nazi Medical War Crimbes
6
Q
Milgram Obedience study
A
Shocks
7
Q
Belmont Report
A
Informed consent, right to withdraw, guidelines for use of deception. Stemmed from Tuskegee syphilis study
8
Q
Institutional Review Board (IRB)
A
any institution receiving federal funding must go through IRB to conduct research with humans
9
Q
Willowbrook Study
A
School for children with mental disabilities, parents who wanted to enroll signed a form to allow their children to be injected with hepatitis. Parents never told they could decline injections.
10
Q
Jewish Chronic Disease Hospital Study
A
Healthy and unhealthy patients injected with cancer cells . participants never gave informed consent and weren’t told they were being injected w/ cancer
11
Q
HIPAA
A
privacy rights of participants
12
Q
Research hypothesis (2 types)
A
testable, concise statement involving the expected relationship between 2 or more variables.
- Nondirectional: Eg: There is a significant relationship btwn amount of sleep and career satisfaction
- Directional: There is a significant positive relationship between . . .
13
Q
Null hypothesis
A
Statement that there is no relationship
14
Q
Alternative hypothesis
A
Developed in order to be eliminated and addresses the question “what else could be causing the results?”
15
Q
Significance level
A
Threshold for rejecting the null hypothesis, with values associated with alpha (typically .001, .01, .05)
16
Q
Statistical significance
A
Cutoff point (critical value)
17
Q
P value
A
likelihood of obtaining a result at least as extreme as the one observed assuming the null hypothesis is true
18
Q
Type I Error
A
(alpha) Occurs when a decision is made to reject a null hypothesis when in fact that null hypothesis is true
19
Q
Type II Error
A
(Beta) Occurs when a decision is made to retain the null hypothesis that should have been rejected bc it’s actually false
20
Q
Power
A
Likelihood of detecting a significant relationship between variables when one is really there
21
Q
Probability sampling
A
Sampling a known population
22
Q
Non probability sampling
A
More common in counseling research. Involves accessing samples of convenience
23
Q
Simple random sampling (probability sampling)
A
Every member of population has equal chance of being selected
24
Q
Systematic sampling (probability sampling)
A
Every nth element is chosen
25
Stratified random sampling (probability sampling)
Population divided into subgroups based on important characteristics (gender, race, etc) and counselor draws randomly from subgroups
26
Cluster sampling (probability sampling)
Counselor identifies existing subgroups and not individual participants.
27
Multi-stage sampling (probability sampling)
2 stage (randomly select 60 schools then 10 classes from each school) 3 stage (select 200 school districts, 20 schools, 10 classes) , etc.
28
Convenience sampling (nonprob sampling)
Most common method. Counselor selects an easily accessible population that most likely doesn't represent the pop of interest
29
Purposeful sampling (nonprob sampling)
Counselor selects sample from a population based on who will be most informative about a topic of interest. Participants selected bc they represent needed characteristics
30
Quota sampling (nonprob sampling)
Counselor draws needed number of participants with the needed characteristic (gender, race) from convenience sample
31
Random selection
Select participants from a population so that every member of the population has an equal chance of being selected
32
Random assignment
Randomly assigning participants to different groups, such as treatment or control group
33
Internal validity
Changes in the DV are due to the effects of the IVs
34
History (internal validity)
Extraneous incidents occur during research. Within or outside the study
35
Selection (internal validity)
Group differences exist before the intervention due to lack of random assignment
36
Statistical regression (internal validity)
Scores of participants who were selected because of their extreme score on a dependent variable (eg: extremely depressed, low achievement) are affected Those with extremely high or low scores will regress toward the mean
37
Testing (internal validity)
Test itself has an impact on individuals, especially when pretests are invovled. Practice effects (memory effects)
38
Instrumentation (internal validity)
Changes in the instrument affect results (computerized, paper and pencil, etc)
39
Attrition (internal validity)
Participants drop out of a research study
40
Maturation (internal validity)
Changes in a participant over time affect the DV
41
Diffusion of treatment (internal validity)
Effects of an intervention are felt by those in another group
42
Experimenter effects (internal validity)
Bias of the investigator influences participant responses. Halo effect (counselors subjective, usually positive perceptions of participant are genearlized to other traits) , Hawthorne effect (presence of investigator affects participant responses)
43
Subject effects (internal validity)
Participants change their behaviors or attitudes based on their understanding of their role as participants. Participants will pick up cues (demand characteristics) from the researcher, or research setting, that motivate them
44
External validity (2 types)
Ability to generalize the results of a study to a larger group
- Population external validity: involves the population to which one can generalize
- Ecological external validity: involves the conditions or settings to which one can generalize
45
Novelty effect (external validity)
New treatment produces positive results just because it is new to participants
46
Experimenter effect (external validity)
halo and hawthorne
47
History by treatment effect (external validity)
experiment is conducted in a particular time period replete with contextual factors that can't be duplicated easily in another setting
48
Measurement of the dependent variable (external validity)
Effectiveness of a program may depend on the type of measurement used in the study
49
Time of measurement by treatment effect (external validity)
Timing of the administration of a posttest may influence results
50
Quantitative Research
Captures relationship between 2 things that can be measured numerically
51
Qualitative Research
Attempts to answer questions about HOW a behavior or phenomenon occurs
52
Mixed Method Research (2 types)
Blends or mixes qualities from quant and qual research
- Concurrent: quant and qual are collected at same time (aka triangulation)
- Sequential design: either quant or qual data is collected first. Exploratory: qual first. Explanatory: quan first
53
Single-Subject research designs (SSRD)
Measure how either receiving treatment or not receiving treatment affects a single subject or group of subjects
54
Descriptive Research
Used to describe a phenomenon and does not involve intervention (treatment). What is and how often something occurs. Usually done as precursor or in conjunction with other research methods
55
Longitudinal Research
Data collection for a particular group over time
56
Cross-sectional research
Examines different groups or cohorts at a particular point in time, with differences in experience being compared
57
Survey Research
Method of collecting quant and qual data
58
Action research
Research that is typically carried out by professional counselors in an effort to improve their own practice/efficiency
59
Pilot study
Smaller than full scale study and is designed to assess feasibility of expanding a study to a much larger scale
60
Nonexperimental Research designs
Quant. Exploratory and descriptive. No intervention involved, no variables or conditions manipulated. Goal is to observe and outline properties of variable.
61
Experimental research designs
Involves intervention (counselor manipulates conditions and variables). Assess cause and effect relationships
62
Descriptive design (nonexperimental research design)
Thoroughly describing a variable at one time (simple descriptive) or over time (longitudinal design).
63
Longitudinal design (trend, cohort, panel)
Trend=assess general population over time, with new people each time data collected. Cohort=assess the same population over time, panel = same individuals over time
64
Comparative design
Investigates group differences for a particular variable.
65
Correlational research design
Describes relationship between two variables
66
Ex post facto research design
(AKA causal-comparative design) involves examining how an IV affects a DV by assessing whether one or more pre-existing conditions possibly caused differences in groups
67
Within-subject design
assess changes that occur within the participants in a group as they experience some intervention
68
Between-group design
Exploring the effects of a treatment or intervention between 2 groups or among more than 2 groups
69
Split-pot design
ASsessing a general intervention on the whole plot and assessing other treatments to subplots within the whole plot
70
Pre-experimental designs
Do not use random assignment. Thus not considered true experimental designs
71
One-group posttest-only design (Pre-experimental designs)
Group receives intervention and change is measured
72
One-group pretest-posttest design (Pre-experimental designs)
Group evaluated before and after intervention
73
Nonequivalent groups posttest-only design (Pre-experimental designs)
No attempt made to begin the study with equivalent groups of participants.
74
True experimental designs (5 types)
(AKA randomized experimental designs) are gold standard for experimental designs: at least 2 groups for comparison and random assignment
75
Randomized pretest-posttest control group design (true experimental)
participants assigned to 2 groups (1 is control) and both are measured before and after intervention
76
Randomized pretest-posttest comparison group design (true experimental)
participants assigned to at least 2 groups, and each group receives a distinct intervention.
77
Randomized posttest-only control group designs (true experimental)
random assignment of participants to treatment or control, and then administer an intervention to one group, then measure outcome
78
Randomized posttest-only comparison group design (true experimental)
with at least 2 groups for comparison and no control group
79
Solomon four-group design (true experimental)
comprehensive true experimental design. using 4 randomly assigned groups, presence of pretest and intervention can be assessed more rigorously
80
Quasi-experimental designs
Useful when it's impossible/inappropriate to randomly assign participants to groups
81
Time series design (quasi-exp)
Repeatedly measuring before/after intervention for one group or control group for comparison
82
Single-Subject Research Designs (SSRDs)
Allow for repeated measures of a target behavior over time for an individual or select group of individuals
83
Descriptive statistics
Organize and summarize data - describe data
84
Frequency distribution
Tabulation of the number of observations (or number of participants) per distinct response for a particular variable
85
Frequency polygon
Line graph of the frequency distribution
86
Histogram
Graph of connecting bars that shows the frequency of scores for a variable
87
Bar graph
Looks similar to histogram but displays nominal data
88
Central tendency
What is the typical score
89
Mean
average
90
Median
Middlemost score. Better to use if there are outliers
91
Mode
Most frequently occuring score (bimodal, multimodal)
92
Variability
How dispersed are scores from a measure of central tendency?
93
Range
Range of data. can be affected by outliers.
94
Interquartile range
eliminates top and bottom quartiles and provides range around median score
95
Standard deviation
most frequently reported indicator of variability for interval or ratio data
96
Variance
Standard deviation squared
97
Skewness
Asymmetrical distribution with data points that don't cluster symmetrically around a mean
98
Kurtosis (3 types)
Meso (normal curve), letpo (tall and thin), platy (flat and wide)
99
Inferential statistics
Used to try and describe results beyond what is garnered from the data alone
100
Degrees of freedom
Number of scores, or categories of a variable that are "free to vary". It's a restrictor
101
Correlation coefficient
Information about relationship between two variables. Is there a relationship? What direction? Is it strong?
102
Spurious correlation
when a correlation overrepresents or underrepresents the actual relationship
103
Regression studies (3 types)
Prediction studies are extensions of correlational studies and known as regression studies
- Bivariate regression: how well scores from independent variable predict scores on dependent variable
- Multiple regression: Involves more than one predictor variable
- Logistic regression: dependent variable is dichotomous
104
T-test (2 types)
Compares 2 means for one variable
- Independent t test: involve comparing 2 independent groups on one dependent variable
- Dependent t test: involve similar groups paired or matched in some meaningful way, or the same group tested twice
105
Analysis of variance (ANOVA)
Involves having at least one independent variable in a study with 3+ groups or levels
106
Post hoc analysis
Allows one to examine every possible pairing of groups for a particular independent variable after one has concluded there are main effects (significant difference among 2 or more groups making up a single independent variable)
107
Analysis of covariance (ANCOVA)
Test includes independent variable as a covariate, or a variable that needs to be statistically adjusted and controlled in order to look at the relationship of the other independent variables and the DV
108
Multiple analysis of variance (MANOVA)
Similar to anova but involves multiple DVs
109
Multiple analysis of covariance (MANCOVA)
Similar to ancova but involves multiple DVs
110
Nonparametric Stats
Used when professional counselors are only able to make a few assumptions about the distribution of scores in the underlying population
111
Chi-square test
Used with 2 or more categorical or nominal variables, where each variable contains at least two categoires.
112
Factor analysis
The purpose is to reduce a large number of variables to a smaller number of factors
113
Meta analysis
allows researcher to combine and synthesize the results of numerous similar studies for a particular outcome or DVs
114
Phenomenology
Used to discover meaning of participants' lived experiences with goal of understanding person
115
Grounded theory
Generate theory that is grounded in data from participants' perspectives for a particular phenomenon
116
Consensual qualitative research (CQR)
approach that combines elements of phenomenology and grounded theory
117
Ethnography
Researcher describes and provides interpretations about the culture of a group or system. Observation
118
Purposive sampling
Obtain information-rich cases that allow for maximum depth and detail regarding a certain phenomenon
119
Saturation
Seek sample sizes that reach point of redundancy of information. Sample until no new data comes in.
120
Convenience sampling
Sampling based on availability or accessibility. Least desirable method. Least trustworthy
121
Maximum variation sampling
Sampling a diverse group and searching for core patterns and individual perspectives based on unique participant characteristics
122
Homogenous sampling
Selecting participants for a specific subgroup with theoretically similar experiences
123
Stratified purposeful sampling
Identify important variables pertaining to a research question and sampling subgroups that best isolate each variable
124
Purposeful random sampling
identify a sample and randomly select participants from that sample
125
Comprehensive sampling
Sampling all individuals within a system
126
Typical case sampling
selecting the "average" participants or those who represent the typical experience for a phenomenon
127
Intensity sampling
identify those with intense but not extreme experiences of a phenomenon
128
Critical case sampling
Sampling those with intense and irregular experiences
129
Extreme or deviant sampling
Looking for the bounds of difference, or those with the most positive and negative experiences
130
Snowball, chain, or network sampling
A pool of participants is derived through obtaining recommendations from earlier participants
131
Criterion sampling
develop criteria and selecting all cases that meet criteria
132
Opportunistic or emergent sampling
Taking advantage of an unexpected opportunity and changing one's research design to include a particular individual in one's pool
133
Theoretical sampling
as theory evolves, sampling those who best contribute information for one's theory
134
Confirming/disconfirming case sampling
Including cases that confirm and add depth to one's evolving theory can also provide exceptions or potentially disconfirm elements of one's theory
135
Qualitative interviewing (3 types)
- Unstructured: no predetermined question
- Semi-Structured: preset interview protocol with flexibility to change/add/remove questions
- Structured: standardized interview protocol and ensures same level of info is collected from each individual
136
Participant observation
Counselors engaging in observations
137
Unobstrusive methods
Collecting photos, videos, documents, archival data, artifacts
138
Contact summary sheet
Single page snapshot of specific contact
139
Document summary form
Attached primarily to unobstrusive data sources (like newspapers or artifacts)
140
Data display
presents organized data in a table format or a figure containing interconnected nodes
141
Inductive analysis
Involves searching for keywords and potential traditions that involves searching for keywords and potential themes from the data without significant preconceived notions of what theory or theories fit the data
142
Trustworthiness
The validity or truthfulness of findings - why others should trust your data collection
143
Credibility (trustworthiness)
Believability of your findings
144
Transferability (trustworthiness)
degree to which data transfers to other contexts and participants
145
Dependability (trustworthiness)
degree of consistency of results over time and across researchers
146
Confirmability (trustworthiness)
reflects that interpretation of data is a genuine reflection of participants' views
147
Needs assessment (program evaluation)
Initial step to explore whether a program is needed for a particular group of individuals
148
Process evaluation (program evaluation)
assessment of an ongoing program to ensure that program activities match plans
149
Outcome evaluation(program evaluation)
Determination of program success by investigating how participants are performing with respect to themselves as well as others who are not involved in the program
150
Efficiency analysis (program evaluation)
comparison of the costs of the program in relation to need, processes, and outcomes.
151
Accountability
Process of providing feedback about a program to its stakeholders
152
Stakeholders
Any individuals involved in or affected by the program
153
Formative evaluations
Ongoing evaluation of a program throughout its implementation to ensure that it is being conducted as planned and acted upon as needed
154
Summative evaluation
involves assessment of the entire program to determine the degree to which program goals and objectives have been met
155
Steps in program evaluation
1. Identify the program to be evaluated
2. plan the evaluation
3. conduct a needs assessment and provide recommendations
4. determine what success is
5. Select data sources
6. Monitor and evaluate the program progress
7. Determine the degree to which a program is successful
8. analyze the program's efficiency
9. continue, revise, or stop program based on findings
156
ABCD model
Audience, Behavior, Conditions, Description
