Research Design, Statistics, & Test Construction Flashcards
1
Q
Quasi-experimental design
A
at least one IV is manipulated, but there is no random-assignment of participants (typically because already in pre-existing groups)
2
Q
Within-subjects design
A
groups compared are correlated or related; three conditions lead to this: repeated measures of same participants, subjects matched prior to assignment to groups, subjects have an inherent relationship (e.g., twins)
3
Q
Latin square
A
most sophisticated form of counterbalancing subjects in a repeated measures design
4
Q
Mixed design
A
includes groups that are both independent and correlated (e.g., patients randomly assigned to two different treatment groups and measured before and after treatment)
5
Q
Idiographic
A
refers to single subject approaches (single or few participants studied intensely); AB, ABAB, multiple baseline, simultaneous treatment, and the changing criterion
6
Q
Nomothetic
A
group approaches to research design (as opposed to single subject)
7
Q
Autocorrelation
A
effect of measuring same person repeatedly; results in highly correlated data; problem of single subject design
8
Q
AB design
A
baseline condition (A) followed by treatment condition (B); most significant problem is threat of history (difficult to determine whether intervention or other event caused change)
9
Q
ABAB design
A
baseline (A) and treatment (B) alternated in ABAB sequence; protects against threat of history; two potential problems: failure of behavior to return to baseline, issues of ethics with removing effective treatment
10
Q
Multiple baseline design
A
treatment is applied sequentially or consecutively across subjects, situations, or behaviors
11
Q
Simultaneous (alternating) treatment design
A
two or more interventions implemented concurrently during the treatment phase that are balanced and varied across time of day
12
Q
Changing criterion design
A
attempt is made to change behavior in increments to match a changing criterion (e.g., slowly reducing number of cups of coffee)
13
Q
Momentary time sampling
A
simply recording whether target behavior is present or absent at moment that time interval ends
14
Q
Whole-interval sampling
A
scoring target behavior positively only if exhibited for full duration of time interval
15
Q
Analogue research
A
evaluates treatment under conditions that only resemble or approximate clinical situations; typically for less severe conditions; tight experimental control but limited generalizability (e.g., grad student clinicians using manual)
16
Q
Clinical trials
A
outcome investigations conducted in clinical settings; often involve methodological compromises and sacrifices
17
Q
Cross-sequential research
A
also called cohort-sequential research; takes several cross sections and follows them over briefer periods of time
18
Q
Stratified random sampling
A
population is first divided into strata (e.g., age levels, income levels, ethnic groups), and then a random sample of equal size from each stratum is selected
19
Q
Proportional sampling
A
individuals are randomly selected in proportion to their representation in the general population
20
Q
Systematic sampling
A
selecting every kth element after a random start, e.g., if 100 out of 1000 persons are needed, every tenth person is selected; needs to be arranged in such a way that it is not biased
21
Q
Cluster sampling
A
identifying naturally occurring groups of subjects (clusters) and randomly selecting certain clusters (e.g., classes or departments at a university, or schools within a particular school district)
22
Q
History
A
threat to internal validity; incidents that intervene between measuring points, either in or outside of the experimental situation; best control is a control group
23
Q
Maturation
A
threat to internal validity; factors that affect the subjects’ performance because of the passing of time (fatigue, maturing); best control is a control group
24
Q
Testing or test practice
A
threat to internal validity; occurs when familiarity with testing affects scores on repeated testing; best control is Solomon Four-Group design
25
Solomon Four-Group design
control for testing threats to validity; divide subjects into four groups: measured pre- and post- and get intervention; measured pre- and post- and don’t get intervention, measured post and gets intervention, measured post and does not get intervention
26
Instrumentation
threat to internal validity; changes in observers or the calibration of equipment; control group corrects for this
27
Statistical regression
threat to internal validity; tendency for extreme scores (scores very much above or below the mean ) to become less extreme (closer to the mean) on retesting, even without any type of intervention; control group controls for this
28
Selection bias
threat to internal validity; caused by non-random assignment; best avoided with random sampling
29
Attrition or experimental mortality
threat to internal validity; differential loss of subjects from the groups; to assess for this, compare subjects who drop out using t-tests on relevant variables
30
Diffusion
threat to internal validity; occurs when no treatment group gets some of the treatment; difficult to eliminate completely, but tighter control over experimental situation can help
31
Construct validity
refers to factors other than the desired specifics of our intervention that result in differences; often lumped under threats to external validity; not measuring what you think you are measuring
32
Attention and contact with clients
threat to construct validity; difficult to tell whether changes are due to treatment or attention
33
Experimenter expectancies
threat to construct validity; cues or clues transmitted to the subjects by the experimenter; Rosenthal effect; can be controlled by masking experimenter to conditions
34
Rosenthal effect
refers to experimenter expectancies
35
Demand characteristics
threat to construct validity; factors in the procedures that suggest how the subject should behave; control by masking subjects to their condition
36
John Henry effect
hreat to construct validity; occurs when persons in a control group try harder than usual in the spirit of competition with the experimental group; control by making sure experimental and control groups do not know about each other and, if not possible, do not give groups any sense of competition
37
Threats to external validity
interfere with generalizability of effects
38
Sample characteristics
threat to external validity; difference between sample and population
39
Stimulus characteristics
threat to external validity; features of the study with which the intervention is associated (e.g., research assessing memory functioning in the laboratory may not be generalizable to memory functioning in naturalistic settings)
40
Contextual characteristics
threat to external validity; conditions in which intervention is embedded; e.g., reactivity
41
Reactivity
subjects behave in a certain way just because they are participating in research and being observed
42
Low power
threat to statistical conclusion validity; diminished ability to find significant results; small sample size and inadequate interventions can contribute
43
Unreliability of measures
threat to statistical conclusion validity; unreliable outcome measure
44
Variability in procedures
threat to statistical conclusion validity; inconsistency in treatment procedures; especially of concern in psychotherapy outcome research
45
Subject heterogeneity
threat to statistical conclusion validity; subject heterogeneity makes it more difficult to find significant differences between groups
46
Varies directly with
as one variable increases so does the other (e.g., a varies directly with b in a=b/c
47
Varies indirectly with
as one variable increases the other decreases (e.g., a varies indirectly with c in a = b/c)
48
Ordinal data
involve tallying people to see which ordered category a person falls into (e.g., likert scale, SES, percentile rank); group means cannot be calculated
49
Interval data
involve obtaining numerical scores for each person, where the score values have equal intervals; no zero score or zero is absolute (e.g., IQ test, t-score, temperature); group means can be calculated
50
Ratio data
involve obtaining numerical scores for each person, where the score values have equal intervals and an absolute zero (e.g., score on EPPP, money in bank, weight, number of children)
51
Standard deviation
average deviation (or spread) from the mean in a given set of scores
52
Variance
standard deviation squared
53
Positive skew
higher proportion of scores in the lower range of values (mode has lowest value, mean highest)
54
Negative skew
higher proportion of scores in the higher ranges of values (mean has lowest value, mode highest)
55
Kurtosis
refers to how peaked a distribution is
56
Leptokurtotic
distribution with a very sharp peak
57
Platykurtotic
distribution that is very flat
58
Criterion-referenced or domain-referenced score
example is percentage correct
59
Norm-referenced score
provides information on how person performed relative to group
60
Standard scores
based on standard deviation from the sample
61
Z-scores
standard scores that correspond directly to standard deviation units; transforming into Z-scores does not normalize a distribution (exact same distribution shape); z score = (score - mean)/SD
62
Z-scores and percentile ranks
-3 = .1, -2 = 2.5, -1 = 16, 0 = 50, 1 = 84, 2 = 97.5, 3 = 99.5
63
Parameters
population values
64
Statistics
sample values
65
Standard error of the mean
average amount of deviation of sample means from the population mean; equal to population SD divided by square root of sample size
66
Central limit theorem
states that assuming an infinite number of equal sized samples (of large enough size) are drawn from the population, and the means of these samples are plotted, a normally distributed distribution of means will result; the mean of the means ( the grand mean) will equal the population mean, and the standard deviation of the means will equal the standard deviation of the population divided by the square root of sample size (the standard error of the mean); allows researcher to calculate whether the obtained mean is most likely due to treatment or experimental effects, or to chance
67
Rejection region
also called region of unlikely values; unlikely researcher will obtain values by chance; size corresponds to alpha level
68
Two factors that contribute to conclusions about statistical significance
treatment effects and chance (sampling error)
69
Type I error
incorrectly reject null hypothesis; likelihood directly corresponds to size of alpha
70
Type II error
incorrectly accept null hypothesis; corresponds to beta
71
Beta
provides probability of making Type II error
72
Power
ability to correctly reject null hypothesis; increased when sample size is large, magnitude of intervention is large, random error is small, statistical test is parametric, test is one-tailed; inversely related to beta (power = 1- beta); direct relationship with alpha
73
Parametric test
three assumptions must be met: data are interval or ratio, homoscedasticity, normally distributed
74
Nonparametric test
used for nominal or ordinal DV
75
Statistic for testing differences, more than one DV
MANOVA
76
Statistics for testing differences, interval or ratio DV
t-test, ANOVA
77
Statistics for testing differences, nominal or ordinal DV
Chi-Square, Mann-Whitney, Wilcoxin
78
Homoscedasticity
similar variability or standard deviations in the different groups
79
Assumption for Chi-Square test
independence of observations
80
Degrees of freedom
number of possible variations in outcomes that can be obtained
81
Degrees of freedom for single sample chi-square
df = # of columns - 1
82
Degrees of freedom for multiple sample chi-square
df = (# rows - 1)(# columns - 1)
83
Degrees of freedom for single sample t-test
df = N - 1
84
Degrees of freedom for matched or correlated samples t-test
df = # of pairs - 1
85
Degrees of freedom for independent samples t-test
df = N - 2
86
Degrees of freedom total for ANOVA
df = N - 1
87
Degrees of freedom within for ANOVA
df within = df total = df between
88
Degrees of freedom between for ANOVA
df between = # of groups - 1
89
Expected frequency for Chi-Square when data are given in each cell
expected frequency for any cell = (sum of row * sum of column)/N
90
F ratio
F ratio = Mean Square between groups/Mean square within groups; typically significant as it gets above 2.0
91
Mean Square
measure of average variability
92
ANOVA post-hoc tests in order of most to least protection against Type I error
in order of most to least conservative: Scheffe, Tukey, Duncan/Dunette/Neuman-Kuels, Fisher’s LSD
93
Two-way ANOVA
when groups are being compared on two IVs; permits analysis of main effects and interaction effects; when interaction is significant, main effects must be interpreted in context of interaction effect
94
Examine whether interaction effects in ANOVA table
add up diagonals in each individual 2x2 set of squares
95
MANOVA
used when there are multiple DVs
96
Coefficient of determination
calculated by squaring correlation coefficient; represents amount of variability in Y that is shared with or explained by X
97
Assumptions of bivariate correlations
linear relationship between X and Y, homoscedasticity, unrestricted range of scores on X and Y
98
Bivariate correlation coefficient for two interval/ratio variables
Pearson r
99
Bivariate correlation coefficient for two ordinal variables
Spearman’s rho, Kendall’s Tao
100
Bivariate correlation coefficient for interval/ratio and true dichotomy
point-biserial
101
Bivariate correlation coefficient for interval/ratio and artificial dichotomy
biserial
102
Bivariate correlation coefficient for two true dichotomie
Phi
103
Bivariate correlation coefficient for two artificial dichotomies
tetrachoric
104
Coefficient for curvilinear relationship between X and Y
Eta
105
Zero-order correlation
examines relationship between X and Y when it is believed there are no extraneous variables affecting the relationship
106
Partial correlation
also called first-order correlation; examines the relationship between the predictor and the criterion with the effect of a third variable removed that is thought to be affecting both variables
107
Part correlation
also called a semi-partial correlation; examines the relationship between the predictor and the criterion with the influence of a third variable removed from only one of the original variables
108
Multivariate tests
involve several predictors and one or more criterions (DVs)
109
Multiplier R
multiple correlation; correlation between two or more IVs and one DV, where Y is always interval or ratio data, and at least one X is interval or ratio data
110
Coefficient of multiple determination
obtained by squaring multiple R; index of the amount of variability in the criterion (Y) that is accounted for by the combination of all the predictors (Xs)
111
Multiple regression
Has multiple predictors
112
Multicollinearity
problem that occurs in a multiple regression equation when the predictors are highly correlated with one another, and therefore essentially redundant
113
Stepwise regression
computer-generated; in forward regression, the computer adds predictor variables one at a time, starting with the predictor that has the highest correlation with criterion outcome; in backward regression, predictor variables are removed one at a time, starting with the variable that contributes the least to criterion outcome; allows for fewest possible predictors
114
Hierarchical regression
researcher controls regression analysis, adding variables in order consistent with theory
115
Canonical R
correlation between two or more IVs and two or more DVs; evaluate relationship between two sets of variables
116
Discriminant function analysis
special case of multiple regression; two or more predictors and one criterion that is nominal (rather than interval or ratio); allows to predict membership in group
117
Loglinear analysis
sometimes referred to as logit analysis; used to predict a categorical criterion based on categorical predictors
118
Approaches for causal modeling
not correlations and regressions; path analysis and SEM
119
Path analysis
applies multiple regression techniques to testing a model that specifies causal links among variables; relies on researcher having developed theoretically-based causal model; straight arrows denote causal relationships, curved denote correlations; path coefficients are analyzed to see if the pattern predicted by the model has emerged
120
Factor analysis
test of structure that extracts as many significant factors from set of data as possible
121
Characteristic root
another name for eigenvalues for factors (indicate strength of factors); less than 1.0 usually not interpreted
122
Factor loadings interpreted
equal to or exceed 0.30
123
Orthogonal rotation
factor rotation in which axes remain perpendicular; results in factors with no correlation
124
Communality
much of a test’s variability is explained by the combination of all the factors; can be calculated in orthogonal rotation; factor loadings squared and added together
125
Oblique rotation
factor rotation in which angle between axes is non-perpendicular and factors are correlated
126
Principal components analysis
subtype of factor analysis; trying to extract factors and there is no empirical or theoretical guidance on the values of the communalities; results in a few uncorrelated factors called components; no prior hypotheses
127
(Principle) factor analysis
communality values ascertained before analysis
128
Classical test theory
also called true score model; total variability = true score variability + error variability
129
Reliability
proportion of true score variability; often symbolized as rxx or rtt; minimum acceptable is 0.80
130
Content sampling error
occurs when a test, by chance, has items that do or do not tap into a test-taker’s knowledge base
131
Time sampling error
occurs when a test is given at different points in time and scores differ because of factors related to passage of time
132
Test heterogeneity error
occurs when a test has heterogeneous items tapping more than one domain
133
Factors affecting reliability
number of items, homogeneity of items, range of scores, ability to guess
134
Four estimates of reliability
test-retest reliability, parallel forms reliability, internal consistency reliability, interrater reliability
135
Coefficient of stability
expression of test-retest reliability
136
Coefficient of equivalence
expression of parallel forms reliability
137
Spearman-Brown prophecy formula
used when calculating split-half reliability; tells us how much more reliable the test would be if it were longer
138
Split-half reliability and speeded tests
split-half reliability inappropriate for speeded tests because only easy items included; preferred test of reliability is alternate forms
139
Power tests
have items that are of varying difficulty level, and subjects are provided sufficient time to complete them all
140
Kuder-Richardson (KR-20 and KR-21) and Cronbach's coefficient alpha
sophisticated forms of internal consistency and reliability; involve analysis of the correlation of each item with every other item in the test; calculated by taking the mean of the correlation coefficients for every possible split-half; KR-20 (vary in difficulty) and KR-21 (consistent difficulty) when items scored dichotomously, Coefficient alpha when not scored dichotomously
141
Standard error of measurement
standard deviation of a theoretically normal distribution of test scores obtained by one individual on equivalent tests; assumed to be consistent across all persons; Smean = SDx * square root (1 - rxx); ranges from 0 (perfectly reliable test) to the standard deviation of the test (not at all reliable test)
142
Content validity
how adequately a test samples a particular content area; quantified by asking a panel of experts if each item is essential, useful/not essential, or not necessary, yet no numerical validity coefficient is derived
143
Criterion-related validity
how adequately a test score can be used to infer, predict, or estimate criterion outcome; calculated by using a Pearson r to correlate the test scores (also known as predictor scores) with criterion scores (also known as outcome scores)
144
Concurrent validity
subtype of criterion-related validity; predictor and criterion measured and correlated at about the same time
145
Predictive validity
subtype of criterion-related validity; delay between the measurement of the predictor and the criterion
146
Standard error of the estimate
average amount of error in estimating each person’s criterion score; standard deviation of a theoretically normal distribution of criterion scores obtained by one person measured repeatedly; Sest = SDy * square root (1-rxy2); ranges from 0 to value of standard deviation of criterion
147
Applications of criterion-related validity coefficient for prediction
expectancy tables, Taylor-Russell tables, decision-making theory
148
Expectancy tables
list the probability that a person’s criterion score will fall in a specified range based on the range in which that person’s predictor score fell
149
Taylor-Russel tables
numerically describe the amount of improvement that occurs in selection decisions when a predictor test is introduced
150
Selection ratio
proportion of available openings to number of applicants
151
Incremental validity
amount of improvement in success rate that results from using predictor test (e.g., if proportion of successful improves from base rate of .4 to .65, incremental validity is .25 or 25%)
152
Three variables that affect incremental validity
criterion-related validity coefficient of the predictor test, the company’s base rate, and the selection ratio
153
Decision-making theory
takes the predictions of performance that were based on the predictor tests and compares them with the actual criterion outcome
154
Item difficulty setting formula
(1.0 + probability of getting item by chance)/2.0
155
Item validity
correlation between item score and criterion score
156
Item-characteristic curve
plot of the relationship between item performance and total score
157
Item response theory
used to calculate to what extent a specific item on a test correlates with an underlying construct; subject’s performance on a test item as representing the degree to which the subject has a latent trait
158
Factors affecting criterion-related validity
range of scores, reliability of predictor, reliability of the predictor and the criterion, criterion contamination
159
Relationship of reliability of predictor and criterion-related validity
test must have some reliability to be valid, but a reliable test does not imply a valid test; validity can be greater than reliability; reliability determines ceiling for validity but is not always greater than validity
160
Correction for attenuation
calculates how much higher validity would be if predictor and criterion were both perfectly reliable
161
Criterion contamination
occurs with subjectively-scored criterion outcomes when the rater is informed of subjects’ predictor scores before assigning them criterion ratings
162
Ways evidence of construct validity can be obtained
factor analysis or multi-trait, multi-method matrix
163
Multi-trait, multi-method matrix
table with information about convergent and divergent validity, both of which are necessary for construct validity
