Test Construction

Question 1

Item Characteristic Curve

Answer 1

A graphical representation of test item’s difficulty, discrimination, and chance of false positive. Difficulty (degree of attribute needed to pass item): indicated by position of curve on the X axis. Discrimination (ability to differentiate between high and low scorers): indicated by slope of the curve. Chance of false positives (probability of getting answer correct by guessing): indicated by the Y-intercept of the curve

Question 2

Criterion-Related Validity Coeffecient

Answer 2

A value that indicates strength of a correlation between test scores and performance on a chosen construct.

Question 3

Test Characteristic Curve

Answer 3

A graphical representation of the expected number of test items a participant answers correctly versus the constructs measured by the test

Question 4

Item difficulty

Answer 4

AKA item difficulty index or ‘p’. Defined as the percentage of examinees that answer the item correctly (how much of the attribute and individual must possess to pass the item).

Question 5

What are the item difficulty (p) ranges?

Answer 5

0 and 1. 0 menas that no one passed the item (too hard) and 1 means that everyone passed (too easy). Average item difficulty should be 0.5

Question 6

With item difficulty, what are the floor and ceiling effects?

Answer 6

Floor effects refers to a test’s ability to distinguish people at the low end of a distribution, while ceiling effects refers to a test’s ability to distinguish people at the high end of a distribution.

Question 7

What is item discrimination?

Answer 7

The ability of the item to unambiguously separate out those who fail from those who pass. Can be visually represented with discrimination as the slope of the curve. Steeper slopes indicate more discrimination.

Question 8

How is item discrimination assessed?

Answer 8

Index D (item discrimination index): difference between the proportion of low-scoreers who answered the item correctly and high-scorers who answered the item correctly.

Question 9

What are the D ranges?

Answer 9

1 to -1; it is desirable to have positive values of D, which would indicate that more high-scoring examinees (rather than low-scoring examinees) answered the item correctly

Question 10

Ratio measure

Answer 10

A level of measurement describing a variable with attributes that have all the qualities of nominal, ordinal, and interval measures as well as a true zero point; measurement of physical objects is an example of ratio measure.

Question 11

Interval measure

Answer 11

A level of measurement describing a variable whose attributes are rank-ordered and have equal distances between adjacent attributes with no true zero point; the Farenheit temp scale s an example of this, because the distance between 17 an 18 is the same as the distance between 89 and 90

Question 12

Nominal scale

Answer 12

A variable whose attributes are simply representations for groups and have no ranked relationship; gender would be example of a nominal scale of measurement because male does not imply more gender than female.

Question 13

Item Response Theory

Answer 13

IRT focuese on determining specific parameters of test items. Makes use of characteristic curves, which provide info about item difficulty, item discrimination, and the probability of false positives.

Question 14

Assumptions of IRT

Answer 14

Single underlying trait, relationship between trait and item response can be displayed in item characteristic curve, and requires large sample size.

Question 15

Computer Adaptive Assessment

Answer 15

Uses IRT; customizes test to the examinee’s ability level.

Question 16

Classical Test Theory

Answer 16

CTT; AKA Classical Measurement Theory, is an approach to testing that assumes that individual items are as good a measure of a latent trait as other items; thus, CTT focuses on the reliability of a set of items. in CTT, item and test parameters are sample’dependent

Question 17

Kappa Coefficient

Answer 17

Measured the degree to which judges agree. Measure of inter-rater reliability. Increases when raters are well-trained and aware of being observed. Applicable only with nominal, ordinal, or discontinuous data.

Question 18

Ranges of Kappa Coefficient

Answer 18

-1 to +1; .80 - .90 indicates good agreement

Question 19

Convergent Validity

Answer 19

Indicates the degree of correlation between two instruments that are intended to measure the same thing

Question 20

Metric Data

Answer 20

A term used to refer to interval/ratio data

Question 20

Continuous Data

Answer 20

A term used to refer to interval/ratio data

Question 21

Internal Consistency

Answer 21

A measure indicating the extent to which items within and instrument are correlated to each other; internal consistency indicates the extent to which the given items measure the same construct

Question 22

Kuder-Richardson Formula 20

Answer 22

A method of evaluating internal consistency reliability; used when test items are dichotomously scored; used when test items vary in difficulty; indicates the degree to which test items are homogenous; falsely elevates internal consistency when used with timed tests

Question 23

Single-Subjects Designs

Answer 23

One or more participants and are focuses on assessing variables within and individual rather than between individuals. They are ideographic (differences within a participant) rather than nomothetic (differences between participants)

Question 24

2 types of single-subject designs

Answer 24

Case study (describes an individual by using tests or naturalistic observation) or experimental (determine how the introduction of a factor affects behavior)

Question 25

Problems with single-subject designs

Answer 25

Autocorrelation (when measured on the same variable multiple times, the variable becomes correlated with itself); Time-intensive (multiple assessments or intense observations are time-consuming); Generalizability (may not generalize); Practice effects (scores may increase from practice)

Question 26

Nomothetic

Answer 26

An approach to personality that focuses on groups of individuals and tries to find the commonalities between individuals.

Question 27

Multicollinearity

Answer 27

Very high multiple correlations among some of all predictors in an equation.

Question 28

Quantitative Research

Answer 28

Systematic empirical exploration or relationships; deductive, rather than inductive. Involves the collection and statistical analysis of quantitative data, whose results can often be generalized.

Question 29

Reliability

Answer 29

Refers to the consistence or repeatability of data; pertains to quantitative research

Question 30

ANOVA

Answer 30

Test for differences in the mean scores of groups based on one or more variables. DV must be continuous and IV must be categorical. Tests the null hypothesis that the means of the group are equal.

Question 31

ANOVA assumptions

Answer 31

Independence of observations (each participant in only one cell); Normality (distribution of scores cluster around the mean with fewer observations fallen farther from the mean; AKA bell-chaped curve); and Homogeneity of Variance (variance of every group is same as variance of every other group, AKA homoscedasticity.

Question 32

2 types of ANOVA

Answer 32

One-Way ANOVA (test the main effect of one IV); or Two-Way ANOVA (tests main effects of first IV (A), second IV (B), and the interaction of the two (A*B))

Question 33

Interaction effect (ANOVA)

Answer 33

The effect of one IV on the DV differs depending on the level of the other IV

Question 34

What are F-ratios?

Answer 34

Ratios of effect variance to error variance. In One-Way ANOVA, there is one F-ratio of the effect of the IV. In a Two-Way ANOVA, there are three F-ratios (main effect A, B, and interaction effect)

Question 35

Advantages of Two-Way Anova over One-Way Anova:

Answer 35

Includes interaction effects; increases power; reduces familywise error rate.

Question 36

Heterogeneity

Answer 36

The violation of the assumption of homogeneity, such that the variances of the groups are not equal. ANOVA is robust to such a violation, if there are no outliers, sample sizes are large and fairly equal, sample sizes within levels are relatively equal, and the hypothesis is two-tailed.

Question 37

Chi-Square Test

Answer 37

Statistical method of testing for an association between categorical variables; specifically, it tests for the equality of expected and observed frequencies or proportions.

Question 38

MANOVA

Answer 38

An extension of ANOVA methods to cover cases where there is more than one DV and where the DVs cannot simply be combined. The MANOVA combines the DVs in such a way as to maximize differences between groups. In addition to identifying whether changes in the IV have a significant effect on the DV, the technique seeks to identify the interactions among the IVs and the DVs, if any.

Question 39

ANCOVA

Answer 39

A general linear model with one continuous DV and one or more IVs, plus a covariate. ANCOVA is a merger of ANOVA and regression for continuous variables. ANCOVA test where IVs have an effect after removing the variance for which one of more covariates account; the inclusion of covariates can increase statistical power because it accounts for some of the variability.

Question 40

Dichotomous/Continuous Variables

Answer 40

Continuous variables assume an intermediate value between two other values and there can be an infinite amount of possible values between those two values. Dichotomous variables have only two values (yes or no)

Question 41

Point-biserial correlation

Answer 41

Examines the relationship between a dichotomous variable and a continuous variable. Can only be used with TRUE dichotomous variables.

Question 42

Biserial correlation coefficients

Answer 42

Examine the relationship between an artificially-created (made form a continuous variable) dichotomous variable and a continuous variable

Question 43

Spearman's rho

Answer 43

This correlation coefficient is used when measuring the relationship between two ranked variables giving a rank-order correlation.

Question 44

Pearson's r

Answer 44

This correlation coefficient is used when measuring the relationship between two continuous variables

Question 45

Eigenvalue

Answer 45

Measures the amount of variance in a set of tests or items that can be accounted for by an underlying factor. Used in factor analysis and principal components analysis. Often converted into percentages to determine percentage of variance in a set of test items accounted for by an underlying factor. Factor analysis will provide same # of eigenvalues as there are items or tests.

Question 46

What do large eigenvalues indicate?

Answer 46

An underlying factor is explaining a large amount of variance in a set of items or tests.

Question 47

Inferential statistics

Answer 47

Deal with formulating conclusions and making inferences from collected data

Question 48

Multiple regression

Answer 48

A manner of regression analysis in which one or more predictor variables are used to predict a single criterion variable.

Question 49

Factor Analysis

Answer 49

A statistical technique that identifies underlying patterns in a data set.

Question 50

Goals of factor analysis

Answer 50

Identify underlying factors that are responsible for variation in a set of items, variables, or tests; Reduce a large set of variables to a smaller number of underlying factors.

Question 51

What are factor loadings?

Answer 51

Produced by factor analysis (along with eigenvalues) and provide a measure of the correlation between an item and an underlying construct. Higher factor loadings indicate that the underlying factor is accounting for a large amount of variance in the item.

Question 52

When is factor rotation used?

Answer 52

To aid in interpretation of factor loadings from a factor analysis

Question 53

Discriminant analysis

Answer 53

A statistical method utilized to predict group membership.

Question 54

Path Analysis

Answer 54

A correlational technique that tests directional hypotheses among multiple IVs and multiple DVs simultaneously.

Question 55

Moderator Variable

Answer 55

Changes the relationship between a predictor and a criterion variable; equivalent to an interaction effect in ANOVA; background variables such as gender, and SES, are common moderators. When moderator variables are present, a test has differential validity (validity differs depending on the level of the moderator, such as whether one is male or female)

Question 56

Cross-Validation

Answer 56

Administering a test to a new sample, one that is different from the original validating sample, so as to evaluate the test's validity on another sample of subjects

Question 57

Criterion Contamination

Answer 57

A misleading increase in a test's validity, in which raters give subjects scores on the criterion variable after being privy to the subject's scores on the predictor variable.

Question 58

Discrete variable

Answer 58

A variable that is measure on either nominal or ordinal scales

Question 59

Shrinkage

Answer 59

A result of corss-validation in which there is a decrease in the validity coefficient due to sample differences.

Question 60

Confounding variable

Answer 60

A variable that affects the dependent or criterion variable, but is of now interest to the researcher

Question 61

One-tailed test

Answer 61

AKA directional test; test for rejection in only one tail; greater chance of rejecting null hypothesis

Question 62

Two-tailed test

Answer 62

AKA non-directional test; tests for rejection in both tails; able to reject null hypothesis in both tails, but each tail has a greater chance of rejecting the null hypothesis

Question 63

T-Score

Answer 63

Standardized score that allows for a participant's score to be compared to the norm group. Mean of 50 with a standard deviation of 10

Question 64

z-score

Answer 64

Mean of 10 and standard deviation of 1

Question 65

What are the %s and T-scores for one standard deviation from the mean?

Answer 65

68% of scores, or T-score between 40-60.

Question 66

What percentage of scores and the T-scores fall within two standard deviations from the mean?

Answer 66

95% or T-scores between 30-70

Question 67

Stanine Score

Answer 67

Mean of 5 and standard deviation of 2.

Question 68

Trend Analysis

Answer 68

An extension of ANOVA. Identifies trends in data when the IV varies from highest to lowest. (Ex. if one group is given 5 mg of meds, a 2nd group gets 10 mg and a third group is given 15 mg.

Question 69

Linear trend

Answer 69

Means are arranged in a line

Question 70

Quadratic trend

Answer 70

Means arranged in a U shape

Question 71

Cubic trend

Answer 71

Means arranged around two points of inflection

Question 72

Quartic trend

Answer 72

Means arranged around three points of inflection

Question 73

Quintic

Answer 73

Means arranged around four points of inflection

Question 74

Multiple correlation (R)

Answer 74

Deals with the correlation between an optimally weighted linear combination of predictors and a criterion. (Multiple regression deals with defining optimal weighting and is a test of prediction).

Question 75

Eta (n)

Answer 75

A universal measure of relationship that can be used regardless of the form of the relationship; it is obtained by computing the variance in Y about any curve of the relationship. Eta is a universal measure of relationship because it (1) applies regardless of the form of the relationship, (2) can be used with either a predicted cure of a relationship or a best-fitting curve obtained after the data are collected, and (3) applies equally well to continuous or categorical independent variables.

Question 76

Type I Error

Answer 76

Rejecting the null hypothesis when it is true. Usually set to 0.05 in the social sciences

Question 77

Type II Error

Answer 77

Failing to reject the null hypothesis when it is false. Related to Type II error is power (1-B), the probability of rejecting the null hypothesis when it is false.

Question 78

Type III Error

Answer 78

Rejecting the null hypothesis, but for the wrong reason. Because of sampling error, two groups can be correctly identified as being significantly different, but the direction of the difference is the opposite of reality. These are relatively rare.

Question 79

As Type I Error becomes smaller,...

Answer 79

Type II Error becomes larger

Question 80

Power is impacted by ____?

Answer 80

Sample size (larger samples increase power); Alpha (smaller alpha levels decrease power, e.g. 0.01 or 0.001 rather than 0.05); Effect size (greater effect sizes increase power, in other words, larger difference between the two groups); and Test used (different statistical tests have more power, two-way ANOVA is more powerful than a one-way ANOVA)

Question 81

Criterion-References Test

Answer 81

Compares the test-taker's performance to an objective standard of achievement. Can be Domain-Referenced (examines the degree to which the test taker has mastered a specific area) or Objectives-Referenced (examines the degree to which the test taker has achieved instructional objectives)

Question 82

Norm-Referenced Test

Answer 82

Compares test-taker's performance to other test-taker's performance. Requires large standardized sample that is representative of population.

Question 83

In multitrait-multimethod matrix, convergent validity is evidenced by ____?

Answer 83

High correlations between measures of the same trait

Question 84

In multitrait-multimethod matrix, divergent validity is evidenced by ____?

Answer 84

Low correlations between measures of different traits

Question 85

Monotrait-Monomethod

Answer 85

Correlation between two tests that measure one trait using one method

Question 86

Monotrait-Heteromethod

Answer 86

Correlation between two tests that measure one trait using different methods

Question 87

Heterotrait-monomethod

Answer 87

Correlation between two tests that measure different traits using one method

Question 88

Heterotrait-heteromethod

Answer 88

Correlation between two tests that measure different traits using different methods

Question 89

Nomological Network

Answer 89

Developed by Cronbach and Meehl (1995) stating that in order to prove that a given measure had construct validity, a "lawful network" for the measure had to be developed; this network includes the theoretical framework for what the instrument is attempting to measure (the construct), an empirical framework for how the construct will be measured (observable manifestations), and the interrelationships among and between the the two frameworks.

Question 90

Predictor variable

Answer 90

Synonymous with IV; the variable that is sued to predict variance in the criterion; plotted on the X axis

Question 91

Criterion variable

Answer 91

Synonymous with DV; variance of the criterion is predicted by the predictor; potted on the Y axis

Question 92

Assessment of relationship between the predictor and criterion

Answer 92

Beta (B) weights (strength of a predictor when all other predictors are held constant), R2 (unique predicitive strength of a predictor); Zero-order correlation (relationship between predictor and criterion ignoring all other predictors); Multicollinearity (i.e. highly correlated predictors, may not reduce predictive ability of predictors)

Question 93

Validity coeffeicient

Answer 93

Correlation between predictor and criterion; squared validity coeffecient indicates the proportion of variance in criterion that is accounted for by the predictor; Greater ranges of scores in both predictor and criterion increases validity coefficient, restricted range decreases validity coefficient; Few validity coeffecients exceed 0.60

Question 94

Conceptual criterion

Answer 94

Theoretical standard that researchers seek to understand

Question 95

Actual criterion

Answer 95

Operational or actual standard that researcher actually assess

Question 96

Criterion deficiency

Answer 96

Portion of the conceptual criterion that is not measured by the actual criterion

Question 97

Criterion relevance

Answer 97

Degree of overlap between the actual criterion and the conceptual criterion

Question 98

Composite criterion

Answer 98

Available criterion measure is a composite of separable attributes

Question 99

Criterion of discrimination

Answer 99

A criterion that inaccurately differentiates between groups, resulting in majority of members being overrepresented in comparison to minority groups

Question 100

True positive

Answer 100

The number of individuals in a given group who exceed cutoff on both predictor and criterion

Question 101

False positive

Answer 101

The number of individuals in a given group who exceed cutoff on predictor but fail to exceed cutoff on criterion.

Question 102

True negative

Answer 102

Number of individuals in a given group who fail to exceed cutoff on both predictor and criterion

Question 103

False negative

Answer 103

The number of individuals in a given group who fail to exceed cutoff on predictor but exceed cutoff on criterion

Question 104

Measurement error

Answer 104

Error in the employed values of a variable due to the presence of distorting influences on the assessment, such as momentary distractions, error in recording or understanding, and influences of other variables on responses to particular items. These are uncorrelated with the "true scores" by definition and treated as "random". A reduction of the correlation coefficient because of error is known as shrinkage.

Question 105

Threats to internal validity

Answer 105

History (any event between pretest and posttest), maturation (natural changes in participants), testing (practice effects), mortality (dropping out), selection, regression effects, demand characteristics

Question 106

AB research design

Answer 106

Simples version of this design in which a baseline (A) is tracked, and then some treatment (B) is applied; if there is a change then the treatment is said to have effect. Weak design because it is subject to many different hypotheses.

Question 107

Time Series Design

Answer 107

AKA quai-experiemental design, refers tot he pretesting and posttesting of one group of subjects at different intervals. The purpose might be to determine long-term effects of treatment, and, therefore, the number of pretests and posts can vary from one to many. Sometimes there is an interruption (follow up test) to assess strength of treatment over time.

Question 108

Correction for Guessing Formula

Answer 108

Usually used for multiple choice exams; Corrected score = R-W/(n-1). R is the number of right answers obtained, W is the number of wrong answers, and n is the number of possible answers per question.