Week 5-6 Validity

Question 1

validity

Answer 1

the degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses of a test

Question 2

four important implications of this definition of validity

Answer 2

1 Validity concerns interpretations and uses of scores
2 Validity is not a property of the test itself
3 Validity is a matter of degree
4 Validity is based on theory and evidence

Question 3

Validity is a crucial basis for:

Answer 3

1 the meaningful interpretation of behavioural research
2 making sound societal decisions based on such research
3 making informed test-based decisions about individuals

Question 4

Content validity:

Answer 4

The degree to which the content of a test is representative of the domain it’s supposed to cover.

Question 5

criterion-related validity

Answer 5

a measure obtained by evaluating the relationship of scores obtained on the test with scores on other tests

Question 6

construct validity

Answer 6

a measure obtained by performing an analysis of:

a) how scores on the test relate to other test scores and measures, and
b) how scores on the test can be understood within some theoretical framework

Question 7

Test Content

Answer 7

This is the match between the content of a test and the content that should be included in the test

Question 8

two types of validity relevant to test content:

Answer 8

1 Content validity

2 Face validity

Question 9

two key threats to content validity:

Answer 9

Construct-irrelevant content

Construct under-representation

Question 10

Face validity

Answer 10

what a test appears to measure to the person being tested, rather than what the test actually measures

Question 11

Content validity can be evaluated only by ________ in a field, whereas face validity must be assessable by __________ •

Answer 11

Experts, non-experts

Question 12

Internal structure

Answer 12

the way the parts of a test are related to each other:
• some tests include items that are highly correlated with each other, forming a single cluster
• other tests include items that fall into two or more clusters

Question 13

Factorial validity concerns the match between the ________ internal structure of a test and the structure the test ________ possess

Answer 13

actual ,should

Question 14

Response processes

Answer 14

there should be a close match between the psychological processes that the respondents actually use when completing a measure, and the process that they should use

Question 15

methods for obtaining validity evidence of the response processes include:

Answer 15

• “think-aloud” procedures
• cognitive interviews
• focus groups
• response times
• eye movements

Question 16

Associations with other variables

Answer 16

the way in which the construct is connected to other relevant psychological variables

• Our theoretical understanding of the construct we are trying to measure should lead us to expect a particular pattern of associations with other variables
• This type of validity evidence emphasises the match between measures predicted and observed associations with other measures

Question 17

Convergent evidence (convergent validity)

Answer 17

the degree to which test scores are correlated with tests of related constructs

Question 18

Discriminant evidence (discriminant validity)

Answer 18

the degree to which test scores are uncorrelated with tests of unrelated constructs

Question 19

Concurrent validity evidence

Answer 19

the degree to which test scores are correlated with other relevant variables that are measured at the same time as the test undergoing validation

Question 20

Predictive validity evidence

Answer 20

the degree to which scores on the test undergoing validation are correlated with relevant variables that are measured at a future point in time

Question 21

consequential validity

Answer 21

refers to the social and personal consequences associated with using a particular test

Question 22

what kind of test is associated with greater consequential validity

Answer 22

a non-biased test

Question 23

three other types of validity that arguably do not fit as strongly within this construct/theory framework:

Answer 23

1 Criterion Validity
2 Induction-Construct Development Interplay
3 Measurement as Theory

Question 24

A criterion

Answer 24

the standard against which a test or test score is evaluated

Question 25

2 examples of criterion validity

Answer 25

Concurrent validity and predictive validity

Question 26

Induction-Construct Development Interplay

Answer 26

bottom up theory development, e.g FFM

Question 27

Measurement as theory

Answer 27

This approach rejects much of the unitary view except the importance attached to constructs and the theoretically based examination of response processes

Question 28

Internal consistency reliability is typically estimated with

Answer 28

Cronbach's α

Question 29

Cronbach's α assumes a) unidimensionality b) multidimensionality

Answer 29

unidimensionality

Question 30

Factorial validity concerns the ___________ ___________ of test scores

Answer 30

internal structure

Question 31

The test items on a unidimensional test would also have the property of ___________ ____________

Answer 31

conceptual homogeneity

Question 32

Is a total test score calculated for a multidimensional tests with uncorrelated dimensions?

Answer 32

no, a score is obtained for each dimension, but the dimensions scores are not combined to compute a total test score

Question 33

The communality for a given variable can be interpreted as

Answer 33

the percentage of variation in that variable explained by the extracted components

Question 34

As a general statement, you want to see communalities that are at least

Answer 34

.04 for items or .09 for subscales

Question 35

Items are less reliable than subscales, so they have _______ communality expectations

Answer 35

lower

Question 36

The sum of the "Initial Eigenvalues" is equal the number of

Answer 36

variables included in the analysis

Question 37

Component loadings refer to the associations between ______ and ____________

Answer 37

items, components

Question 38

Component loadings vary between

Answer 38

-1 and +1

Question 39

The size of the loading indicates the degree of association between a _______ and a ______________

Answer 39

item, component

Question 40

Component loadings further from 0 indicate ____________ associations

Answer 40

stronger

Question 41

A positive loading indicates that people who respond with a high score on an item have a ________ level of the underlying component

Answer 41

high

Question 42

A negative loading indicates that people who respond with a high score on an item have a _____ level of the underlying component

Answer 42

low

Question 43

As a general statement, useful component loadings are either _____ or ____ or greater

Answer 43

.20 (Items) or .30 (subscales)

Question 44

Simple structure occurs when each item is strongly linked to

Answer 44

one and only one component

Question 45

One way to help achieve simple structure is to

Answer 45

rotate the solution

Question 46

should you always rotate the solution?

Answer 46

Yes

Question 47

How many components do you need to rotate the solution?

Answer 47

two or more

Question 48

If you extract only one component, then there will only be the

Answer 48

"Component Matrix"

Question 49

If you have a minimum of 5 variables per factor and the communalities all exceed _____, then a sample size of about should be sufficient

Answer 49

.20, 150

Question 50

There are no simple guidelines to follow, because the size of the sample required will be dependent upon two main factors:

Answer 50

- the amount of communality associated with the variables (higher communality means less sample size required) - the number of variables per factor (higher number of variables per factor means less sample size required)

Question 51

4 Methods of evaluating construct validity

Answer 51

1 Focussed associations 2 Sets of correlations 3 Multitrait-Multimethod Matrices 4 Quantifying Construct Validity

Question 52

3 Factors affecting validity coefficients

Answer 52

1 True associations 2 Measurement error 3 Restricted range

Question 53

The interconnections between a construct and other related constructs are known collectively as a

Answer 53

nomological network

Question 54

validity coefficient

Answer 54

correlation coefficient between a test score (predictor) and a performance measure (criterion)

Question 55

Example of a focused association

Answer 55

SAT and first year university marks

Question 56

Validity generalization studies are intended to evaluate the predictive utility of a test's scores across

Answer 56

a range of settings, times, situations, etc

Question 57

Sets of correlations

Answer 57

large nomological networks incorporating a wider variety of other constructs, with differing levels of association with the main construct

Question 58

validity generalization

Answer 58

is a process of evaluating a test's validity coefficients across a large set of studies, comparable to a meta analysis

Question 59

Is the judgement about the degree to which the pattern of coefficients matches the expectation a) objective b) subjective

Answer 59

A subjective

Question 60

Validity generalization studies can essentially address three questions:

Answer 60

1 estimate the average level of predictive validity across studies 2 estimate the degree of variability associated with the validity coefficients 3 identify sources of systematic variability in the validity coefficients

Question 61

a correlation between two scores may conflate two sources of variance

Answer 61

* trait variance (the good stuff) | * method variance (the bad stuff)

Question 62

large correlations between different traits using the same measurement method suggests that the correlations are simply due to

Answer 62

a response style (i.e., method variance)

Question 63

using MTMMM we want a lot shared trait variance, particularly identical traits measured using

Answer 63

different methods

Question 64

The multitrait component of the study refers to

Answer 64

administer the questionnaire measuring the trait of interest in addition to other measures such as a measure of impulsivity, conscientiousness, and emotional stability

Question 65

the multimethod component refers to

Answer 65

measuring the trait of interest and the additionally trait using multiple methods, e.g self report and acquaintance report

Question 66

Sources of variance for Heterotrait-Heteromethod correlations

Answer 66

Nonshared trait variance, nonshared method variance

Question 67

The most stringent test in a MTMM analysis is to determine whether the ________________ _________________- correlations are "meaningfully" larger than the ____________- ________________ correlations

Answer 67

monotrait-heteromethod, heterotrait-monomethod

Question 68

one of the limitations associated with MTMMM

Answer 68

There are no clear guidelines to evaluate the differences in the mean correlations

Question 69

At the very least, the ___________ ___________ correlations need to be larger than the ______________ _____________correlations

Answer 69

monotrait-heteromethod,heterotrait-monomethod

Question 70

Quantifying construct validity (QCV)

Answer 70

this procedure requires researchers to predict the magnitude of the correlation between their measure of interest and their selected criteria

Question 71

3 advantages of the QCV approach

Answer 71

1 It forces researchers to consider carefully the expected pattern of convergent and discriminant associations that would make theoretical sense 2 It forces researchers to make explicit quantitative predictions about the pattern of associations 3 It provides a single value reflecting the overall "goodness-of-fit" between the predicted and actual associations

Question 72

using the QVC, why would a low correlation between the predicted and actual associations not necessarily reflect poor validity

Answer 72

because the predicted associations may simply be a poor reflection of a construct's nomological network

Question 73

If a test's or a criterion's reliability is much lower than .70, there are two options:

Answer 73

1 disregard—or reduce the weight given to—a validity coefficient based on poor reliability 2 adjust the validity coefficient to account for measurement error

Question 74

a correlation between two variables can be reduced if the range of scores in one or both variables is

Answer 74

artificially limited or restricted