Flashcards in Class Notes Deck (72):

1

## if one level of IV is assigned via randomization

### considered experimental -- as consumer be keenly aware of which level is randomized -- e.g., randomized on tx but not gender -- 2 x 2 factorial design

2

## any time you adminster the same measure to the same participants (to collect data on DV) at a later date (at any point -- could be 5 minutes apart)

### Repeated measures design

3

## subsumed within repeated measures -- generally speaking, typically more than 20 observations (e.g., clinical trials) no longer need 20 for statisical methods to hold (so need at least 10) -- tend to occur in fixed intervals depends on what you're collecting data on -- logorythmic data collection - time period continues to get bigger (exponentially) between observations

### Time-series design

4

## Time-series design

5

## Difference between repeated measures and time series design

### number of observations (collected data on DVs)

6

## In non-equivalent group design one should

### control with pre-test

7

## Threats that concern ______ relate to statistical conclusion validity

### integrity of treatment itself

8

## Threats that concern ______ relate to internal validity

### making comparisons between tx groups

9

## What does randomization do?

### "Ensure" that participant groups are equal prior to treatment

10

## What does randomization not do?

### Ensure anything that happens after treatment -- possibility of history effects

11

## Important issues to look for in time series designs

###
* change in intercept (level)

* slope

* stability of effect (continuous or discontinuous effect)

* delayed vs immediate effect (instantaneous vs delayed -- when you see effects taking place)

12

## Standard error is based on

### sample of samples

13

## What type of research is meta-analysis?

### Ex-post facto

14

## Assessment is defined as

###
* Overarching, sampling behavior

* In contrast to research when we assess people

15

## Measurement is defined as

###
* Establishing quantitative rules for assigning numbers to represent attributes of persons

* Attributes of people, not to people

* Distinction between observations and inferences

* Must think: How representative is this of behaviors outside of this context?

16

## Test, scales, and measures are defined as

### Objective, quantitative measurement using standardized procedures; psychometric properties of scores essential

17

## What are Rating protocols?

###
Taxonomies, classification and rating systems done by an observer (usually)

18

## What is Evaluation?

### Assessing the congruency between what is expected and actually occurs (formal to informal, may be quantitative) Chen, 1990

19

## What is Clinical assessment?

### Less formal, typically not fully standardized or quantitative

20

## What is a Scale?

###
* often used interchangeably (not always) with measure, questionnaire or test

* Some say questionnaire is less formal

* Assumed to be assessing a single construct or domain

21

## What is a construct?

### Trait, domain, ability, latent variable, theta 0

22

## What is theta?

### Item Response Theory (IRT) uses this to talk about the construct itself -- latent variable

23

## What are the differing types of item responses?

###
1. Dichotomous

2. Polytomuos

3. Graded responses

24

## What are dichotomous item responses?

### Two levels (true or false, yes or no, etc.)

25

## What are polytomous item responses?

### Three or more levels, often ordered but not always

26

## What are graded item responses?

###
More than 2 ordered response options

All graded responses are polytomous but not all polytomous items are graded items

27

## What is Classical Test Theory (CTT)?

###
* Total sums of squares partitioned into true score variance vs error score variance

* Partitioning variance

28

## What is Modern Test Theory = Item Response Theory (IRT)?

###
Has to do with probability

* What's the probability that someone will respond in a certain way?

* Not only assigning where the individual is on the construct

29

## What is Standard Error of Measurement (SEM or SEm) ?

###
* Estimate of extent to which an observed score deviates from true score

* Create confidence interval

* Probability that an individual's true score lies within a range

30

## What is reliability?

### * How consistently does a scale measure what it is designed to measure?

31

## What are types of reliability?

###
1. Test-retest

2. Parallel forms

3. Split halves

4. Internal consistency

5. Inter-rater

32

## Parallel forms

###
* Administer once in different forms and see how they correlate

33

## Split-halves

###
* See how one half of test correlates to other half within each participant

* But need twice as many items (think statistical power and N)

34

## Three types of validity

###
1. Content

2. Criterion

3. Construct

35

## What is content validity?

###
* Extent to which items are representative of or sampled from content domain being measured

* Most important in tests in classes

36

## What are limitations of content validity?

###
* Often non-statistical approach is used -- ask panel of experts

* Often relies upon face validity - does this look like it tests what it's supposed to test

37

## What are two types of criterion validity?

###
1. Predictive

2. Concurrent

38

## What is predictive validity?

### Before and after -- look at outcomes

39

## What is concurrent validity?

### Measure all items at same time

40

## What are limitations of concurrent validity?

###
* Often use 1-item scales

* Lack validity and reliability

* Extremely limited variance

41

## What is construct validity?

###
* How well scores measure a specific trait or construct

* Requires a priori specification and operationalization of the construct

42

## Steps in developing CTT test construction

###
1. Qualitative research

2. Explicate theorizing

3. Define constructs

4. Generate item pool

5. Content validity study

6. Derivation sample

7. Cross-validation study

8. Subsquent studies testing evidence of construct validity

43

## What are limitations of CTT?

###
1. Cannot change anything about the scale

2. Standard error of measurement is presumed constant across levels of the construct, items, and scores

3. Statistics are sample dependent

44

## What are main characteristics of IRT?

###
1. Probablisitc

2. Statistical mathematical theory

3. Scaling items as well as people on latent trait

4. Wide variety of IRT models (dichotomous--more common-- and polytomous)

45

## What are basic assumptions of IRT?

###
1. Well-known constructs and well-established tests

a. know dimensionality

b. know validity

c. know what's correct and incorrect

2. extensive item banks and data banks

3. large samples (>3000)

46

## In contrast to assumptions of IRT, what is typical of CPY instrumentation?

###
1. New scales

2. Constructs are not well known or defined

a. confounds

b. unknown dimensionality

c. ordered rating scale data (likert)

3. non-existent item pool; no data banks

4. small sample sizes (in best cases n = ~300-400)

47

## What is measurement invariance?

### Cross-cultural applications of tests, scales, and measures

48

## 4 questions to ask regarding measurement invariance

###
To what extent...

1. Can a construct be conceptualized equivalently across cultures?

2. is the same construct being measured equivalently across cultures?

3. can mean scores be compared equivalently across cultures?

4. can measures of association (correlation) be compared equivalently across cultures?)

49

## Multiple regression

###
* Find lines that best fit the data instead of planes (as above)

* Linear composite of IVs to best explain DVs

* Combination of weights on IVs constitutes linear composites

49

## Confirmatory Factor Analysis (CFA)

###
* forces the data into this model then assess how well the model fits the data

* Also assess how scales correlate with one another (Orthagonal or Oblique - allows correlation)

50

##
Exploratory Factor Analysis (EFA)

###
* Misapplication of statistical procedure for measurement development

* Analogous to doing "atheoretical" research

* No constraints on data

* See how many factors come out ("sem-magical")

* interpret the factors that come out based on the data that they're based on

* Do a CFA because we have implicit theorizing

51

## Things to look for in articles relating to measurement

###
1. Look for ceiling/floor effects (Determine possible range for scale or subscale. How many items, what's the rating scale? If mean + or - 1 SD is highly skewed -- No longer have adequate prediction of probability of error when you have skewed scores)

2. Type 1 and Type 2 error rates are not protected

* "I adapted this test"

* Changed the rating scale, items, instructions, order, etc. -- changed anything

* Computing cronbach's alpha is insufficient to protect against these threats

* Different rating scales for same scale/sub-scale

52

## Analysis of variance and multiple regression

### are identical!

53

## Multiple regression =

### 1 DV, always univariate

54

## Multivariate =

### multiple IVs and multiple DVs

55

## Multiple regression in linear model

### y = a + b1x1+b2x2+...bkXk + e

56

## Multi-collinearity

### cannot interpet the overlap of explained variance between variables in multiple regression

57

## Part correlation

### unique variance explained by one predictor in the model, controlling for other predictors in the model

58

## If summing all zero order correlations and the value exceeds 1.0...

### by definition you have profound multicolliniarity (x1 and x2 are highly correlated)

59

## Moderation

###
* Answers for whom or when does this relation apply?

* Affects magnitude and/or strength of relation

* Easier to think about as high low, but better to use continuous

60

## Interactions refer to moderation or mediation effects?

### Moderator

61

## If interaction term is significant...

### main effects are meaningless

62

## Mediation

### *How or why a relation exists

63

## Full mediation

### Mediator variable completely accounts for relation between IV and DV

64

## Partial mediation

### Explains much of, but not all, of the relation between IV and DV (IV and DV path alone still exists but is nearer to non-significance)

65

## MAXMINICON

###
1. Maximize experimental variance

2. Minimize error variance

3. Control extraneous variance

66

## MSH = SSH/dfH

### Mean square variance (want to increase)

67

## MSE = SSE/dfe

### Mean square error (want to decrease)

68

## Maximize experimental variance by

###
1. Ensure maximum variability in Y due to X

2. Make treatments as different as possible, but realistic

69

## Control extraneous variance a priori by

###
1. Homogenize on the confounding variable (restriction of range)

2. Match participants on all relevant conditions

3. Randomly assign participants to treatment conditions

70

## Control extraneous variance a priori or post hoc by

###
1. Build a blocking variable into the design to control the confound

2. Covary any confounding variable (analysis of covariance)

71