3: Foundations of Quantitative Measurement

Question 1

In the social sciences, quantitative and qualitative approaches are typically associated with what?

Answer 1

Deep philosophical differences in epistemology.

Question 2

What is the classical test theory measurement model? What concepts does it underpin?

Answer 2

Observed score = true score + error.

Underpins concepts of reliability and validity.

Question 3

Levels of specificity in variables are predicated on constructs, operational definitions, and measures. Define each.

Answer 3

Construct: a psychological concept that is not directly observable. No tangible existence outside person’s mind.

Operational definition: clear, measurable definition of a construct based on theory. May capture only a portion of entire construct.

Measures: clearly defined set of procedures for obtaining a measure of the construct of interest. Must be clear and precise enough to be replicated by other scientists.

Question 4

What is construct validity?

Answer 4

How well you translated/transformed construct into a functioning, operating reality - measuring what you wish to measure.

Question 5

In what way is it useful to think of construct validity?

Answer 5

As an umbrella variable that encompasses all other forms of validity.

Question 6

What is content validity? Provide an example. What can improve it?

Answer 6

Does measure cover all aspects of underlying construct?

E.g., measure missing some of depression DSM-5 components may lack content validity.

Multiple measurements of construct can improve content validity.

Question 7

What is face validity? Provide an example.

Answer 7

Extent measure ‘appears’ to measure underlying construct.

E.g., item “nervous” has face validity for measuring anxiety, item “jealous” does not.

Question 8

When studying older adults, some items for personality disorders may have poor _____ due to developmental changes.

Answer 8

Face validity.

Question 9

What is criterion validity?

Answer 9

How well measure correlates with established “gold standard” measures of the same construct.

Question 10

List the two subtypes of criterion validity. Provide examples.

Answer 10

Concurrent validity: “at the same time”(e.g., correlate questionnaire with a clinical interview).

Predictive validity: “predicting the future”(e.g., does your anorexia measure predict future weight loss).

Question 11

Two concepts are specific to criterion validity for clinical psychology and medical diagnosis. What are they?

Answer 11

Sensitivity: how well it picks out people with the disorder (i.e., few false negatives).

Specificity: how well it avoids diagnosing healthy people with a disorder (i.e., few false positives).

Question 12

How does one calculate sensitivity and specificity using signal detection theory?

Answer 12

Sensitivity: hit / (hit + miss)

Specificity: correct rejection / (false alarm + correct rejection)

Question 13

When finding optimal cut-off score to use to get the best balance of sensitivity and specificity via an ROC curve, what indicates such?

Answer 13

Larger values on the y-axis indicate better sensitivity (% of hits). Smaller values on the x-axis indicate better specificity (% of correct rejections).

Question 14

As sensitivity increases, what happens to specificity?

Answer 14

Decreases.

Question 15

Define convergent and discriminant validity.

Answer 15

Convergent validity: correlate with other measures that it should be related to.

Discriminant: not correlate with measures that it should not correlate with.

Question 16

Unreliability is the amount of _____ in the measurement.

Answer 16

Error.

Question 17

What is test-retest reliability? When is it generally more useful?

Answer 17

Is the measure consistent over time - do scores stay more or less the same when repeatedly measured?

Useful for constructs which are theorized to be stable (i.e., personality) rather than transient states (e.g., fear).

Question 18

Test-retest reliability is often assessed with what? What do higher values indicate?

Answer 18

Correlation coefficient. Higher values indicate higher test-retest reliability.

Question 19

Describe the concept of internal consistency.

Answer 19

Used to assess a questionnaire with multiple items. Do all the items in the questionnaire more or less measure the same thing?

Question 20

The statistic most commonly used to measure internal consistency is Chronbach’s Alpha α. Conceptually, it’s calculated as a function of what two things? How can you increase it?

Answer 20

Number of items, the average intercorrelation among all the items.

Increase number of items or remove items that are very weakly correlated with other items.

Question 21

Describe the concept of inter-rater reliability. When is it used?

Answer 21

Two or more trained coders independently review data, provide their ratings. Ideally, ratings from all coders are similar.

Used when scores are derived from a trained coder looking at raw data.

Question 22

What are the two nominal scales used to determine inter-rater reliability?

Answer 22

% agreement.

Cohen’s Kappa: more conservative, controls for agreement which occurs by chance alone.

Question 23

What are the two ordinal, ratio, or interval scales used to determine inter-rater reliability?

Answer 23

Pearson correlation.

Intraclass correlation: more conservative, more complex calculations for different situations.

Question 24

What are the rules of thumb when evaluating reliability and validity stats that determine whether stats are:

• good
• acceptable
• marginal
• poor

Answer 24

Good: 0.80 (reliability); 0.50 (validity)

Acceptable: 0.70 (reliability), 0.30 (validity)

Marginal: 0.60 (reliability), 0.20 (validity)

Poor: 0.50 (reliability), 0.10 (validity)

Question 25

What are the four types of basic hypotheses?

Answer 25

Descriptive: what is X like?

Descriptive/comparison: does group 1 differ from group 2?

Correlation: do X and Y covary?

Psychometric: is a measurement reliable and valid?

Question 26

What are the three most common hypotheses in published psychology research?

Answer 26

Mediation: X leads to M (mediator), which in turn leads to Y.

Moderation: relationship between X and Y varies depending on the value of the moderator, M.

Incremental validity: X1 predicts Y over and above another known predictor (X2).