W9 - Reliability & Validity Flashcards by Holly Tomlinson

Define reliability

Consistency of measurements, an ind’s performance on a test or the absence of measurement error.

How well did you know this?

Not at all

Perfectly

Classical Test Theory

Spearman 1904

O = T + e

O = Observed score 
T = True score 
e = Error

How well did you know this?

Not at all

Perfectly

What are the 2 types of error measurement?

Systematic error

Random error

How well did you know this?

Not at all

Perfectly

Define systematic error

Consistent error which biases the true score + doesn’t affect reliability

How well did you know this?

Not at all

Perfectly

Define random error

Unpredictable error which biases the true score + does affect reliability

How well did you know this?

Not at all

Perfectly

Ways to minimise error

Train researcher to ensure proficient use of instrument

Repeats

Compare data from 2+ researchers

Careful design of study protocol

Consider choice of instrument

Calibrate instrument

How well did you know this?

Not at all

Perfectly

Common technique used to assess relative reliability across time/researchers/writers…

Pearsons correlation coefficient

Higher correlation = ⬆️ reliability

How well did you know this?

Not at all

Perfectly

How can relative reliability be assessed?

Through the test-retest reliability

= Assess the stability of the measurements on different occasions.

How well did you know this?

Not at all

Perfectly

What is used when doing the test-retest reliability?

2 tests: Pearsons correlation coefficient

2 or + tests: Intraclass correlation coefficient (ICC)

How well did you know this?

Not at all

Perfectly

Inter-rater reliability

Reliability / consistency across raters

Correlating the scores obtained from a group of participants by 2 or + researchers

How well did you know this?

Not at all

Perfectly

What does internal consistency stand for?

Reliability across different parts of a measurement instrument

i.e items within a sub-scale on a questionnaire

How well did you know this?

Not at all

Perfectly

How is internal consistency assessed?

Using Chronbach’s alpha reliability coefficient

Values range from 0-1

Closer to 1 = higher reliability

How well did you know this?

Not at all

Perfectly

List some terms used for absolute reliability

Also known as measures of absolute reliability

Technical error of measurement

SE of measurement

Coefficient of variation

Limits of agreement

How well did you know this?

Not at all

Perfectly

Define validity

Extent to which a test/instrument measures what its supposed to measure.

How well did you know this?

Not at all

Perfectly

What are the types of validity?

Validity of measurement

Validity of a study

How well did you know this?

Not at all

Perfectly

What comes under validity of measurement?

Face validity

Content

Construct validity

Criterion

How well did you know this?

Not at all

Perfectly

Validity of measurement

What comes under criterion validity?

Concurrent

Predictive

How well did you know this?

Not at all

Perfectly

What comes under validity of a study?

Internal

External

Define face validity

Whether the method of data collection obviously involves the factor being measured.

Define content validity

If the instrument adequately covers the domain of interest.

Define construct validity

Assesses extent to which an instrument accurately measures hypothetical constructs

What are the ways of assessing construct validity?

Convergent validity

Discriminant validity

ASSESSING CONSTRUCT VALIDITY

Convergent validity

Scores on an instrument to those on a similar measure

ASSESSING CONSTRUCT VALIDITY

Discriminant validity

Scores on an instrument are NOT related to those from an instrument which assess a different construct.

Criterion-related validity

Looks at whether the scores on an instrument are related to scores on a previously validated measure.

What are the ways of criterion-related validity?

Concurrent validity Predicative validity

CRITERION-RELATED VALIDITY Concurrent validity

Scores collected at roughly the same time

CRITERION-RELATED VALIDITY Predicative validity

Criterion instrument completes at a later date

Commonly used technique to assess criterion-related + construct validity

Pearsons Correlation Coefficient

Can an instrument be reliable but not valid?

Yes As it could be consistently measuring the wrong thing

Can an instrument be valid but not reliable?

Internal validity

Refers to the ability to attribute changes in the dependent variable to the manipulation of the independent variable

External validity

Refers to the ability to generalise the results of a study to other settings + other individuals

Threats to internal validity

Maturation (Age/Growth) ? Selection bias ? Expecting certain results ? Measurement + equipment? - can be overcome by freq calibration Mortality ? (w/drawal / drop out)

Threats to internal validity How can expecting certain results be avoided?

Blinding / double blind study

Threats to external validity

Reactive or interactive effects of testing Interaction of selection of bias + treatment Reactive effects of experimental arrangements Multiple-treatment interference

THREATS TO EXTERNAL VALIDITY How does reactive or interactive effects of testing have an influence?

Pre-test makes a participant more aware or sensitive to the treatment

THREATS TO EXTERNAL VALIDITY How does interaction of selection bias + treatment have an influence?

Treatment is only effective in the group selected

THREATS TO EXTERNAL VALIDITY How does Reactive effects of experimental arrangements have an influence?

Treatments effective in lab may not transfer to the real world

THREATS TO EXTERNAL VALIDITY How does Multiple-treatment interference have an influence?

Effects of a previous treatment may influence subsequent ones

What is the definition of relative reliability?

The degree to which data maintain their position in a sample with repeated measurements

What is the definition of absolute reliability?

The degree to which repeated measurements vary for individuals

3. Which of the following describes test-retest reliability? a. Consistency across items b. Consistency across raters c. Consistency across time points d. None of the above

Consistency across time points

When scores on an instrument are related to scores on a previously validated measure, which type of validity has been established?

Criterion-related validity