week 6: Reliability and validity

Question 1

what is measurement error?

Answer 1

obtained score = true score +- error

Question 2

what are 3 common factors for measurement error?

Answer 2

1) situational contaminants
2) response set biases
3) Transitory personal factors

Question 3

what are situational contaminants?

Answer 3

data is affected by conditions which are produced such as
- friendliness of researchers
- location of data gathering
- environment factors : light, temperature

Question 4

Explain response set biases 3

Answer 4

1- extreme responses
2- acquiescence response: agree with everything
3- social desirability response

Question 5

explain transitory personal factors

Answer 5

data affected by temporary states such as hunger, mood. factors can affect measurement, anxiety can increase heart rate.

Question 6

what are the 3 strategies to reduce error?

Answer 6

1) standardisation
2) anonymity
3) train interviewers

Question 7

explain the 3 strategies to reduce error

Answer 7

standardisation:
collect data at same place and time
assure respondents of ample time given

anonymity:
answer without fear of judgement

train interviewers:
behave the same
assess participants readiness

Question 8

what is reliability?

Answer 8

The consistency with which an instrument
measures the target attribute.

Question 9

what is validity?

Answer 9

The degree to which an instrument measures what it is
supposed to be measuring.
4
Body T
Not BP
Pain scale
Pain level
Anxiety level
Polit & Beck 2014

Question 10

what are the 3 aspects of reliability

Answer 10

Stability
Internal Consistency
Equivalence

Question 11

what is stability - reliability test?

Answer 11

Stability
* The extent to which scores are similar on two
separate administration of an instrument

Question 12

how do u assess for stability reliability test?

Answer 12

Assessed through test-retest reliability procedure
* The same instrument is given twice to the same group of people
* The reliability is the correlation between the scores on the two tests
* The scores on the two tests are not identical but most differences are small
* Reliability coefficient (r), a numeric index that quantifies an instrument’s
reliability can be computed.
0.7 to 0.8 is good
 More appropriate for fairly enduring characteristics (e.g. self-
esteem, personality, IQ test

Question 13

what is internal consistency reliability test?

Answer 13

Internal consistency （homogeneous)
* The extent that all the subparts of the
instrument measure the same trait.
* Appropriate for most multi-item instruments.
* Evaluated by administering instrument on one
occasion.
* The most widely used reliability approach

Question 14

how to evaluate internal consistency reliability test?

Answer 14

Internal consistency
* Evaluated by Cronbach’s alpha (coefficient alpha)
 Acceptable level = 0.7 – 0.9

Question 15

what does it mean if:

 Cronbach’s alpha too low -
 Cronbach’s alpha too high

Answer 15

low:-Items are measuring different traits.
too high: – Redundancy of items

Question 16

what is equivalence reliability test?

Answer 16

Equivalence
* Concerns the degree to which two or
more independent observers or
coders agree about the scoring on an
instrument.
* Assessed by comparing observations
or ratings of two or more observers.
* A high level of agreement between
the raters indicates a good
equivalence of the instrument.
Reliability

Question 17

how to assess for equivalence reliability test?

Answer 17

Assessed through Inter-rater (interobserver) reliability
procedure: which having two or more trained
observers/coders watching an event simultaneously, and
independently recording data according to the
instrument’s instructions.
* A index of agreement is calculated.
* Cohen’s Kappa (κ) is used to measure inter-rater reliability for
categorical outcomes. ( 0.6)
* Intraclass Correlation Coefficient (ICC) is used to measure inter-
rater reliability for continuous measures. ( 0.7)

Question 18

what are the 2 major aspects of validity?

Answer 18

Major aspects of validity
* Content Validity
* Criterion-Related Validity

Question 19

what is content validity?

Answer 19

Content validity
* Concerns the degree to which an instrument
has an appropriate sample of items for the
construct being measured.
* Adequacy of content of the instrument in
providing full coverage of the concepts of
interest.

Question 20

how to assess for content validity ?

Answer 20

An instrument’s content validity is necessarily based on
judgment by an expert panel.
* Have experts rate items on a four-point scale
1 = not relevant 2 = somewhat
3 = relevant 4 = very relevant
* A formal content validity index (CVI) across the
experts’ ratings of each item’s relevance.
* The CVI for the instrument is the proportion of items
rated as either 3 or 4. A CVI score of 0.90 or better
indicates good content validity.

Question 21

what is criterion validity test?

Answer 21

Criterion-related validity
* The extent to which an instrument is correspond to some
external criterion of the variable of interest.
* External criterion:
* A gold standard or well-established valid measure for the variable
of interest.

Question 22

what is the difference between concurrent validity and predictive validity?

Answer 22

Concurrent
Validity
To reflect the same incident of behavior
with a criterion measure at the same time.
* Administer the tested instrument together
with the criterion measure.

Predictive
Validity
* To predict subjects’ responses in the
future
* Criterion measure is used to assess
subjects’ response in the future.