Brainscape's Knowledge GenomeTM


Top Flashcards (Certified)
All Flashcards

Research Stats > Week 6: Reliability, Validity, Epidemiologic Analysis and Dichotomizing Treatment Effect > Flashcards

Week 6: Reliability, Validity, Epidemiologic Analysis and Dichotomizing Treatment Effect Flashcards

1
Q

What is reliability?

A

Extent to which a measurement is consistent and free from error

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

All reliability scores have…

A

signal and noise

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What is signal?

A

true score

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What is noise?

A

error

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Reliability is the ratio of…

A

signal to noise

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

relative reliability

A

ratio of total variability of scores compared to individual variability within scores
unitless coefficient
ICC and kappa

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

absolute reliability

A

indicates how much of a measured value is likely due to error
expressed in the original unit
SEM is commonly used

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Standard error of measurement (SEM) for relative measure of reliability

A

ICC (and kappa)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Standard error of measurement (SEM) for absolute measure of reliability

A

SEM

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Most common types of reliability

A

test-retest, inter-rater, intra-rater, internal consistency

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

inter-rater

A

2+ or more raters who measure the same group of people

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

intra-rater

A

the degree that the examiner agrees with himself or herself
2+ measurements on the same subjects

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

in measurement validity, the test should…

A

discriminate, evaluate, and predict

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

reliability is a __________ for validity

A

prerequisite

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

content validity

A

establishes that the multiple items that make up a questionnaire, inventory, or scale adequately sample the universe of content that defines the construct being measured

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Criterion-related Validity

A

establishes the correspondence between a target test and a reference or ‘gold’ standard measure of the same construct

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

concurrent validity

A

the extent to which the target test correlates with a reference standard taken at relatively the same time

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

predictive validity

A

the extent to which the target test can predict a future reference standard

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

construct validity

A

establishes the ability of an instrument to measure the dimensions and theoretical foundation of an abstract construct

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

convergent validity

A

the extent to which a test correlates with other tests of closely related constructs

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

divergent validity

A

the extent to which a test is uncorrelated with tests of distinct or contrasting constructs

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

quantifying reliability: ‘old approach’

A

pearson’s r
assesses relationship
only 2 raters could be compared

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

Quantifying reliability: ‘modern’ approach

A

intraclass correlation coefficients (ICC)
cohen’s kappa coefficients
both ICCs and kappa give single indicators of reliability that capture strength of relationship plus agreement in a single value

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

ICC

A

values from 0 - 1.0
measures degree of relationship and agreement
can be used for > 2 raters
interval/ratio data

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
ICC types
six types depending on purpose, design, and type of measurements
26
ICC type is defined by
two numbers in parentheses ex: ICC (2,1), ICC (model, form)
27
model 1
raters are chosen from a larger pop; some subjects are assessed by different raters (rarely used)
28
model 2
each subject assessed by the same set of raters
29
when is model 2 used
for test-retest and inter-rater reliability
30
model 3
each subject is assessed by the same set of raters, but the raters represent the only raters of interest
31
when do you use model 3
used for intra-rater reliability or when you do not wish to generalize the scores to other raters
32
ICC forms
second number in parentheses represents number of observations used to obtain reliability estimate
33
form 1
scores represent a single measurement
34
form k
scores based on mean of several (k) measurements
35
ICC interpretation
no absolute standards
36
ICC > 0.90
best for clinical measurements
37
ICC > 0.75
good reliability
38
ICC < 0.75
poor to moderate reliability
39
cronbach's alpha (a)
represents correlation among items and correlation of each individual item with the total score between 0.70 to 0..90
40
if cronbach's alpha is too low it means
not measuring the same construct
41
if cronbach's alpha is too high it means
redundancy
42
agreements of reliability for categorical scales are
diagonal
43
disagreements of reliability for categorical scales are
all other parts of the table
44
percent agreement
simply how often raters agree range of 0% to 100%
45
kappa coefficient
proportion of agreement between raters after chance agreement has been removed can be used on both nominal and ordinal can be interpreted like ICC
46
weighted kappa
best for ordinal data can choose to make 'penalty' worse for larger disagreements weights can be arbitrary, symmetric or asymmetric
47
kappa = <.4
poor to fair
48
kappa = .4 - .6
moderate
49
kappa = .6 - .8
substantial
50
kappa = .8 - 1.0
excellent
51
what does a diagnostic test do
focuses the examination identify problems assist in classification
52
diagnostic test are all about
probabilities and limiting uncertainty
53
pre-test probability
before any testing takes place
54
post-test probability
outcome of the test
55
clinical prediction rules (CPR)
combinations of clinical findings predictions quantifies the contributions of a set of variables to diagnosis, prognosis, and likely response to treatment
56
concurrent validity is
sensitivity/specificity correlation coefficients
57
predictive validity is
correlation coefficients regression
58
what is sensitivity
proportion of people WITH the disease who have a positive test result LEFT COLUMN, TOP BOX out of 100
59
what is specificity
proportion of people WITHOUT the disease who have a negative test result RIGHT COLUMN, BOTTOM BOX out of 100
60
SpPin
test with HIGH specificity Positive helps rule a condition IN
61
SnNout
test with HIGH sensitivity Negative helps rule a condition OUT
62
PPV (positive predictive value)
patients with positive tests divided by all patients with positive test results
63
NPV (negative predictive value)
patients with negative tests divided by all patients with negative test results
64
Likelihood ratios
quantifies the test's ability to provide persuasive information NOT influenced by prevalence ranges from 0 to infinity
65
LR is 0 - 1
decreased probability of disease/condition of interest
66
LR = 1
no diagnostic value; null value
67
LR > 1
increased probability of disease/condition of interest farther from 1 = more likely
68
LR+ =
sensitivity/(1-specificity)
69
LR- =
(1-sensitivity)/specificity
70
diagnostic test is positive = what likelihood ratio
LR+
71
diagnostic test is negative = what likelihood ratio
LR-
72
LR+ : > 10
large and often conclusive shift
73
LR+ : 5 - 10
moderate shift
74
LR+ : 2 - 5
small: sometimes important
75
LR+ : 1 - 2
small: rarely important
76
LR- : < 0.1
large and often conclusive shift
77
LR- : 0.1 - 0.2
moderate shift
78
LR- : 0.5 - 0.2
small: sometimes important
79
LR- : 0.5 - 1
small: rarely important
80
case-control and cohort studies are...
intended to study risk factors association between disease and exposure
81
what is an example of a exposure?
cervical manipulation, smoking, running > 20 mi/wk
82
what is an example of disease or outcome?
cancer, stroke, knee OA
83
cohort studies subjects are selected based on
exposure
84
cohort studies are usually
prospective but can be retrospective!
85
case-control studies are selected based on
whether or not they have a disorder
86
case-control studies are usually
retrospective
87
Relative Risk is in
cohort studies (two 'o's')
88
Odds ratios are in
case-control studies (a and o, 'at odds')
89
RR or OR = 1
null value no association between an exposure and a disease if 1 is in CI then it is not significant
90
RR or OR > 1
positive association between an exposure and a disease exposure is considered to be harmful
91
RR or OR < 1
a negative association between an exposure and a disease exposure is protective
92
experimental event rate
% patients in experimental group with bad outcome
93
control event rate
% patients in control group with bad outcome
94
number needed to treat
how many patients you have to provide treatment to in order to prevent one bad outcome - the closer to 1 the better
95
Number Needed to Treat (NNT)
NNT = 1/ARR
96
if NNT = 1.0 it means
need to treat 1 patient to avoid one adverse outcome
97
if NNT = 10 it means
need to treat 10 patients to avoid one adverse outcome
98
do we want a small or big NNT?
SMALL
99
Number needed to harm (NNH)
NNH = 1/ARI (ARI = EER - CER)
100
if NNH is 1.0 it means
we need to treat 1 patient to cause an adverse outcome
101
if NNH is 10 it means
we would need to treat 10 patients to cause an adverse outcome
102
do we want a small or big NNH
BIG

Research Stats (4 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2025 Bold Learning Solutions. Terms and Conditions