testing, validity, reliability

Question 1

measurement:

Answer 1

administering a test for purpose of score and information

Question 2

evaluation:

Answer 2

interpreting a score, place value on score, make a decision based on the results

Question 3

construct:

Answer 3

theoretical representation of a characteristic; a concept that can be defined but NOT directly measured
ex: motivation, teamwork

Question 4

test battery:

Answer 4

series of tests put together to answer a question

Question 5

examples of constructs in kinesiology

Answer 5

• body image
• balance
• fatigue
• power
• body composition
• metabolic health
  etc.

Question 6

5 aspects to consider for ethical practice:

Answer 6

1. fairness
2. privacy and confidentiality
3. data ownership and protection
4. safety
5. participant experience

Question 7

consequential validity:

Answer 7

group differences in test scores due to bias or measuring skills that aren’t part of construct

Question 8

construct underrepresentation:

Answer 8

construct defined to narrowly, resulting in lack of inclusion of all important components

Question 9

test bia:

Answer 9

meanings and implications of tests scores are different for a particular subgroup than the rest of test-takers

Question 10

4 ways to mitigate risk/ ensure client is fully informed:

Answer 10

1. emergency action plan
2. informed consent
3. pre-participation screening
4. MSK screening

Question 11

5 stages of testing

Answer 11

1. test selection
2. preparation
3. administration
4. data processing
5. decision-making and feedback

Question 12

5 important characteristics of test selection

Answer 12

reliability/ objectivity
validity
sensitivity
practicality
participant burden

Question 13

4 challenges in test selection

Answer 13

• test protocols evolve
• tests may be experimental, not routine
• field tests may be more valid for test construct but provide unreliable data
• subjects may not listen/ follow/ remember instructions

Question 14

4 main sources of measurement error:

Answer 14

1. test
2. test taker
3. test
4. built environement

Question 15

ways to reduce measurement error:

Answer 15

calibrate equipment
clear pretest instructions
control environment
use a valid and reliable test
train testers
provide warm up

Question 16

administration: typical testing sequence (8)

Answer 16

1. informed consent, preparticipation screening
2. non-fatiguing tests ex: body comp, flexibility
3. agility
4. max power and strength
5. sprint
6. local muscular endurance
7. fatiguing anaerobic power/ capacity
8. aerobic power/ capacity (after sufficient rest)

Question 17

why interpret data/ results

Answer 17

• to indicate client’s performance compared to norms/ set of standards
  multiple trials: can use best score or mean score

Question 18

criterion score:

Answer 18

measure used to indicate person’s ability:
- should be most accurate measure of construct
obtained by:
- participation
- known valid criterion
- expert judges

ex: pass/fail, proficient/ not

Question 19

reliability:

Answer 19

consistency/ repeatability of observation; degree to which repeated measurements of trait are reproducible under same conditions (test is free from measurement error)

Question 20

validity:

Answer 20

“most important test characteristic”
degree to which a test measures what it’s supposed to:
- reliability
- relevance
- appropriateness of score

Question 21

can a test be valid but not reliable?

Answer 21

no

Question 22

3 stage process of validity:

Answer 22

1. definitional (what should test measure)
2. confirmatory (how well does test measure construct)
3. theory-testing (do results match up w definition)

Question 23

logical validity:

Answer 23

does test measure construct of interest?
- assessed by interviews w experts in subject
*subjective, weak type of evidence
ex: single leg static balance

Question 24

construct validity:

Answer 24

degree to which test measures hypothetical construct, typically through comparison of results to behaviour
- not best form of evidence
- usually abstract ex: generosity
- statistical procedures to confirm theory
ex: would vert jump or stair climb test best measure power in volleyball

Question 25

3 types of construct validity:

Answer 25

1. known differences (test to see if actual differences exists b/n populations) 2. convergents (2+ tests measure same construct) and discriminant (2+ tests that measure different construct) 3. factor analysis

Question 26

criterion validity:

Answer 26

relationship b/n scores on test that measure construct and recognized standard/ criterion ex: run test and compare to gold standard - use 1+ statistical test - R > or = to 0.80 (+ or -)

Question 27

examples of suitable criterion measures

Answer 27

- expert judge ratings - tournament standings - predetermined criteria - future outcome

Question 28

criterion test:

Answer 28

DIRECT measure of fitness component ex: aerobic fitness, strength, etc. or INDIRECT protocol ex: body composition, anaerobic power

Question 29

concurrent criterion validity:

Answer 29

criterion is measured at ~the same as the alternate measure and scores are compared ex: those w/ fastest 1.5 mile run should have highest VO2 max scores

Question 30

predictive criterion validity:

Answer 30

new protocol is subsequently used to predict performance - criterion measured some time later ex: equation to predict VO2 max from 1.5 mile run should equal subject's actual VO2 max score - often uses regression analysis

Question 31

what does regression analysis allow you to predict?

Answer 31

a continuous dependent variable from a number of independent variables assumptions: - # of subjects should be 5:1 - data is normally distributed - linear relationship b/n variables

Question 32

can you have reliability without validity?

Answer 32

yes

Question 33

stability reliability:

Answer 33

measured w same instrument on 2 separate occasions - scores shouldn't really change - beware of maturation/ practice effects

Question 34

3 factors contributing to low stability reliability:

Answer 34

1. test taker (may be injured/ fatigured/ trained to improve) 2. test (different measuring instrument, issue w/ calibration) 3. tester (may be difficult to standardize if test requires jusdgement)

Question 35

internal-consistency reliability:

Answer 35

measures collected in single day/ session - at least 2 trials of test ex: skin folds - consistent rate of scoring by test takers throughout a test or trial to trial

Question 36

differences between stability and internal consistency reliability:

Answer 36

time factor: cognitive learning will occur between administrations changes in day to day performance: - stability = unaffected - internal consistency = major source or error coefficients are not comparable

Question 37

when is a test considered objective? objectivity depends on (3):

Answer 37

consistency b/n 2+ judgements of same performance, scorer's personal opinion and bias eliminated 1. competency of judges 2. clarity of scoring system 3. degree to which judge can assign scores

Question 38

when is ICC used and what is the required R value for excellent reliability? below avg reliability?

Answer 38

"intraclass correlation coefficient" used w small samples w/ data from 2+ sessions excellent: = or > 0.75 below avg = < 0.40

Question 39

coefficient of variation (Cv):

Answer 39

degree of variation b/n testing trials in participant's repeated measurements - used to compare variability of measurements

Question 40

how to calculate Cv? acceptable Cv %?

Answer 40

Cv = (standard deviation)/(mean) x 100 Cv = < 10% (sometimes 5?)

Question 41

when can we expect reliability?

Answer 41

- testing environment favourable for good performance - people are motivated, ready to be tested, informed, familiar - person administering test is trained & competent

Question 42

what might happen if same test is done on repeated days?

Answer 42

- familiarization and cognitive learning - low stability reliability * increased time b/n tests may produce more reliable response

Question 43

why and when do we calibrate?

Answer 43

to confirm accuracy of measurement at least every 6 months (unless otherwise indicated by manufacturer)

Question 44

What does a Bland-Altman plot show? can be used to estimate?

Answer 44

- shows level of agreement b/n 2 methods/ plots difference in measurements against mean of measurements - used to estimate reliability AND establish concurrent validity - closer dots = better correlation