Short Answer Questions Flashcards
1
Q
According to the PBA/APS guidelines on the use and interpretation of psychological tests what technical information relating to a test should a psychologist take into account when interpreting the information derived from a test?
A
- understand nature of construct(s) underlying test score
- understand basic psychometric principles + procedures
- understand technical properties + limitations
- context + purpose + integrate test results with other important info about the individual
- importance of communicating CIs
- monitor and periodically review continuing effectiveness of tests you use
BASED ON ACCEPTED STANDARDS OF USE
- clear directions for admin/scoring
- info about purpose, SEM, validity + reliability
- test is valid for your purpose and sub-population
- acceptable validity + reliability in well-designed, independently replicated studies
- ensure it’s valid for your context/sub-group
- adequate reliability
- appropriate norm/reference group > need info about norm sample + procedure
2
Q
Apart from technical information relating to any tests used, what other considerations should a psychologist bear in mind when conducting a psychological assessment – according to the PBA/APS guidelines on the use and interpretation of psychological tests?
A
- all relevant info: language proficiency, medications/health problems, cultural background
- age, gender, language, background, social class (important for norms!)
- affect and mental state
- client’s previous experience with same/similar test(s)
- own competence: training, knowledge, expertise
- select words carefully in a report
- understand the limits of the test results
3
Q
According to Strauss and Smith (2009) discussion of test validity has been dominated by two main themes. Describe these themes and give examples of each approach.
A
- all validity can be thought of as contributing to construct validity
CONSTRUCT VALIDITY
- degree to which a test measure what it claims to measures
- theoretical understanding, latent variable or latent construct modelling
- rship to other relevant constructs + latent variables
- simultaneous process of measre and theory validation
- subsumes content and criterion validity
- EG: Personality 5 Factor Model - a statement of theory + (when used to describe the psych meaning in a test) a statement of construct validity of a test that measures 1+ of the five factors
CRITERION-RELATED
- practical evaluation of the usefulness of the test in the relevant population(s)
- eg. correlation b/w test score and criterion variable + t-tests to determine mean diff b/w criterion groups on test scores
- concurrent eg: depression test scores (BDI) vs. DSM consensus of MDD diagnosis
- predictive eg: poor scores on memory test predict future care needs in patient with diagnosis of alzhiemers
- once this is established, calc Se and Sp
4
Q
According to Strauss and Smith (2009) what contemporary research method is best suited to evaluating the multi-trait multi-method approach to construct validity?
A
- CFA
- analyses MTMM matrices
- allows for simultaneous evaluation of convergent and discriminant validity + contribution of method variance to observed rships
- allows direct comparison of alternative models of rships among constructs
- statistical tools are available to do this and are increasingly accessible to researchers
5
Q
In practice two members of the family of standard errors of measurement are most useful. What are they and how are they interpreted?
A
STANDARD ERROR OF PREDICTION
- to observe change in one individual’s score over time
- can produce CIs to predict future scores
STANDARD ERROR OF ESTIMATION
- use to get CIs around PTS
- to determine likely location of individual’s score on test scale + to compare observed score with population parameters
6
Q
Describe the Predicted True Score and explain its use with examples.
A
- PTS = theoretically correct score
- CIs placed around for an individual score
- most important to consider when Rxx (reliability) is relatively low
- provides the center of the distribution of scores that would be observed if an individual were tested many times with the same test (assuming no real change in scores)
- PTS will always be closer to mean than observed score (how much closer depends on reliability of test: more reliability = PTS closer to observed score)
EXAMPLE
- observed score 130
- test: M 100, SD 15
- if reliability 0.7 > PTS is 121
- if reliability 0.9 > PTS is 127 (closer to observed)
7
Q
Describe some reasons why knowledge of the reliability of the score derived from a test aids interpretation of that test.
A
- provides info on proportion of variance in observed score that is due to variance in true score, rather than error variance
- helps determine predicted stability of an individual’s score over time
- helps determine change (worsen/improve) in condition b/w one testing occasion and another
- determines the distro of one person’s score (less reliable = wider distro)
- while reliability does not imply validity, reliability does place a limit on the overall validity of a test
- known reliability of two variables allows us to estimate max validity correlation
8
Q
In applied research, correlations between a pair of tests scores are commonly interpreted in terms of convergent and discriminant validity. How does the reliability of either or both test score affect the interpretation of this kind of correlation?
A
- reliability can have an attenuating effect on validity coefficients
- validity correlations are diluted/weakened/underestimated by measurement error
- disattenuation provides for a more accurate estimate of the correlation between the parameters by accounting for this effect
- when considering convergent and discriminant validity coefficients, they should always be interpreted in their dis-attenuate form
- attenuation formula: corrects observed validity coefficient on basis of reliability
- corrected Rxy = Rxy / sqrt( Rxx x Ryy )
- can manipulate formula to determine max validity correlation b/w 2 variables
- assume corrected Rxy = 1
- known reliability of 2 variables allows us to estimate max validity correlation
9
Q
Describe the use of the Spearman-Brown prophecy formula for test development.
A
- relates psychometric reliability with test length
- SBPF allows a test developer to determine the reliability of a new test, as expanded by a factor of n
- tells you how the reliability of a test will change after expanding the test length
- produces a negatively accelerating growth curve with increasing test length (“diminishing returns” function)
- should use formula to determine optimal trade-offs b/w reliability and time devoted to data collection
10
Q
What do we mean by relying on “clinical judgement” to form professional opinions?
A
- using past clinical experience + intuitive learning experiences
- some say it is just anecdotal evidence
- drawing knowledge on basis of causal inferences, drawn from clinical samples you have experienced
- knowledge accumulated more or less informally over (many) years of clinical practice
- use of intuitive (generalizing from the particular) logic to guide clinical practice and de-emphasizing objectivity based on good test standardization and norms
- repudiating scientific methods in clinical research and clinical decision making
- no formal data analysis
- collected haphazardly
11
Q
Describe some of the common biases which may influence learning from clinical experience.
A
- representativeness heuristic: identify a client’s symptoms as conforming to or understanding of a disorder (pigeonholing) and then assuming that we understand the client and their prognosis
- confirmation bias: bc we are experts in normality, we make decisions in-line with abnormality
- availability heuristic: experiences that are vivid in memory are most likely to be invoked to explain clinical observations
- overconfidence, under-use of base rates etc.
- hindsight bias: tendency to view events as more predictable than they really are
- ontological-epistemological identity fallacy: human behaviour is complex and therefore our study and understanding of it must be complex (can’t be simple)
- Kreapelinian hierarchy: if it’s psychopathology, it can’t be brain disease + if it’s brain disease, it can’t be psychopathology (but overlap actually more common than not, but this comorbidity is dealt with very poorly in our diagnostic classification systems)
12
Q
Outline some of the recommended methods to improve the accuracy of clinical judgement.
A
- actively consider alternative outcomes
- minimise the role of memory
- incorporate base rates and prevalence rates into diagnostic thinking
- do your own diagnostic validity and treatment outcome research
- avoid using assessment/predictive techniques with unsatisfactory/unknown reliability + validity
- don’t assume that you understand a client > always obtain follow-up data
- develop models of diagnostic judgment and decision-making
- adopt an evidence-based approach > read literature for quality of evidence, interpret clinical importance of the evidence, apply high quality evidence to patients
- train psychs to be aware of own biases + limitations of their clinical judgment
- ALWAYS be cautious in clinical opinions
13
Q
Grove and Meehl’s (1996) describe the results of a meta-analysis of the empirical literature comparing clinical with statistical prediction. Briefly summarize the findings of the meta-analysis.
A
- meta-analysis of empirical literature comparing clinical with statistical prediction since 1920
- studies of predicting health-related phenomena (eg. diagnosis) or human behaviour
- 136 studies, 167 distinct comparisons b/w the two methods of prediction
- all had empirical outcomes of 1+ human-based prediction and 1+ statistical prediction
- 64 favoured stat prediction
- 64 showed equivalent accuracy
- 8 favoured clinician (but no reason why, random, not concentrated in any one predictive area or represent any one type of clinician, no obvious characteristics in common)
- according to “total evidence rule: most plausible explanation of these deviant findings is a combo of random sampling errors + clinicians’ informal advantage in being privy to more data than actuarial formulae
- less studies favouring clinicians than would be expected due to chance if clinician and statistical were equivalent
- clinician experience makes little/no diff in predictive accuracy BUT type of stat prediction did make a diff (best = weighted linear prediction)
- OVERALL: stat prediction methods superior to clinical prediction
14
Q
“I Cannot Use Actuarial Prediction Because the Available (Published or Unpublished) Code Books, Tables, and Regression Equations May Not
Apply to My Clinic Population”
Summarize Grove and Meehl’s (1996) responses to this objection.
A
- relies on notion that slightly non-optimal stat parameters due to validity generalisation would liquidate the superiority of the stat over clinical model > no evidence for this + does not make mathematical sense for cases where the actuarial method’s superiority is high
- if stat predicts something with 20% greater accuracy than clinicians in a no. of studies around the world + no reason to think that your patient is extremely unlike all other psych outpatients = improbable that clinicians are so superior that the stat method would reduce efficacy to level of clinicians
- cannot simply assume that if an actuarial formula works in several outpatient psych pops, and each one does better than the local clinicians or better, the formula will not work well in one’s own clinic
- all clinicians have diff training in diff settings with diff supervisors etc. > subject to same validity generalisation concern as stat methods, if not more so
- some think it unethical to apply a stat predictive system to their specific client without having validated it > but this is strange coming from people who rely on anecdotal evidence daily to make big decisions given all the literature says that anecdotal evidence is untrustworthy
- if re-validating predictor equation/table is deemed necessary, this is quick (just have to record hits/misses from someone else’s discriminant function) and then make a new predictor system if it doesn’t work well (quick, doesn’t need fancy maths)
15
Q
“Statistical Predictionists Aggregate, Whereas We Seek to Make Predictions for the Individual, so the Actuarial Figures Are Irrelevant in Dealing With the Unique Person”
Summarize Grove and Meehl’s (1996) responses to this objection.
A
- they say: actuarial figures give probs that are relevant for making predictions with an individual
SCENERIO
- medical illness > radical surgery
- you would ask if it work, how risk, % works, how many die etc
- doctor: why are you asking me about stats? you are a unique individual, nobody is exactly like you, do you want to be a mere statistic? what diff does % make anyway?
- physician cannot tell you beforehand into which group (success/failure) you will surely fall, but probs are still important for decisions
- claim cannot be made: stats give mere probabilities (average results/aggregates), whereas in dealing with the unique person one will know exactly what will befall this person
- could assign client to patient categories and get the probability of the event > resulting proportions would differ depending on which reference class was used
- there are as many probabilities as there are reference classes > choose the narrowest reference class with no. of cases big enough to provide stable relative frequencies available
- hardly any clinical decisions (using formal or informal procedures) that people claim to be absolutely certain
- clinicians focus on cases they could have saved an actuarial mistake + ignore the obvious point that any decision policy (unless infallible) will involve making some mistakes
- research should be done to show that clinicians to realise that, in general, they do not do as well as the actuarial method + then realise how they can improve upon the equation once in a while by clear-cut “broken-leg” countervailings > note there should be a high threshold for countervailings though
