L6. factorial, divergent and convergent validity Flashcards
Factor analysis and what it does
- also known as data reduction analysis
- determines whether the items in a test actually measure a single dimension
1: Helps us clarify the number of factors within a set of items (or indicators)
2: Helps us determine the nature of the associations among the factors
3: Helps us determine which items are linked to which factor, which facilitates the interpretation of those factors
Factorial validity
- Factorial Validity is relevant to determining whether the scores of a measure correspond to the number and nature of dimensions of theorized dimensions that underlie the construct of interest.
- factorial analysis is used to help determine this
- pertains to internal structure of test
- loading is the way an item contributes to the variance of an attribute in a test study
dimensionality
uni, multi, multi
implications of dimensionality
1: Unidimensional test
Consists of items that all measure one, single attribute
2: Multidimensional test (uncorrelated) Consists of items that measure two or more dimensions that are unrelated to each other
3: Multidimensional test (correlated)
Consists of items that measure two or more dimensions that are correlated with each other (positively or negatively)
- scoring
- evaluation
- use of the test scores
multidimensional tests with correlated dimensions can produce a variety of scores
- subtest score: based on the items of a single dimension
- area scores: combining several subtest scores but not all
- total score, all of the subtests within an inventory
for reliability and validity you must conduct these analysis for all the score that you use from a test
- all subtests
- all area scores
- all total scores
eg intelligence scoring w multiple dimensions
component vs factor analysis
factor
- most people use factor these days although very similar
- the dimension causes the factors to correlate positively with each other
- includes uniqueness (variance not due to dimension)
- the loadings tend to be smaller but the correlation between factors larger in factor analysis
component
- items contribute variance to dimension
uncorrelated multidimensions
- some personality tests have uncorrelated dimensions
- for example the NEPO-PI R: the 5 factor model
- its inappropriate to calc a total score for all of them
Principle component analysis
scree plot
eigenvalues
commonality
component loadings
simple structure
component corrs
sample size requirements
Scree plot
- helps determine how many factors to extract
- if below ‘break’ line don’t investigate
- consists of eigenvalues (numerical representations of components with respect to their size) ordered from smallest to largest
eigenvalues
- In a separate table, Jamovi outputs the eigenvalues and the percentage of variance associated with each.
- Jamovi calls ‘eigenvalues’ SS Loadings (i.e., Sum of Squared Loadings).
commonality
- Uniqueness= the % of variance associated with a particular variable that was NOT included in the analysis
- 1 – Uniqueness value x 100 = percentage that that component of variance has been accounted for
Communality = the % of variance associated ‘ ‘ INCLUDED in the analysis
- Calculate by squaring the factor loadings associated with an item
- Want at least .04 or .09
.04 or greater for items- if lower would get it out of the analysis and then redo it
.09 or greater for subscales
- Items are less reliable then subscale - hence they have a lower communality expectation
component loadings
- useful if .2+ for items or .3+ for subscales
simple structure
- Simple structure refers to the degree to which an item (or scale or any variable included in the analysis) is associated with only one substantial loading on a single dimension (i.e., component) and negligible loadings on the remaining dimensions.
- to help achieve a simple structure use oblimin or promax in jamovi (only 2 components or more)
component corrs
- if 0 means they’re unrelated
sample size requirements
depends on:
1. amount of commonality associated w variables, the higher commonality the less sample size needed
2. the number of variables per factor, higher variables = less size
when conducting research we do which analysis first:
- relibility ICR’s
- factorial validity
- convergent and divergent vals
nomological network
- represents the pattern of effects between variables and constructs
- interconnections between constructs are collectively known as a nomological network
4 methods for evaluating convergent and divergent validity
1. focused associations
—> validity generalisation
2. sets of correlations
3. multitrait-multimethod matrices
4. quantifying construct validity
- Focused Associations
- Correlation between test scores + specified criterion are ‘make or break’
- E.g. correlation between SAT scores + first year Uni marks -if does not correlate, it’s a ‘break’ for that test, SAT helps uni’s select students, is standardised, similar to intelligence test but somewhat more focused on crystallised intelligence (vocabulary )
- For SAT to justifiably interpret as a valid indicator of uni performance, must actually correlate with uni marks. Found a .55 correlation between SAT and uni grades –>
- validity coefficient - if large, have more confidence in using the test for its intended purpose ( do not know exactly how large)
Validity Generalisation –
- validity genralisation studies evaluate the predictive utility of a test’s scores across a RANGE of SETINGS, times, situations
e.g. SAT + UNI grade correlation has been estimated based on 110,000 students from more than 25 uni’s
this addresses 3 things:
1. The average level of predictive V across studies
2. The degree of variability (SD in the validity coefficients) associated with V coefficients
3. Identify sources of systematic variability in the V coefficients
- In practice validity studies included fewer than 400 P’s, so assume that –> test should be valid in a diff scenario to where it was tested e.g. measure of, leadership may be useful for the manager of a bank but not for the manager of a construction industry
TB Example
- 25 studies of conscientiousness and job performances - have diff results as there a diff types jobs
- Sets of Correlations
- Once coefficients are estimated, placed in a table for visual inspection
- Subjective judgements- just eyeballing the pattern of coefficients and whether it is consistent with construct V in that case - Multitrait- Multimethod Matrices
- More SYSTEMATIC + trying to decompose trait - congruent variance and method variance (that is sometimes congruent and not)
- Tries to overcome the fact that a correlation between 2 scores may conflate/ combine 2 sources of variance:
1 = TRAIT VARIANCE (Good) - WANT this to be LARGER than method variance
2= METHOD variance (Bad)
- need at least 3 different methods
- NO guidelines for correlations + RARE, time consuming + expensive
- MT -HM > HT – MM = Monotrait herteromethod > heterotrait monomethod (MTMM) (HTMM)
- Lacks scientific approach to evaluating the patterns of correlations - Quantifying Construct Validity (QCV)
Western + Rosenthal (2003)
- researchers PREDICT the size correlations between their measure of interest + their selected criteria
* Then ESTIMATE the correlation between these = ‘actual correlations’
* THEN correlation between predicted + estimated are ESTIMATED has to be POSITIVE
Example - Measure of Social Motivation
* Persons desire to make positive impressions on other people
* Study has a group of professors ‘guess’ correlation between social motivation and 12 other self-report measures of personality e.g. agreeableness, need to belong
* Take the average of the estimates
* Get responses to questionnaires + estimate the empirical correlations + place in a table
Limitations of QCV
* Not sure how large correlation is sufficient
Factors that Affect a Validity Coefficient / Factors affecting Validity
* Are Pearson Correlations = affected by STATISTICAL + MEAUSREMENT issues
* Validity revolves around correlations
