L3: Individual Differences & validation

Question 1

define interpersonal skills

Answer 1

skills related to social sensitivity, relationship building, working w others, listening, and communication

Question 2

define incremental validity

Answer 2

how much a new test or predictor adds value byeond what is already measured by existing predictors.
- empirical
aka it shows whether adding a new test (like SJTs) improves the ability to predict an outcome (job performance- compared to traditional tests (cognitive ability tests)

Question 3

why do SJTs predict long-term success?

this is important for its validity as a predictor for future success

Answer 3

• they assess soft skills like communication, empathy, and decision making
• these skills remain stable over time & are crucial for job performance

Question 4

define general procedural knowledge

Answer 4

the knowledge someboy has acquired about effective & ineffectives courses of trait related behaviour in situations like those described in the SJT
in LIevens study, general procedural knowledge reltaes to students procedural knowledge about (in)effective behaviour in interpersonal situations (with patients) as depicted in the SJT items

Question 5

what are the implications of LIevens study?

Answer 5

• SJT are valuable for student seleciton alongside cognitvie tests
• predicting post academic success is important, not just academic success
• long term validity of SJTs supports their use in professional admissions

Question 6

define reliability

Answer 6

consistency of a test or measurement. so whether the measure is:
- dependable
- stable
- consistent over time
- gives the truest picture of someone’s abilities/characteristics

Question 7

how do you test reliability?

Answer 7

• correlation coefficient methods
• test retest, coeff of stabiility
• parallel/alternate forms, coeff of equivalence
• coeff of stability & equivalence: combines different sources of error
• internal consistency (alpha)
• interrater reliability

Question 8

define correlation coefficient

Answer 8

degree of consistency/agreement between 2 sets of independently derived scores (r)

Question 9

define internal consistency

Answer 9

degree to which the items in one test are intercorrelated
split half: split test into 2 equivalent halves
chronbachs alpha: mean of all possible split half
most commonly used
ex: all math problems in a test should measure math ability

Question 10

define interrater reliabilty

Answer 10

agreement between raters on their rating of some dimensiion
ex: olympic judges giving similar scores for a gymnast

Question 11

what is parallel forms reliability? and example?

Answer 11

consistency across different versions of the test
ex: 2 versions of the SAT should give similar results

Question 12

what is test retest reliablity & an example

Answer 12

consistency over time
ex: taking an IQ test twice & getting similar scores

Question 13

what is a “good” reliability?

Answer 13

• depends on what you want to do with the scores
• the more important the decision (like life or death for reliability of fMRI scan) the more precise the measure needs to be. generally for research in social sciences >.7 and for selection decisiosn >.9

Question 14

what does validity look at?

Answer 14

whether a measure is:
- actually measuring the construct its supposed to measure &
- whether the decisions based on that measure are correct

Question 15

define content validity

Answer 15

The extent to which test items cover the intended
performance domain

Question 16

how do you measure content validity?

Answer 16

• rational examination of the manner in which the performance domain is sampled by the predictor
• SMEs: degree of agreement among them regarding how essential a particular item/test is (if more than 1/2 SMEs say an item is essential, that item has at least some content validity, CVI formula)
• difficult for more abstract constructs

Question 17

define criterion validity

Answer 17

• How well a test predicts real-world outcomes.
• empirical relationship between predictor & criterion (performance measure)
• subtypes: predictive & concurrent

Question 18

what is predictive criterion validity vs concurrent criterion validity?

Answer 18

predictive: measures how well a test predicts future success
(test scores collected before the criterion like college entrance exam)
concurrent: measures how well a test correlates w current performance (test scores & criterion data collected at the same time like job skills test)

Question 19

how do you measure predictive validity?

Answer 19

• use stats to demonstrate the actual relationship between predictors & criteria
• for ex: linear realtionship Y = a + bX + c
  1. measure candidates on predictor during selection (like conscientiousness)
  2. select candidates without using the results (need to validate first)
  3. obtain measurement of criterion performance later -> time period depends on type of job & how much training is needed (approx 6m)
  4. assess the strength of the relationship (stats)

Question 20

how do you measure concurrent validity?

Answer 20

both predictor & criterion data gathered from same employees (incumbents)
- cross sectional study

Question 21

what are some issues affecting validity?

Answer 21

• range restriction

Question 22

define meta analysis

Answer 22

combines multiple studies to determine overall validity

Question 23

what are the pros & cons of meta analysis

Answer 23

Pros: Resolves inconsistencies, finds general patterns.
Cons: Affected by publication bias (only strong results get published), study differences, and data quality.

Question 24

what is range restriction?

Answer 24

if only a specific group is tested, the validity appears lower than it really is
ex: if a study only includes top students, the entrance test might seem weak at predicting performance

Question 25

what are the 3 types of range restriction?

Answer 25

Direct range restriction – If only high-scoring applicants are selected, we underestimate the test’s true predictive power. Indirect range restriction – If a selection method is based on a different variable (e.g., motivation letters instead of grades). Natural attrition – If weak or strong performers quit early, reducing data availability.

Question 26

define construct validity

Answer 26

Whether a test actually measures the concept it’s supposed to measure. - subtypes: convergent & discriminant validity

Question 27

what is convergent validity & discriminant validity

Answer 27

- C: should be related to scores on other measures of the same construct - D: should be unrelated to scores on instruments that are not supposed to be measures of that construct (theoretically distinct)

Question 28

how can we generalize the findings between predictor & criterion from all the different studies?

Answer 28

meta analysis

Question 29

what are 2 examples of individual difference tests used in personnel psych to predict performance

Answer 29

- general mental ability (or cognitive ability): best predictor of job success, especially for complex jobs - the big 5: conscientiousness strong predictor of job success (but too much of it -> perfectionism, rigidity). job complexity affects how much personality matters

Question 30

why is intelligence, as measured in general mental ability (GMA), a strong predictor of performance?

Answer 30

higher GMA = faster learning, better problem solving

Question 31

define face validity

Answer 31

Whether a test looks like it measures what it should (even if it might not). - not technical but may affect applicants reactions to test (motivation, sense of fairness)

Question 32

why is measurement of individual differences important?

Answer 32

- there are many individual level differences (personality, motivation, skills etc) - we aim to describe this variability & understand it, explain it, and predict it - measurement is a tool that helps us achieve that goal - also important in HR

Question 33

what is the importance of measurement of indivdiual differences in HR?

Answer 33

- decision making: personnel decisions, evaluation of employees - HR specialists need to be able to: select & use psych measurements, interpret the results, communicate these results to others can affect careers of individuals, so essential for HR specialists to understand the application of psych measures. need to judge how effective they are.

Question 34

how do we assess adequacy of measurement of individual differences?

Answer 34

1. Are we actually measuring what we want to measure? 2. Are we measuring this consistently? 3. Does our measure of the predictor (e.g. personality) actually predict future performance?

Question 35

define test

Answer 35

any psych measurement instrument, technique or procedure that systematically mesures a sample of behaviour (eg interview, presentation, performance test etc)

Question 36

how do you develop a test?

Answer 36

1. item generation (steps 1-4) 2. pilot test (steps 5-7) 3. post pilot activities (steps 8-10)

Question 37

how does item generation go in test dev?

Answer 37

1. determine purpose (are u trying to evalute performance? select applications? doing research?) 2. define attribute (whats the content, so which constructs should/shouldnt be included) 3. develop measure plan (formatting, what will the anchors be) 4. write items (include some reverse items): double than what u need (if u want 10 item scale, dev 20 items), be specific & concrete simple, avoid negation

Question 38

how do you pilot test in test dev?

Answer 38

5. pilot test w representative sample 6. feedback from pilot test sample on test-perceptions & item clarity -> content validity 7. item analysis (distractor analysis, item difficulty, item discrimination) -

Question 39

what is distractor analysis?

Answer 39

frequency of each incorrect item (should be approx equal)

Question 40

what is item difficulty?

Answer 40

nr who answer correct/total (approx 5)

Question 41

what is item discrimination?

Answer 41

how well does the item serve to discriminate between better vs worse performers

Question 42

what are post pilot activities in test dev?

Answer 42

8. select items (good to have normal distribution- not too hard & not too easy) 9. determine reliability & validity 10. reivse & update items

Question 43

in what 3 ways can testing be systematic?

Answer 43

1. content: items are systematically chosen from behavioural domain to be measured 2. administration: standardized procedures so that each time the test is given, same directions, same recording of answers, w same time limits & as little distractions as possible 3. scoring: rules are specified in advance

Question 44

whats the point of systematic testing

Answer 44

to minimize contamination on test scores

Question 45

what are potential issues when designing a test?

Answer 45

- cost - face validity - contamination on test scores - interpretation of the test results by examiner

Question 46

how do you choose predictors?

Answer 46

- **what** should you measure?: job analysis give u clus about what variables are related to job success (ex: negotiation skills) - **where** & **how** do you find the measure youre looking for? : does it already exist or do i have to create it? only create when unavailable then run a pilot and assess the items (difficulty, discirmination etc)

Question 47

what is measurement in psychology?

Answer 47

the process of assigning numbers to objects or events according to rules - goal is to quantify individual differences in traits like intelligence, personality, job performance etc

Question 48

what are some examples of psych measures?

Answer 48

- cognitive ability tests - personality inventories (like big 5) - situational judgment tests (SJTs) - interviews

Question 49

what are the 4 different scales of measurement?

Answer 49

- nominal - ordinal - interval - ratio

Question 50

what is the nominal scale of measurement?

Answer 50

- categorizes data without any order ex: male vs female, eye colour etc

Question 51

what is the ordinal scale of measurement?

Answer 51

ranks data but does not indicate precise differences - ex: ranking in a race: you know 1st place is better than 2nd but u dont know if the time difference between them is equal

Question 52

what is the interval scale of measurement?

Answer 52

- equal intervals between values but no true zero (so also no 2x as much statements) - like temperature: difference between 20 and 30 c is same as between 30 and 40 c + but zero doesnt mean "no temperature" (theres no true zero); IQ scores

Question 53

what is the ratio scale of measurement?

Answer 53

like interval (equal intervals between values) but with an absolute zero ex: weight, height, age

Question 54

which type of data analysis can be conducted with nominal scales?

Answer 54

count, mode

Question 55

which type of data analysis can be conducted with ordinal scales?

Answer 55

median, percentiles, rank order correlation

Question 56

which type of data analysis can be conducted with interval scales?

Answer 56

means, sds, correlation, regression

Question 57

which type of data analysis can be conducted with ratio scales?

Answer 57

all math operations, including ratios & percentages

Question 58

what scale of measurement is often applied in psychology?

Answer 58

many psych traits are measured on interval scales, even tho they dont have a true zero

Question 59

what are the 7 concrete steps of selecting/creating the right measure | 7 steps

Answer 59

1. Determine the Measure’s Purpose – What are you trying to assess? 2. Define the Attribute – Is it cognitive ability, job performance, or personality? 3. Develop a Measure Plan – What type of questions will be used? 4. Write Items – Ensure the questions or tasks are clear and relevant. 5. Conduct a Pilot Study & Traditional Item Analysis – Test the measure before full implementation. 6. Use Item Response Theory (IRT) to analyze the questions – Identify good vs. bad items. 7. Revise & Update Items – Remove ineffective or biased questions.

Question 60

how do you use item response theory to conduct comprehensive item analyses?

Answer 60

IRT is used to evaluate individual test items based on difficulty, discrimination, and guessing

Question 61

define item difficulty

Answer 61

Determines how hard a question is.

Question 62

define item discrimination

Answer 62

Measures how well a question differentiates between high- and low-ability test takers.

Question 63

define guessing factor

Answer 63

Some multiple-choice items can be answered correctly just by luck so need to account for this

Question 64

what does generalizability theory (G theory) say?

Answer 64

it assess reliablity in mesurement by analyzing MULTIPLE sources of error to give a more complete picture of a tests accuracy ex: a test score might be influenced by time errors (morning vs evening test taking conditions), rating bias (some interviewers might be stricter), item variance (some questions might be harder than others) G theory helps separate these factors to improve measurement accuracy

Question 65

what is the scale coarseness problem? solution?

Answer 65

some measurement tools lack precision cause they dont offer enough response options ex: survey w only agree or disagree solution: use continuous or fine grained scales where possible

Question 66

what are some upcoming trends in psych measurement?

Answer 66

changes in how we measure individual differences - AI in testing - gamification - mobile & online assesments - new biometric & neuroscientific approaches (eye tracking, brain scans etc)

Question 67

what is the relationship between reliablity & validity?

Answer 67

a test must be reliable to be valid, but a reliable test is not necessarily valid ex: a rule that gives different lengths for the same object is unreliable, but if it consistenly gives wrong lengths, its reliable but not valid Validity≤ sqrt (Reliability) low reliability places a ceiling on valiidty. even if a test is somewhat valid, if it has low reliabliity, its usefulness is limited

Question 68

what is the point of cross validation?

Answer 68

helps ensure a test remains valid across different samples

Question 69

how do you conduct cross validation

Answer 69

1. conduct an initial validity study 2. apply the test in a new group & see if results remain the same 3. if the validity coefficient drops, this is called **shrinkage**

Question 70

why is cross validation important?

Answer 70

- prevent overfitting: test working well in one study but not in real world settings - ensures predictiv epower remains stable across different samples

Question 71

what is synthetic validation? when is it used?

Answer 71

using previous research & job analysis to infer test validiity used when local validation is not possible (small organisation often lacks neough data to conduct full validation studies) ex: instead of testing 1k salespeople, use existing data from other companies to support a tests validity