Selection Flashcards

Question

Yu & Kuncel (2020)

Answer 1

Comparing random weighting schemes with expert judgments 2 general ways to go about making judgments in selection: (a) mechanical methods, statistically combine predictors (b) holistic/clinical methods, human expert, subjective judgment Mechanical methods outperform human judgment However, individuals may have concerns about using purely mechanical methods with no human input to determine weights (gets at concerns of face validity, applicant reactions to the process) Compared the following: Predictor weighting (regression) - random weighting = consistently applied, inconsistently applied - optimal regression weighting Found: experts outperformed inconsistent weights (meaning everything was completely random within and across candidates); consistent random weights (I.e., assigning a random weight to each predictor in the same way across all candidates) reliably outperformed expert judgment for hiring decisions across 3 datasets implication: experts do not make judgments completely inconsistently and are aware, to some extent, of what information is most valuable. However, their inconsistency in combining information does drastically damage their accuracy. authors suggest to develop decision making systems that help control consistency or to manage consistency by aggregating multiple expert judgments

Answer 2

Personnel selection: a review of ways to maximize validity, diversity, and the applicant experience review of personnel psychology: criterion related and content validity subgroup differences Adverse impact Bias Applicant reactions cost and time effects on unit and firm-level outcomes development decisions: use multiple selection procedures, add structure, contextualize selection procedures, minimize cognitive load, review selection procedures, limit faking opportunities, consider gamifying selection procedures, consider pre-test explanations/practice tests/coaching, share target KSAOs to applicants, allow for retesting weight prediction and criteria, consider AI, score banding future research: bias and opportunity in AI, selection across cultures, selection for gig work and telework

Answer 3

Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Mechanical / algorithmic = applying an algorithm or formula to each applicant’s scores Holistic / clinical = data are combined using judgment, insight, or intuition, rather than an algorithm or formula that is applied the same way for each decision Mechanical methods outperform holistic methods when combining criteria see also Yu & Kuncel (2020)

Answer 4

woo compares validity and reliability, bias, and fairness of grad school assessment methods (GRE, undergrad gpa, personal statements, resume, letters of rec, interviews); finds that other assessment methods than GRE have limited psychometric evidence and may be plagued by sociocognitive rater biases; suggests future steps Gomez et al. (2023) - commentary to Woo et al. Argue that Woo et al. engage in epistemic oppression by failing to cite germane research, erroneously do not evaluate the GRE's construct validity, and do not consider Black American's humanity by casually talking about George Floyd's murder Kuncel & Worrell (2023) - commentary to Woo et al. Extend upon next steps based on Woo et al.'s findings; invest in education for gifted and talented students at all levels of education; consider broader skills than overall cog ability like psychosocial skills and more *narrow* cognitive skills [specific]

Answer 5

summarizes points of divergence and convergence between authors; suggests next steps argues that Gomez and Boykin missed the main point of the argument (that other methods of graduate school admission assessment are likely even MORE unfair than the GRE and potentially are less valid and reliable) agree that additional predictors should also be used, but future research needs to evaluate psychometric properties of these other predictors

Answer 6

Why minority recruiting doesn’t often work, and what can be done about it conduct a number of simulations to suggest that targeted recruiting must explicitly model applicant qualifications as part of the recruiting process. important to focus on increasing minority applicant qualifications rather than just # of minority applicants to increase workplace diversity recruiting intervention x race x applicant qualifications model: - designed to enhance the probability of applying among more qualified minority applicants, more so than it enhances the probability of applying among majority applicants and also among less qualified minority applicants

Answer 7

Social science strategies for managing diversity: I-O opportunities to enhance inclusion white paper that lists individual (employee, manager) and organizational (hiring, div training, and perf management) suggestions for enhancing inclusion individuals (a) employees: provide support (listen), confront bias (intervene) (b) managers: beware of own bias blind spots, question assumptions, be a role model for inclusion organizations (a) hiring: job related measurement, consider full range of competencies, address stereotype threat (b) diversity training: include multiple group rather than single group focus; use multiple learning techniques; awareness AND behavioral goals; integrate into larger strategic diversity initiatives (c) performance management: provide more information, increase time, increase accountability

Answer 8

employment discrimination book; provides review of legal landscape key topics burden shifting = legal framework used in discrimination cases to guide how evidence is presented and evaluated, and who is responsible for providing evidence at each step (a) adverse impact burden shifting framework - legal steps adverse impact = neutral practices disproportionately harm protected group members, focuses on consequences of employment decisions, no intent required; typically assessed with applicant flow data - phase 1: challenger must demonstrate a particular employment practice causes discrimination in question - phase 2: company must demonstrate that challenged practice is job related and consistent with business necessity (data from job analysis typically used as evidence) -phase 3: plaintiff must prove then an equally valid, job related practice exists with less, or no, adverse impact (b) disparate treatment burden shifting framework - legal steps - disparate treatment = intentionally treating individuals differently based on their group membership in a protected group - phase 1: the prima facie (establishing foundation for case with evidence) case for disparate treatment requires sufficient direct or indirect evidence that plaintiffs belong to a protected class, are qualified for the position, are subject to negative employment decision, others not in the protected group are treated more favorably - phase 2: burden is shifted to employer, defense must articulate (not prove) that a legitimate reason exists for the alleged discriminatory practice - phase 3: plaintiff must prove by direct or indirect evidence that the organization's reason for their decision is a pretext for discrimination

Answer 9

chapter in adverse impact analysis textbook provides an overview of practical significance testing for adverse impact reviews data, statistical inference, type I and II errors, 1 or 2 tailed tests z-test, chi-square test, FET, LMP Assumptions: independence, large sample size, correct sampling model choosing among tests, multiple comparisons limitations of statistical significance tests, using confidence intervals

Answer 10

chapter in adverse impact analysis book provides an overview of statistical significance testing for adverse impact presents model for jointly considering practical AND statistical significance statistical significance --> statistical precision (e.g., confidence intervals) --> practical significance impact ratio, phi coefficient, odds ratio absolute difference in selection rates, h statistic, shortfall statistic when to use which effect size and how to interpret each

Answer 11

A theory of adverse impact proposes a theoretical model of adverse impact that includes factors like SES, exposure to test content, and exchange motivation performance irrelevant race related variance in tests contributes to adverse impact shows a 3-part venn diagram of test variance (red), race variance (yellow), and job performance variance (blue). what they are discussing is the overlap of test and race variance (orange) that does not include the overlap of job performance variance (brown) --> performance-irrelevant race-related variance in test scores = r^2PIRV racial subgroup differences are NOT uniform across cognitive subtests, with crystallized intelligence showing much larger racial differences than fluid intelligence subtests (via cognitive speed) proposes solutions for IOs to tackle: long term (change systems and structures of quality control and opportunity, but this is a huge undertaking and should be seen as a distal goal) medium term (test development) short term (predictor weighting, criterion weighting, recruiting)

Answer 12

Differential validity and differential prediction of cognitive ability tests: understanding test bias in the employment context Differential validity = show that observed and operational validities of cognitive ability tests are about 10-20% lower for African Americans and Hispanic Americans than for Whites differential prediction shows slope and intercept differences, whites have a slightly higher intercept than African Americans and Hispanic Americans, meaning that non-White job performance is typically OVERpredicted should be noted that most differential validity and prediction data is from the 1980s/earlier, many have used the general aptitude test battery (GATB) and do not provide adequate information on statistical artifacts future research needs to look at issues from indirect range restriction need research on what causes predictive bias... range restriction? psychometric characteristics of test or criterion? contextual influences (e.g., stereotype threat)? true differences in cognitive ability?

Answer 13

Human Resource Selection Good textbook cite regarding the following chapters: job performance concepts and measures, job analysis, legal issues in selection, recruitment of applicants human resource measurement in selection, reliability of selection measures, validity of selection procedures Application forms (biodata and training/experience evaluations), reference and social media checks, selection interviews, ability tests, personality assessment, simulation tests, test for CWBs, strategies for selection decision making

Answer 14

Breaugh, 2013 Theory suggests that providing realistic information about a job during recruitment should result in new employees having job expectations met, based on the assumption that an RJP allows people who do not perceive a strong person-job fit to withdraw from the process In turn, met expectations --> lower turnover and higher job satisfaction employer's recruitment actions can influence the interest of prospective job applicants in a job opening and the ability or individuals the org hires, their diversity, their job performance, and their retention

Answer 15

weights for each predictor are statistically determined to simultaneously optimize two (or more) criteria (e.g., job performance and diversity) as compared to only optimizing one criterion (job performance) when regression weights are estimated in the weights analysis De Corte et al. (2007) introduced this method in selection and highlighted that after using incumbent data, it's necessary to cross-validate with applicant data Song et al. (2017) found that when pareto-optimal weights were applied to an applicant sample, both expected diversity and job performance outcomes decreased (diversity and validity shrinkage) Sample size. Validity shrinkage and diversity shrinkage both decrease when sample size increases. (>100) In any case, pareto optimization still outperformed the unit weighted solution Rupp et al. (2020) provided a user friendly demonstration of how to use pareto-optimization suggest that the metrics need to be considered carefully in light of legal issues legal counsel should be sought out to examine the legal intricacies current applicants should not be used to determine pareto optimal weights (instead use a calibration sample a priori)

Answer 16

Related to Sackett et al (2022) well understood that the goal of validation of selection procedures used for predicting job performance = estimation of OPERATIONAL validity in an applicant sample, using a criterion measure that is free of measurement error (unrealistic) OBSERVED validity estimates are underestimates in the presence of range restriction and measurement error in the criterion; because of this, corrections for range restriction and measurement error were developed in early psychometrics, used to obtain 'better' estimates of operational validity meta-analysis = ideal approach - obtain estimate of operational validity for each study and cumulate the findings - in order for this to work correctly, we'd need to know the amount of criterion measurement error and range restriction for each study. - here, we use an artifact distribution: estimates of reliability for the criterion measure may be available for a subset of the collected studies. the mean and variance of the artifact distribution of reliability estimates are obtained --> correct the mean and variance of the full set of observed validity estimates using this distribution issues: assumes subset studies used to build the artifact distribution are randomly drawn from studies included in the meta; independence of artifacts if looking at multiple

Answer 17

Goldberg (2005) - Similarities between recruiters and applicants positively influence application attraction and selection decisions Avery & McKay (2006) - minority job seekers respond more positively to messages about diversity, including descriptions of diversity philosophies or diversity management policies, and often seek out information when making job choice decisions Williamson et al. (2008) - non-minority job seekers are more attracted to organizations that express a value for diversity, suggesting effectiveness of practice across demographic groups

Answer 18

cites to use: Van Iddekinge et al. (2016) - facebook profiles of college students and followed up with them in new jobs, SMA ratings were unrelated to supervisor's performance ratings Roulin and Levashina (2019) - LinkedIn in two studies. 1: Profiles that are longer, include a picture, and have more connections are rated more positively. Raters were fairly consistent. 2: itemized LinkedIn assessment is more effective than a global assessment. social media profiles are commonly looked at by potential employers to inform selection decisions. some studies demonstrate that SMA contains sufficient interrater reliability, convergent validity with traditional methods, and small yet significant criterion related validity (Roulin and Levashina 2019) SMA can lack both convergent and criterion related validity (Van Iddekinge et al. 2016) there may be potential for subgroup differences in SMA ratings, empirical research has supported the possibility that this leads to unfair discriminatory selection decisions (Van Iddekinge et al., 2016) applicants may respond negatively to SMA and believe it is an invasion of privacy

Answer 19

cites to use: Arthur et al. (2003) - provides AC dimensions: consideration/awareness of others, communication, drive, influencing others, organizing and planning, problem solving, stress tolerance Sackett & Dreher (1982) - exercise variance is way more prominent in ratings than dimension variance (bad) Kuncel & Sackett (2014) critiqued these findings and argued that overall dimension ratings, which are a composite measure of individual dimension ratings rather than exercises, explain the most variance in assessor ratings Sackett (2021) remarked that he originally got it wrong in 1982, as dimensions DO reliably and validly explain variance in ratings

Answer 20

Gatewood et al. (2019) Biodata is historical information that represents the applicants past behaviors and past experiences (at work, education, family, community involvement; separate from measures like personality or values). It is empirically developed and scaled in a way to maximize prediction Use of biodata has declined since the 70s because many of the components reflect the same aspects of personality tests and require considerable resources to develop (technical expertise and large sample) based on the idea that past behaviors are good predictors of future behaviors; an applicants previous experiences will predict how they perform on the job ideally, items would be developed based on data from a job analysis, have validity evidence, and have been screened for possible discrimination impact against protected groups concerns about accuracy / implications for validity and reliability

Answer 21

Automated video interview personality assessments: reliability, validity, and generalizability investigations organizations are increasingly using AVIs to screen applicants, but AVIs lack supporting evidence this paper developed AVIs that use verbal, para-verbal and nonverbal behaviors extracted from the video interviews ---> assessing big 5 results: AVI personality assessments exhibited stronger evidence of validity when trained on when trained on interviewer reports (compared to trained on self-reports from applicants) when cross-validated in other samples AVI personality assessments trained on interviewer reports [as outcome] had mixed evidence of reliability, exhibited consistent convergent and discriminant relations; the model used predictors that appear to be conceptually relevant to the focal traits and predicted academic outcomes little evidence of reliability or validity for AVIs trained from self-report applicant data

Answer 22

Reducing subgroup differences in personnel selection through the application of ML Main point: Collectively, the studies in this article illustrate that ML is unlikely to be able to resolve the issue of adverse impact, but it may assist in finding incremental improvements [personal comment: this gets at the idea that ML is likely to be most beneficial when it comes to automating selection procedures and text analysis stuff] STUDY 1: fairness aware ML algorithms (designed to optimize for predictive accuracy while limiting adverse impact of predictions) that statistically eliminate subgroup differences must create predictive bias *mathematically*, which may reduce validity and penalize high-scoring racial minorities STUDY 2: statistically removing subgroup differences (by oversampling higher performing minorities during ML training stage) only slightly reduced adverse impact ratios of resulting ML model, but also slightly reduced model accuracy (convergent validity in this study)

Answer 23

Improving measurement and prediction in personnel selection through the application of ML ML can score audio constructed responses with as much reliability and criterion-related validity as humans ML can be used to predict multiple outcomes simultaneously (e.g., productivity and turnover) but gains over traditional methods are small with highly structured data [may not be worth all the extra effort]

Answer 24

A simulation of the impacts of ML to combine psychometric employee selection system predictors on performance, adverse impact, and number of dropped predictors In a large scale set of simulations, ML does not greatly predict beyond traditional methods like regression, UNLESS samples are small relative to the number of parameters (n to k ratio is less than 3) - BUT there are many nuanced findings where ML may be better, such as when item-level models are used most consistently valuable improvement from adopting ML over traditional regression was from dropping predictors rather than by improving prediction the future of ML: - potential of ML for selection is unlikely to be realized in selection systems focusing on the combo of scales composites from preciously validated psychometric tests - INSTEAD, it will likely be realized in *unconventional design scenarios*, such as the use of individual items to make multiple trait inferences, or with novel data formats like text, image, audio, video, or behavioral traces

Answer 25

The Relative Importance of Task, Citizenship, and Counterproductive Performance to Global Ratings of Job Performance: A Policy-Capturing Approach 3 component model Task performance → activities formally recognized as part of a job Contextual performance → discretionary behaviors that contribute positively to organization’s environment Ex. organizational citizenship behaviors (OCBs) Counterproductive performance → voluntary bc harming well-being of organization or its members Ex. CWB, deviant policy-capturing approach to understand what is the relative importance to managers of task performance, OCB, and CWB in performance ratings policy capturing presents individuals with different scenarios to capture the weight that decision makers place on different cues within the scenarios Can help to understand the relative importance of a set of factors to a person/decision-maker/employee found that performance raters fell into 3 clusters: (1) weighting task performance highest, (2) CWB highest, or (3) equal and large weights to task performance and CWB (citizenship was generally given less weight across the board)

Answer 26

AI-based tools in selection: Considering the impact on applicants with disabilities Our research suggests that people with disabilities face many barriers to finding employment. Could technologies based on AI used in selection be one of those barriers? Or is it possible that AI technologies help people with disabilities find employment more easily? **Games should be designed with multiple modes of signaling to ensure they can be used by applicants with either visual or auditory disabilities, giving all applicants a full opportunity to perform well and thereby perceive the selection tool as providing fair decisions.** Tips Choose vendors and software carefully Provide options for requesting accommodations Make choices that work for all applicants Inform applicants of technology requirements Learn about algorithmic auditing Help applicants get the information they need Learn about the legal context for use of selection technologies Like any new approach to employee selection, the adoption of AI based tools needs to be carefully considered. On the one hand, some tools may offer features that some applicants with disabilities appreciate and that allows them to more fairly present their talents in the selection process. On the other hand, these tools may create additional barriers for other applicants with disabilities thereby leading to less inclusive employment opportunities

Answer 27

Conscientiousness assessments for people with attention-deficit/hyperactivity disorder: Measurement properties and potential issues FOR to be work-oriented helped level out some of the differences between ADHD and non ADHD individuals important to look at facets of conscientiousness, as not all facets are as important to all jobs (e.g., cautiousness, compared to achievement striving) [side note: leaders have all sorts of conscientiousness levels] hiring professionals should strive to use more job-relevant conscientiousness self-report measures in selection procedures, both in terms of item contextualization and the use of achievement-striving and dutifulness facets.

Answer 28

Examining personality testing in selection for neurodiverse individuals Area for Future Research 1: Are there differences as to how neurotypical and neurodivergent individuals react to personality testing in preemployment settings? Area for Future Research 2: What factors affect neurodivergent reactions to personality testing in preemployment settings? Area for Future Research 3: Is there evidence of test bias or measurement bias on personality tests used for selection across neurotypical and neurodivergent individuals? Area for Future Research 4: Is there evidence of AI on personality tests for neurodivergent individuals?

Selection Flashcards

(52 cards)