Lecture 5: Deveopment of a study plan

Question 1

One-stage designs

Answer 1

decisions are made after all measures have been administered and thus, all information is available and considered

Question 2

Single screen

Answer 2

a singular test result justifies all ensuing decisions
(generally not advisable)

Question 3

Non-sequential test battery

Answer 3

Every applicant takes the entire test battery
- Decision is made after all info has been integrated according to the a priori formulated decision rules

Question 4

What are three strategies of a multi-stage desing?

Answer 4

1. Pre-reject strategy
2. Pre-select strategy
3. Completely sequential strategy

Question 5

Pre-reject strategy

Answer 5

A selection of measures is used, but not everyone takes all measures.
1) 1. Test - applicants are dropped who fail to reach a certain score (the rest of applicants take further test)
2) After 2. testing - final rejection or positive diagnosis

Question 6

Pre-select strategy

Answer 6

• applicants are already terminally accepted after the 1st test
• everybody else is subjected to futher testing

Question 7

Completely sequential strategy

Answer 7

combination of pre-rejection and pre-selection strategy
1) After 1. test applicants are catgeorized in three groups (pre-selected, pre-rejected and further tests)
2) For poeple taking further test: Further accepted (positive diagnosis) or further rejected.

Question 8

What should be considered in one-stage and multi-stage study plans?

Answer 8

1. Selection rate
2. Base rate
3. Criterion validity
4. Interactions of these variables.

Question 9

Selection rate

Answer 9

• the rate of applicants who are hired.
  (b+d)/(a+b+c+d)

Question 10

Hit rate

Answer 10

From the selected people, which are suited
b/(b+d)

Question 11

If a high criterion validity is given …

Answer 11

… the scatter plot is more narrow -> the error areas for a and d get smaller (good for the hit rate)

Question 12

What happens if you shift the selection rate? (case for higher and lower cutoff)

Answer 12

• It makes only one of the errors smaller
• Higher cutoff:Less suitable people get accepted, more unsuitable poeple get rejected.
• Lower cutoff: More suitable people get accepted, less unsitable get rejected.

Question 13

Base rate

Answer 13

the proportion of suitable candidates
(a+b)/(a+b+c+d)
- hits that can be expected upon random selection.

Question 14

Interaction between base rate and hit rate

Answer 14

high base rate (and good criterion validity) are helpful for achieving a high hit rate (suitable people are selected)

Question 15

Taylor-Russel Tables - what are they good for?

Answer 15

For estimating hit rates

Question 16

What can we take from the taylor-russel table?

Answer 16

• the higher the base rate, the higher the hit rate
• the less poeple selected, the better the hit rate
• VALIDITY is also very influential.

Question 17

General advise for planning the assessment

Answer 17

1. Order of tests
2. Formalia
3. Preparation for test instructors

Question 18

Order of tests (What is important)

Answer 18

• exhaustion of testees
• practice and transfer effects

Question 19

Formalia (What is important when testing?)

Answer 19

• ensure all organisational aspects are in order
• create friendly and open atmosphere
• give feedback during testing?

Question 20

Preparation for test instructors (what is important)

Answer 20

• often more than one person involved
• intensive preparation vital
• check for prejudice against testees
• control and regulate sympathy
• ensure to be open for all results at all times.

Question 21

Why do we often see in practice one-stage designs?

Answer 21

1. Due to time
2. Due to financial constraints

Question 22

What are tests of the one-stage design?

Answer 22

• single screen
• non-sequential test battery

Question 23

What are examples for a singular screen (1) and a non-sequential test battery (2) ?

Answer 23

1. Interview
2. An Assessment center.

Question 24

What would be a pre-rejection strategy - applied to the real world?

Answer 24

1. You send in your application documents (first poeple are dropped based on this)
2. Further selection or rejection based on a following interview.

Question 25

What are advances of a multi-stage design (1) vs. of a one-stage desing (2)

Answer 25

1. different measures are used
2. Less time, less money :)

Question 26

What does criterion validity have to do … with selection, base rate etc. ?

Answer 26

• when criterion validity is given, the plot becomes more narrow
• then the a and d error becomes smaller
• the hit rate becomes larger.

Question 27

What is one of the onliest reasons how both of the errors can become smaller?

Answer 27

-increasing criterion validity
- hence, errors become smaller and the hit rate become larger.

Question 28

What can reduce one of the errors?

Answer 28

• Shifting the selection rate

Question 29

What does the multi-stage plan tell us about selection rate (1) and base rate (2)
(in regard to validity)

Answer 29

1) Use a lower cutoff it test (scores) has a low reported validity - hence, less poeple selected.
2) If test (scores) valid, increase base rate.

Question 30

What is desirable for achieving a high hit rate?

Answer 30

• high base rate
• good criterion valdity