10. Evaluation (people/projects)

Question 1

personal evaluations

Answer 1

(individual level)
Recognise people’s achievements
Encourage future performance

Question 2

program evaluations

Answer 2

(systems level)
Identify how a policy or intervention is working
Identify areas for improvement/refinement
Decisions to continue/abandon program

Question 3

what is the process of person evaluation?

Answer 3

career planning (describe career and work goals) -> planning proposal (create goals expected outcomes) -> performance agreement (agree on performance indicators, targets, timeframe) -> performance evaluation (compare indicators with targets) -> feedback discussion (revise and update career goals) -> back to career planning

Question 4

Why do we evaluate people?

Answer 4

Assign rewards (e.g., bonuses)
Identify where help/training is needed
Identify when people can take on greater challenges and responsibilities (e.g., promotion)
Reinforce good behaviours
Extinguish bad behaviours

Question 5

what do we evaluate?

Answer 5

performance
effectiveness
prodctivity

Question 6

what do we evaluate in performance?

Answer 6

task-related behaviour

Question 7

what do we evaluate in effectiveness?

Answer 7

evaluation of standard of performance

Question 8

what do we evaluate in productivity?

Answer 8

cost of achieving level of effectiveness

time, money, burnout

Question 9

what is Bartram’s GREAT EIGHT competency model?

Answer 9

Leading/deciding
supporting/cooperating
interaction/presenting
analysing / interpreting
creating/conceptualising
organising/executing
adapting/coping
enterprising/performing

Question 10

who evaluates?

Answer 10

supervisor 
other peers
clients
subordinates
team members

goes in a circle - 360 degrees feedback

Question 11

how do we evaluate?

Answer 11

with objective measures and judgemental measures

Question 12

what are the objective measures of evaluation?

Answer 12

Quantitative measure of production, e.g., sales, outputs
Academics:
- Research (papers published; article views; number of citations, h-index, i-10 index…)
- Teaching (unit ratings; number of units taught)

Question 13

what are the judgemental measures of evaluation?

Answer 13

allows consideration of context factors not captured by objective measures
e.g., supervisor’s overall impression
rating compared to other employees, influenced by perceived difficulty of job or other contextual information

Question 14

Why do we abandon annual performance reviews?

Answer 14

due to measurement and psychological factors

Question 15

measurement factors for abandoning annual performance reviews

Answer 15

annual reviews are disconnected from timescales of work
multiple factors (and people) contribute to one’s performance

Question 16

psycholigical factors for abandoning annual performance reviews

Answer 16

Competition v. collaboration -
Performance rankings set up competitive mindsets
Doesn’t satisfy needs for learning and growth-
Fostered by more immediate feedback
More frequent communication = more informative feedback
Shifts from debating performance ratings to discussing development opportunities

Question 17

what should person evaluations be?

Answer 17

Developmental
Tied to organisational objectives
Specific
Sufficiently frequent

Question 18

how should person evaluations be sufficiently frequent?

Answer 18

by evaluating according to:
according to a person’s experience (less experience, more frequent)
task timeline (shorter task timelines, more frequent)
timely for decision-making

Question 19

types of evaluation

Answer 19

proactive
clarificative
interactive
monitoring
impact

Question 20

proactive evaluation

Answer 20

Assessing need for a program

If program required, assessing how it has been done elsewhere

Question 21

clarificative evaluation

Answer 21

Understand rationale and plausibility of a proposed program prior to commitment

Question 22

interactive evaluation

Answer 22

Inform stakeholders about issues and concerns with a planned program
Use knowledge and expertise to improve a planned program

Question 23

monitoring evaluation

Answer 23

Assessing program processes

Is process of implementing program on track?

Question 24

impact evaluatin

Answer 24

Identify the program/intervention outcomes

Inform decisions about future programs, e.g., whether to use approach in other settings

Question 25

the steps to program evaluation

Answer 25

engage stakeholders (understand key issues) -> describe program (goal & purpose expected effects) -> design evaluation (Method, users, agreements) -> gather evidence (indicators, sources, quality, quantity, practicality) -> justify conclusions (analyse, interpretation, recommendations) -> tweak or revise program (implement lessons) -> back to engage stakeholders

Question 26

What are the general standards of program evaluations

Answer 26

utility feasibility proprietary accuracy

Question 27

utility standard for program evaluations

Answer 27

Information will serve user needs Measures are directly related to program goals It is clear what the findings can show

Question 28

feasibility standard for program evaluations

Answer 28

Information is practical to collect Use of resources in proportion to the program/issue Methods require skills within organisation’s capacities

Question 29

proprietary standard for program evaluations

Answer 29

Information is obtained ethically Respect the rights and welfare of those involved Adheres to relevant laws and workplace policies

Question 30

accuracy standard for program evaluations

Answer 30

Information is accurate Reliability: indicators produce consistent responses Validity: Indicators fairly represent the construct/idea

Question 31

How does one describe the program?

Answer 31

What is the main purpose of your specific program/intervention? For example… - change behaviours directly - change attitudes - change social norms - change public awareness - Change knowledge Evaluation needs to be linked to your purpose

Question 32

what are the ideal features for evaluation measures?

Answer 32

1. Derived from the theory driving the intervention 2. Includes measures of… - the intervention itself (whether it was implemented correctly) - its impact (consequences) 3. Include both immediate and longer-term consequences 4. Valid -measures are clearly related to the theoretical construct 5. Reliable e.g., survey items are rated consistently – highly correlated with each other 6. Uses multiple methods to make up for weaknesses of any single type of measure

Question 33

what are the types of measures for gathering evidence?

Answer 33

``` Behaviours Behavioural intentions Attitudes Perceptions - Knowledge - Social norms -> Descriptive: reports of observed behaviours (what people actually do) -> Injunctive: shared expectations about what people should/should not do -Awareness ```

Question 34

what is the hierarchy for selecting intention, attitude and norm measures in gathering evidence?

Answer 34

1. USE existing scales when suitable 2. ADAPT existing scales when possible 3. DEVELOP new scales when there are no suitable/adaptable existing scales

Question 35

what are the issues in creating items for gathering evidence?

Answer 35

leading or biased questions eg. “how much do you dislike getting up early in the morning” double-barrelled questions (unless topics very closely aligned) eg. “please rate the extent to which you enjoy lectures and tutorials” use scales likely to match the detail people are likely to use eg. 5-, 7-, 9-point scales; percentages

Question 36

what are the direct measures for measuring norms in the theory of planned behaviour?

Answer 36

descriptive | injunctive

Question 37

descriptive measures of measuring norms

Answer 37

(what others do) many people who are important to me do… people in my life whose opinions I value do…

Question 38

injunctive measures of measuring norms

Answer 38

(what others expect of you) many people who are important to me think that I should… it is expected of me that I will… people in my life whose opinions I value would approve of…

Question 39

what are the belief-based measures for measuring norms in the theory of planned behaviour?

Answer 39

strength | motivation to comply

Question 40

strength measures of measuring norms

Answer 40

(measurement similar to injunctive) My family thinks that I should… usually tied to distinct reference groups

Question 41

motivation to comply measures of measuring norms

Answer 41

E.g., generally, how much do you want to do what your family thinks you should do?

Question 42

what is a common multi-method form of research?

Answer 42

Field studies

Question 43

what rare field studies?

Answer 43

a general term for methods emphasising observing and interacting with people in natural contexts

Question 44

what do field studies include?

Answer 44

observation inquiry breach

Question 45

what is observation in field studies?

Answer 45

observe/record | could be observing behaviour directly, or records of behaviour (eg. transcripts of conversations)

Question 46

what is inquiry in field studies?

Answer 46

interview people in relevant situations | gain knowledge about context as well as people

Question 47

what is breach in field studies?

Answer 47

record reactions to unexpected actions | e.g., ethnomethodology – reactions to norm-breaking