Professional Capability - Evaluating Impact Flashcards
What is Evaluation?
Multi level, systematic method for collecting, analyzing and interpreting data to confirm the effectiveness of talent development initiatives. Efficacy depends on demonstrating the value of the investment.
Evaluation is one way to document whether the investment achieved the desired outcomes. Programs are based on objectives specifying what they must accomplish and in what time frame
Purposes of Evaluation
1) determine business impact, cost benefit ratio, ROI for the solution
2) objectives - they were met and how well?
3) instructional strategies: assess the effectiveness and appropriateness of the content
4) performance assessment: reinforcing learning by using a test or similar way
5) facilitator feedback:
6) participants feedback:
7) Learning retention: assessing on the job environment
Ralph Tyler Goal Attainment Method
Earliest design process incorporating Evaluation of learning experiences based on objectives
Ralph Tyler Goal Attainment Method
Model poses 4 questions
- What objectives learners should achieve?
- What Learning activities will assist learners to achieve these objectives?
- How should the curriculum be organized?
- How should learners achievement be measured?
It helps to evaluate the success of curriculum
Measurement Process
Measure: a standard used to evaluate the degree of quality of the results of a solution
Measurement: a part of the research, is the process of quantifying assessment data and providing the necessary info required to make sound decisions about an issue or a situation. Measurements define or quantify specific attributes of an observation.
Measurement Process
TD professionals should identify the desired outcomes before designing an evaluation plan. To do this, they must gather, summarize and interpret the data generated by the assessment process to determine the root cause.
The root cause could be the process, lack of resources, lack of info, lack of motivation, health related issues or a need for KSA. Based on these results they will determine the best solutions and write objectives.
Measurement Process
Consideration for selecting the measurement process;
Nature of the solution
Characteristics of the learners
Focus on the outcome
Measurement Process
The evaluation process;
Use the assessment data to identify evaluation outcomes and goals
Develop an evaluation design and strategy
Select and construct measurement tools
Analyze and report data
Output Models
A program is a set of resources and activities toward one or more common goals typically under the direction of a single manager or management team.
Program evaluation is the systematic assessment of program results and if possible the assessment of how the program caused them.
Results may occur at several levels; Reaction to the program What was learned What was transferred to the job The impact on the organization
Output Models
Evaluation includes ongoing monitoring of programs as well as one time studies of program process or effects.
Program evaluation assess the effect of a learning program
Learning transfer evaluation measures the success of the learner’s ability to use when they have on the job
Formative vs Summative Evaluation
Formative Evaluation
It occurs throughout the design of any TD solution.
Purpose of the evaluation is to improve the draft learning program and increase the likelihood that it will achieve the objectives.
TD professionals should use a formative evaluation while the learning program is being developed and use this info to revise the learning program immediately to make it more effective.
During the evaluation, they should ensure the learning program is understandable, accurate, current and functional.
Formative evaluation could include but if not limited to pilot test, beta test, technical reviews with SMEs, production reviews and stakeholder reviews.
Formative vs Summative Evaluation
Summative Evaluation:
Occurs after a TD solution has been delivered.
This one focuses on the results or impact of the TD solutions to provide evidence about the value of a program.
Measures participants reactions, the effect on business goals, the initiative costs and the stakeholder’s expectations.
This evaluation measures the outcome and could include but not limited to standardized tests, reaction forms, stakeholder satisfaction surveys and the final ROI.
Kirkpatrick Evaluation- 4 Levels
1) Reaction: reaction sheets, (smile sheets) word of mouth feedback to the instructor, managers or other employees
Measures the degree to which participants find the program favorable, engaging and relevant to their jobs
Kirkpatrick Evaluation- 4 Levels
2) Learning: evaluates a learner’s mastery of the program content, knowledge or performance tests-determine participant’s ability, observation of skills or behavior by the learner
Measures the degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment
Kirkpatrick Evaluation- 4 Levels
3)Behavior: evaluates a learner’s transfer of knowledge and skills to the job and the extent to which they have applied what they learned. Comprehensive continuous performance monitoring based on behaviors, Manager assessment, self assessment, observation
Measures the degree to which participant apply what they learned during the program when they are back on the job
Kirkpatrick Evaluation- 4 Levels
4) Results: evaluate whether target outcomes occur, often viewed as a program’s organizational effect
This can be accomplished only if well defined targeted outcomes are identified prior to designing the program
Productivity measures, cost or expense, employee turnover, engagement
Measures the degree to which targeted outcomes occur as a result
Phillips ROI Methodology
6 types of collected data First level 4 is similar to Kirkpatrick 1) Reaction and planned action 2) Learning 3) Application and implementation 4) Business impact 5) ROI 6) Intangible measures
Phillips ROI Methodology
The methodology includes the use of;
1) standard values: those that are already accepted in the organization
2) historical costs: (ex:cost of an unexpected absence) represents the measures being converted have cost the organization in the past
3) input from internal or external experts on a particular measure
4) participant estimates
5) supervisor and manager estimates
6) link to other measures that have already been converted to monetary value, a technique used when placing value on customer value on customer or employee satisfaction
7) TD staff estimates
Phillips ROI Methodology
Isolating effects and accountability:
Includes a step which answers the question: How do we know our training is what caused the results?
Using a control group is one way to answer this question, not always feasible
Some other approaches; trend line analysis, forecasting methods, use of experts
The Brinkerhoff Case Method
The SCM (success case method) was developed to assess impact of organizational solutions. It involves identifying and examining the most and least successful cases in a program.
The Brinkerhoff Case Method
5 key steps of SCM;
1) focus and plan: a success case study
2) create an impact model: that defines what success should look like
3) design and implement; a survey to search for best and worst cases
4) interview and document; success cases
5) communicate; finding, conclusion and recommendation
SCM is a useful approach for documenting stories of impact that can be shared with stakeholders and used to develop an understanding of the factors that enhanced or impeded program success
Balanced Score Card
Measures effectiveness of a solution, initiative or practice from 4 perspectives
Balanced Score Card
4 perspectives
1) customer perspective : did the solution, initiative or practices meet the customer’s need or expectation?
2) innovator and learning perspective: did users gain the needed skills or knowledge
3) internal business perspective: did the solution, initiative or practice have an effect back on the job?
4) financial perspective: did the solution have a financial payoff?
Evaluation Approaches
Cost benefit analysis: measures monetary gains and losses
Culturally responsive evaluation: holistic framework for centering evaluation in culture
Developmental evaluation approaches: useful in complex or uncertain environments such as innovation, radical program redesign or crisis
