Monitoring and evaluation Flashcards
(33 cards)
How does OECD define evaluation?
The systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results.
What are the aims of an evaluation?
To determine:
- the relevance and fulfilment of objectives
- development efficiency
- effectiveness
- impact
- sustainability
Why is an evaluation important?
- It is a useful tool to gather info to improve ongoing and future projects and the services offered to stakeholders and beneficiaries, as well as for generating ideas and innovation for future projects.
- Can provide evidence-based assessments of the worth or significance of a project
- Can provide project management with recommendations for improving the performance and for best use of resources
- Indicates not only that information on the performance of a particular project will be available, but also that the agency is committed to improve performance and maximize results
- Promotes transparency and accountability which, in turn, assist donors in their decision-making and about future project funding and can increase stakeholder buy-in
What types of evaluation are there and how are they defined?
They are defined by the timing of the evaluation - the purpose - who conducts it - and the methodology used.
Timing: ex-ante, real-time (early stages of implementation), midterm, final, ex-post.
Actors involved: internal vs external / Individually vs Jointly
What is a formative evaluation?
It is an evaluation conducted during implementation for the purposes of improving performance. It is intended to help Project Managers adjust and improve project implementation based on the evaluator’s findings and stakeholders’ suggestions and needs.
Real-time evaluations are linked to formative evaluations - similar to performance monitoring.
What is a summative evaluation?
It is conducted at the end of a project, for the benefit of stakeholders not directly involved in the management of the implementation (e.g. donors). It provides insights about the effectiveness of the project and gives the stakeholders the opportunity to use best practices identified during the evaluation.
What is an impact evaluation?
It attempts to determine the entire range of the project, including the positive and negative, primary and secondary long-term effects produced by the project - directly or indirectly, intended or unintended. Often ex-post.
What is an outcome evaluation?
An attempt to assess the extent to which a project was successful in producing change (e.g. if the indented institutional changes were produced by a project)
What is a process evaluation?
It assess how the results of a project were achieved and the different systems put in place for implementation (e.g. if participatory methods were used; if the organisation implementing the project responded effectively to change)
When are evaluations planned?
- During project development: the purpose, timing, intended use and methodology of the evaluation must be planned + a budget line for evaluation in the proposed budget, which means that consultants fees or staff costs, travel, data collection costs and data analysis costs must be included.
- Prior to conducting the evaluation.
Who can do an evaluation?
- An external individual or firm who is recruited to conduct the evaluation.
- An internal independent evaluator, who is a staff member from outside of the implementing office, not involved in the design or implementation of the project.
- Internal self-evaluation from staff involved with the project.
What does planning involve when it comes to external evaluators?
Estimating how many days the evaluator will require for:
- initial document and literature review
- travel
- initial meetings with the management team
- preparation of the inception report
- data collection and analysis
- presentation of initial findings
- preparation of the draft report
- revisions and finalization of the evaluation report
What is an inception report?
An inception report of an evaluation is prepared by an evaluator after an initial review of relevant documentation. It sets out the conceptual framework to be used in an evaluation, the key evaluation questions and methodology, including information on data sources and collection, sampling and key indicators. The inception report also includes a timeline for the evaluation project and drafts of data collection instruments.
What is important in the planning stage when using an internal independent evaluator?
- Travel expenses
- Salary
- Duration of assignment, and budget
What is the key planning tool prior to the implementation of the evaluation?
TOR - terms of reference.
It should build on the work done by the project developer during the initial planning stage and elaborate it in greater detail.
It should provide clear and detailed specifications of the roles and responsibilities of the parties involved in the evaluation.
In case of external evaluators, the TOR is important element of the contract.
What should the TOR include?
- Evaluation context: political, economic, social, environmental, legal context + project itself.
- Evaluation purpose: why is the evaluation conducted, why is it conducted at that time, who are the intended audience, how will it be used.
- Evaluation scope: What will the evaluation cover, which time period will be assessed, which components or phases, which geographical area, should something be excluded.
- Evaluation criteria: the criteria that will be addressed.
- Methodology: the method for data collection and analysis.
- Deliverables: the products that should be generated at various stages of the evaluation process + responsibility, e.g. inception report, initial presentation of findings, draft evaluation report, final evaluation report.
- Time schedule: the dates by which tasks have to be done or products have to be handed over + responsibility
- Role specification of the various parties involved
- Budget: details the resources available to conduct the evaluation + the cost of each item.
Cross-cutting themes should be addressed throughout the TOR.
Which evaluation criteria are widely adopted in development- or humanitarian oriented projects?
- Relevance: the extent to which a project’s objective and intended results remain valid and pertinent. Focused on intentions of the project rather than the actual achievements. Examines the relationships between inputs, activities, outputs, objectives + if the hypothesis and assumptions remain valid.
- Efficiency: how well human, physical and financial resources are used to undertake activities and how well these resources are converted into outputs. HOW the outputs are realized, not their effects.
- Effectiveness: assess the extent to which a project achieves its results. If activities have led to outputs which have led to outcomes which has realized objective. Considers if beneficiaries have used the services and products provided and if this has led to intended outcomes. All internal and external factors that influenced the achievement of outputs and outcomes.
- Impact: the positive, negative, primary, secondary long-term effects produced by a project, directly or indirectly, intentionally or unintentionally.
- Sustainability: The durability of the project’s results or the continuation of the projects benefits once external support ceases. Ex-post.
In humanitarian interventions:
connectedness: extent to which activities of short-term emergency nature are conducted in a manner that takes longer-term and interconnected problems into account
coverage: extent to which humanitarian assistance reaches all major populations affected and the impacts of humanitarian actions on different population and subgroups
coherence: extent to which there is a consistency across policies (security, developmental, trade etc), and the extent to which all policies take into account humanitarian and HR considerations.
What does the evaluator do?
- Document review of e.g. reports, deliverables, national strategies and plans, project documents, and any other documentation deemed relevant.
- He or she submits a draft inception report, or evaluation matrix, which provides a detailed description of their understanding of the TOR and how they will conduct the evaluation
- Data collection
- Analyse data
Quality control during an evaluation
- After receipt of the inception report; assess work, correct understanding
- During data collection; periodically monitor, check up on meetings and surveys and target numbers, ensure logistical arrangements
- During the presentation of initial findings; identify and address misinterpretations or gaps
- After receipt of the draft evaluation report; it should be ensured that the report describes the project and context, the eval. purpose and scope, the methodology and data sources and limitations, that findings are presented clearly and based on evidence, that conclusions and recommendations are stated clearly and based on evidence
How to use evaluations?
- Disseminate evaluation findings; learn from them, circulate reports widely
- Incorporate findings into practice; review to identify lessons learned and good practices
Stages in the implementation phase
- Project planning
- Project start-up
- Project execution
- Project closure
Planning for project implementation
Plan and monitor: activities results budget risks
Lack of planning and monitoring –> extensions, underspending or overspending, unachieved or underachieved results, poor reporting, negative evaluations, risk of relationships and reputation
Detailed work plan
A simple tool to plan and monitor a project’s activities. Developed already at the project proposal stage by the project developer. The PM then takes the same workplace and makes it more detailed. Should include not only the project activities identified in the results matrix but all activities and tasks related to all phases of implementation + a section for ongoing monitoring, evaluation and reporting activities + additional sub-activities/tasks, assigned to staff member, so detailed concrete levels of each activity.
Balance in level of detail.
So: Activity + sub-activities/tasks Responsible person Status Comments Time plan & sequence
What are activities or tasks that need to be completed before others can be undertaken called?
Sequential or linear.
