Guide For The Development of A Project Evaluation Plan
Guide For The Development of A Project Evaluation Plan
Guide For The Development of A Project Evaluation Plan
2005
The Office of Learning Technologies (OLT) would like to express its appreciation to Michel Gouault who greatly assisted the development of this guide.
TABLE OF CONTENTS
1. 2. 3. 4.
WHAT IS EVALUATION AND WHY EVALUATE A PROJECT?............................................................6 PLANNING A PROJECT EVALUATION .....................................................................................................7 WHY PERFORM AN EVALUATION?.............................................................................................................................7 WHO IS THE EVALUATION REPORT INTENDED FOR? ..................................................................................................7 WHAT WILL THE EVALUATION COVER? ....................................................................................................................8 HOW WILL THE EVALUATION BE PERFORMED? .........................................................................................................9 WHO WILL PERFORM THE EVALUATION? ................................................................................................................11 WHO DOES WHAT? ...............................................................................................................................................12 WHEN MUST THE EVALUATION REPORT(S) BE COMPLETED?...................................................................................12 HOW MUCH WILL IT COST? ...................................................................................................................................12
5. 6. 7. 8.
APPROVING THE EVALUATION PLAN...................................................................................................13 REPORTING ON EVALUATION RESULTS..............................................................................................14 DISSEMINATING EVALUATION RESULTS ............................................................................................15 A FEW PRACTICAL CONSIDERATIONS .................................................................................................15
1.
INTRODUCTION
This guide deals with the evaluation of pilot projects funded by the Office of Learning Technologies (OLT). These projects are varied and cover a wide range of activities integrated within an implementation schedule and a budget. These projects also vary according to their scale, their specificity and the scope of their goals and objectives. The content of this guide may be adapted to the conditions and the context of each project. Even though these characteristics have an effect on the evaluation, the basic principle of an evaluation remains the same: to ensure an optimal relationship between the goal to be achieved, at a reasonable cost, and the resources (human, technical and financial) that are used. By measuring the projects results, an evaluation allows to learn and to draw lessons in order to ensure that the goals and objectives of a project are achieved or to plan for other projects. This evaluation guide is not a recipe book. Its main purpose is to introduce the sponsors of projects submitted to the OLT to the evaluation of a project, and to help them design and develop an evaluation plan to be submitted as part of their pilot project proposal. Its content may be adapted to each project.
2.
The terminology used in project evaluation may sometimes seem complicated. So we invite you to familiarize yourself with the following concepts which we hope will lead to a better understanding of what project evaluation is and what its not. First of all, here is the definition of a few terms related to project proposals submitted to the OLT ... Project goal(s): a general statement of desired outcomes to be achieved over a specified period of time (the reasons for which the sponsor wishes to undertake the project). Project objective(s): the essential and long-term benefits towards which efforts are directed and for which outputs are to be produced. Output: products, activities or services stemming from the project and delivered to the projects target population. They are also the specific results obtained from the management of inputs. Input: activities and resources (human, material, financial) used to carry out activities, produce outputs and achieve results. Results: the consequences or changes directly attributed to the activities of the project. The results achieved may be measured with respect to the inputs, outputs, goals and objectives of the project.
Evaluation is the systematic collection and analysis of information on the actual performance of a project. Its aim is to analyze the relevance, progress, success and cost-effectiveness of the project. An evaluation compares planned results with the actual results of a project. It is a diagnostic tool. Monitoring is a continuing management exercise. Its aim is to supervise the accounting and administrative processes of a project. When implementing a project, monitoring deals almost exclusively with the conversion of inputs into outputs. This exercise will help evaluate if what was supposed to be done really is. Adjustments to the project are possible when monitoring is done throughout the project management life cycle. Auditing is an independent inspection of operations. It deals primarily with accounting and financial issues. Whereas auditing is periodic and independent of project management, monitoring is continuous and is incorporated into project management. Performance measures are indicators that provide information (either quantitative or qualitative) on the extent to which the results of a project have been achieved. Measuring is the operation which primarily consists in associating symbols to objects, events or people following precise rules. We often confuse evaluation with measures used to evaluate. Any activity which aims at interpreting results, or data obtained from measures, are part of an evaluation. To assure that the evaluation process leads to good decisionmaking, it must rest on correct and precise measures. Qualitative measuring aims at collecting data in order to describe and evaluate a situation or an activity. Qualitative measuring tends to be more anecdotal. Case studies are a good example. Quantitative measuring aims at collecting data in order to measure (through numbers and statistics) the range or the scope of an activity. Examples of quantitative measures include the number of end users in a project, their age or education level. Quantitative measures are often obtained through surveys. Efficiency refers to producing planned outputs within budgetary limits and established deadlines. For example: was the implementation of the project well managed? Were the inputs provided and managed at the least cost? Would there have been better ways of achieving the same results at a lower cost or in less time? What could be done to change or improve the projects efficiency and what would have been the financial effects? Effectiveness refers to achieving planned results and contributing to attain established goals and objectives. For example: to what extent was the delivery of inputs consistent with what was originally planned? To what extent were the projects planned outputs produced? To what extent were the projects objectives achieved? How would a greater production of outputs better achieve the projects goals and objectives? Have the projects goals been achieved? Impact refers to the intended or unintended, negative or positive, consequences of a project, some of which happen only some time after the end of the project. For example: what happened as a result of the implementation of the project? What were the consequences and the effects of the project on women, disadvantaged rural dwellers, and disabled persons targeted by the project? What were the unintended outcomes? What are the probable long-term consequences of the project? Why did the intended outcomes not happen? What lessons can be drawn from the project?
helps to incorporate the views of users and recipients into the conduct of the project; makes available information collected throughout the evaluation process (and not only upon completion of an activity or of the project), regularly informing stakeholders involved in the evaluation process, and in the project itself, of the results of the evaluation; allows verification of whether or not the project sponsor and partners are doing what they had originally set out to do; can offer ideas on how to plan for future activities and help other groups working in the same area through dissemination of evaluation results.
Even though an evaluation involves the intervention of an external observer (who is often perceived as someone who makes a judgment on the success or the failure of a project), its purpose is not and must not be to blame anyone. On the contrary, when an evaluation is performed in a spirit of cooperation, it helps to understand why and how certain activities worked well and others did not. It can also help to increase the knowledge and capacity of stakeholders regarding evaluation and to empower them by providing the tools to plan for future activities. Any rigorous evaluation must observe certain rules, but these need not be complicated. At the very least, there needs to be an evaluation plan, accompanied by an implementation schedule, which shows the type of information to be collected and why, the methods to be used to collect the required information and from whom this information will be collected. The evaluation plan must also clearly identify the roles and responsibilities of stakeholders involved in the process. It must also show the sequence of the different stages of the evaluation process.
4.
The preparation and planning phase of an evaluation helps to clarify the requirements and expectations regarding the evaluation, in order to capitalize on it as much as possible and to make it a useful and constructive exercise. Here are some basic questions to ask when planning a project evaluation:
Knowing who the evaluation report is intended for is a fundamental issue that must be addressed at the outset of the planning phase of an evaluation, since it will obviously affect the design, methodology and schedule of the evaluation. This phase should also include an overview of the means by which evaluation results will be disseminated. The evaluation report is first intended for the project sponsor who will use it for the management of their project. It should also be shared with the partners involved in the project. Furthermore, it will be included in the project documentation to be provided to OLT. So it is important to clearly understand the needs and requirements of a funder with respect to an evaluation. In the case of the OLT, these needs and requirements are specified in the guidelines for project proposals. A funder usually favours the following evaluation criteria: the satisfaction of end-users or project participants, as well as the efficiency of project activities. However, regardless of their evaluation approach, all funders want to know how the funds they provide are used. It is also important to remember that the project sponsor, his partners and the funders all want the project to succeed.
In addition to measuring the results of a projects intended outcomes: to what extent have the projects goals and objectives been met, to what extent do the impacts and the benefits of the project contribute to
the growth and development of the target communities and of the general population:, the evaluation must also address any results or outcomes that were not originally planned. A project may have unintended results that may be beneficial or undesirable, and it may also have positive and negative outcomes.
Confidentiality Confidentiality is an important aspect of any evaluation, especially the confidentiality of files that are used and of people who are interviewed. From the outset, those involved in the evaluation process should establish rules governing confidentiality and ensure that they are observed. Performance and success measures (indicators) The performance and success indicators of a project are, in some way, the purpose of a project. Once a project has been defined, it is assumed that its potential implementation will result in changes to a certain situation (e.g. online networking of a target community in order to provide learning opportunities that meet the communitys needs). More specifically, performance and success indicators are the measure of planned changes that result from the activities of a project. Thus, indicators are a series of assumptions that can be measured both quantitatively (e.g. the number of participants who have achieved their learning objectives) and qualitatively (e.g. the level of satisfaction of participants). Success indicators should be identified by the project sponsor, his partners and the external evaluator as soon as the projects goals and objectives have been established. When indicators are identified at the outset, they also help to clarify the projects goals and objectives: what initial data are available (benchmarks) that will later help to measure progress and the identification of outcomes obtained as a result of the projects implementation? What changes are planned with respect to the needs of the target community? What will be the final outcome of these changes? etc... The development of indicators allows the identification of the results expected if the project is implemented and if the conditions for its success are met. If they are correctly developed and applied, performance and success indicators will: determine what changes have occurred and their scope; compare changes that were originally planned with those that have actually occurred; assess the impact of those changes with respect to the goal of the project; examine the relationship between inputs and outputs and perform a cost-benefit analysis.
In order to be relevant and useful, indicators must be: Linked to available data: once the projects success indicators have been identified, one must ensure that the data required for these indicators can be collected at the appropriate time. The usefulness of an indicator is limited by the availability of data. If data are not available, it will be necessary to find or develop other indicators. Objectively verifiable: the assessment of project outcomes should be based on data collection methods that can produce the same results, regardless of who performs the evaluation. The main focus of an evaluation is not so much on the use of quantitative or qualitative measures, but rather on determining if selected indicators of change are in fact objectively verifiable, whether they are quantitative or qualitative.
10
Valid: they must measure results that relate to the project and its activities; Specific and independent: they must be appropriate for each level of the projects implementation. For example, an output indicator can not be used to prove that the projects goal has been achieved. Indicators vary according to the different levels of a project: (a) for inputs: the type of resources and expenses for each activity and the target dates; (b) for outputs: the order of magnitude of the results of outputs produced to achieve the objectives and the expected completion dates; (c) for project objectives: the objectively verifiable conditions that can demonstrate and measure important elements related to the achievement of objectives; and (d) for the project goal: the objectively verifiable measure that demonstrates the level of achievement of the goal. Measurable: the projects implementation levels must be specifically and realistically set out in terms of quantity, quality and time.
11
and financial management issues to the context of the project and the task to be performed? does the evaluator have the ability to plan and complete evaluation activities? will the evaluator be able to encourage stakeholders to participate efficiently in evaluation activities and to organize their participation? are there potential risks for conflicts of interest between the evaluator and the project? is the evaluators professional background acceptable to the project sponsor and his partners?
as a general rule, the cost of evaluation should be between 8% and 10% of the total project cost. In order to calculate evaluation costs, a complete list of all tasks related to the evaluation must be prepared. This is the project evaluation plan. This plan provides information necessary for developing the project evaluation budget and allows project sponsors to realistically calculate evaluation costs.
5.
The OLT requires that an evaluation plan be submitted along with project funding applications. The following is a sample plan that can be adapted to the specifics of a project and the selected methodology. Prepare a five-column table: Objectives/activities to be evaluated: these must be specific, measurable and linked to project objectives. Performance/success indicators: these must provide information (quantitative or qualitative) on the extent to which the project is achieving (or has achieved) its stated objectives. Performance/success measurement method: the measurement tools and the method (for example: face-to-face interviews, self-administered questionnaire) for collecting information to assess planned results. Responsibility: the person or persons responsible for completing the designated activity or task Schedule: the dates showing either the duration of the designated evaluation activity or the deadline for completing it.
Objectives / activities Performance / Success indicators : Performance / success measurement method Responsibility Schedule
1 2 etc.
13
The evaluation plan must also show: interim evaluations and end-of-project evaluation; how interim evaluations (preliminary results) will be used to make any required and appropriate adjustments to the project; and how interim and final results will be presented to the OLT.
6.
In order to be truly effective, an evaluation must include the lessons that can be drawn from a project. It must also allow decision-makers to determine if it is advisable to continue the project, to modify it, to abandon it, or to duplicate it elsewhere. Any evaluation report should include the following elements: at the front-end, a brief summary of the evaluation and an outline of recommendations; the objectives of the evaluation; the parameters of the evaluation (available means, constraints or restrictions); description of the methodology used; an evaluation of inputs; an evaluation of processes; an evaluation of outputs; an evaluation of the projects impact in relation to the projects goal and ultimate objective; a summary of project results (findings), including lessons drawn from the project and what could have been done differently; potential follow-ups (recommendations); and any relevant attachments (questionnaires used, analytical grids, evaluation plan, etc...). The evaluation plan must include the preparation of interim evaluation reports and a final report all submitted to OLT. It is important to remember that an evaluation is not a purely speculative process. It must be possible to draw practical conclusions and meaningful lessons from it.
14
7.
Lessons learned from a project evaluation are always useful, and not only for the project sponsor and their partners or for the OLT, because some of the conclusions drawn from the practical experience can also be applied to other similar projects. In any case, the results of an evaluation must be disseminated according to a dissemination plan developed in cooperation with project partners and the OLT.
8.
Each project to be evaluated has its own characteristics and each external evaluator has his or her own values. The sponsor of a project should consult an external evaluator as soon as a project proposal has been drafted. Even before developing an evaluation plan, the project sponsor and the external evaluator should meet together: (a) Ensure that the external evaluator has access to all information relevant to the planning and the conduct of the evaluation (documents, files, agreement with the OLT, activities and resources of the project sponsor and his partners, etc.), and that he or she is provided with the financial and human resources required for the evaluation. (b) Review the projects internal logic, including (a) planned measures of the projects inputs, outputs, goal and objectives, (b) causal links between those four levels, (c) external situations and factors that will affect those links, (d) specific and objectively verifiable progress indicators that will later help measure or verify the extent to which planned outputs were achieved and the projects objectives and ultimate goal were met. This point is fundamental and provides the external evaluator with a basic conceptual framework. (c) Review the availability of socio-economic data before the start of the project, since these outline the activities or conditions that the project is designed to change. These data need to be collected before the project is implemented. They are also essential in order to measure the progress achieved. Without this information, it is impossible to measure changes or effects resulting from the project. (d) Review and validate the schedule of planned activities set out in the projects action plan, which provides for the conversion of inputs into outputs and facilitates the monitoring and evaluation of progress achieved. (e) Ensure that the goals of the evaluation are clear and that they specify the nature, quantity and quality of the information to be gathered, so that the external evaluator knows who wants what and for what purpose.
(f) Identify other practical aspects: administrative or operational constraints, the evaluation budget,
15
environmental factors (material, political, social and cultural), as well as human resources that are available or needed.
(g) Anticipate how participants in the evaluation may perceive the external evaluators role and how
these perceptions could lead to concerns or a lack of confidence about the evaluation process. It is important to regularly inform those involved in the evaluation process and in the project about the results of periodic evaluations, in order to allow discussion of these results and of their significance for the ongoing work, the projects implementation and the achievement of the projects goals and objectives.
16
Resources:
Evaluation Manual UNESCO Internal Oversight Service (no date) A short introduction to the basics of evaluation, including a list of references. UNESCO, 7 Place de Fontenoy, 75352 Paris 07 SP, France http://www.unesco.org/ios/eng/evaluation/tools/outil_01e.htm galement disponible en franais sous le titre : Manuel de Rfrence http://www.unesco.org/ios/fre/evaluation/tools/outil_f.htm Guide to Project Evaluation: A Participatory Approach Population Health Directorate, Health Canada (1996) A guide offering a simple, comprehensive framework for project evaluation. It includes definitions of evaluation terms, annotated bibliography, worksheets and interview guides. This publication can be made available on computer diskette, large print, audio-cassette, or Braille upon request. National Clearinghouse on Family Violence, Family Violence Prevention Unit, Public Health Agency of Canada, Health Canada (Address Locator: 1907D1),Ottawa, ON K1A 1B4 http://www.phac-aspc.gc.ca/ncfv-cnivf/familyviolence/html/fvprojevaluation_e.html galement disponible en franais sous le titre : Guide d'valuation de projet : Une dmarche participative http://www.phac-aspc.gc.ca/ncfv-cnivf/violencefamiliale/html/fvprojevaluation_f.html Glossaries Lexicon for Results-Based Management and Accountability Treasury Board of Canada Secretariat (2002) A glossary of evaluation terms with French equivalents, many terms are defined in the governmental context. Centre of Excellence for Evaluation, Results-based Management Directorate, 300 Laurier West, Ottawa, ON K1A 0R5 http://www.tbs-sct.gc.ca/eval/pubs/RMAF-CGRR/rmafcgrr05_e.asp galement disponible en franais sous le titre : Lexique pour la gestion et la responsabilisation axes sur les resultants http://www.tbs-sct.gc.ca/eval/pubs/RMAF-CGRR/rmafcgrr05_f.asp Glossary of Key Terms in Evaluation and Results Based Management/ Glossaire des principaux termes relatifs lvaluation et la gestion axe sur les rsultats Organisation for Economic Co-operation and Development (2002) A tri-lingual glossary in English, French and Spanish 2 rue Andr Pascal, F-75775 Paris Cedex 16, France http://www.oecd.org/dataoecd/29/21/2754804.pdf
17