Guide For The Development of A Project Evaluation Plan

Download as pdf or txt
Download as pdf or txt
You are on page 1of 17

GUIDE FOR THE DEVELOPMENT OF A PROJECT EVALUATION PLAN OFFICE OF LEARNING TECHNOLOGIES (OLT) HUMAN RESOURCES DEVELOPMENT CANADA

2005

The Office of Learning Technologies (OLT) would like to express its appreciation to Michel Gouault who greatly assisted the development of this guide.

TABLE OF CONTENTS

1. 2. 3. 4.

INTRODUCTION..............................................................................................................................................4 PROJECT EVALUATION CONCEPTS AND TERMINOLOGY ............................................................4

WHAT IS EVALUATION AND WHY EVALUATE A PROJECT?............................................................6 PLANNING A PROJECT EVALUATION .....................................................................................................7 WHY PERFORM AN EVALUATION?.............................................................................................................................7 WHO IS THE EVALUATION REPORT INTENDED FOR? ..................................................................................................7 WHAT WILL THE EVALUATION COVER? ....................................................................................................................8 HOW WILL THE EVALUATION BE PERFORMED? .........................................................................................................9 WHO WILL PERFORM THE EVALUATION? ................................................................................................................11 WHO DOES WHAT? ...............................................................................................................................................12 WHEN MUST THE EVALUATION REPORT(S) BE COMPLETED?...................................................................................12 HOW MUCH WILL IT COST? ...................................................................................................................................12

5. 6. 7. 8.

APPROVING THE EVALUATION PLAN...................................................................................................13 REPORTING ON EVALUATION RESULTS..............................................................................................14 DISSEMINATING EVALUATION RESULTS ............................................................................................15 A FEW PRACTICAL CONSIDERATIONS .................................................................................................15

1.

INTRODUCTION

This guide deals with the evaluation of pilot projects funded by the Office of Learning Technologies (OLT). These projects are varied and cover a wide range of activities integrated within an implementation schedule and a budget. These projects also vary according to their scale, their specificity and the scope of their goals and objectives. The content of this guide may be adapted to the conditions and the context of each project. Even though these characteristics have an effect on the evaluation, the basic principle of an evaluation remains the same: to ensure an optimal relationship between the goal to be achieved, at a reasonable cost, and the resources (human, technical and financial) that are used. By measuring the projects results, an evaluation allows to learn and to draw lessons in order to ensure that the goals and objectives of a project are achieved or to plan for other projects. This evaluation guide is not a recipe book. Its main purpose is to introduce the sponsors of projects submitted to the OLT to the evaluation of a project, and to help them design and develop an evaluation plan to be submitted as part of their pilot project proposal. Its content may be adapted to each project.

2.

PROJECT EVALUATION CONCEPTS AND TERMINOLOGY

The terminology used in project evaluation may sometimes seem complicated. So we invite you to familiarize yourself with the following concepts which we hope will lead to a better understanding of what project evaluation is and what its not. First of all, here is the definition of a few terms related to project proposals submitted to the OLT ... Project goal(s): a general statement of desired outcomes to be achieved over a specified period of time (the reasons for which the sponsor wishes to undertake the project). Project objective(s): the essential and long-term benefits towards which efforts are directed and for which outputs are to be produced. Output: products, activities or services stemming from the project and delivered to the projects target population. They are also the specific results obtained from the management of inputs. Input: activities and resources (human, material, financial) used to carry out activities, produce outputs and achieve results. Results: the consequences or changes directly attributed to the activities of the project. The results achieved may be measured with respect to the inputs, outputs, goals and objectives of the project.

.... and a few concepts ....

Evaluation is the systematic collection and analysis of information on the actual performance of a project. Its aim is to analyze the relevance, progress, success and cost-effectiveness of the project. An evaluation compares planned results with the actual results of a project. It is a diagnostic tool. Monitoring is a continuing management exercise. Its aim is to supervise the accounting and administrative processes of a project. When implementing a project, monitoring deals almost exclusively with the conversion of inputs into outputs. This exercise will help evaluate if what was supposed to be done really is. Adjustments to the project are possible when monitoring is done throughout the project management life cycle. Auditing is an independent inspection of operations. It deals primarily with accounting and financial issues. Whereas auditing is periodic and independent of project management, monitoring is continuous and is incorporated into project management. Performance measures are indicators that provide information (either quantitative or qualitative) on the extent to which the results of a project have been achieved. Measuring is the operation which primarily consists in associating symbols to objects, events or people following precise rules. We often confuse evaluation with measures used to evaluate. Any activity which aims at interpreting results, or data obtained from measures, are part of an evaluation. To assure that the evaluation process leads to good decisionmaking, it must rest on correct and precise measures. Qualitative measuring aims at collecting data in order to describe and evaluate a situation or an activity. Qualitative measuring tends to be more anecdotal. Case studies are a good example. Quantitative measuring aims at collecting data in order to measure (through numbers and statistics) the range or the scope of an activity. Examples of quantitative measures include the number of end users in a project, their age or education level. Quantitative measures are often obtained through surveys. Efficiency refers to producing planned outputs within budgetary limits and established deadlines. For example: was the implementation of the project well managed? Were the inputs provided and managed at the least cost? Would there have been better ways of achieving the same results at a lower cost or in less time? What could be done to change or improve the projects efficiency and what would have been the financial effects? Effectiveness refers to achieving planned results and contributing to attain established goals and objectives. For example: to what extent was the delivery of inputs consistent with what was originally planned? To what extent were the projects planned outputs produced? To what extent were the projects objectives achieved? How would a greater production of outputs better achieve the projects goals and objectives? Have the projects goals been achieved? Impact refers to the intended or unintended, negative or positive, consequences of a project, some of which happen only some time after the end of the project. For example: what happened as a result of the implementation of the project? What were the consequences and the effects of the project on women, disadvantaged rural dwellers, and disabled persons targeted by the project? What were the unintended outcomes? What are the probable long-term consequences of the project? Why did the intended outcomes not happen? What lessons can be drawn from the project?

3. WHAT IS EVALUATION AND WHY EVALUATE A PROJECT?


Evaluation
Evaluation is an important part of project management. It consists in measuring the effects of the project. Its goal is to learn from the evaluated project, in order to better understand it and to improve on it. Evaluation consists of: describing the flow of a project and its activities; identifying the progress achieved and the results obtained through the implementation of the project, by collecting appropriate data and submitting it to a comprehensive and systematic analysis; making a value judgment on the results identified and comparing them with established objectives and in accordance with predetermined criteria; and using the process to gain a better understanding of the project or of its completed activities, and drawing lessons that could potentially change ongoing activities in order to better align them with the projects goals. Evaluation allows project sponsors and their partners to become aware of: their perceptions of the goals and objectives of the project, its activities, its flow and the use of resources to bring it to fruition, the overall results achieved as well as the impact and outcomes of the overall project and its activities. Any project that has an implementation plan also has targets to be met and conditions by which to achieve them. The evaluation criteria are set by outlining these explicitly. On the basis of a set of inputs (resources), evaluation involves determining and proving whether planned results (outputs) have been achieved or not, and if these outputs have actually led to the attainment of the established goals and objectives of a project: If inputs.... then (on the condition that certain assumptions are met).... outputs ...... then (on the condition that....) ... achievement of the projects goal (objectives).

Why evaluate a project?


Project evaluation links the funding to results achieved and: offers a common and concrete view of activities to be implemented or currently underway; provides an opportunity to take a step back, to reflect on the conduct of activities that form part of a project and on the reasons for implementing them; helps to move a project forward by providing a clear and specific direction, by operationalizing the implementation plan, by allowing for a greater ability to make required changes and improve the delivery of a project, and by learning from the experience of implementing a project; helps to identify and understand the causes of a problem and find ways of resolving it; 6

helps to incorporate the views of users and recipients into the conduct of the project; makes available information collected throughout the evaluation process (and not only upon completion of an activity or of the project), regularly informing stakeholders involved in the evaluation process, and in the project itself, of the results of the evaluation; allows verification of whether or not the project sponsor and partners are doing what they had originally set out to do; can offer ideas on how to plan for future activities and help other groups working in the same area through dissemination of evaluation results.

Even though an evaluation involves the intervention of an external observer (who is often perceived as someone who makes a judgment on the success or the failure of a project), its purpose is not and must not be to blame anyone. On the contrary, when an evaluation is performed in a spirit of cooperation, it helps to understand why and how certain activities worked well and others did not. It can also help to increase the knowledge and capacity of stakeholders regarding evaluation and to empower them by providing the tools to plan for future activities. Any rigorous evaluation must observe certain rules, but these need not be complicated. At the very least, there needs to be an evaluation plan, accompanied by an implementation schedule, which shows the type of information to be collected and why, the methods to be used to collect the required information and from whom this information will be collected. The evaluation plan must also clearly identify the roles and responsibilities of stakeholders involved in the process. It must also show the sequence of the different stages of the evaluation process.

4.

PLANNING A PROJECT EVALUATION

The preparation and planning phase of an evaluation helps to clarify the requirements and expectations regarding the evaluation, in order to capitalize on it as much as possible and to make it a useful and constructive exercise. Here are some basic questions to ask when planning a project evaluation:

WHY perform an evaluation?


It is not easy to accurately evaluate a project when each stakeholder has a different perception of it and intends to use the results in his or her own way. When an external evaluator knows exactly who wants what and for what reasons, he or she can then perform an evaluation that meets the requirements of all stakeholders. The first thing to do, for the sponsors and the external evaluator, is to clearly identify the goals of a project evaluation. The first meeting with the external evaluator will help identify: a) the purpose of the evaluation (what we want to know and why), b) the criteria for determining the projects success, and c) the context in which the evaluation work is to be done.

WHO is the evaluation report intended for?


7

Knowing who the evaluation report is intended for is a fundamental issue that must be addressed at the outset of the planning phase of an evaluation, since it will obviously affect the design, methodology and schedule of the evaluation. This phase should also include an overview of the means by which evaluation results will be disseminated. The evaluation report is first intended for the project sponsor who will use it for the management of their project. It should also be shared with the partners involved in the project. Furthermore, it will be included in the project documentation to be provided to OLT. So it is important to clearly understand the needs and requirements of a funder with respect to an evaluation. In the case of the OLT, these needs and requirements are specified in the guidelines for project proposals. A funder usually favours the following evaluation criteria: the satisfaction of end-users or project participants, as well as the efficiency of project activities. However, regardless of their evaluation approach, all funders want to know how the funds they provide are used. It is also important to remember that the project sponsor, his partners and the funders all want the project to succeed.

WHAT will the evaluation cover?


This is the most important aspect of an evaluation. The goal of the evaluation must primarily refer to the framework of the projects action plan: what do we want to know ... and why? What type of information would be useful and why? This phase allows the project sponsor and the partners to further examine the work to be done and to determine how to measure its efficiency. Once the project stakeholders and the external evaluator have identified the purpose of the evaluation, the criteria for determining the projects success and the conditions under which the work is to be done, they have the necessary information to set the goals and objectives of the project evaluation. These goals and objectives are NOT the same as those of the project: the goals and objectives of a project specify the intended changes resulting from the implementation of the project, whereas the goals and objectives of an evaluation address the description and the measure of the results and outcomes of a project. Thus, the goals of an evaluation generally state what is expected from the project evaluation, and the objectives of an evaluation are specific and measurable activities to be accomplished by certain dates. Example for a given project: Project goal: Project objectives: Evaluation goal: Evaluation objectives: to provide a learning project to a target group. (a) to design the program, (b) to deliver the project and its services. to evaluate the relevance, efficiency and impact of the project. (a) to assess if the program meets the needs of the target group, (b) to measure the outcomes of the project on participants.

In addition to measuring the results of a projects intended outcomes: to what extent have the projects goals and objectives been met, to what extent do the impacts and the benefits of the project contribute to

the growth and development of the target communities and of the general population:, the evaluation must also address any results or outcomes that were not originally planned. A project may have unintended results that may be beneficial or undesirable, and it may also have positive and negative outcomes.

HOW will the evaluation be performed?


This is the method that the external evaluator will use and which is set out in the evaluation plan. The main quality of an evaluation plan is its flexibility. It must be realistic and take into account available financial and human resources as well as the availability of reliable data. The choice of data collection methods depends on a number of factors: what type of information is needed; who will it be collected from; who is the evaluation intended for; what is the purpose of the evaluation; what are the requirements for quantitative data; what data needs to be collected from users or participants? It is important to remember that evaluation consists of determining the value of results obtained and that quantitative data alone are not sufficient to determine if an activity or a project is good or bad. Any evaluation done during the course of a project should determine the status of the progress of implementation and of outputs obtained to date, as well as examine the projects ultimate goal and objectives, their merits and their relevance.

Confidentiality Confidentiality is an important aspect of any evaluation, especially the confidentiality of files that are used and of people who are interviewed. From the outset, those involved in the evaluation process should establish rules governing confidentiality and ensure that they are observed. Performance and success measures (indicators) The performance and success indicators of a project are, in some way, the purpose of a project. Once a project has been defined, it is assumed that its potential implementation will result in changes to a certain situation (e.g. online networking of a target community in order to provide learning opportunities that meet the communitys needs). More specifically, performance and success indicators are the measure of planned changes that result from the activities of a project. Thus, indicators are a series of assumptions that can be measured both quantitatively (e.g. the number of participants who have achieved their learning objectives) and qualitatively (e.g. the level of satisfaction of participants). Success indicators should be identified by the project sponsor, his partners and the external evaluator as soon as the projects goals and objectives have been established. When indicators are identified at the outset, they also help to clarify the projects goals and objectives: what initial data are available (benchmarks) that will later help to measure progress and the identification of outcomes obtained as a result of the projects implementation? What changes are planned with respect to the needs of the target community? What will be the final outcome of these changes? etc... The development of indicators allows the identification of the results expected if the project is implemented and if the conditions for its success are met. If they are correctly developed and applied, performance and success indicators will: determine what changes have occurred and their scope; compare changes that were originally planned with those that have actually occurred; assess the impact of those changes with respect to the goal of the project; examine the relationship between inputs and outputs and perform a cost-benefit analysis.

In order to be relevant and useful, indicators must be: Linked to available data: once the projects success indicators have been identified, one must ensure that the data required for these indicators can be collected at the appropriate time. The usefulness of an indicator is limited by the availability of data. If data are not available, it will be necessary to find or develop other indicators. Objectively verifiable: the assessment of project outcomes should be based on data collection methods that can produce the same results, regardless of who performs the evaluation. The main focus of an evaluation is not so much on the use of quantitative or qualitative measures, but rather on determining if selected indicators of change are in fact objectively verifiable, whether they are quantitative or qualitative.

10

Valid: they must measure results that relate to the project and its activities; Specific and independent: they must be appropriate for each level of the projects implementation. For example, an output indicator can not be used to prove that the projects goal has been achieved. Indicators vary according to the different levels of a project: (a) for inputs: the type of resources and expenses for each activity and the target dates; (b) for outputs: the order of magnitude of the results of outputs produced to achieve the objectives and the expected completion dates; (c) for project objectives: the objectively verifiable conditions that can demonstrate and measure important elements related to the achievement of objectives; and (d) for the project goal: the objectively verifiable measure that demonstrates the level of achievement of the goal. Measurable: the projects implementation levels must be specifically and realistically set out in terms of quantity, quality and time.

WHO will perform the evaluation?


The overall responsibility for the evaluation of a project must be assigned to an external evaluator, even though those directly involved in the development and implementation of a project play an important role in the evaluation process, with respect to project proposals submitted to the OLT. The external evaluator must not be connected in any way to the project (external consultant from a university, a research group or a consulting firm) for the following reasons: the task of evaluation requires time, constant attention and special skills; calling on an external evaluator who is not linked to the project sponsor or his partners is a way of ensuring neutrality, objectivity and professionalism. It also meets accountability requirements with respect to the use of OLT funds; an external evaluator will challenge the project sponsor and his partners to think critically about the development and implementation of the project, lessons to be learned and potential changes to the project. Note: Contracts for any one contractor in excess of $25,000 must be awarded through a competitive process involving at least three bids. If the external evaluator is involved with the project sponsor and his partners in coordinating data collection and analysis, then the quality of the evaluation will depend largely on the participation of project stakeholders and on the integration of the evaluation process into their ongoing activities. However, the external evaluator is responsible for supervising the evaluation, both during the course of the project and after its completion, according to the schedule set out in the project implementation plan. Here are a few questions to ask when choosing an external evaluator: Based on their previous experience, will the evaluator be able to adapt his knowledge of technical

11

and financial management issues to the context of the project and the task to be performed? does the evaluator have the ability to plan and complete evaluation activities? will the evaluator be able to encourage stakeholders to participate efficiently in evaluation activities and to organize their participation? are there potential risks for conflicts of interest between the evaluator and the project? is the evaluators professional background acceptable to the project sponsor and his partners?

WHO does WHAT?


It is important to define the roles and responsibilities of stakeholders in the project evaluation, with respect to both the external evaluator and others involved in the process: who will refine the evaluation plan? Who will collect the required data and supervise the collection? Who will analyze the data? Who will coordinate available resources (budget and expertise, time and organizational constraints)? Who will report on the results? Except for the external evaluator, the participation of stakeholders in the project evaluation should correspond to their level of involvement in the project. It could be useful to establish an evaluation committee in order to ensure the smooth conduct of the evaluation process, to motivate stakeholders and to provide a feedback process.

WHEN must the evaluation report(s) be completed?


Evaluation is an ongoing process which should ideally be initiated during the development and planning phase of a project. In any case, however, it is essential that the evaluation process begin as soon as the project goals and objectives have been established. Evaluation can be done either during the course of a project (e.g. when one of the critical phases has been completed) or after the completion of a project (evaluation of achievements and outcomes and of the projects impact). The external evaluator will need to estimate the time required for each phase of the project evaluation, in order to determine the duration of the evaluation process and to ensure that deadlines for reporting to the OLT are met. The contribution agreement indicates when the evaluation report must be provided to OLT. The project sponsor should consult an external evaluator as soon as a project proposal has been drafted.

HOW MUCH will it cost?


A funding application to the OLT must include an evaluation budget. The costs of all tasks deemed necessary to evaluate the project need to be estimated, including the external evaluators fees, wages paid to project staff for evaluation activities, travel and meeting expenses, office equipment, stationery, typing services, photocopying, documentation and printing, as well as other requirements specific to each project. Budget estimates for a project evaluation vary according to the nature and complexity of each project, but, 12

as a general rule, the cost of evaluation should be between 8% and 10% of the total project cost. In order to calculate evaluation costs, a complete list of all tasks related to the evaluation must be prepared. This is the project evaluation plan. This plan provides information necessary for developing the project evaluation budget and allows project sponsors to realistically calculate evaluation costs.

5.

APPROVING THE EVALUATION PLAN

The OLT requires that an evaluation plan be submitted along with project funding applications. The following is a sample plan that can be adapted to the specifics of a project and the selected methodology. Prepare a five-column table: Objectives/activities to be evaluated: these must be specific, measurable and linked to project objectives. Performance/success indicators: these must provide information (quantitative or qualitative) on the extent to which the project is achieving (or has achieved) its stated objectives. Performance/success measurement method: the measurement tools and the method (for example: face-to-face interviews, self-administered questionnaire) for collecting information to assess planned results. Responsibility: the person or persons responsible for completing the designated activity or task Schedule: the dates showing either the duration of the designated evaluation activity or the deadline for completing it.
Objectives / activities Performance / Success indicators : Performance / success measurement method Responsibility Schedule

1 2 etc.

13

The evaluation plan must also show: interim evaluations and end-of-project evaluation; how interim evaluations (preliminary results) will be used to make any required and appropriate adjustments to the project; and how interim and final results will be presented to the OLT.

6.

REPORTING ON EVALUATION RESULTS

In order to be truly effective, an evaluation must include the lessons that can be drawn from a project. It must also allow decision-makers to determine if it is advisable to continue the project, to modify it, to abandon it, or to duplicate it elsewhere. Any evaluation report should include the following elements: at the front-end, a brief summary of the evaluation and an outline of recommendations; the objectives of the evaluation; the parameters of the evaluation (available means, constraints or restrictions); description of the methodology used; an evaluation of inputs; an evaluation of processes; an evaluation of outputs; an evaluation of the projects impact in relation to the projects goal and ultimate objective; a summary of project results (findings), including lessons drawn from the project and what could have been done differently; potential follow-ups (recommendations); and any relevant attachments (questionnaires used, analytical grids, evaluation plan, etc...). The evaluation plan must include the preparation of interim evaluation reports and a final report all submitted to OLT. It is important to remember that an evaluation is not a purely speculative process. It must be possible to draw practical conclusions and meaningful lessons from it.

14

7.

DISSEMINATING EVALUATION RESULTS

Lessons learned from a project evaluation are always useful, and not only for the project sponsor and their partners or for the OLT, because some of the conclusions drawn from the practical experience can also be applied to other similar projects. In any case, the results of an evaluation must be disseminated according to a dissemination plan developed in cooperation with project partners and the OLT.

8.

A FEW PRACTICAL CONSIDERATIONS

Each project to be evaluated has its own characteristics and each external evaluator has his or her own values. The sponsor of a project should consult an external evaluator as soon as a project proposal has been drafted. Even before developing an evaluation plan, the project sponsor and the external evaluator should meet together: (a) Ensure that the external evaluator has access to all information relevant to the planning and the conduct of the evaluation (documents, files, agreement with the OLT, activities and resources of the project sponsor and his partners, etc.), and that he or she is provided with the financial and human resources required for the evaluation. (b) Review the projects internal logic, including (a) planned measures of the projects inputs, outputs, goal and objectives, (b) causal links between those four levels, (c) external situations and factors that will affect those links, (d) specific and objectively verifiable progress indicators that will later help measure or verify the extent to which planned outputs were achieved and the projects objectives and ultimate goal were met. This point is fundamental and provides the external evaluator with a basic conceptual framework. (c) Review the availability of socio-economic data before the start of the project, since these outline the activities or conditions that the project is designed to change. These data need to be collected before the project is implemented. They are also essential in order to measure the progress achieved. Without this information, it is impossible to measure changes or effects resulting from the project. (d) Review and validate the schedule of planned activities set out in the projects action plan, which provides for the conversion of inputs into outputs and facilitates the monitoring and evaluation of progress achieved. (e) Ensure that the goals of the evaluation are clear and that they specify the nature, quantity and quality of the information to be gathered, so that the external evaluator knows who wants what and for what purpose.
(f) Identify other practical aspects: administrative or operational constraints, the evaluation budget,

15

environmental factors (material, political, social and cultural), as well as human resources that are available or needed.
(g) Anticipate how participants in the evaluation may perceive the external evaluators role and how

these perceptions could lead to concerns or a lack of confidence about the evaluation process. It is important to regularly inform those involved in the evaluation process and in the project about the results of periodic evaluations, in order to allow discussion of these results and of their significance for the ongoing work, the projects implementation and the achievement of the projects goals and objectives.

We wish you a successful evaluation.

16

Resources:
Evaluation Manual UNESCO Internal Oversight Service (no date) A short introduction to the basics of evaluation, including a list of references. UNESCO, 7 Place de Fontenoy, 75352 Paris 07 SP, France http://www.unesco.org/ios/eng/evaluation/tools/outil_01e.htm galement disponible en franais sous le titre : Manuel de Rfrence http://www.unesco.org/ios/fre/evaluation/tools/outil_f.htm Guide to Project Evaluation: A Participatory Approach Population Health Directorate, Health Canada (1996) A guide offering a simple, comprehensive framework for project evaluation. It includes definitions of evaluation terms, annotated bibliography, worksheets and interview guides. This publication can be made available on computer diskette, large print, audio-cassette, or Braille upon request. National Clearinghouse on Family Violence, Family Violence Prevention Unit, Public Health Agency of Canada, Health Canada (Address Locator: 1907D1),Ottawa, ON K1A 1B4 http://www.phac-aspc.gc.ca/ncfv-cnivf/familyviolence/html/fvprojevaluation_e.html galement disponible en franais sous le titre : Guide d'valuation de projet : Une dmarche participative http://www.phac-aspc.gc.ca/ncfv-cnivf/violencefamiliale/html/fvprojevaluation_f.html Glossaries Lexicon for Results-Based Management and Accountability Treasury Board of Canada Secretariat (2002) A glossary of evaluation terms with French equivalents, many terms are defined in the governmental context. Centre of Excellence for Evaluation, Results-based Management Directorate, 300 Laurier West, Ottawa, ON K1A 0R5 http://www.tbs-sct.gc.ca/eval/pubs/RMAF-CGRR/rmafcgrr05_e.asp galement disponible en franais sous le titre : Lexique pour la gestion et la responsabilisation axes sur les resultants http://www.tbs-sct.gc.ca/eval/pubs/RMAF-CGRR/rmafcgrr05_f.asp Glossary of Key Terms in Evaluation and Results Based Management/ Glossaire des principaux termes relatifs lvaluation et la gestion axe sur les rsultats Organisation for Economic Co-operation and Development (2002) A tri-lingual glossary in English, French and Spanish 2 rue Andr Pascal, F-75775 Paris Cedex 16, France http://www.oecd.org/dataoecd/29/21/2754804.pdf

17

You might also like