Identifying M&E Indicators

Monitoring & Evaluation Guidelines

United Nations World Food Programme Office of Evaluation

What is an Indicator

How to check the Link between the Operation Design and the M&E elements 6 What is Beneficiary Contact Monitoring 11 Selecting Appropriate Indicators What is meant by Disaggregating or Stratifying and how is It done 14 17

Guidelines

Identifying M&E Indicators

Overview Introduction. The purpose of this module is to explain what an indicator is and what an
indicator should be like in WFP.

Why is this Module important?

Indicators play a key role in (M&E) because they enable staff to measure achievements and changes connected to WFP operations. This module is important because it defines what an indicator is and explains an indicators function in M&E. It provides concrete guidance on how to ensure that the indicators stated in the Logical Framework are clearly formulated and logically related to operation design a precondition for an efficient results-oriented M&E strategy. It also explains what Beneficiary Contact Monitoring (BCM) is and how it is used in the monitoring of WFP operations. WFPs criteria for selecting indicators are outlined, and examples of performance indicators at different levels in the results hierarchy are given.

What does this Module aim to achieve?

This module has the following objectives:

Explain what an indicator is and how it is used in M&E. Describe the critical relationship between M&E and operation design, and explain what steps should be followed in order to ensure that this link is clearly articulated, Define BCM and explain how it is used in M&E, Explain how to select appropriate indicators. Explain what stratifying and disaggregating mean in relation to sampling, data collection, indicators and analysis including the requirements for monitoring and evaluating WFPs Commitments to Women and explain the rationale for stratifying prior to data collection, as well as during analysis.

What should be reviewed before starting?

What is RBM Oriented M&E How to design a Results-Oriented M&E Strategy for EMOPs and PRROs How to design a Results-Oriented M&E Strategy for Development Programmes

Section Titles and Content Headings

What is an Indicator Introduction What is an Indicator An Example of the Function of Indicators in a Logical Framework Matrix Examples of Appropriate and Inappropriate Indicators in terms of reflecting Targets How to check the Link between the Operation Design and the M&E elements Introduction The Standard Logical Framework Matrix and how it relates to M&E The main Contents of the Logical Framework Matrix How to check the Design Logic in a Logical Framework

Guidelines

How to check the M&E Elements in a Logical Framework An Example of how to check the Design Logic in a Logical Framework An Example of Distinct and Separate Results Hierarchy Levels and Design Elements An Example of SMART Indicators within Each Level of the Results Hierarchy What is Beneficiary Contact Monitoring Introduction What is BCM? BCM and the Logical Framework An Example of BCM Leading Indicators and Their relation to Outcome Indicators in Development Operations, PRROs and EMOPs Selecting Appropriate Indicators Introduction Indicators in WFP How to select Indicators Example of SMART Indicators for a School Feeding Activity Logical Framework What is meant by Disaggregating or Stratifying and how is It done Introduction What is Stratification and what is Disaggregation Stratification Requirements for the M&E of WFPs Commitments to Women Example of Pre-stratification of a Probability Sample Example of Stratification of a Non-probability Sample Examples of the Stratification Factors that are listed in Indicators

Guidelines

What is an Indicator
Introduction. This section explains what an indicator is and how it is used in
monitoring and evaluation (M&E).

What is an Indicator
WFP defines an indicator as a quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement or to reflect the changes connected with a WFP operation. As such, an indicator is simply a measurement. Indicators are compared over time in order to assess change. In the logical framework approach, an operation is broken down into design elements such as inputs, activities, outputs, outcomes and impacts. Separate indicators for each of these elements (or levels) are used to measure performance. Indicators play the critical role of informing management as to whether an operation is being implemented as planned and achieving the desired results as articulated in its logical framework. As such, indicators are sometimes referred to as performance indicators.

An indicator may be disaggregated by any number of factors. For WFP, gender and age group are critical factors for the disaggregation of indicators. Other potentially important factors by which indicators may be disaggregated are operation-specific and should be defined in terms of the subgroups that are to be compared with one another. When listing an indicator in the logical framework, be sure to state the factors by which it must be disaggregated (e.g. school enrolment rate by grade and gender). In addition, be sure that the indicator is non-directional (e.g. neither positive nor negative), as it is simply a measure. Target achievements for indicators should be listed in the first column of the logical framework (e.g. the internal logic) or in a separate column.

Guidelines

An Example of the Function of Indicators in a Logical Framework Matrix

What the operation will do; what it seeks to achieve Logical framework hierarchy Impact The higher objective to which this operation, along with others, is intended to contribute Outcome The outcome of an operation; the changes in beneficiary behaviour, systems or institutional performance caused by the combined output strategy and key assumptions Outputs The actual deliverables; what the operation can be held accountable for producing Activities The main activity clusters that must be undertaken in order to accomplish the outputs Output indicators that measure the goods and services finally delivered by the operation Inputs/resources Budget by activity; monetary, physical and human resources required to produce the outputs People, events, processes, sources of data monitoring system for validating implementation progress Risks regarding implementation and efficiency People, events, processes, Risks regarding design effectsources of data supervision iveness and monitoring system for validating operation design How performance will be measured Factors outside management control that may affect project performance Assumptions and risks

Performance indicators (Impact)

Means of verification

Indicators (increasingly stand- The programme evaluation ardised) to measure prosystem gramme performance (Outcomes) Measures that describe the accomplishment of the outcome; the value, benefit and return on the investment

Risks regarding strategic impact

People, events, processes Risks regarding programmeand sources of data for organ- level impact ising the operations evaluation system

Examples of Appropriate and Inappropriate Indicators in terms of reflecting Targets

Appropriate indicator, which is non-directional Outcome Reduce malnutrition of children under 5 years of age by 20 percent (column 1 of logical framework) Percentage of children under 5 who are acutely malnourished (column 2 of logical framework) Inappropriate indicator, which is directional and reflects a target Outcome Reduce malnutrition of children under 5 years of age (column 1 of logical framework) 20 percent decrease in acute malnutrition among children under 5 (column 2 of logical framework)

Guidelines

How to check the Link between the Operation Design and the M&E elements
Introduction. This section clarifies the critical relationship between M&E and operation
design, and provides the steps to follow to ensure that this link is clearly articulated.

The Standard Logical Framework Matrix and how it relates to M&E

The primary purpose of M&E is to measure the degree to which an operation design is implemented as planned and how successfully it achieves its intended results. The operation design describes how inputs and activities will result in outputs delivered by WFP and its partners, and how the operation designers believe these outputs will, in turn, result in desired outcomes and impacts. The relationship between each of these levels is described in a logical framework hierarchy for the operation and represents a hypothesis concerning how the operation, starting with the initial resources or inputs that are available, will bring about the desired results. When a results-based approach to design is used, the desired outcomes or impacts are identified first, then the outputs needed to achieve those outcomes, and then the inputs and activities needed to deliver those outputs. The logical framework approach produces a matrix (see following page), which combines the concepts of results-based management (RBM), results-based operation design and M&E.
What the operation will do; what it seeks to achieve How performance will be measured Factors outside management control and that may affect project performance Assumptions and risks

Logical framework hierarchy Impact

Performance indicators (Impact)

Means of verification

The higher objective to which Indicators (increasingly stand- The programme evaluation this operation, along with oth- ardised) to measure prosystem ers, is intended to contribute gramme performance Outcome The outcome of an operation; the changes in beneficiary behaviour, systems or institutional performance caused by the combined output strategy and key assumptions Outputs The actual deliverables; what the operation can be held accountable for producing Activities Output indicators that measure the goods and services finally delivered by the operation Inputs/resources People, events, processes, sources of data monitoring system for validating implementation progress (Outcomes) Measures that describe the accomplishment of the outcome; the value, benefit and return on the investment

Risks regarding strategic impact

People, events, processes Risks regarding programmeand sources of data for organ- level impact ising the operations evaluation system

People, events, processes, Risks regarding design effectsources of data supervision iveness and monitoring system for validating operation design

The main activity clusters that Budget by activity; monetary, must be undertaken in order physical and human reto accomplish the outputs sources required to produce the outputs

Risks regarding implementation and efficiency

Guidelines

The main Contents of the Logical Framework Matrix

Each of the 4 columns in the Logical Framework is described in the following paragraphs. The first and fourth columns articulate operation design and assumptions, while the second and third columns outline the M&E performance measurement indicators and means in order to test whether or not the hypothesis articulated in the operation design holds true. Column 1: This column outlines the design or internal logic of the operation. It incorporates a hierarchy of what the operation will do (inputs, activities and outputs) and what it will seek to achieve (purpose and goal). Column 2: This column outlines how the design will be monitored and evaluated by providing the indicators used to measure whether or not various elements of the operation design have occurred as planned. Column 3: This column specifies the source(s) of information or the means of verification for assessing the indicators. Column 4: This column outlines the external assumptions and risks related to each level of the internal design logic that is necessary for the next level up to occur.

How to check the Design Logic in a Logical Framework

To check the design logic of the logical framework, review and test the internal and external logic (columns 1 and 4, respectively) and the feasibility of the operations logical framework. Test the logic beginning with inputs and move upwards towards the impact using an if (internal logic) and (external logic) then (internal logic at the next level) logic test. Where necessary, adjust the logical framework to overcome logic flaws or unfeasible/unlikely relationships among various levels of the logical framework hierarchy. If no logical framework exists for the operation, consult the Logical Framework Guidelines. Specifically check that the following conditions hold: Inputs are necessary and sufficient for activities to take place Activities are necessary and sufficient for outputs that are of the quality and quantity specified and that will be delivered on time. All outputs are necessary, and all outputs plus assumptions at the output level are necessary and sufficient to achieve the outcome. The outcome plus assumptions at the outcome level are necessary and sufficient to achieve the impact. The impact, outcome, and output statements are not simply restatements, summaries or aggregations of each other, but rather reflect the resulting joint outcome of 1level plus the assumptions at that same level. Each results hierarchy level represents a distinct and separate level, and each logical framework element within a results hierarchy level represents a distinct and separate element. The impact, outcome, activities, inputs and assumptions are clearly stated, unambiguous and measurable. Impacts and outcomes are stated positively as the results that WFP wishes to see. Outputs are stated positively in terms of service/product delivery. The assumptions are stated positively as assumptions, rather than risks, and they have a very high probability of coming true.

How to check the M&E Elements in a Logical Framework

Check that the following conditions hold in the logical framework: Indicators for measuring inputs, activities, outputs, outcome and impact are specific, measurable, accurate, realistic and timely (SMART) (column 2).

Guidelines

Beneficiary contact monitoring (BCM) indicators are identified for the purpose of tracking progress between outputs and outcomes and are noted at the outcome level. 2 levels within 1 logical framework do not share the same indicator (if they do, the indicator at 1 level is not specific enough to that level or the design logic between levels is flawed). The unit of study (e.g. individuals, children, households, organisations) in the numerator and, where applicable, the denominator of each indicator are clearly defined such that there is no ambiguity in calculating the indicator. The means of verification for each indicator (column 3) are sufficiently documented, stating the source of the data needed to assess the indicator (be sure that sources of secondary data are in a useable form).

An Example of how to check the Design Logic in a Logical Framework

The following diagram is an example of testing the internal and external logic of a nutrition projects logical framework using the if-and-then logic test. If we deliver the outputs through our planned activities and using the planned inputs, and our assumptions at the output, outcome and impact levels hold true, then the desired outcome will occur and lead to the desired impact.

Guidelines

An Example of Distinct and Separate Results Hierarchy Levels and Design Elements

Guidelines

An Example of SMART Indicators within Each Level of the Results Hierarchy

Once the design levels have been clarified and simplified, choosing SMART indicators is a relatively straightforward task. Note how each indicator is designed to measure the element in the corresponding row only.
Results hierarchy Impact Increased completion of primary education by girls Performance indicators A. % of students dropping out between one grade and another by gender and grade (between grades) Numbers of girls completing grade 6 and grade 9 Number of girls enrolled at the beginning of each academic year % of girls absent for 3+ days/ month Number of rations distributed to girl students per semester/per academic year

B. Outcome Increased enrolment of female students by an average of 10% per year over 5 years A.

Increased attendance by enrolled female A. students Output Dry take-home rations distributed (on average per year) to female students who are enrolled and attending classes A.

10

Guidelines

What is Beneficiary Contact Monitoring

Introduction. This section defines Beneficiary Contact Monitoring (BCM) and explains
how it is used in monitoring and evaluation (M&E).

What is BCM?
BCM is defined as a systematic investigation to monitor beneficiaries womens, mens, girls and boys perceptions of an operation. BCM Focuses on beneficiary access to, use of and satisfaction with outputs by seeking feedback directly from the women, men and children who are the target group for an operation. Concerns the transition between outputs and outcomes. Provides managers with an indication of progress towards the achievement of an operations outcomes. Uses a variety of techniques and data collection methods.

It applies the following logic: If members of the target group do not have access to operation outputs, they will not experience any benefit. If members of the target group have access to, but have chosen not to use, the outputs, they will not experience any benefit. If members of the target group are using operation outputs, but are not satisfied with the services or facilities they are receiving, they are unlikely to use them in the longer term, and therefore their experience of benefits will be limited. In the logical framework matrix, BCM seeks to gauge progress in the transition from service delivery (outputs) to benefits (outcomes). The indicators used in BCM are therefore referred to as leading indicators. BCM indicators should be included in the logical framework at the outcome level. It must be noted that beneficiaries reaction, however positive, is only a proxy indicator of the improved situation that an operation is intended to bring about. A proxy indicator is an indicator that is substituted for another indicator that would be hard to measure directly.

11

Guidelines

BCM and the Logical Framework

The leading indicators for BCM should be included in the logical framework as the last indicators listed under column 2 at the outcome level.
Results Hierarchy Impact Performance Indicators Means of Verification Assumptions and Risks

Outcome The benefits derived by the beneficiaries Transition from service delivery to benefit Beneficiary Contact Monitoring Indicators Mens, womens and childrens access to, use of and satisfaction with the outputs provided by the operation

Outputs The services provided by the operation Activities

12

Guidelines

An Example of BCM Leading Indicators and Their relation to Outcome Indicators in Development Operations, PRROs and EMOPs
BCM is concerned with beneficiaries perspectives of access to, use of and satisfaction with outputs. This important qualitative and contextual characteristic of BCM complements more quantitative outcome indicators.
Development School feeding activity desired outcome: Increase access to basic education for boys and girls Outcome indicators: Net enrolment rate disaggregated by gender PRRO Supplementary feeding desired outcome: Reduce mortality and morbidity risk in children under 5 Outcome indicators: EMOP General food distribution desired outcome: Save lives through provision of adequate food Outcome indicators:

Number of children under 5 suffering from malnutrition Mortality rate of children under 5

Crude mortality rate Under-5 mortality rate

BCM indicators:

BCM indicators: Access: % of target households aware of supplementary feeding programme Use: % of target households participating in supplementary feeding programme Satisfaction; mothers perception of quality and adequacy of food supplement

BCM indicators: Access: % families registered for food distribution Use: number of registered men, women, girls and boys consuming target ration Satisfaction: mens, womens and childrens opinions about composition and quality of ration

Access: % of parents fathers and mothers aware of value of sending girls to primary school Use: % of target households with girls and boys enrolled in participating schools Satisfaction: degree of girls and boys appreciation of canteen food/dry rations (from very low to very high)

13

Guidelines

Selecting Appropriate Indicators

Introduction. The purpose of this section is to explain how to select appropriate
indicators.

Indicators in WFP
A generic set of indicators cannot be prescribed because indicators must be developed specifically for each operations design and in line with the articulation of the operations design elements (e.g. inputs, activities, outputs, outcomes, impact) as shown in the logical framework. However, many WFP operations around the world share a similar design, and the Office of Evaluation (OEDE) has developed a list of potential indicators categorised for each level of the logical framework hierarchy - for many types of operations. There is no need to think up a completely unique and original set of indicators, but rather to pick from the list a concise and focused set and, if required, adjust them to suit the operation that is being monitored and evaluated.

How to select Indicators

Indicator selection usually takes place during the design process and is reflected in the operations logical framework matrix (column 2). Indicators should be specific, measurable, accurate, realistic and timely (SMART). This acronym provides a detailed set of criteria for assessing the appropriateness of potential indicators. Each of the indicators identified must satisfy the following conditions. Specific An appropriate indicator measures only the design element (output, outcome or impact) that it is intended to measure and none of the other elements in the design. Many indicators are related to every design element (since all the elements within a design are related), but few are specific measures of performance for each and every element. For example, if the target output of a food-for-work (FFW) project is to construct 20 wells, the specific indicator to measure this will be the number of wells constructed. Time spent in water collection is a related, but not specific, measure. Similarly, the number of hours spent in FFW activities is a related, but not specific, indicator. In the case of an emergency operation (EMOP) with the desired impact of reducing acute malnutrition among children under 5 years of age, a related, but not specific, indicator might be the number of meals consumed by children under 5. The specific indicator would be the percentage of under-5 children who are acutely malnourished (< -2 standard deviation weight-for-height). Because the design must treat each level in the results hierarchy, and each design element in the level, as a separate and distinct element, the appropriate indicator at one level (or for 1 design element) cannot be the appropriate indicator for another. If an indicator is shared at 2 levels or between 2 design elements, either one of the indicators is not specific enough or the design logic is flawed. Measurable An appropriate indicator is measurable and clearly defines the measurement such that 2 people would measure it in the same way. For quantitative proportions or percentages this means that both the numerator and the denominator must be clearly defined. For quantitative whole num14

Guidelines

bers and qualitative data it means defining each term within the indicator such that there can be no misunderstanding as to the meaning of that indicator. This is critical for ensuring that the data collected by different people at different times are consistent and comparable. Examples of indicators that are not measurable include the percentage of households that are food-secure (food-secure is not defined precisely) and the percentage of women with increased access to health services (access is not defined precisely). The critical means of ensuring that indicators are measurable is to define all the terms within the indicator, even those for which a general agreement about meaning may be shared among staff members. Accurate Some indicators are more accurate measures than others. For example, measuring the weightfor-height of children under 5 years of age will yield a more accurate figure for the percentage of acutely malnourished (wasted) children than will measuring the mid-upper arm circumference (MUAC). Again, note the need to define clearly what is meant by 'acutely malnourished' in terms of measurement (previous criteria). Similarly, a 7-day dietary recall will yield a more accurate measure of food consumption than will asking the average number of meals that were consumed over the last month. However, the accuracy criteria must be balanced with the other criteria, taking into consideration the resources available for M&E in WFP. Realistic The indicators selected must be realistic in terms of their ability to collect the data with the available resources. Some indicators present major problems for data collection owing to the cost or skills required (e.g. anthropometric surveys, large-scale sample surveys). Being realistic in planning what information can be collected ensures that it will, in fact, be collected. This is an important factor to consider and may lead to compromises on other criteria. Timely Indicators must be timely in several aspects. First, they must be timely in terms of the time spent in data collection. This relates to the resources that are available - staff and partner time being critical. If it takes 2 days to collect dietary recall data from 1 household, this indicator is probably inappropriate. Second, indicators must reflect the timing of collection. For example, a FFW activity constructing water dams may have an indicator of time spent in water collection as a measure of whether or not the desired outcome has been achieved. However, the indicator must reflect the seasonal differences, as well as the operation activities, that affect water availability. An appropriate indicator may disaggregate by dry and wet season. Finally, the time-lag between output delivery and the expected change in outcome and impact indicators must also be reflected in the indicators that are chosen. This time-lag can be significant, especially for Country Programmes (CPs) aimed at poverty reduction. Some more general guidelines for indicator selection, based on commonly found mistakes, include the following: Do not state the target achievement in the indicator itself: The indicator is simply a measurement and, as such, should be non-directional (e.g. neither positive nor negative). Targets should be listed either in the first column of the logical framework - as part of the operation's internal logic - or as a separate column. Do not select too many indicators: Managers have a tendency to ask for too much information, assuming that the more they know the better prepared they will be. The result is often information overload. Instead, information needs must be related directly to decisionmaking roles and levels of management - field managers require more detailed information, while aggregated and summarised data are used at higher levels. The selection of indicators should reflect this through the specification of a minimum set of information. There is a tendency for staff and partners to want to capture every nuance and to identify all the pos15

Guidelines

sible indicators during the design of an operation. A brief reminder about the cost and time needed to collect and analyse the data usually brings the focus back to the minimum set of information needed. Do not select indicators that are unnecessarily complex: Some indicators present major problems for data collection in terms of the skills or resources required. For example, household income data can be complex and expensive to collect. Alternative indicators to consider are patterns of expenditure or household characteristics such as the materials used to construct the house. Qualitative indicators (e.g. wealth ranking) can also convey complex information, perhaps less accurately but accurately enough for most data needs. Do not overconcentrate on physical progress indicators: Information about food stocks and distribution is vitally important within a WFP operation, but it does not provide sufficient information on the performance of the operation. Identifying these indicators is relatively straightforward. However, information about the results of an operation is also needed, and the selection of indicators at these levels is slightly more complex. To some extent, the logical framework mandates the identification of indicators at the outcome and impact levels, making it an ideal shared framework for operation design and M&E. In addition, BCM provides a means of tracking the likelihood of achieving the desired outcomes and impact.

Example of SMART Indicators for a School Feeding Activity Logical Framework

In the following example, notice how each indicator measures only the design element that is in its corresponding row. When the design is sufficiently and coherently articulated in the logical framework (e.g. through clear logic and no overlapping design elements) it is much easier to choose the appropriate indicators.
Results Hierarchy Impact Decreased drop-out rate and increased completion of primary education by girls Performance Indicators A. B. C. Outcome Increase enrolment of female students by an average of 10% per year over 5 years Increase attendance by enroled female students Output Dry take-home rations distributed (on average per year) to female students who are enroled and attending classes A. % of students dropping out by gender and grade during academic year % of students dropping out from 1 grade to another grade per gender and grade (between grade) # of girls completing Grade 6 and Grade 9 # of girls enrolled each year at the beginning of academic year % of students absent 3+ days/ month by gender # of rations distributed to girl students per semester/per academic year # of distributions conducted per academic year # MT Wheat, Vegatble Oil received by project # of staff/consultants Amount of funds

A. A.

Activities Inputs

3 Dry take-home ration distributions held A. per academic year (target 3) 78305 MT Wheat, 4195 MT Vegetable Oil A. B. C.

16

Guidelines

What is meant by Disaggregating or Stratifying and how is It done

Introduction. This section explains what stratifying and disaggregating mean in
relation to sampling, data collection, indicators and analysis, including the requirements for monitoring and evaluating WFPs Commitments to Women. In addition, it provides the rationale for stratifying prior to data collection, as well as during analysis.

What is Stratification and what is Disaggregation

Whenever a comparison is made between 2 groups regardless of how the groups have been defined the data regarding those groups are being stratified or disaggregated. In WFP, important factors for stratification include age group and gender, reflecting WFPs Commitments to Women. Other commonly found variables for stratification are geographic location (village, district, province, etc.) and wealth group. The factors by which data can be stratified (i.e. by splitting a larger group into 2 strata e.g. men and women or many strata e.g. 5 age groups) are endless and should be selected on the basis of the analytic needs of the operation. When stratification is carried out during the analysis stage it is known as disaggregation. The concept of stratification must be applied prior to data collection, while the sampling strategy is being devised, in order to ensure that enough data is collected about each of the groups that are being compared. If this does not occur, there may be too few sample units from 1 or more of the stratified groups to allow valid conclusions to be drawn. The stratification concept is applicable to both probability and non-probability sampling. In probability sampling, a separate sampling frame is developed for each stratum (or grouping of units) and the selected sample size is applied to each stratum. The same rules apply in cases of nonprobability sampling. Each time an additional stratum is added, the sample size doubles. The decision to carry out pre-stratification during sampling should therefore only be made after careful consideration of the additional costs associated with adding strata. The best practice is to list the factors for stratification in the indicators. This ensures that critical pre-stratification needs are considered prior to choosing a sample. It also ensures that poststratification (or disaggregating) occurs during analysis.

Stratification Requirements for the M&E of WFPs Commitments to Women

As part of its Commitments to Women, WFP has made a commitment to generate and disseminate data that is disaggregated by gender. All WFP monitoring and reporting will specify: Mens and womens percentage shares of the resources received from food distribution (e.g. disaggregated output indicators); Mens and womens shares of benefits by category of activities (e.g. disaggregated beneficiary contact monitoring [BCM], outcome and impact indicators); The percentage of positions held by women in the planning and management of food distribution (e.g. disaggregated activity and output indicators). All M&E information must assess whether WFPs gender commitments have been adhered to and the reasons for any failure to do so. All indicators must also be computed to reflect WFPs gender commitments (i.e. they should be disaggregated by gender).

17

Guidelines

Example of Pre-stratification of a Probability Sample

Separate estimates of acute malnutrition are desired for each of 3 districts in order to compare them. A sample size of 210 children is needed from which to draw valid estimates of nutritional status (i.e. 95 percent confidence in an estimate with +/- 10 percentage points in either direction) and infer the results to the larger population. Therefore, a sample of 210 is taken in each district to allow for separate estimates, at the defined level of confidence and precision, from each district. This results in a total sample size of 630, which means that the overall estimate (for all the districts combined) will be more precise (based on 630 instead of 210 units).

Example of Stratification of a Non-probability Sample

10 villages are to be included in a sample for conducting focus group discussions. Just before setting off for the villages, several staff members point out that having mixed gender focus group discussions will prevent womens participation owing to cultural roles in public meetings. It is decided that focus group discussions must be held with men and women separately in order to allow each genders varying experiences regarding participation in the food-for-work (FFW) activity to be compared. Consequently, 2 focus group discussions (with a total of 20 participants) are held in each of the 10 villages.

Examples of the Stratification Factors that are listed in Indicators

Results hierarchy Impact Decreased drop-out rate and increased completion of primary education by girls Performance indicators A. B. % of students dropping out by gender and grade (during the academic year) % of students dropping out between one grade and another by gender and grade (between grades) Numbers of girls completing grade 6 and grade 9 Number of girls enrolled at the beginning of each academic year % of students absent 3+ days/ month by gender Number of rations distributed to girl students per semester/per academic year

C. Outcome Increased enrolment of female students by an average of 10% per year over 5 years A.

Increased attendance by enrolled female A. students Output Dry take-home rations distributed (on average per year) to female students who are enrolled and attending classes A.

18

Guidelines

Module Summary What has been covered in this module? This module defined what an indicator is and explained its function in M&E. It provided concrete guidance on how to ensure that the indicators stated in the Logical Framework are clearly formulated and logically related to operation design a precondition for efficient M&E design. It also defined what BCM is and described how it should be used in the monitoring of WFP operations. WFPs criteria for selecting indicators were outlined, and examples of performance indicators at different levels in the results hierarchy were given.

What additional resources are available?

For further information the following modules and resources might be useful: Choosing Methods and Tools for Data Collection Going to the Field to collect Monitoring and Evaluation Data How to consolidate, process and analyse Qualitative and Quantitative Data Reporting on M&E Data and Information for EMOPs and PRROs Reporting on M&E Data and Information for Development Programmes The WFP Indicator Menu

19

Identifying M&E Indicators

United Nations World Food Programme Office of Evaluation and Monitoring Via Cesare Giulio Viola, 68/70 - 00148 Rome, Italy Web Site: www.wfp.org E-mail: [email protected] Tel: +39 06 65131