Partnership for Growth (PFG), 2011-2015, Mid-Term USAID Evaluation.

PARTNERSHIP FOR GROWTH EL SALVADOR – UNITED STATES (2011–2015) Mid-Term Evaluation Final Report September 2014 September 30, 2014 Final Report PARTNERSHIP FOR GROWTH MID-TERM EVALUATION EL SALVADOR USG and GOES Goal Leads Scorecard Meeting in San Salvador, El Salvador (May 2014) Optimal Solutions Group, LLC Mark Turner Christabel Dadzie Robert Burke Gerard Martin Rigoberto Stewart Partnership for Growth El Salvador Mid-Term Evaluation Report ii TABLE OF CONTENTS LIST OF TABLES ............................................................................................................................. VI LIST OF FIGURES ........................................................................................................................ VIII ACKNOWLEDGEMENTS ................................................................................................................... X ACRONYMS .................................................................................................................................... XI EXECUTIVE SUMMARY ............................................................................................................... XIII Purpose of Evaluation ............................................................................................................................ xiii Methodology .......................................................................................................................................... xiv Findings, Conclusions, and Recommendations ...................................................................................... xv 1. INTRODUCTION .......................................................................................................................... 1 2. 1.1 Background and Context............................................................................................................... 1 1.2 Purpose and Scope of the Evaluation ............................................................................................ 2 1.3 Evaluation Questions .................................................................................................................... 2 METHODOLOGY ...................................................................................................................... 3 2.1 Evaluability ................................................................................................................................... 3 2.2 Goal-Selection Process for In Depth Review of 7 Goals .............................................................. 3 2.2 Methods of Data Collection and Data Analysis ............................................................................ 5 2.2.1 3. Data Collection Approaches ................................................................................................. 5 2.3 Data Analysis ................................................................................................................................ 7 2.4 Evaluation/ Study Limitations ...................................................................................................... 8 KEY GENERAL FINDINGS ON THE EL SALVADOR PFG INITIATIVE .................................... 10 3.1 Overall Advantages of the PFG Initiative ................................................................................... 10 4. CROSS-CUTTING QUESTION ONE: WHAT ARE THE ADVANTAGES AND/OR DISADVANTAGES OF THE PFG WHOLE-OF-GOVERNMENT APPROACH TO DEVELOPMENT ASSISTANCE? .............. 13 4.1 5. Findings on Advantages of PFG WGA....................................................................................... 14 4.1.1 Implementation of Whole of Government Approach by USG PFG Staff ............................ 15 4.1.2 Implementation of Whole of Government Approach by GOES PFG Staff .......................... 19 4.1.3 Challenges/ Disadvantages of PFG WGA .......................................................................... 20 4.2 Conclusions and Lessons Learned .............................................................................................. 21 4.3 Course Corrections and Recommendations ................................................................................ 22 CROSS-CUTTING QUESTION TWO: TO PARTNERSHIP FOR GROWTH U.S. GOVERNMENT STAFF, AS WHAT EXTENT HAS AFFECTED THE WORKLOAD OF NATIONAL GOVERNMENT AND Partnership for Growth El Salvador Mid-Term Evaluation Report iii COMPARED TO THE WORKLOAD CREATED BY TRADITIONAL FORMS OF DEVELOPMENT ASSISTANCE DELIVERY? ............................................................................................................... 25 5.1 Findings....................................................................................................................................... 25 5.2 Conclusions and Lessons Learned .............................................................................................. 32 5.3 Course Corrections and Recommendations ................................................................................ 33 6. CROSS-CUTTING QUESTION THREE: WHAT CONTRIBUTIONS HAS NON-ASSISTANCE MADE TO THE PFG PROCESS, AND HOW CAN IT BE UTILIZED MOVING FORWARD? .............................. 34 6.1 Findings....................................................................................................................................... 34 6.2 Conclusions and Lessons Learned .............................................................................................. 38 6.3 Course Corrections and Recommendations ................................................................................ 38 7. COUNTRY-SPECIFIC QUESTION ONE: THE CONSTRAINTS ANALYSIS DOES NOT IDENTIFY REMEDIES TO ADDRESS THE BINDING CONSTRAINTS TO GROWTH. FOR EACH OF THE CONSTRAINTS, ARE THE GOAL-LEVEL COMMITMENTS SET FORTH IN THE JCAP ALONE CAPABLE OF ACHIEVING THE CONSTRAINT-LEVEL OBJECTIVES AND OUTCOMES? ................... 40 7.1 Findings....................................................................................................................................... 41 7.2 Conclusions and Lessons Learned .............................................................................................. 46 7.3 Course Corrections and Recommendations ................................................................................ 46 8. COUNTRY-SPECIFIC QUESTION TWO: THE PFG MODEL PLACES EMPHASIS ON EVIDENCEBASED DECISION MAKING AND FACT-BASED MONITORING. IS QUANTITATIVE AND OBJECTIVELY VERIFIABLE INFORMATION BEING USED TO MANAGE JCAP IMPLEMENTATION IN ORDER TO ACHIEVE AND MEASURE RESULTS? .............................................................................................. 48 8.1 Findings....................................................................................................................................... 49 8.2 Conclusions and Lessons Learned .............................................................................................. 55 8.3 Course Corrections and Recommendations ................................................................................ 57 9. COUNTRY-SPECIFIC QUESTION 3: AT THE MID-TERM, ARE THE PERFORMANCES OF THE PFG INTERVENTIONS ON TARGET AND CREATING THE NECESSARY OUTPUTS TO ACHIEVE THE DESIRED OUTCOMES? ............................................................................................ 65 SELECTED 9.1 Findings....................................................................................................................................... 67 9.2 Conclusions and Lessons Learned .............................................................................................. 79 9.3 Recommendations and Course Corrections ................................................................................ 79 10. OVERALL SUMMARY AND CONCLUSION .............................................................................. 81 11. ADDENDUM – CASE STUDIES OF 7 SELECTED GOALS ..................................................... 82 11.1 Crime and Insecurity Constraint (4 Case Studies) ...................................................................... 82 Security Goals 1&2 ............................................................................................................................. 82 Partnership for Growth El Salvador Mid-Term Evaluation Report iv Security Goal 4 ................................................................................................................................... 96 Security Goal 11: .............................................................................................................................. 103 Security Goal 12 ............................................................................................................................... 108 11.2 Low Productivity in the Tradables Sector (3 Case Studies)...................................................... 113 Tradables Goal 3 .............................................................................................................................. 113 Tradables Goal 5 .............................................................................................................................. 121 Tradables Goal 6 .............................................................................................................................. 127 ANNEXES ................................................................................................................................. 132 ANNEX 1 - EVALUATION STATEMENT OF WORK .................................................................... 132 ANNEX 2 - WORK PLAN ........................................................................................................... 152 ANNEX 3 - ADVANTAGES AND DISADVANTAGES FOR DATA COLLECTION METHODS FOR THIS EVALUATION .............................................................................................................................. 154 ANNEX 4 - DATA COLLECTION INSTRUMENTS ........................................................................ 157 4.1 Stakeholder Types for Interview Guides 1-5 ............................................................................ 157 4.2 Semi-structured Interview Guides 1-5 ............................................................................................ 158 4.3 Confidential Online Survey ............................................................................................................ 176 ANNEX 5 - TABLE OF PFG GOALS AND LINES OF ACTION ..................................................... 187 ANNEX 6 – PFG CONSTRAINTS AND GOAL-LEVEL INDICATORS ............................................ 196 ANNEX 7 - LIST OF REFERENCE AND SOURCES – SELECTED GOALS ..................................... 200 Security References .............................................................................................................................. 200 Tradables References ............................................................................................................................ 202 ANNEX 8 - LIST OF REFERENCES AND SOURCES - GENERAL .................................................. 204 Partnership for Growth El Salvador Mid-Term Evaluation Report v LIST OF TABLES Table 2.1: Number of stakeholder interviews ............................................................................................... 6 Table 2.2: Evaluation Questions Asked per Respondent Type in Semi-Structured Interviews .................... 6 Table 2.3: Web-based survey, by response rate and government affiliation ................................................ 7 Table 4.1: Interview responses referencing advantages of PFG WGA....................................................... 14 Table 4.2: Interview Responses Referencing Disadvantages of PFG WGA .............................................. 15 Table 5.1: Respondents' gender .................................................................................................................. 27 Table 5.2: Hours per week dedicated to planning and developing PFG (Based on survey responses only) .................................................................................................................................................................... 31 Table 6.1: Examples of direct quotes about non-assistance in the El Salvador PFG initiative, provided by survey respondents ...................................................................................................................................... 35 Table 6.2: Number of interviewees who provided examples of non-assistance activities in PFG El Salvador Initiative ....................................................................................................................................... 36 Table 8.1: Summary of Availability of Quantifiable and Objectively Verifiable Information. Crime and Insecurity (Selected Goals Only) ................................................................................................................ 59 Table 8.2: Summary of Availability of Quantifiable and Objectively Verifiable Information. Low Productivity and Tradables. (Selected Goals’ Only)................................................................................... 61 Table 9.1: PFG - Comparative overview of semi-annual scorecards issued since the start of PFG: November 2011– May 2014 ....................................................................................................................... 68 Table 9.2: Status of Selected Crime and Insecurity Constraint LOAs at Mid-Term (Selected Goals Only) .................................................................................................................................................................... 72 Table 9.3: Status of Selected Low Productivity and Tradables Constraint LOAs at Mid-Term (Selected Goals Only) ................................................................................................................................................. 74 Table 11.1: Public satisfaction with the performance of the judicial system and the National Police in El Salvador and other countries in Central America in 2010 and 2012 .......................................................... 86 Table 11.2: Percent of Salvadorans who are satisfied with the performance of police, households vs. microenterprises, 2012 and 2013 ................................................................................................................ 87 Table 11.3: Percent of Salvadorans who are satisfied with the performance of judges and the court, households vs. microenterprises, 2012 and 2013 ........................................................................................ 88 Table 11.4: Summary Status of Goal 1-2 at Mid-term................................................................................ 94 Table 11.5: Number of reported crimes in public transportation in El Salvador (2012-2013) ................... 98 Partnership for Growth El Salvador Mid-Term Evaluation Report vi Table 11.6: Summary Status of Goal 4 at Mid-term ................................................................................. 102 Table 11.7: Summary Status of Goal 11 at Mid-term ............................................................................... 107 Table 11.8: Summary Status of Goal 12 at Mid-Term.............................................................................. 112 Table 11.9: Summary Status of Tradables Goal 3 at Mid-term ................................................................ 117 Table 11.10: Summary Status of Tradables Goal 5 at Mid-term .............................................................. 125 Table 11.11: Summary Status of Tradables Goal 6 at Mid-term .............................................................. 130 Partnership for Growth El Salvador Mid-Term Evaluation Report vii LIST OF FIGURES Figure 2.1: Evaluation Designed Process to Select Goals for In-Depth Review ......................................... 4 Figure 3.1: Did PFG bring changes, compared to other approaches to development assistance intended to affect economic growth? (Based on survey responses only) ...................................................................... 10 Figure 3.2: Is PFG meeting its goal of advancing economic growth in El Salvador? (Based on survey responses only)............................................................................................................................................ 11 Figure 4.1: USG and GOES interview respondents’ perceptions of PFG WGA advantages* ................... 15 Figure 5.1: Changes in workload resulting from PFG involvement: USG and GOES (Based on survey responses only)............................................................................................................................................ 26 Figure 5.2: Changes in workload resulting from PFG involvement: USG and GOES leadership (Based on survey responses only) ................................................................................................................................ 26 Figure 5.3: Changes in workload resulting from PFG involvement: USG and GOES non-leadership (Based on survey responses only) ............................................................................................................... 27 Figure 5.4: Changes in workload resulting from PFG involvement, by gender (Based on survey responses only) ............................................................................................................................................................ 28 Figure 5.5: Change in workload resulting from PFG, by task (Based on survey responses only).............. 29 Figure 5.6: Change in workload resulting from PFG by task – USG and GOES (Based on survey responses only)............................................................................................................................................ 29 Figure 5.7: Staff involved in planning and development of PFG (Based on survey responses only)......... 30 Figure 5.8: Perceived Effectiveness of PFG Compared to Traditional Development Assistance Approaches (Based on survey responses only) ........................................................................................... 31 Figure 5.9: Perceived Effectiveness of PFG Compared to Traditional Development Assistance Approaches – USG and GOES (Based on survey responses only) ............................................................. 32 Figure 6.1: Have you seen non-assistance tools being used in the PFG activity with which you are or were involved? (Based on survey responses only) .............................................................................................. 37 Figure 6.2: Respondents Awareness of Non-Assistance Activities by Stakeholder Type (Based on survey responses only)............................................................................................................................................ 37 Figure 8.1: Are the appropriate indicators being used to allow for transparent, accountable, and fact-based monitoring of the PFG? (Based on survey responses only) ........................................................................ 52 Figure 8.2: Are the appropriate indicators being used to allow for transparent, accountable, and fact-based monitoring of the PFG? (Based on survey responses only) ........................................................................ 53 Partnership for Growth El Salvador Mid-Term Evaluation Report viii Figure 9.1: Responses about whether or not PFG Selected Interventions are on Target (Based on survey responses only)............................................................................................................................................ 67 Figure 9.2: Interviewee Responses concerning the status of PFG at Mid-Term......................................... 67 Figure 11.1: Public satisfaction with the performance of judicial system and National Police in El Salvador and other countries in Central America in 2010 and 2012 .......................................................... 85 Figure 11.2: Percent of Salvadorans who are satisfied with the performance of the National Police, households vs. microenterprises, 2012 and 2013 ........................................................................................ 87 Figure 11.3: Percent of Salvadorans who are satisfied with the performance of judges and the court, households vs. microenterprises, 2012 and 2013 ........................................................................................ 88 Figure 11.4: Convictions per 100,000 Adults (2010, 2011, 2012) ............................................................. 89 Figure 11.5: Number of reported crimes in public transportation in El Salvador (2012-2013) .................. 98 Figure 11.6: Public perception of safety on public transport routes in El Salvador (2012 – 2013) ............ 99 Figure 11.7: Number of municipalities with crime prevention councils (2008-2012).............................. 104 Figure 11.8: CTS and ICPS Prison population and prison capacity* for El Salvador and other Central American Countries 2010-2013 ................................................................................................................ 109 Figure 11.9: Global competitiveness index: higher education and training.............................................. 114 Figure 11.10: Global competitiveness index: labor market efficiency ..................................................... 114 Figure 11.11: Foreign direct investment as percent of GDP ..................................................................... 122 Figure 11.12: FutureBrand country brand ranking ................................................................................... 122 Figure 11.13: Global competitiveness index: business sophistication ...................................................... 128 Partnership for Growth El Salvador Mid-Term Evaluation Report ix ACKNOWLEDGEMENTS The development and success of this evaluation was a result of a collaborative effort between the authors and many other people, who we would like to acknowledge. First, a depth of gratitude goes to the USAID points of contact in Washington D.C., Alex Gainer, Alice Brooks, and Mileydi Guilarte who reviewed the reports and provided valuable feedback to the team. Special thanks goes to our point of contact in El Salvador, Jennifer Johnson who spent many hours providing documentation, responding to questions of the evaluators, and organizing meetings, including high level meetings and site visits, for the evaluation to be successful. We would also like to thank the GOES point of contact – Mauricio Ruano, for his support in gathering the evaluation documentation and providing valuable comments to the team. A big thank you also goes to the Troika at USAID, MCC and State Department who met with the evaluation team and provided thorough feedback on the final evaluation report. Likewise, we are very grateful for the time, information, and insights provided by our 115 interviewees, including government officials, implementers, and independent experts, whose contributions are at the heart of this evaluation. The authors are also grateful to the support team at Optimal – Claire Everhart, Joanne Perodin and Omotunde Okesanya for their tireless efforts on this evaluation, and to the hard work of our local consultants, Camila Flores and Marcella Veneziani, whose local knowledge expertise and contributions were instrumental to the success of the evaluation. Partnership for Growth El Salvador Mid-Term Evaluation Report x ACRONYMS AGO Attorney General’s Office ASI Asociación de Industriales de El Salvador DOJ Department of Justice DOS Department of State ENPV National Violence Prevention Strategy FBI Federal Bureau of Investigation FDI Foreign Direct Investment Fiscalía The Prosecutor’s Office FONDEPRO Production Development Fund GOES Government of El Salvador GDP Gross Domestic Product ICITAP International Criminal Investigative Training Assistance Program ICT Information and Communication Technology INL International Narcotics and Law Enforcement Affairs IUDOP The Instituto Universitario de Opinión Publica JCAP Joint Country Action Plan LOAs Lines of Action M&E Monitoring and Evaluation MCC Millennium Challenge Corporation MINEC Ministry of Economy MINED Ministry of Education NGO Non-Governmental Organization OECD Organization for Economic Cooperation OPDAT Office of Overseas Prosecutorial Development, Assistance & Training PFG Partnership for Growth PNC National Civil Police PROESA La Agencia de Promoción de Inversiones de El Salvador RLA Resident Legal Advisor SEED Scholarship for Education and Economic Development SME Small and Medium Enterprises Partnership for Growth El Salvador Mid-Term Evaluation Report xi SOW Statement of Work TA Technical Assistance UCA Centroamericana José Simeón Cañas University UMEP Updated Methodology and Evaluation Plan U.S. United States USAID United States Agency for International Development USG United States Government UST Early Solution Units UTE Executive Technical Unit of the Justice Sector VMT Vice Ministry of Transportation WGA Whole of Government Approach Partnership for Growth El Salvador Mid-Term Evaluation Report xii EXECUTIVE SUMMARY The Partnership for Growth (PFG) aims to achieve accelerated, sustained, and broad-based economic growth in partner countries, including El Salvador and the Philippines, through bilateral agreements between the United States Government (USG) and the partnering countries’ national governments. Using principles set forth in President Barack Obama’s September 2010 Presidential Policy Directive on Global Development, the PFG requires rigorous, joint analyses of countries’ individual constraints to growth to develop joint action plans to address the most pressing of these constraints and to establish high-level mutual accountability for the goals and activities selected to alleviate them. The countries of El Salvador, the Philippines, Tanzania, and Ghana were selected as the first group of countries in which the United States (U.S.) and partner governments would attempt to structure new PFG initiatives, with selection based, in part, on each of country’s record of accomplishment in implementing ongoing Millennium Challenge Compacts (MCC). In November 2011, the USG and the Government of El Salvador (GOES) signed a Joint Country Action Plan (JCAP) that identified two primary constraints to Salvadoran economic growth and 20 goals aimed at easing those constraints. Each goal is associated with multiple lines of action (LOAs) to be undertaken by the GOES and the USG. It is believed that carrying out the agreed-upon LOAs will lead to goal achievement, which, in turn, will mitigate the effects of currently binding constraints and accelerate and sustain El Salvador’s rate of broad-based, inclusive economic growth. Purpose of Evaluation Optimal Solutions Group, LLC (Optimal) was contracted to conduct a mid-term evaluation of the PFG initiative in El Salvador. According to the evaluation statement of work (SOW), this evaluation serves two purposes. First, the evaluation analyzes whether the PFG process demonstrates improvements over pre-PFG assistance approaches. In particular, the evaluation team examined the extent to which the PFG’s wholeof-government approach (WGA) and Constraints Analysis (CA) led to a change in the way USG delivered development assistance. The second purpose of the evaluation consists of two parts: (i) determining whether PFG efforts “have been developed in such a way as to allow for the eventual determination of their impact in addressing the identified constraints and desired outcomes.” (ii) determining whether “measurable elements of PFG are moving in the right direction, are considered necessary and sufficient to achieve PFG goals, and are contributing to national interests through the integration and coordination of work done by both governments.”1 The evaluation was guided by the following six evaluation questions as outlined in the statement of work. 2 1 2 Evaluation Statement of Work, 2013. pg. 7-8 Evaluation Statement of Work, 2013. pg. 9-10 Partnership for Growth El Salvador Mid-Term Evaluation Report xiii Cross-Cutting Questions 1) What are the advantages and/or disadvantages of the PFG whole-of-government approach to development assistance? 2) To what extent has PFG affected the workload on national government and U.S. government staff, as compared to the workload created by traditional forms of development-assistance delivery? 3) What contributions has “non-assistance” made to the PFG process, and how can it be utilized moving forward? Country-Specific Questions—El Salvador 1) For each of the constraints, are the goal-level commitments set forth in the JCAP capable of achieving the constraint-level objectives and outcomes? 2) Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? 3) At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? Methodology The evaluation team employed a mixed-methods approach for data collection and analysis of the El Salvador PFG activities to inform this mid-term evaluation. This approach included an extensive desk review of all available project documentation and monitoring data; an evaluability assessment and selection of goals for in-depth case study; interviews with PFG architects 3 and leadership in Washington D.C.; a site visit to El Salvador to conduct individual and group semi-structured, face-to-face interviews; telephone interviews with stakeholders who were not available in person; and an online survey administered to USG and GOES PFG staff. The evaluation team conducted a qualitative research evaluation, which reviewed, collated, and analyzed the perceptions gathered. Quantitative approaches were also used to analyze and verify information provided by PFG in situations when the documentation allowed for this methodology. The evaluation team employed triangulation to verify findings from various vantage points. Upon completion of the analysis, the evaluation team developed this report that details the findings and conclusions and provides recommendations and course corrections that could be employed by the El Salvador PFG partners at midterm. These findings, conclusions, and recommendations are provided in response to the six evaluation questions stipulated in the SOW. 3 Architects refer to individuals who were part of designing and mapping out the overall PFG initiative, which led to the selection of the four countries that then initiated and implemented their own initiatives. See Annex 4 (Stakeholder Types) of this report. Partnership for Growth El Salvador Mid-Term Evaluation Report xiv Findings, Conclusions, and Recommendations Overall Advantages and Disadvantages of PFG In general, the evaluation team found that PFG has aided in positive movement of policy reforms in El Salvador. The initial PFG planning through the Constraints Analysis (CA) activity laid a good foundation for implementing the overall initiative. Additionally, PFG increased leverage for implementing USG development activities in El Salvador. Cross-Cutting Question 1: What are the advantages and/or disadvantages of the PFG Whole-ofGovernment Approach to development assistance? Findings - Advantages of PFG WGA within USG and GOES PFG Staff • • • • • • The CA and JCAP facilitated WGA dynamics. Within the USG, the WGA has focused and targeted human and operational capacity related to the policy and programmatic objectives of PFG. WGA has been promoted due to the purposeful design that has grouped lead agencies together under individual goals, who would otherwise not have worked together. WGA has also promoted increased understanding of the work done by USG agencies within the security constraint, because staff that would typically work in silos, currently work together. PFG has improved cooperation among GOES agencies. WGA has led to increased coordination both within and between GOES and USG. Challenges/ Disadvantages of PFG/WGA • • • • WGA coordination activities require significant staff time. WGA coordination is more limited across goals than within goals. Lessons learned from implementing WGA within specific goals are not systematically shared across goals. GOES leadership’s commitment and ownership of PFG was sometimes limited. Conclusions • • • • • PFG WGA represents a positive change in development assistance compared to previous approaches. WGA is most likely to be effective if purposefully incorporated into the design of the initiative and backed by high-level leadership. WGA promoted collaboration both within governments and across governments. WGA coordination is time consuming. There is insufficient inter-goal (and LOA) coordination, particularly related to the security constraint. Partnership for Growth El Salvador Mid-Term Evaluation Report xv Recommendations • • • Provide stronger guidance for inter-goal collaboration. A bilateral PFG management, coordination, and monitoring and evaluation (M&E) unit should be put in place to effectively supervise and guide the various elements of the PFG initiative. Increase discussions and trainings to inform PFG staff how various goals (and LOAs) feed into the larger PFG initiative. Develop performance measurements that track the role of WGA coordination activities in promoting effectiveness in implementing the PFG initiative. Cross-Cutting Question 2: To what extent has Partnership for Growth affected the workload of national government and U.S. government staff, as compared to the workload created by traditional forms of development-assistance delivery? Findings • • • • The initiation and implementation of the PFG initiative resulted in an overall increase in the workload of both USG and GOES staff. PFG increased the workload primarily for stakeholders involved in planning. Perceptions of increased workload did not vary between USG and GOES or by gender. Survey respondents believe that increased workload is associated with perceived effectiveness of PFG and WGA. Conclusions • PFG has unambiguously increased the workloads of both USG and GOES staff. However, given that there is no measurable system for tracking workload, it was difficult to measure whether or not this increased workload differed from prior development assistance delivery methods. Recommendations • • • Consider reducing the number of coordination meetings to reduce staff workload. Identify management staffing that could assist in reducing the workload of PFG staff across the Initiative. Identify and promote use of limited resources more efficiently to reduce overall workload. Cross-Cutting Question 3: What contributions has “non-assistance” made to the PFG process, and how can it be utilized moving forward? Findings • • The most frequently cited form of non-assistance involved the diplomatic engagement of senior embassy staff with their GOES counterparts on policy or institutional development issues. The concept of non-assistance 4 is unclear to many PFG stakeholders. 4 USAID defined non-assistance tools as those including “diplomatic engagement, convening authority, and other forms of nonmonetized assistance to engage both governmental and non-governmental stakeholders in support of catalytic policy change and development priorities. Partnership for Growth El Salvador Mid-Term Evaluation Report xvi • • The business opportunities created by PFG and its value in changing public perception are not known and leveraged to their potential. Overall, non-assistance efforts have had mixed success in El Salvador. Conclusions • • • Embassy leadership has provided the clearest example of the successful use of non-assistance in El Salvador. The lack of familiarity of USG and GOES Goal Leads and implementers with the concept of nonassistance means that opportunities to employ non-assistance are being missed. The value of promoting PFG goals, through non-assistance activities is not always evident. Recommendations • • • Conduct training on the diverse forms non-assistance takes and its value to a broad array of beneficiaries. Consider increasing PFG El Salvador staffing geared towards non-assistance. Engage civil-society organizations and the private sector (both large investors and small and medium enterprises (SMEs)) systematically and periodically outside of routine project implementation activities in order to increase understanding of PFG goals and activities and to obtain input on ways to improve PFG. Country-Specific Question 1: For each of the constraints, are the goal-level commitments set forth in the JCAP capable of achieving the constraint-level objectives and outcomes? Findings • • • • • • A formal theory of change to clarify the interdependencies between LOAs and goals in relieving the constraint was not developed. The JCAP is considered a central guiding tool, although it is also considered overly ambitious. The initial selection of activities and LOAs was not based on a theoretical framework; however, subsequent project and program designs and implementation at the LOA-level have contributed to tightening the focus on objectives and outcomes. The ability to tackle constraints and attain goals is subject to the influences of a broad range of factors beyond the domain of PFG, including the politics of the host country The JCAP did not take into account interactions between goals. The JCAP M&E Addendum’s call for the formation of goal-level implementation teams to design, monitor, and update goal-level work plans was not fully executed. Conclusions • • • Given the absence of explicit theories of change, it is not possible to provide a theoretical assessment of the ability of goals selected in the JCAP to affect constraint-level outcomes. Because the constraints and goals are subject to a broad range of political and external factors, achieving constraint level objectives and outcomes cannot depend exclusively on goal-level performance. The theories of change implied in the JCAP— one for tradables, and another for security— are considered to be interdependent. Partnership for Growth El Salvador Mid-Term Evaluation Report xvii • The fact that bilateral goal-level implementation teams and goal-based work plans were not systematically created, makes it more difficult to determine how and to what extent goals are relevant to constraint-level objectives. Recommendations • • • Consider reviewing and revising (as applicable per goal) the JCAP geared towards better specifying how goal-level commitments can attain constraint-level outcomes. At mid-term, systematically create, for each goal, implementation teams and work plans, as required by the M&E Addendum to chart the next 2.5 years of PFG implementation. (Re)define indicators for LOAs and, when necessary, reword their corresponding sections within the JCAP in order to ensure that the indicators are all attainable and can ultimately lead to the realization of constraint-level outcomes. Country Specific Question 2: Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? Findings • • • • • • • The El Salvador PFG produced an M&E Addendum to the PFG at the initiation of PFG, which served as an effective foundational document for M&E of the initiative. Conforming to the Addendum, the El Salvador PFG also successfully introduced a twice yearly bilateral public scorecard process to track PFG progress. The nature of the two sets of indicators used to track the goals related to the security and tradables constraints are different. Monitoring goal progress with PFG scorecards is primarily a negotiated and only partly factbased process although still a useful management tool. Generally, PFG stakeholders felt that better indicators could have been selected to monitor PFG, but they did not readily provide alternatives. Effective M&E at the LOA-level was mostly performed under United States Agency for International Development (USAID) guidance. The M&E process for PFG at the goal and the LOA-levels has evolved, becoming more rigorous over time. Conclusions • • • Generally, while indicators do exist for goal and LOA-level tracking, methods of tracking varied among GOES and USG implementers. Implementers working with USAID track their LOAs through USAID-required tracking systems, but these are not always specific to PFG. While it was generally believed that at least some M&E indicators could be improved, alternate indicators were not readily identified. Partnership for Growth El Salvador Mid-Term Evaluation Report xviii Recommendations • • • • • Rely less on high level constraint-level indicators and more on an analysis of how goals (and goal-level indicators) impact the constraints, to infer progress of the PFG. In the interest of transparency, acknowledge that the scorecard process is an on-going negotiation that will most likely become more consistent and, thus, more objective over time. Actively engage all PFG partners in a continuing exercise to improve tracking and reporting of outcome indicators. Identifying and refining M&E indicators should also be an on-going process. Provide training among PFG staff to leverage existing USAID and MCC rigorous M&E practices. Country-Specific Question 3: At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? Findings • • • Responses varied concerning whether PFG interventions are on track at mid-term. The seven selected goals are largely, but often not completely, on target according to the case study findings. No goal-level implementation teams and goal-level work plans as promised by the M&E addendum were identified. Conclusions • • • Discussions about goals and LOAs do not include results-level indicators and targets. A periodic review process, such as the El Salvador scorecard process, do not address goal-level and constraint-level indicators, as well as any discrepancies between LOA indicators and goallevel indicators in tracking progress. Implementation teams and work plans were not developed for all goals. Therefore, LOA-level progress tracking (except for USAID-led goals) is being done in a less rigorous manner than one might expect based on the M&E addendum. Recommendations • • Leverage the change in GOES administration at the PFG midterm to conduct a goal by goal review of the PFG interventions assessing how LOA performance directly relates to the goallevel indicators to achieve expected outcomes. Establish implementation teams and require them to develop LOA work plans. Overall Summary and Conclusion Overall, the evaluation team found the PFG El Salvador initiative to have made great progress in developing a true partnership, where both governments have a seat at the table in decision making. Further, WGA in El Salvador is to be applauded in its ability to have focused the El Salvador initiative, and promote efficiency. Overall, objectively verifiable information is collected within the initiative. While improvements can be made to monitor data in a more systematic manner, leveraging a structured bilateral management team and with goal-level work plans, the progress of several PFG selected interventions provides evidence of the effectiveness of the overall initiative at mid-term. Partnership for Growth El Salvador Mid-Term Evaluation Report xix 1. INTRODUCTION 1.1 Background and Context The Partnership for Growth (PFG) aims to achieve accelerated, sustained, and broad-based economic growth in partner countries, including El Salvador and the Philippines, through bilateral agreements between the United States Government (USG) and the partnering countries’ national governments. Using principles set forth in President Barack Obama’s September 2010 Presidential Policy Directive on Global Development, the PFG requires rigorous, joint analyses of countries’ individual constraints to growth to develop joint action plans to address the most pressing of these constraints and to establish high-level mutual accountability for the goals and activities selected to alleviate them. The Presidential Policy Directive on Global Development, which recognizes that global economic development “is vital to U.S. national security and is a strategic, economic, and moral imperative for the United States,” calls for the elevation of “development as a central pillar of [U.S.] national security policy, equal to diplomacy and defense.” 5 The directive is based on the premise that “where leaders govern responsibly, set in place good policies, and make investments conducive to development, sustainable outcomes can be achieved.” 6 The directive calls for • elevating broad-based and sustainable economic growth; • increasing the focus of resources, policy tools, and engagement in support of select countries and sub-regions where the conditions are right to sustain progress; • increasing investment and engagement in development-focused innovation; • underscoring the importance of country ownership and responsibility; and • reorienting the USG approach to prioritize partnerships from policy conception to implementation. The countries of El Salvador, the Philippines, Tanzania, and Ghana were selected as the first group of countries in which the U.S. and partner governments would attempt to structure new PFG initiatives, with selection based, in part, on each country’s record of accomplishment in implementing ongoing Millennium Challenge Compacts. In November 2011, the USG and the Government of El Salvador (GOES) signed a Joint Country Action Plan (JCAP) that identified two primary constraints to Salvadoran economic growth and 20 goals aimed at easing those constraints. Each goal is associated with multiple lines of action (LOAs) for a grand total of 153 LOAs (see Annex 5) to be undertaken by the GOES and the USG. It is believed that carrying out the agreed-upon LOAs will lead to goal achievement, which, in turn, will mitigate the effects of currently binding constraints and accelerate and sustain El Salvador’s rate of broad-based, inclusive economic growth. 5 Remarks by the president at the Millennium Development Goals Summit, September 22, 2010. http://www.whitehouse.gov/thepress-office/2010/09/22/remarks-president-millennium-development-goals-summit-new-york-new-york 6 Partnership for Growth Fact Sheet. November 2011. http://www.state.gov/r/pa/prs/ps/2011/11/177887.htm Partnership for Growth El Salvador Mid-Term Evaluation Report 1 1.2 Purpose and Scope of the Evaluation The evaluation of the PFG initiative in El Salvador serves two purposes. As detailed in the statement of work (SOW), the evaluation first assesses: whether the PFG process demonstrates improvements over pre-PFG assistance approaches. In particular, the evaluation will examine the extent to which the PFG’s whole-of-government approach (WGA) and Constraints Analysis led to a change in the manner of USG delivery of development assistance and whether these changes demonstrated improvements in terms of operational efficiency, selection, coordination, design and management of development interventions, and ultimately increased the probability and effectiveness of assistance efforts in achieving verifiable results. The findings and conclusions of this part of the mid-term evaluation will help decision makers determine whether PFG indicates an improved model for providing assistance and whether it portrays a higher probability of achieving desired development results. 7 The second purpose of the evaluation is to determine the following: (i) whether PFG efforts “have been developed in such a way as to allow for the eventual determination of their impact in addressing the identified constraints and desired outcomes.” 8 (ii) “whether or not [measurable elements of PFG] are moving in the right direction, are considered necessary and sufficient to achieve PFG goals, and are contributing to national interests through the integration and coordination of work done by both governments.” 9 1.3 Evaluation Questions As outlined in the SOW, six principal questions guided the evaluation. The evaluation team also developed sub-questions, as appropriate, which further probed specific topics to evoke in-depth responses. The six questions are listed below. Cross-Cutting Questions 1) What are the advantages and/or disadvantages of the PFG whole-of-government approach to development assistance? 2) To what extent has PFG affected the workload of national government and U.S. government staff, as compared to the workload created by traditional forms of development-assistance delivery? 3) What contributions has “non-assistance” made to the PFG process, and how can it be utilized moving forward? 7 Statement of Work for Partnership for Growth (PFG) Mid-term Evaluation: El Salvador and The Philippines, pp. 1–2. IBID 9 IBID 8 Partnership for Growth El Salvador Mid-Term Evaluation Report 2 Country-Specific Questions—El Salvador 4) For each of the constraints, are the goal-level commitments set forth in the JCAP capable of achieving the constraint-level objectives and outcomes? 5) Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? 6) At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? The evaluation team’s findings from the country-specific portion of the mid-term evaluation will be crucial in informing decisions related to continuing PFG. Per the SOW, the findings will provide “tangible input to the national government and USG entities for identifying obstacles and optimizing PFG implementation in the field, allowing for country program course corrections where feasible and needed in order to enhance the likelihood of achieving sustainable, cost-effective and measurable results.” 10 2. METHODOLOGY The evaluation team employed a variety of data collection approaches to that would respond to the six key evaluation questions for the Partnership for Growth (PFG) mid-term evaluation. The overall evaluation utilized a mixed-method approach to both data collection and analysis to ensure that the findings would provide useful information for future PFG initiatives and evaluations. 2.1 Evaluability The statement of work required the evaluation team to assess the evaluability of the PFG initiative’s Joint Country Action Plan (JCAP) as a whole and the evaluability of specific goals and their corresponding LOAs. Through a document review, the team assessed the evaluability of the PFG initiative and its goals by first determining the following: (i) the availability of data; (ii) the problem diagnostic and baseline situation; (iii) the causal logic of activities, objectives, and outcomes; and (iv) the intended beneficiaries. Once the team confirmed that the initiative as a whole and most of its goals could be evaluated, it developed a goal-selection process, as it was not possible to evaluate all goals. 2.2 Goal-Selection Process for In Depth Review of 7 Goals Given that there are 20 goals (14 under the security constraint and six under the tradables constraint) and 153 LOAs (see Annex 5), an in-depth review of all goals for this evaluation was not possible due to time, budget, information, and documentation constraints. The SOW, therefore, required the evaluation team to select and evaluate at least two goals, one per constraint. To conduct this assessment, the evaluation team endeavored to select goals that would best represent PFG themes, focusing on constraint and subconstraints, subject matter, and multiagency cooperation. Also, goals were chosen to reflect the diversity of partnering agencies and implementing partners and the extent to which their LOAs represented 10 IBID Partnership for Growth El Salvador Mid-Term Evaluation Report 3 initiatives formed under PFG. Figure 2.1 below provides an overview of the two-stage goal-selection process. Figure 2.1: Evaluation Designed Process to Select Goals for In-Depth Review Phase one: Subconstraint and WGA/interagency representativeness Low Productivity in Tradables Constraint Crime and Insecurity Constraint Institutional Strengthening (Goals 1-7) Crime and Violence Prevention (Goals 8-14) Loosening Constraints on Capital Formation (Goals 2, 3, 4) Improving the Business Environment (Goals 1, 5) Promoting Innovation and Internationalization (Goal 6) WGA Representativeness: Maximum Number of (USG & GOES) Agencies Participating Phase two: Other criteria relevant to the selection of goals under the crime and insecurity constraint Maximum Representation across Lead (USG & GOES) Agencies Multi-Goal Overarching Program Representativeness Maximum Representation Across Implementing Partners PFG Representativeness (Representative of Activities Initiated with PFG) The evaluation team selected the following 7 goals for in depth review: Goals Related to the Crime and Insecurity Constraint Goal 1-2: Professionalize justice sector institutions and improve criminal justice practices and procedures to make them more effective in combating crime and insecurity in El Salvador, as well as enhance the public perception of these government institutions. Goal 4: Facilitate economic growth by ensuring El Salvador’s labor force is protected from crime while transiting to and from work and ensuring that the public transportation service providers serving the labor force are protected from crime. Partnership for Growth El Salvador Mid-Term Evaluation Report 4 Goal 11: Prevent crime and violence in key municipalities of El Salvador and support reforms as outlined in components 2 (Social Prevention of Violence and Crime) and 5 (Institutional and Legal Reform) of the National Policy for Justice, Public Safety, and Violence Prevention. Goal 12: Reduce overcrowding in prisons, thereby allowing the Salvadoran prison system to safely, securely, and humanely manage an increasing population. Goals Related to the Low Productivity in Tradables Constraint Goal 3: Improve the quality of the education system in order to create a more highly qualified and technologically skilled labor force. Goal 5: Support a strategy for attracting and promoting foreign direct investment (FDI) and making El Salvador a more attractive place for foreign investment. Goal 6: Surmount low productivity of tradables by transforming factors of production of the tradables sector through the implementation of strategies to improve innovation and quality and a focus on the international market. 2.2 Methods of Data Collection and Data Analysis 2.2.1 Data Collection Approaches The evaluation team began the data collection and review process with a preliminary informationgathering and extensive desk review. The team then conducted sixteen in-person interviews in Washington, D.C. with U.S.-based stakeholders and then traveled to El Salvador for twenty days beginning on March 31, 2014, where it conducted 115 interviews. Evaluation, trade, and security subjectmatter experts conducted the interviews. The remainder of this section is a summary of the data collection and analysis approaches undertaken in this evaluation. A detailed description of the methodology is provided in a separate report—the Updated Methodology and Evaluation Plan (UMEP). All data collection and analysis activities followed USAID Evaluation Policy: January 2011. 11 Desk Review: The evaluation team conducted a desk review based on the program documentation received primarily from the United States Agency for International Development (USAID) for the PFG initiative as a whole, as well as documents that pertained exclusively to the El Salvador interventions. All reviewed information was categorized per the SOW’s guidance in a desk review matrix and used to identify key themes of each of the six evaluation questions (see Annex 7 for document reference list). Semi-Structured Interviews: The evaluation team conducted semi-structured interviews with key PFG stakeholders including high-level leadership, leadership, architects, Goal Leads, implementers, and independent experts in both the United States and in El Salvador. USAID identified the initial list of stakeholders, and the evaluation team then developed interview protocols suitable for each of the above 11 USAID (2011, January). “Evaluation Policy” http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Partnership for Growth El Salvador Mid-Term Evaluation Report 5 categories of stakeholders. The interviews were semi-structured in nature, thus allowing for an open framework, conversational communication, and probing questions. The initial list of stakeholders to interview included 45 GOES, 64 USG, and 102 LOA implementers and independent experts. A total of 115 interviews were conducted (31 with GOES stakeholders, 39 with USG stakeholders, and 45 with independent implementers and experts). Table 2.1 lists the number of intended interviewees, the actual number, and the response rates by government, affiliation, and respondent’s primary role. The overall response rate was 52 percent, the response rate for GOES respondents was 69 percent, and the response rate for USG respondents was 61 percent. Table 2.1: Number of stakeholder interviews Stakeholder Type Leadership Architects Goal Leads Government Sub-Total Government Total Stakeholder Type United States Government # of intended # of respondents respondents 30 17 (57%) 15 4 (27%) 19 18 (95%) 64 39 (61%) Government of El Salvador # of intended # of respondents respondents 18 9 (50%) 5 4 (80%) 22 18 (82%) 45 31 (69%) 70 (64%) Independent Stakeholders # of respondents 32 (52%) 13 (32%) 45 (44%) # of intended respondents Implementers 61 Experts 41 Total 102 Total Number of 115 Respondents Note: Response rate = # of respondents / # of intended respondents. Response rates are in parentheses next to the number of respondents. Table 2.2 below lists the cross-cutting and country-specific questions that were asked of each stakeholder type (see Annex 4 for the semi-structured interview guides per stakeholder type). Table 2.2: Evaluation Questions Asked per Respondent Type in Semi-Structured Interviews Stakeholder Type CCQ1 CCQ2 CCQ3 CSQ1 High-level leadership X Leadership X X X X Architect X X X X Goal Leads X X X X LOA implementers X Independent experts X X X NOTE: CCQ means cross-cutting question and CSQ means country-specific question. CSQ2 X X X X X CSQ3 X X X X Web-based Survey: The web-based survey, which included 19 questions, was administered directly to PFG staff (GOES and USG), including leadership, PFG architects, goal leads and government implementers. However, members of high-level leadership were not surveyed. Table 2.3 below provides the number of stakeholders who received the survey and those who responded, by government affiliation. Partnership for Growth El Salvador Mid-Term Evaluation Report 6 Table 2.3: Web-based survey, by response rate and government affiliation Government Affiliation GOES USG Government Total # of intended respondents # of respondents 46 129 175 26 82 108 Response Rate 57% 64% 62% Note: USG respondents represent USG staff based in El Salvador and also in Washington D.C. (or other posts and used to work in El Salvador) who were part of PFG design, management and/or implementation. Out of the 19 survey questions, 4 were applicable to multiple evaluation questions; a majority (10 out of 19) was applicable to Cross-Cutting Question 2, concerning workload changes; and none were applicable to Country-Specific Question 3. The overall response rate was 62 percent, with a notable variance in response rates between roles and affiliation. For example, the USG response rate was 64 percent (82 responses out of 129 solicitations) while the GOES response rate was 57 percent (26 responses out of 46 solicitations). Annex 4 lists the questions included in the web-based survey. 2.3 Data Analysis The data analysis utilized a mixed-methods approach. The evaluation team, in collaboration with USAID, determined the analysis techniques that were most suitable for answering the six evaluation questions. The team took the necessary steps to ensure that the data analysis produced accurate results and identified any limitations of the analysis. Specifically, prior to conducting data analysis, the evaluation team developed a UMEP report (as previously indicated) that detailed the purpose of the evaluation, questions required by the SOW, data quality assurance procedures, data entry, data cleaning procedures, types of analyses, and limitations to the data analysis. While the evaluation team did undertake quantitative analysis, the methods in this evaluation were largely qualitative and comparative. Interview and web-based survey data were cross-referenced with information obtained during desk reviews, where possible, in order to understand and verify empirical evidence. Specifically, interviews gathered information about stakeholders’ perceptions as they applied to the SOW’s six evaluation questions. These data were distilled into themes that underscored the applicable evaluation questions. The evaluation team recognized the importance of producing valid findings and properly documenting its findings. To this end, the team used a multi-pronged process to ensure that data were analyzed in an accurate and unbiased manner, particularly the data from the semi-structured interviews. This quality control approach involved: • • • • training coders; using a layered coding process—interview summary notes were reviewed by two primary reviewers and a senior reviewer; conducting daily coder meetings; and performing periodic senior-review spot checks (through evaluation and subject-matter experts) to reconcile inconsistencies. The evaluation team also used different data verification methods, including validating summary notes and triangulation. Validating summary notes involved reviewing audiotapes, checking the accuracy of the Partnership for Growth El Salvador Mid-Term Evaluation Report 7 notes, and, if necessary, augmenting the written summary notes used by the coders. The evaluation team also validated the accuracy of data by triangulating across multiple data sources and modes when possible. The evaluation team’s triangulation methods consisted of collecting the same type of information from different sources using different methods. Data collected from the desk reviews, semistructured interviews, and web-based surveys of various stakeholders were compared as much as possible in order to strengthen the reliability of the analysis. Throughout the evaluation, the data received were carefully collated, reviewed continuously, summarized, and synthesized. This systematic approach helped to identify trends and establish the authenticity, validity, and reliability of the reported findings. Semi-structured interviews were coded by evaluation question, question-specific themes, and perceived sentiment. Web-based survey results were aligned with the interview results by respondent and evaluation question. The team also used NVivo and Microsoft Excel coding tools to code interview findings and Microsoft Excel and Stata to conduct online survey-data cleaning and quantitative analysis. It is important to note that for the stakeholder interviews—which provided the majority of information concerning the PFG design, initiation, and implementation—the evaluation team developed modular interview guides for PFG staff and key stakeholders: high-level leadership, leadership, architects, Goal Leads, LOA implementers, and independent experts. Sometimes a respondent would fulfill more than one role; for example, a staff member who was considered leadership because he or she played a managerial role in the PFG structure could also be a Goal Lead, or a Goal Lead could also be an LOA implementer. Despite the multiple roles an individual may play, only one interview guide could be used for the hour-long interviews. Consequently, interview instruments were administered based on a respondent’s primary role as understood by the evaluation team. During analysis, the evaluation team’s approach included counting stakeholders more than once, as many stakeholders fulfilled more than one role. Therefore, quantitative counts provided for leadership, Goal Leads and implementers would reflect a “double count”. 2.4 Evaluation/ Study Limitations Identified study limitations include the following: Lack of counterfactual: Assessing PFG’s performance without a readily available counterfactual or a valid metric of PFG’s alignment with WGA was a limitation to the overall study. Respondents with years of experience were able to compare PFG to more traditional approaches to development, but respondents with little experience had no point of reference with which to compare. Also, the perceptions of those who had more experience and were able to make a comparison may have reflected recall bias. Perceptions gathered through semi-structured interviews: Interviews were structured to address evaluation questions, which provided flexibility for further exploration of themes. Although this approach was a strength, it also presented a weakness in that all responses were subjective, and because the interviews made up the majority of the data collection, the majority of the findings are based on perceptions and not necessarily facts. The online survey included a limited number of queries: The online survey was administered to PFG architects, leadership, and Goal Leads, affording respondents an opportunity to relay their candid impressions of PFG. To promote a high response rate, the online survey was as short and specific as Partnership for Growth El Salvador Mid-Term Evaluation Report 8 allowable. Although respondents were provided with an opportunity to elaborate if they wished, questions were largely closed-ended and did not provide the detail that an interview would. The dichotomous (“yes/no”) and ordinal (“good/average/bad”) nature of most of the variables allowed for quantification and direct comparison. However, these advantages were partially offset by the fact that the evaluator (who designed the survey questions) directed and limited the lines of inquiry. As with the interviews, although they provided another dimension for comparison in developing findings, these responses were also based on perception. Data verification: Ideally, evaluations should be based on “analyzed facts, evidence and data supported by strong qualitative or quantitative evidence and not anecdotes, hearsay or people’s opinions.”12 With information collected from hundreds of sources (108 online survey respondents and 115 interviewees), the evaluation team has confidence in the strength of the qualitative data collected and analyzed. Multiple information sources facilitated cross-validation, allowing the evaluation team to make statements that are supported by the opinions of multiple respondents. However, even the online survey was no more than the quantification of respondents’ opinions. In contrast to opinion, “facts” are characterized as information that can be objectively described in an identical fashion by different observers.13 Rigorous PFG tracking information and monitoring data for each of the goals and LOAs would have been a more robust source to confirm perceptions. However, the documents received did not always provide the required level of rigor for each goal and corresponding LOA. Even scorecards, which were to provide an objective rating of goal progress, were primarily negotiated values and, therefore, did not rise to the level of fact. Although this situation presented a limitation to the evaluation’s findings, when possible, factually based evidence was used to corroborate perceptions. Language: Given that the official language in El Salvador is Spanish, it was the preferred language for conducting the evaluation in El Salvador. The evaluation team included bilingual staff members and local translators and note takers. Interviews with GOES representatives and key stakeholders were conducted in Spanish unless otherwise requested. The online surveys were made available in English and Spanish. Conducting the evaluation in two languages could pose a risk, when translating and analyzing findings. Therefore, the evaluation team developed a field plan, which included various strategies to be taken to ensure that translations were reliable. 12 USAID, November 2012. “How-To Note: Preparing Evaluation Reports.” http://www.usaid.gov/sites/default/files/documents/1870/How-to-Note_Preparing-Evaluation-Reports.pdf 13 Because “facts” should be described identically by different observers, any survey of “fact” would produce no variance; a degenerate distribution. Therefore, survey findings cannot be viewed as “facts.” Partnership for Growth El Salvador Mid-Term Evaluation Report 9 3. KEY GENERAL FINDINGS ON THE EL SALVADOR PFG INITIATIVE This chapter details the key findings from the evaluation—interviewees’ and survey respondents’ overall perceptions of the PFG initiative. These findings provide important context for the entire evaluation. Chapters 4 through 10 then provide an in-depth discussion of the six evaluation questions. For each evaluation question, an analysis of the responses to the question, findings, conclusions/ lessons learned, and recommendations are provided. 3.1 Overall Advantages of the PFG Initiative Finding 1: PFG represents an improvement over previous development assistance strategies. Based on data collected through the online survey only, the evaluation team found that respondents perceived PFG to be an improvement over other development assistance strategies. As shown in figure 3.1 below, a total of 73 percent of respondents believed that PFG was an improvement over other forms of development assistance: 55 percent saw it as “an improvement” and 18 percent saw it as a “significant improvement.” Figure 3.1: Did PFG bring changes, compared to other approaches to development assistance intended to affect economic growth? (Based on survey responses only) Online survey respondents also shared their views on whether the PFG initiative is meeting its ultimate goal of advancing the economy in El Salvador. As illustrated in figure 3.2 below, 43 percent of respondents agreed that PFG is meeting its goal, with 8 percent stating that they strongly agree and 35 percent stating that they agree that PFG has been a catalyst for promoting economic growth. The perception of PFG’s effectiveness on overall economic growth varied.14 14 This finding is discussed in further detail in Country Specific Question 3, which examines whether the PFG initiative is on target. Partnership for Growth El Salvador Mid-Term Evaluation Report 10 Figure 3.2: Is PFG meeting its goal of advancing economic growth in El Salvador? (Based on survey responses only) The team obtained further insight into the two questions above—whether PFG was an improvement over other approaches for development assistance and whether it was meeting its goal in El Salvador—through interviews. In answering these two questions, interviewees frequently mentioned three topics in relation to the PFG: positive movement on policy reforms, the importance of the Constraints Analysis, and additional leverage provided by the initiative. Below is a summary of the findings related to each of these topics. Finding 2: PFG has aided in the positive movement of policy reforms. Overall, interviewees believed that PFG engaged senior management and section heads from the USG embassy in a timely manner to promote critical policy and institutional reforms related to the PFG initiative. Policy reform, including legislation required to implement policy reforms is central to many PFG goals. All of the USG and about half of the GOES leadership identified the strong leadership of the United States (US) Ambassador and the engagement of other senior USG embassy officials as a major catalyst for key policy reforms. One respondent stated, “From my perspective, PFG has succeeded in effectively branding the U.S. government’s development assistance in El Salvador. In addition, requiring senior-level commitment by the Government of El Salvador to [implement] the action plan gives us more leverage in pushing for them to live up to these goals of programs.” Finding 3: Initial PFG planning through the CA activity laid a good foundation for the PFG initiative. Most interviewees also considered that the objectivity of the CA, which was the initial defining exercise upon which PFG constraints were selected, was important in providing a focus for the USG assistance effort. As a USG official involved with the initial El Salvador PFG process recalled, “There was some resistance… as we had to phase out what did not relate to the two constraints and was falling by the wayside. Selecting things and not selecting others was the hardest part.” USAID, in particular, had to give up some of its jurisdiction (or “equities”) but continued to be a major player. Some USG PFG leadership representatives also observed that more constraints might have re-introduced a sector-like approach, creating fewer incentives for interagency coordination. However, some interviewees did question the final Partnership for Growth El Salvador Mid-Term Evaluation Report 11 choice of constraints, with their focus on economic outcomes, which made social, environmental, and health outcomes seem secondary. Partnership for Growth El Salvador Mid-Term Evaluation Report 12 4. CROSS-CUTTING QUESTION ONE: WHAT ARE THE ADVANTAGES AND/OR DISADVANTAGES OF THE PFG WHOLE-OF-GOVERNMENT APPROACH TO DEVELOPMENT ASSISTANCE? The whole-of-government approach (WGA) was initially utilized within fragile and emerging countries, 15 motivated by the evolving threat of terrorism and the desire to mitigate risks and adverse outcomes by ensuring that information is shared between (and among) agencies.16 As evidenced by PFG, the concept of WGA has also been leveraged more broadly within the international development arena. WGA can be defined as an aspiration to achieve horizontal and vertical coordination between stakeholders in a particular policy area in order to eliminate situations in which different policies undermine each other and to use scarce resources better. 17 Similarly, the Australian Management Advisory Committee’s Connecting Government report (2004) says of WGA: “Whole of government denotes public services agencies working across portfolio boundaries to achieve a shared goal and an integrated government response to particular issue.” 18,19 WGA has been utilized widely due to the several benefits that accrue from practicing this approach. For example, WGA has been recognized as a system for enhancing “the quality of services and benefit[ing] participating organizations by offering better processes, improved relationships, and greater capacity to respond to local needs, as well as more efficient use of resources.”20 Based on the definitions of WGA above, the three main components needed to demonstrate an effective WGA include: • • • a shared interest/objectives by multiple organizations and/or agencies; leadership that promotes WGA within management and coordination; and accountability mechanisms for fostering the approach. Overall, based on the evaluation team’s observations, interviews, and survey responses, the team concluded that the El Salvador PFG initiative’s WGA demonstrates the first two criteria above but has room for improvement in establishing the third. The evaluation team analyzed the advantages and disadvantages of the WGA as a core component of the PFG El Salvador initiative as well as PFG’s alignment with WGA. According to the evaluation statement of work, specific to PFG: Within the U.S. government, the term “whole of government” reflects efforts to align each agency’s activities to achieve a common objective. PFG calls upon the US 15 Organization for Economic Co-operation and Development (OECD) Whole of Government Approaches to Fragile States Report (2006). 16 Hammond, T. (2004) Why is the intelligence community so different (difficult?) to redesign? Paper presented at the SOGconference, University of British Colombia, Vancouver, June 15–17). 17 Christensen, T., & Laegreid, P. (2007). The Whole-of-Government Approach to Public Sector Reform. Public Administration Review, 67(6):1059-1066 18 Australian Management Advisory Committee’s Connecting Government report (2004) 19 Ling, Tom. (2002). Delivering Joined-Up Government in the UK: Dimensions, Issues and Problems. Public Administration 80 (4): 615 – 42 20 Humpage, L. (2005). Experimenting with a ‘Whole of Government’ Approach: Indigenous Capacity Building in New Zealand and Australia. Policy Studies, 26(1): 47-66 Partnership for Growth El Salvador Mid-Term Evaluation Report 13 Government (USG) and partner countries to be more comprehensive and creative in our development work—to reach beyond aid to all the instruments that both governments can bring to bear to connect and amplify the impact of current investments and unlock growth potential.21 Information about the advantages and disadvantages of the PFG’s WGA were collected through semistructured interviews conducted primarily with PFG staff members, including members of leadership, architects, and Goal Leads from USG and GOES. Independent experts were also asked about their perception of PFG’s performance to date as compared to more traditional approaches in international development. Additionally, the evaluation team also administered an online survey that allowed respondents to provide their impressions of WGA, although it was largely in the context of the additional workload that interagency cooperation requires. 4.1 Findings on Advantages of PFG WGA Overall, more interview respondents characterized PFG WGA as advantageous (49 out of 75) than having no advantages (6 out of 75 respondents) 22 (see table 4.1). More advantages were noted by USG personnel (78 percent) than by GOES personnel (54 percent) as illustrated in figure 4.1 subsequently. Table 4.1: Interview responses referencing advantages of PFG WGA No. of advantages of PFG WGA mentioned At least one None USG GOES Leadership Goal Leads Expert s 13 (76%) 1 (6%) 3 (18%) 15 (83%) 0 (0%) 3 (17%) 0 (0%) 1 (100%) 0 (0%) USG TOTAL 28 (78%) 2 (5%) Leadership Goal Leads Experts 2 (22%) 1 (11%) 6 (67%) 11 (61%) 1 (6%) 6 (33%) 8 (67%) 2 (16.5%) 2 (16.5%) GOES TOTAL USG and GOES Total Answers 21 (54%) 49 4 (10%) 6 No 6 (17%) 14 (36%) 20 Response 36 39 Total 17 18 1 9 18 12 75* Interviewed (100%) (100%) ∗ While a total of 115 persons were interviewed, only 75 were asked this question (CCQ1). See tables 2.1 and 2.2 for details. 21 Statement of Work – Partnership for Growth Mid-Term Evaluation: El Salvador and The Philippines, September 2013, pg 10. The acronym WGA itself was not very well known by Salvadoran respondents, but most correctly identified it as interagency coordination. 22 Partnership for Growth El Salvador Mid-Term Evaluation Report 14 Figure 4.1: USG and GOES interview respondents’ perceptions of PFG WGA advantages* *Based on 75 respondents. See table 4.1 for details. Table 4.2 below suggests that Salvadoran interview respondents realized more disadvantages of PFG WGA than their USG counterparts did, but overall only 19 of the 75 respondents mentioned any disadvantages. Table 4.2: Interview Responses Referencing Disadvantages of PFG WGA No. of Disadvantages of PFG WGA mentioned At least one None No Response Total Interviewed 4.1.1 USG Leadership 4 (23%) 10 (59%) 3 (18%) 17 (100%) Goal Leads 2 (11%) 13 (72%) 3 (17%) 18 (100%) Experts 1 (100%) 0 (0%) 0 (0%) 1 (100%) USG TOTAL 7 (19%) 23 (64%) 6 (17%) 36 (100%) GOES Leadership 2 (22%) 1 (11%) 6 (67%) 9 (100%) Goal Leads 4 (22%) 8 (45%) 6 (33%) 18 (100%) Experts 6 (50%) 4 (33%) 2 (17%) 12 (100%) GOES TOTAL 12 (31%) 13 (33%) 14 (36%) 39 (100%) USG and GOES Total Answers 19 (25%) 36 (48%) 20 (27%) 75 (100%) Implementation of Whole of Government Approach by USG PFG Staff Finding 1: The Constraints Analysis and Joint Country Action Plan (JCAP) facilitated WGA dynamics. The review of the Constraints Analysis itself was not part of the statement of work for the mid-term evaluation. However, the Constraints Analysis does provide context for the development of the PFG and the establishment of its principles, targeted constraints, related goals, and LOAs, and came up during most of the interviews. Among architects and leaders involved in the initial stages of the PFG process, the Constraints Analysis was often mentioned as necessary for promoting the WGA. Four of the seven USG architects identified the Constraints Analysis as an essential part of the PFG system, promoting the initiation of WGA. All interviewees who explicitly commented on the Constraints Analysis suggested that it was a very useful exercise, explaining that beginning the PFG process with an objective technical Partnership for Growth El Salvador Mid-Term Evaluation Report 15 analysis of the constraints to economic growth gave the process credibility. As evidenced by a USG leader’s testimony, which was representative of other USG perceptions, the Constraints Analysis helped focus the attention of both governments on a finite number of central development objectives. The technical exercise also provided the opportunity to bring together USG agencies, catalyzing coordination and cooperation as the PFG progressed. Additionally, all three independent experts who discussed the advantages of PFG WGA indicated that having a variety of stakeholders participate in an evidenced-based, organized technical process promoted collaboration during project initiation and implementation. One respondent shared that initially Salvadoran counterparts were skeptical about the US providing the methodology; however, they later realized its virtue and appreciated its ability to produce a uniform approach to addressing obstacles to growth and launching the PFG initiative. Finding 2: Within the USG, the WGA has focused human and operational resources on the policy and programmatic objectives of the PFG. From the point of view of USG agency officials, the PFG has unequivocally improved coordination in several aspects of USG development assistance in El Salvador. Eight out of nine members of USG leadership in El Salvador who responded to the WGA interview question shared this sentiment. A USG leader and architect shared the following: The interagency [cooperative system] seems like a normal thing to do, but the reality is that you seldom see clear lines of coordination [in our aid approach], and the PFG approach was meant to change this. The whole idea of the El Salvador PFG was to guarantee a coordinated approach to development among USG agencies in the first place, as much for the benefit of ourselves [and] the GOES, but also other donors. Another leader within the security sector stated that as a result of PFG, WGA has led to a change in how agencies such as the National Security Agency, the Federal Bureau of Investigation (FBI) and the Bureau of International Narcotics and Law Enforcement (INL), which would normally work in silos, now participate in leadership meetings and work collaboratively with other USG agencies. Such coordination has prompted related agencies to focus more keenly on common objectives and to enhance implementation coordination. Further, a member of the PFG USG leadership shared a commonly held view that because of PFG, the level of coordination and collaboration within the Embassy in El Salvador had increased, if compared with other missions. Fourteen of the Goal Leads who were interviewed also shared the sentiment that PFG WGA has promoted more collaboration within USG agencies than previous development assistance practices had. For example, a USG leader explained that because of the PFG WGA, USG agencies are currently working together toward the same goal. Four Goal Leads further expressed that USG interagency collaboration has assisted GOES to understand that responses to queries would be coordinated no matter which USG agency is contacted; this knowledge has improved working relationships and the overall system of providing development assistance. Also, because PFG was agreed upon by the executive branch of GOES, the USG Goal Leads are able to refer their GOES Goal lead counterparts to agreements that were made at high levels and cannot be easily ignored. Such a system of buy-in at the highest levels helps to ensure cooperation both within and across government agencies. As another USG Goal Lead stated, “the Partnership for Growth El Salvador Mid-Term Evaluation Report 16 main advantage [is that] [PFG WGA] gives a perception of something a little bit more urgent to be done, and it’s agreed upon at a higher level.” Still another Goal Lead from USAID explained, “the PFG has forced us to work better among ourselves and coordinate among our programs and arrive more organized at their [the GOES] door, while it also [provides] more clarity to the GOES on what we do and how we do it.” Finding 3: WGA has led to increased coordination within USG and as a result, has encouraged coordination among USG Partners. In addition to improved coordination within USG as a result of WGA, all leadership officials and Goal Leads provided examples of PFG WGA leading to improved coordination with other entities. In particular, they discussed increased coordination between the USG and GOES and increased coordination between the USG and other donor agencies. This increased cooperation could be interpreted as an indirect benefit of WGA. Essentially, given that the USG team has realized the benefits of collaboration, they have made a concerted effort to use this method when working with other entities in El Salvador. Additionally, a USG Goal Lead explained, “to the extent that the Salvadoran side had to mirror how the [US] Embassy worked on PFG, the effect was to encourage an unprecedented and entirely positive level of interagency cooperation within the GOES.” One Goal Lead working in the education sector explained that the most important achievement regarding PFG had to do with improved coordination among all Salvadoran development donors, explaining that when using more traditional development approaches, similar donors tended not to communicate. However, once PFG Goal 10 (improve educational opportunities for in-school and out-of-school youth in targeted high-risk municipalities with high crime rates) was initiated with the Ministry of Education (MINED), for instance, various funders collaborated to successfully implement the inclusive schools approach. Further, an architect shared extensive changes that have been realized in the way the US and GOES interact on security goals, although the extent to this increased interaction differs depending on the respective goal. Finding 4: PFG WGA has been promoted due to the purposeful design that has grouped lead agencies together under individual goals, who would otherwise not have work together. The evaluation team found that a conscious effort was made to incorporate the WGA into the programmatic aspects of the PFG from the onset to guarantee that each principal USG agency active in El Salvador was assigned responsibilities as Goal Leads or co-Goal Leads. One high-level U.S. Department of State (DOS) leadership official who was directly involved with the initial stages of the El Salvador PFG indicated that this was a conscious decision made to promote interagency collaboration. This decision ensured that agencies assigned to different goals under the PFG initiative worked together. Nevertheless, the evaluation team found that the positive outcomes of the WGA were most evident within individual goals, but not at the inter-goal level, i.e., when different USG agencies operated under the same goal. For example, under Goal 4 of the security constraint (ensuring protection from crime and extortion in El Salvador’s public transportation service), not only did typical law enforcement agencies such as the INL, ICITAP and Office of Overseas Prosecutorial Development, Assistance & Training (OPDAT) coordinate better on various task-related LOAs, they also coordinated with USAID (through its Transparency program), which has responsibility for an additional LOA under this same goal. Typically, these agencies would have worked in isolation; however, they now coordinate efforts because they were purposefully assigned to the same goal. Partnership for Growth El Salvador Mid-Term Evaluation Report 17 USAID and INL’s work on community policing is another example of coordination between agencies working on the same goal under the security constraint (Goal 9: support the PNC to strengthen its service orientation). Prior to PFG, different development approaches existed without much coordination or dialogue between the two agencies, as each had its own vision of community policing. Further, the INL’s mandate allows it to implement some activities that USAID cannot, such as donating computers and other hardware to the police. Therefore, by collaborating, the two agencies together can be more effective in achieving their goals and helping the communities with which they work. According to Goal Leads, this positive change in implementation under PFG has also led to improved coordination and dialogue, allowing the two agencies to complement each other’s mandates and better understand and respect what each agency is bringing to the table. Work on Goal 10 (improving educational opportunities for in-school and out-of-school youth in high-risk municipalities) also presents another example of increased collaboration within the security sector as a result of the PFG El Salvador initiative’s purposeful pairing. The LOA of this goal built on pre-PFG USAID work. However, with its inclusion under the security constraint, USAID had to more explicitly relate and coordinate these activities with crime and violence prevention approaches and activities, in particular assigned activities implemented by the INL under this goal. The evaluation team found through interviews that USAID officials involved in Goal 10 perceived this change as an enriching and positive development, which has led to new partnerships and a more comprehensive approach to conducting their work. The tradables constraint presents a less pronounced situation, as fewer agencies are involved with the tradables constraint, and their missions are more closely aligned. Nonetheless, a number of specific examples of improved coordination were offered by interviewees as a result of bringing these agencies together. For example, for Tradables Goal 6 (surmount low productivity of tradables), USAID, the Department of Agriculture, and the Customs Service worked to facilitate the entry of Salvadoran exports into the U.S. by enabling Salvadoran producers and exporters to meet phytosanitary, documentation, and other standards through a ‘one-stop-shop’. The implementation of this ‘one-stop shop’ has created a better experience for beneficiaries. Finding 5: Within the security constraint, WGA has enhanced USG agencies’ understanding of overlapping goals. Stakeholders working within the security constraint indicated that USG-security-related agencies (INL, Department of Justice (DOJ), ICITAP, OPDAT and FBI) typically work mostly within silos, given the sensitive nature of their work. Thus while not all of their work can lend itself to WGA, this sector of respondents indicated that PFG’s WGA had enriched their work in terms of collaboration with other entities within USG. It also promoted improved efficiencies where possible and better understandings of what other USG agencies are doing in the same field. For instance, a Goal Lead from INL stated, “We already coordinated and continue to coordinate as the law enforcement group, but that does not include USAID. Due to the PFG, however, we now also coordinate with USAID and other non–law enforcement agencies.” This increased interagency coordination with non-law enforcement agencies has led to enhanced coordination with implementing crime prevention activities between USAID and law enforcement agencies. Similarly, among USAID officials, awareness about the work of law enforcement agencies has increased, and there is less hesitation to reach out for partnership and synergies. Partnership for Growth El Salvador Mid-Term Evaluation Report 18 The team found that although WGA is viewed as essential and inherent to PFG, some PFG goals cannot easily follow a WGA model. For instance, Security Goal 14 (extradition) is mainly pursued on a case-bycase basis and often involves highly sensitive information that simply does not lend itself to an interagency approach. That said, the evaluation team found that the FBI very much appreciates being part of the PFG because it feels that non-law-enforcement agencies, such as USAID, now better understand its work and objectives and how the work is relevant to the other security sector goals as well. The FBI also expressed that it now has a better understanding of what non-law enforcement agencies bring to the security and justice sector reform agenda. 4.1.2 Implementation of Whole of Government Approach by GOES PFG Staff Finding 6: PFG has improved cooperation among GOES agencies. GOES leadership and Goal Leads appreciate the intra-governmental coordination and cooperation that result from PFG. Two out of the eight GOES leaders who responded to this question mentioned that as a result of PFG WGA, they have become more organized, and their agencies are more coordinated. They both noted that because of PFG WGA, all agencies within GOES that work with USG have coordinated under a common goal. Six of the fifteen GOES Goal Leads who responded to this question specifically identified an increase in coordination within GOES as a direct advantage of PFG WGA. All of the respondents working on both the security and tradables constraints expressed similar sentiments about PFG WGA’s ability to improve coordination within GOES. A GOES security Goal Lead reported that as a result of PFG WGA, the council, district attorney, and attorney general collaborated in responding to security-related goals. Another explained that the increased coordination has led to empowering the middle-level Goal Leads to be much more proactive and to work with higher level-officials to complete projects. In addition, the PFG-related meetings of Goal Leads and LOA implementers improved GOES coordination of activities. PFG also aided in the creation of taskforces comprised of different agencies to work on a single problem. One such example is the Transportation Task Force created under Goal 4 (facilitate economic growth by ensuring El Salvador’s labor force is protected from crime while transiting to and from work) to combat crimes within the public transportation system. This taskforce brought together personnel from the National Civil Police (PNC), the Ministry of Transportation, and the Prosecutor’s Office (Fiscalía) to combat the problem of robbery and extortion on public buses. Similarly, under Tradables Goal 3 (improve the quality of the educational system and improve skills training), Salvadoran training institutions and the Ministry of Labor worked together to match students’ skills to employers’ needs. Finding 7: PFG WGA has led to increased coordination between GOES and USG and ultimately led to increased efficiency. All ten GOES Goal Leads who responded to this question indicated that coordination has increased between GOES and USG as a result of PFG WGA. For example, all USG and GOES Goal Leads and coGoal Leads indicated that their respective agencies are coordinating their activities in a more uniform manner and more frequently, as compared to approaches undertaken prior to PFG. Based on interviewee testimony, such coordination has led to increased efficiency. One Goal Lead indicated that USAID now has a greater presence within the El Salvador justice system because of PFG WGA. Another Goal Lead shared that the increase in coordination has led to increased decision making power for the GOES. The Partnership for Growth El Salvador Mid-Term Evaluation Report 19 respondent stated that in the past, USG imposed projects based on its own judgment, but currently with PFG, responsibility for decision making is shared between USG and GOES. It was also apparent from the interviews with GOES leadership and Goal Leads who shared their views in response to this question that although coordination improved between USG and GOES agencies as a result of PFG, GOES could not singularly focus on PFG. GOES leaders noted that although PFG encompassed virtually all USG assistance activities in El Salvador, GOES had many functions outside of PFG and a range of other bilateral and multilateral development programs with different priorities and approaches. 23 Finding 8: Other WGA advantages provided by GOES leadership and Goal Leads through interviews. • • • • • 4.1.3 El Salvador’s selection as one of four countries implementing the PFG approach has led to increased interest in El Salvador’s economy and has enhanced both domestic and international private sector interest in El Salvador. Mid-level leadership with the GOES now has the ability to interact with higher level leadership in GOES and present their projects and concerns more freely. As a result of the PFG initiative, the GOES has now learned ways to better collaborate with its various partners, such as improving dialogue with the Salvadoran private sector. As a result of improved collaboration and communication, the Salvadoran private sector has become a priority for both governments. Improved coordination has led to better alignment of responsibilities and accountability among GOES staff. MINED has adopted a more efficient system to promote security-related education needs. Challenges/ Disadvantages of PFG WGA Only 19 respondents out of 75 in the semi-structured interviews mentioned any disadvantages associated with PFG WGA (see Figure 4.1 and table 4.2). Therefore, because the universe for response is minimal, this section details all the disadvantages discussed by interviewees, regardless of the number of respondents who discussed them. Finding 9: Time is needed to truly implement WGA coordination activities. In using the WGA, a lot of time was devoted to developing consensus, which meant that setting up goals in preparation for implementation took longer than it would for projects and programs that do not require WGA and increased coordination and consensus. This could be viewed as a challenge due to the prolonged time it took to begin implementation. One Goal Lead explained that given that relations were new with the new GOES administration, it took some time before consensus could be reached on how to move forward. Another Goal Lead expressed a similar sentiment, stating, “one thing is to accept the job as it is and have expectation to get things done as they are. However, [given that] funds are from existing funding streams, not directed towards PFG, additional coordination had to occur to decide how to fund project implementation.” The same Goal Lead further explained that similar to funding complications, 23 The level of effort is discussed in more detail in the section responding to workload changes. Partnership for Growth El Salvador Mid-Term Evaluation Report 20 there were also goals for other funding sources and PFG goals that needed to be coordinated. Even though sometimes the goals overlapped, it took time to ensure that Goal Leads appropriately adhered to PFG requirements. Time was also needed for internal coordination and meetings, which usually occurred once a week (and at initiation, even more frequently). Finding 10: WGA coordination is more limited across goals than within goals (i.e. among LOAs of the same goal) and lessons learned are not systematically shared across goals. Based on the interviews and the review of project documentation, the team discovered that coordination, synergy, and other aspects of WGA were not as widespread across goals as they were within goals. The evaluation team gathered that this can be attributed to the project being in the first two years of implementation, which has essentially been a start-up period when each goal and its team spent a significant time orienting themselves in order to implement LOAs. As one of the implementers stated, “coordination among all those who work on the same topic—crime prevention at a local level—could still be improved. Various agencies are now involved with peaceful coexistence plans for schools (Planes Escolares de Convivencia), but we do not sufficiently share the findings, and thus risk constantly having to reinvent the wheel.” The evaluation team also found that the scorecard elaboration process did not provide sufficient opportunity for knowledge sharing across goals. During the Goal Lead scorecard meeting that the evaluation team witnessed, findings per goal were shared with all the other goals, but each Goal Lead had only three minutes to present progress related to the goal. The short presentations for each goal were also not followed by an overall presentation on WGA, on the potential for synergies among goals, or on the overall status of the PFG initiative based on the progress (or regress) of the respective goals. 4.2 Conclusions and Lessons Learned PFG WGA represents a positive change in development assistance compared to previous approaches. The evaluation team found almost unanimous agreement among the USG and GOES partners that the WGA has positively changed the delivery of US assistance and has enhanced the effectiveness of the PFG. As one of the main components of the PFG approach, WGA has been effective in its ability to push the initiative forward, bringing together stakeholders within the individual governments and across the two partner governments. Based on information provided by the interviewees, El Salvador has both directly and indirectly benefitted from WGA, which should be continued. However, there are some goals for which it is difficult to practice WGA, particularly within the security sector. Nevertheless, from the findings gathered, the evaluation team concludes that WGA is essential to achieving the ambitious development goals of PFG within a limited period of time. WGA is most likely to be effective if purposefully incorporated into the design of the initiative and backed by high-level leadership. The evaluation team found that WGA in El Salvador had been ingrained in the PFG process from its onset and was first promoted by the US Ambassador. Through her leadership, PFG WGA was promoted among US agency leadership, and this practice then trickled down to staff. Similarly, on the GOES side, the initiative was first introduced by high-level leadership with a trickle-down effect over time. There is also evidence that WGA was purposefully integrated into the Partnership for Growth El Salvador Mid-Term Evaluation Report 21 defining processes such as the Constraints Analysis, the JCAP, and the selection of Goal Leads, and this fostered collaboration even before the entire staff was introduced to the concept. WGA promoted collaboration both within governments and across governments. The evaluation team found that WGA fostered collaboration both within individual governments and across the partner governments. Within governments, there was a new sense of collaboration, particularly among agencies that would otherwise work in isolation. The result of this increased coordination was an understanding and appreciation of all agencies working within common goals. Further evidence demonstrated that the El Salvador and United States government agencies in El Salvador increased their work together because of the introduction of WGA and its underlying ability to promote coordination. WGA coordination is time consuming. While coordination leads to some efficiencies, it is also time consuming. This conclusion is further elaborated in the section devoted to Cross-Cutting Question 2, which addresses increases in workload as a result of PFG. This finding was raised as a disadvantage for WGA, but only by three respondents. The evaluation team concludes that while WGA may be time consuming, the responses that highlight an appreciation of coordination outweigh those that mention increased time, meaning that the benefits of the increased time spent to promote WGA outweigh the costs. There is insufficient inter-goal (and LOA) coordination, particularly among the security (majority) constraint. Principally, the WGA functioned at the goal level, where each goal subsumed its corresponding LOA. Coordination among goals was infrequent. Although the WGA approach at goal level was, to a certain extent, an obvious and intuitive approach—partly as a result of JCAP-imposed tasks to work together—collaboration needed to promote how goals working together lead to promoting effectiveness of the overall PFG initiative is unclear. Such guidance was insufficiently provided on the USG and GOES sides. 4.3 Course Corrections and Recommendations Provide stronger guidance for inter-goal collaboration. Now that most goals (and corresponding LOAs) are underway, the potential to increase synergy across goals can be capitalized. As noted in the findings and conclusions, while PFG Goal Leads collaborate to implement their respective goals, there is little coordination and collaboration across multiple goals. The evaluation team believes that there are ways in which the PFG initiative would benefit if its staff and key stakeholders were aware of how other goals (and corresponding LOAs) feed into the larger PFG initiative. To enhance this process, the evaluation team recommends that a small bilateral PFG management, coordination, and monitoring and evaluation (M&E) unit be put in place. Currently, PFG management for the USG is housed within USAID with one main PFG coordinator; GOES also has one PFG coordinator. Additionally, the following teams exist in the initiative—a USG Principals Group; an overall security and tradables constraint lead on both USG and GOES sides; and USG and GOES goal leads for each goal. The evaluation team believes that establishing a Principals Group on the GOES side with regular internal meetings and further regular joint meetings for these two counterparts (USG and GOES Principal Groups) could be beneficial. The above teaming mechanisms could be improved further, given the vastness of the initiative, by establishing a joint (bilateral) management team with members from both USG and GOES who would oversee the PFG initiative. All PFG staff and stakeholders should be made aware of this team and their roles and responsibilities (which are discussed throughout the report). The team recommends Partnership for Growth El Salvador Mid-Term Evaluation Report 22 that a conscious effort be made to identify and share good practices and lessons learned across goals; also highlighting how indicators are monitored and progress tracked. Such discussions can take place during management meetings, implementation team meetings, and scorecards meeting. Develop performance measurements that track the role of coordination activities in promoting effectiveness. The evaluation team concluded that WGA has led to increased coordination within the El Salvador PFG initiative, which has ultimately led to efficiencies. However, the evaluation team was not able to identify direct performance measures that gauge the quality, scope, and depth of coordination. For such an initiative, it is possible, particularly over time, that frustration among management and staff could arise if indicators cannot be identified. Consequently, the evaluation team recommends that PFG El Salvador management develop performance indicators that define WGA coordination activities and track how these activities have led to enhanced effectiveness and/or efficiency. Suggestions of possible indicators are provided below. There are challenges inherent in measuring the performance of a concept such as coordination. For example, when personnel are gathered, one might consider using attendance as a quick performance indicator (which currently does occur); however, participating in meetings by itself does not lead to improved outcomes, so this is not an adequate performance indicator (in fact, the increased time for attending meetings was identified as a disadvantage of PFG WGA). The PFG teams both within USG and GOES will need to collaboratively identify key indicators that measure how the teams have been more effective. The following examples of potentially useful indicators could be considered: • • • • • Develop a set agenda for coordination meetings: Ensure that agenda items are reviewed and tracked during each meeting, and action items are reviewed at the onset of each meeting and tracked in terms of progress or regress. Introduce accountability measures: Establish repercussions for not participating in these meetings or not following through on agenda and/or action items. Identify and inform staff and key stakeholders of responsibilities for the overall management of PFG coordination: Ensure that all PFG staff members are aware of the management personnel within each government (and across governments). Staff should also be aware of their respective responsibilities related to fostering coordination activities. Establish (and update) an overarching work plan to track progress on coordination: The PFG initiative uses scorecards to track indicators of project (goals and LOA) performance. However, based on the findings, it is evident that the PFG initiative is more than a compilation of programs, and instead a government-to-government initiative. Therefore, in addition to tracking the progress of projects, it would be key to track the progress of the non-project initiatives such as coordination activities, publicity of coordination, and other activities such as non-assistance (discussed in a subsequent section). Such tracking could occur through PFG group listservs or newsletters (some of which already exist). Identify and track activities/ actions that demonstrate coordination: As a result of the PFG initiative, several activities could promote coordination. For instance, the evaluation team witnessed the Goal Leads scorecard meeting, which demonstrated coordination across governments. Meetings such as these should be included in reporting on coordination activities. Partnership for Growth El Salvador Mid-Term Evaluation Report 23 The above bullet points are examples, but key performance indicators would be best identified through a collaborative process that includes all PFG partners on the leadership level (within GOES and USG) to identify the most relevant components of coordination and map out how coordination (and other WGA elements) affect the specific PFG initiative in El Salvador. As discussed in the previous recommendation, a newly established bilateral PFG management team could be assigned to facilitate the tasks indicated in this recommendation. Partnership for Growth El Salvador Mid-Term Evaluation Report 24 5. CROSS-CUTTING QUESTION TWO: TO WHAT EXTENT HAS PARTNERSHIP FOR GROWTH AFFECTED THE WORKLOAD OF NATIONAL GOVERNMENT AND U.S. GOVERNMENT STAFF, AS COMPARED TO THE WORKLOAD CREATED BY TRADITIONAL FORMS OF DEVELOPMENT ASSISTANCE DELIVERY? To determine whether PFG has led to an increase in workload for staff, the evaluation team gathered data through interviews and the web-based survey and based its findings on respondents’ perceptions. Performance indicators on time usage or timesheets pre-PFG through PFG implementation would have provided a more accurate assessment of how PFG has affected GOES and USG workload, but these data were not collected. Moreover, it may be difficult for respondents to compare the PFG workload to the workload pre-PFG due to recall bias for those with more than three years of involvement with PFG or possibly no point of comparison for those with less than three years of PFG involvement. Nevertheless, the data gathered via semi-structured interviews and web-based surveys provide insight about changes in government staff workloads that are attributable to PFG and the perceived effectiveness of changes in workload. The USG findings reflect perceptions of both staff in El Salvador and Washington DC. It is important to note that some of the Washington DC staff are involved in overarching PFG management/activities in other PFG countries, yet their responses are included here in the El Salvador report. This may influence the findings on change in workload under PFG given that many Washington staff members were involved in the planning of PFG at the outset. However, the evaluation team assumes that because the survey was specific to the El Salvador initiative, the responses provided by all stakeholders, as much as possible, relate to their affiliation and work with the El Salvador initiative. As shown in table 2.3 in the previous methodology section, 82 USG and 26 GOES PFG staff responded to the survey questions on workload changes. Details are provided below. 5.1 Findings Finding 1: PFG has resulted in an increase in the workload of USG and GOES staff. As illustrated in figure 5.1 below, respondents confirmed that PFG led to an increase in workload for USG and GOES staff. Among USG respondents, 51 percent indicated that their workloads increased somewhat, and 29 percent indicated that their workloads increased significantly. The findings among GOES respondents differed– 35 percent indicated that their workloads increased somewhat and 19 percent indicated that their workloads increased significantly. No respondents reported a decrease in workload resulting from involvement in PFG. Partnership for Growth El Salvador Mid-Term Evaluation Report 25 Figure 5.1: Changes in workload resulting from PFG involvement: USG and GOES (Based on survey responses only) Of these same respondents, those in leadership roles experienced greater increases in workload than those in non-leadership roles. All of the USG respondents in leadership roles reported that their workloads increased somewhat or significantly, while 88 percent of GOES respondents in leadership roles reported that their workload increased somewhat or significantly (figure 5.2). By comparison, 76 percent of USG respondents in non-leadership roles reported that their workloads increased somewhat or significantly, while 43 percent of GOES respondents in non-leadership roles reported that their workloads increased somewhat or increased significantly (figure 5.3). Figure 5.2: Changes in workload resulting from PFG involvement: USG and GOES leadership (Based on survey responses only) Partnership for Growth El Salvador Mid-Term Evaluation Report 26 Figure 5.3: Changes in workload resulting from PFG involvement: USG and GOES non-leadership (Based on survey responses only) Finding 2: There was no discernible difference in workload by gender. Thirty-nine percent of USG respondents were female, in comparison to 19 percent of GOES respondents (see table 5.1). Nineteen (19) percent of GOES respondents did not report gender as compared to only 1 percent of USG respondents. Table 5.1: Respondents' gender Association N Male Female No answer USG 82 60% 39% 1% GOES 26 62% 19% 19% Total 108 60% 34% 6% Figure 5.4 suggests that there was no discernible difference in workload attributed to PFG by gender. Fifty seven (57) percent of female respondents reported that their workload increased somewhat or significantly, while 58 percent of male respondents reported that workload increased somewhat or significantly. Partnership for Growth El Salvador Mid-Term Evaluation Report 27 Figure 5.4: Changes in workload resulting from PFG involvement, by gender (Based on survey responses only) Finding 3: It is unclear whether PFG is crowding out traditional forms of development-assistance delivery. As shown above, 74 percent of all respondents reported that their workload had increased since they became involved with PFG, with 47 percent reporting that it had “increased significantly.” Interestingly, the modal number of reported hours worked per week (40) was the same for those who reported significant increases in workload due to PFG as those who reported that their workload stayed about the same. On face value, this suggests that PFG activities overtook non-PFG activities. Alternatively, one could argue that non-PFG activities became aligned to PFG to the point that “normal work” activities could not easily be separated from “PFG work” activities. However, it is common practice within the United States federal government to not record actual time worked; as a result, the team suspects that the hours reported in the web-based survey may understate the actual number of hours worked and possibly overstate how much PFG activities are possibly crowding out traditional forms of development assistance activities. If a more accurate workload tracking mechanism was put into place, it may be possible to gauge how initiatives like PFG affect staff workloads as well as offset or take over other activities. Finding 4: PFG has prompted a significant increase in work specifically focused on coordination, both intra- and inter-governmental. PFG activities were categorized into the following activities: 1) coordination with colleagues within my government; 2) coordination with colleagues in other (partner) governments; 3) monitoring progress of PFG tasks; 4) communicating on PFG with my superiors and senior leadership in my government; 5) managing PFG activities; 6) designing and/or procuring PFG activities; 7) other administrative tasks. According to figure 5.5, below, the increased workload due to PFG activities was distributed fairly evenly across the range of activities; however, “coordination with colleagues within my government” constituted the largest increase in workload in comparison to other PFG activities. This confirms the findings in Partnership for Growth El Salvador Mid-Term Evaluation Report 28 Cross-Cutting Question 1, which indicates that PFG WGA requires more time due to coordination and collaboration activities. Secondarily, the coordination with other governments, monitoring progress of PFG tasks, and communicating with senior leadership accounted for large shares of the increased workload due to PFG. Figure 5.5: Change in workload resulting from PFG, by task (Based on survey responses only) It appears that PFG has prompted the USG to expend more hours than GOES on intra-government coordination (see figure 5.6). GOES staff has expended relatively more hours, due to PFG, in designing and/or procuring PFG activities. Figure 5.6: Change in workload resulting from PFG by task – USG and GOES (Based on survey responses only) Partnership for Growth El Salvador Mid-Term Evaluation Report 29 The semi-structured interview results confirmed the perceptions gathered from these surveys. For example, during an interview, a Goal Lead noted, “It requires more workload to coordinate the activities of different agencies.” Another interviewee explained that most goals involved the activities of two or more agencies from USG as well as several agencies on the GOES side. For example, Security Goal 4 (reduce the Impact of Crime on Public Transportation) involved USG DOJ, INL, ICITAP, OPDAT and USAID and the GOES PNC, the Attorney General’s Office (AGO), and the Vice-Ministry of Transportation. Overall, the time devoted to interagency cooperation, particularly on procedures for moving forward, was substantial, as reported by multiple interviewees. Survey findings support the assertion that multiagency coordination was one of the more time-consuming tasks associated with PFG. On average, respondents spent 5 hours per week coordinating with their colleagues, and 75 percent spent up to six (6) hours on this task (see table 5.2). By comparison, with the exception of “Managing PFG Activities” (6 hours per week), all other tasks required, on average, about three (3) hours per week. Finding 5: PFG has resulted in an increase in workload particularly for stakeholders involved in planning. Forty-four percent of USG respondents were involved in the planning and development of PFG as compared to 35 percent of GOES respondents. USG staff spent a disproportionate effort in planning PFG, than in conducting other PFG activities, as demonstrated in figure 5.7 below. Figure 5.7: Staff involved in planning and development of PFG (Based on survey responses only) Table 5.2 illustrates that 35 percent of USG leaders and 28 percent of non-leaders spent 16 or more hours per week in planning PFG. By comparison, 20 percent of GOES leaders and only 9 percent of GOES nonleaders dedicated 16 or more hours per week in planning PFG. Note that a disproportionate number of GOES non-leaders did not respond to these questions. Partnership for Growth El Salvador Mid-Term Evaluation Report 30 Table 5.2: Hours per week dedicated to planning and developing PFG (Based on survey responses only) USG Hours per Week GOES Leaders Goal Leads Leaders Goal Leads Zero 6% 8% 0% 5% 5 hours or less 25% 42% 20% 33% 6 to 10 hours 28% 12% 20% 10% 11 to 15 hours 6% 4% 40% 5% 16 to 20 hours 9% 12% 0% 5% Greater than 20 25% 16% 20% 5% No Answer 0% 6% 0% 38% Finding 6: Respondents shared that increased workload is associated with perceived effectiveness of WGA and PFG. Similar to innovations in other industries, PFG has required the “innovators” and “early adopters” in USG and GOES to invest upfront, resulting in significant increases in workload and possibly overtaking former traditional forms of development-assistance delivery. It is still too early to assess PFG’s efficiency or its effectiveness, but this analysis provides insights to stakeholders’ perceived effectiveness. Figure 5.8 illustrates that PFG stakeholders who invested more time into PFG activities perceived that PFG was a significant improvement compared to traditional development assistance approaches. For example, more than 70 percent of PFG stakeholders who reported that PFG was a significant improvement over traditional approaches also dedicated 16 or more hours per week to PFG activities. In comparison, less than 30 percent of PFG stakeholders who reported that PFG was a significant improvement over traditional approaches worked less than 16 hours per week on PFG activities. Figure 5.8: Perceived Effectiveness of PFG Compared to Traditional Development Assistance Approaches (Based on survey responses only) Partnership for Growth El Salvador Mid-Term Evaluation Report 31 These same trends generally held when analyzing this same association by governmental affiliation (see figure 5.9). Disproportionately more GOES respondents who had invested 16 or more hours per week reported perceiving PFG as an improvement over traditional approaches than USG respondents who had invested comparable levels of effort. Interestingly, less than 5 percent of UGS respondents did not respond to these questions but more than 35 percent of GOEs respondents did not respond to these questions. Figure 5.9: Perceived Effectiveness of PFG Compared to Traditional Development Assistance Approaches – USG and GOES (Based on survey responses only) 5.2 Conclusions and Lessons Learned PFG has unambiguously increased workloads for both USG and GOES staff; 80 percent of USG survey respondents indicated that PFG increased their workloads either somewhat or significantly as compared to 54 percent of GOES respondents. There was no discernible difference in workload by gender, and it is unclear if the increased workload has overtaken efforts related to traditional forms of developmentassistance delivery or whether PFG has simply replaced former development assistance delivery. The evaluation team’s findings unambiguously demonstrate that PFG has prompted a significant increase in both intra- and inter-governmental coordination, a finding that aligns with data gathered in response to Cross-Cutting Question 1. It may be too early to assess the effectiveness of PFG, but survey results suggests that staff in both the USG and the GOES who invested a significant amount of time into PFG also believed that PFG is a significant improvement over former development assistance practices. Thus, increased workload seems to have translated into increased development efficiency. Partnership for Growth El Salvador Mid-Term Evaluation Report 32 5.3 Course Corrections and Recommendations Consider reducing the number of coordination meetings to reduce staff workload. Based on the evaluation findings, it is evident that the PFG created a heavier workload for USG and GOES staff. To the extent that it could be accomplished without eroding the benefits of the WGA, the number of required meetings should be reduced. However, care should be taken to ensure that reducing the number of meetings does not jeopardize interagency cooperation and mutual accountability. Identifying which meetings to maintain would also demand reflection from the involved parties, as the payoff of a meeting is not always obvious. For example, although it can be argued that at least some meetings represent an inefficient use of time, it can also be argued that meetings promote interagency cooperation and mutual accountability. The simple act of convening a meeting raises the visibility of agencies and promotes awareness of each agency’s unique and complementary capabilities. Moreover, an agency’s perception of its role as part of a team can function as a motivator. As part of a team, both its successes and failures are open to the scrutiny of the agency’s teammates. Identify management staffing that could assist in reducing the workload of PFG staff across the Initiative. Make an effort to ensure that sufficient resources are available to manage PFG activities without detracting from other worthwhile commitments. For instance, if staff members spend 3 to 6 hours per week on PFG, then 3 to 6 hours per week on other, non-PFG responsibilities may be neglected. It would be prudent to have, from the start, sufficient staff to manage the daily functions of PFG, coordinate across agencies and governments, and facilitate the monitoring of PFG data to track progress. While PFG currently has both a USG and GOES coordinator and various teams within the leadership, the evaluation team believes that establishing a bilateral management team (detailed in Cross Cutting Question 1 previously) to oversee the various aspects of PFG would be useful for reducing the overall workload of other PFG staff. Identify and promote use of limited resources more efficiently to reduce overall workload. The PFG—and the WGA, in particular—represent a relatively new approach to development assistance. Encouraging staff closest to the project to identify processes that work well and those that do not could help USG and its partners use their human capital more productively in the future. Identifying promising processes—i.e., those that appear to work well—involves mapping inputs to desired outcomes, first theoretically, and then empirically. Although this would be best accomplished at project inception and iteratively reevaluated, the identification of promising processes can and should be an on-going process (further detailed in Country-Specific Questions 1 and 2). By adequately tracking activities that lead to increased workloads as the PFG progresses, areas can be identified for redirecting workload or teams can be reassigned to create a more balanced workload among PFG staff. Partnership for Growth El Salvador Mid-Term Evaluation Report 33 6. CROSS-CUTTING QUESTION THREE: WHAT CONTRIBUTIONS HAS NON-ASSISTANCE MADE TO THE PFG PROCESS, AND HOW CAN IT BE UTILIZED MOVING FORWARD? Particular to this evaluation, USAID defined non-assistance tools as those including “diplomatic engagement, convening authority, and other forms of non-monetized assistance to engage both governmental and non-governmental stakeholders in support of catalytic policy change and development priorities.” Further, the evaluation team views non-assistance tools to include the following: • • • one-on-one engagement by government officials (the Ambassador, the Deputy Chief of Mission or other members of senior staff); public information activities; and stakeholder engagement. The evaluation team sought to identify whether non-assistance, as defined above, has made any contributions to the PFG initiative in El Salvador and how it can be utilized moving forward. Leadership, Goal Leads, and independent experts within both USG and GOES were asked about non-assistance in semi-structured interviews and an online survey. Documentation received from USG and GOES (through the USAID point of contact) prior to the evaluation team’s fieldwork in El Salvador was also carefully reviewed for evidence of non-assistance activities and used to inform the evaluation. Based on the information gathered, which is detailed subsequently, the evaluation team found that while non-assistance activities do exist within the PFG initiative in El Salvador, there is no clear system for documenting these activities and similarly no clear understanding of the term among all PFG stakeholders. 6.1 Findings Finding 1: The most frequently cited form of non-assistance involved the diplomatic engagement of senior U.S. embassy staff with their GOES counterparts on policy or institutional development issues. As discussed in the context of the country-specific research questions, the PFG is not an initiative solely managed by the executive branches of the two governments. It also requires the support of the Salvadoran legislature and polity. In this context, about 90 percent of USG and Salvadoran interviewees who knew about non-assistance mentioned the critical importance of the U.S. Ambassador’s support for PFG. The Ambassador is extremely influential and known widely among Salvadorans. One non-governmental respondent said, “Th[is] Ambassador is the Ambassador closest to the Salvadoran population that we have had.” The Ambassador has not hesitated to use her considerable influence to advance policy and institutional reform. Because policy reform, legislative reform, and institutional reform are critical parts of numerous PFG goals, this mobilization of senior USG officials in support of PFG has been critical to PFG’s successes. From responses made specifically about the Ambassador, it is evident that she highly prioritized PFG and its approach within the Embassy from its initial stage until now. She not only ensured that PFG was at the heart of all discussions within the mission, but also managed to promote it in discussions about policy decision making with GOES, using non-assistance to promote positive outcomes for the initiative. Partnership for Growth El Salvador Mid-Term Evaluation Report 34 Finding 2: Accurate examples of non-assistance provided by respondents through the online survey. The results of the semi-structured interviews were supported by the open-ended survey responses. Table 6.1 provides more evidence of accurate examples of non-assistance provided through the online survey responses. Table 6.1: Examples of direct quotes about non-assistance in the El Salvador PFG initiative, provided by survey respondents • • • • • • • • • • • • • • • • • • • • • Use of Embassy and GOES power to convene civil-society discussions Use of Embassy/GOES power to enhance earned media coverage of PFG events Ambassador’s direct involvement in specific issues, such as the Access to Information Institute GOES ministerial direct involvement in specific issues, such as the Crime and Violence Prevention Strategy Diplomatic engagement to advance PFG legislation Increase in dialogue between the private sector and the government Ambassador Aponte has moved forward a number of issues I deal with in my project through her convening power, including bringing the Growth Council back together. Diplomatic engagement by embassy officials on key PFG legislative initiatives, such as assets forfeiture Public-private engagement and dialogue, such as Growth Council, FCS, liaising with business community Use of USG ‘good offices’ to promote PFG with other donors to join the initiative ‘Good offices’ to foment meetings that might not have happened otherwise Convening of authority by U.S. Ambassador for donor coordination Institutional reform to increase public transparency and support the new Institute of Access to Public Information Engagement of private companies in crime and violence prevention, such as the approval and launching of the National Violence Prevention Strategy (ENPV) to prevent crime and violence at the municipal level. Passing of the private-public-partnership law Treasury technical assistance in the airport project Enlisting FEMA and FAA in airport safety & emergency planning State Department engagement All new scholarship under Fulbright, UGRAD and IVLP is mainly in a field relevant to the PFG Focus on infrastructure and air transportation Non-assistance expressed through links between American universities such as the University of Arizona, Don Bosco University, Stanford University, among others, as well as with US NGOs and Foundations Finding 3: The concept of non-assistance is unclear to many PFG stakeholders. The evaluation team concluded from interviews that the concept of non-assistance is not well known, even within the embassy. In 90 percent of interviews outside of USG and GOES senior leadership, the term “non-assistance” had to be defined for the respondents. Table 6.2, below, provides data on the number of respondents, by stakeholder type, who were able to address questions concerning the types of non-assistance in the El Salvador PFG initiative, helping to illustrate that most PFG staff, particularly within GOES, were not aware of non-assistance activities in El Salvador. Further, as shown in table 6.2, USG leadership (82 percent) was the stakeholder group most knowledgeable about the term non-assistance and most able to provide clear examples of non-assistance Partnership for Growth El Salvador Mid-Term Evaluation Report 35 activities within the PFG initiative. Among GOES leadership, fewer respondents (45 percent) were able to provide examples of non-assistance activities in El Salvador, with 55 percent unable to provide examples of non-assistance. One of the GOES senior officials provided detailed information and examples of nonassistance, stating that from the beginning of PFG, there was an emphasis placed on what can be termed “non-assistance,” considering that in a partnership such as this the agreement goes beyond any financial resources. As a result, non-assistance activities were mentioned in various PFG initiation documents agreed upon by GOES and USG including • • • • • Joint declaration of principles for both governments; Joint action plan for both countries (JCAP); Monitoring and evaluation reports; Order 169 for the creation of the National Growth Council; and Press conference to launch PFG. Table 6.2: Number of interviewees who provided examples of non-assistance activities in PFG El Salvador Initiative Interview Responses USG Leadership GOES Goal Leads Leadership Goal Leads USG and GOES Totals At least 14 13 4 3 34 one (82%) (72%) (45%) (17%) (55%) response 3 5 5 15 28 No (18%) (28%) (55%) (83%) (45 %) response 17 18 9 18 62 Total NOTE: Some stakeholders’ responses about non-assistance may not have been accurate. Therefore, the information provided is an estimate to inform the reader of the stakeholder groups that were able to provide information on nonassistance activities within the El Salvador initiative. Findings from the online survey question about non-assistance provide a different perspective than the interviews did. When asked if respondents had witnessed non-assistance, overall 78 percent of survey respondents claimed to have witnessed non-assistance within PFG activities in El Salvador, while approximately 15 percent had not (see figure 6.1). However, as illustrated in figure 6.2 subsequently, when responses were reviewed by stakeholder type, it was evident that much fewer GOES respondents (65 percent) were aware of non-assistance activities than USG (82 percent). Partnership for Growth El Salvador Mid-Term Evaluation Report 36 Figure 6.1: Have you seen non-assistance tools being used in the PFG activity with which you are or were involved? (Based on survey responses only) Figure 6.2: Respondents Awareness of Non-Assistance Activities by Stakeholder Type (Based on survey responses only) Finding 4: Non-assistance efforts have met with mixed results in El Salvador. Exerting US government influence with GOES officials to resolve political gridlocks have been very effective. Once these activities have occurred (when appropriate) the public affairs activities to raise the Salvadoran public’s awareness of PFG have been somewhat effective in raising awareness of PFG but not in publicizing the goals of the PFG, the progress it has made, and how these outcomes will positively affect the lives of Salvadorans. During the interviews, the evaluation team gathered that the USG has also not used its convening authority to engage civil society, small and medium enterprises, and the community of international investors enough to advance PFG among these groups. Consequently, the PFG is missing opportunities to raise support for PFG that could be instrumental in ensuring the successful implementation of activities. Partnership for Growth El Salvador Mid-Term Evaluation Report 37 6.2 Conclusions and Lessons Learned Embassy leadership provided the clearest example of the successful use of non-assistance in El Salvador. The Ambassador was cited as the most visible proponent spearheading non-assistance activities within PFG in El Salvador. However, non-assistance can be affected beyond diplomacy; therefore, other opportunities for non-assistance within the PFG could be explored to compliment the work of the Ambassador and high-level officials. USG and GOES Goal Leads and implementers’ lack of familiarity with the concept of nonassistance means that opportunities to employ non-assistance will be missed. If many PFG stakeholders have limited familiarity with the concept of non-assistance, it is unlikely that they will know that it is to be encouraged and that opportunities for its utilization—in its many forms—should be actively sought out. Non-assistance activities should be made known to external stakeholders. Opportunities exist to leverage non-assistance activities to promote informed expectations for both the investor community, foreign and domestic, and the Salvadoran citizenry. 6.3 Course Corrections and Recommendations PFG staff should receive training about the diversity of non-assistance tools and their value to a broad array of beneficiaries. Given non-assistance’s importance within the PFG approach, the evaluation team recommends that PFG, possibly through the PFG management team suggested in a previous recommendation, should conduct training(s) related to non-assistance and continue these training as new staff are brought in. Particularly given that the concept of non-assistance frequently presented confusion among stakeholders, a first step to developing training activities would be for the term and definition to be clearly defined. During these training, there should be a concerted effort to identify past examples of non-assistance and activities that will lend themselves to non-assistance in the future. Such training would aid PFG staff in identifying opportunities for non-assistance and effectively translating examples of non-assistance to the larger stakeholders within their goals and LOAs. Non-assistance requires increased staffing. Based on the findings about non-assistance activities and the need to bring more awareness to various stakeholders, the evaluation team recommends that PFG El Salvador consider increasing staffing geared towards non-assistance. Increased staffing will be needed because workloads will invariably increase if attempts to leverage non-assistance are increased in the future. The team is not necessarily suggesting that additional staff need to be hired. Within the existing structure, it would be useful to designate individual(s) who would be responsible for identifying nonassistance opportunities, especially related to public information and stakeholder mobilization within the PFG initiative. Also, it would be beneficial to work with the relevant stakeholders to highlight and report on such activities in a uniform manner. In the previous section in response to Cross-Cutting Question 1, the evaluation team suggested having a core management team devoted to PFG activities; the evaluation team believes that identifying areas for promoting non-assistance could also be a task for the core management team. This work would include some of the ideas provided above such as facilitating training of new PFG staff on the importance of non-assistance and identifying ways to conduct and promote non-assistance. Partnership for Growth El Salvador Mid-Term Evaluation Report 38 Engaging civil society, the private sector, and other stakeholders can increase publicity for and the success of PFG. Civil society, private sector, and other stakeholders’ input and support for PFG programs and objectives could be improved. It is clear from the team’s findings that external stakeholders are not aware of the goals and objectives of PFG. The evaluation team recommends systematic, periodic engagement of civil-society organizations and the private sector (both large investors and small and medium enterprises [SMEs]) to increase their understanding of PFG goals and activities and to obtain their input on ways to improve the initiative, as applicable. Ultimately greater awareness about PFG, could lead to greater/wider stakeholder mobilization and increased public perception concerning the positive effects of the PFG in general, or of more specific PFG activities (e.g. within the security section, public information campaigns about what a prisoner classification system is, and how it is being introduced or modernized with support from the PFG, may contribute to overcome a public opinion locked into the belief that either “nothing” is being done, or that prison reform is only about building more prisons) is essential to promoting the economic growth of El Salvador. Partnership for Growth El Salvador Mid-Term Evaluation Report 39 7. COUNTRY-SPECIFIC QUESTION ONE: THE CONSTRAINTS ANALYSIS DOES NOT IDENTIFY REMEDIES TO ADDRESS THE BINDING CONSTRAINTS TO GROWTH. FOR EACH OF THE CONSTRAINTS, ARE THE GOAL-LEVEL COMMITMENTS SET FORTH IN THE JCAP ALONE CAPABLE OF ACHIEVING THE CONSTRAINTLEVEL OBJECTIVES AND OUTCOMES? This evaluation question (and the subsequent one) is directly related to the second purpose of the PFG evaluation as outlined in the statement of work: “evaluate whether PFG efforts have been developed in such a way as to allow for the eventual determination of their impact on addressing the identified constraints and desired outcomes.” Therefore, the analysis and findings gathered in response to this question are within this context. Given the complexity of development outcomes and the uncertainty of exogenous factors, the evaluation team has concluded that this evaluation question cannot be answered empirically, that is, based on constraint-level outcomes achieved to date. Consequently, the following theoretical question will be addressed instead: Are the individual goals (and the corresponding LOAs) necessary, and are they sufficient to achieve the constraint level outcomes? The evaluation team has concluded that the selected goals are generally aligned with their respective constraints; however, it is difficult for the team to gauge the necessity and sufficiency of individual goal-level commitments in the JCAP in achieving the constraint-level outcomes. This is especially true of the security area because while the Constraints Analysis prioritized ‘tradables’ as a set field of economic development to focus the El Salvador PFG, it did not prioritize a set area(s) for security, e.g. organized crime or justice sector administration, and essentially posited the entire security sector governance area as the constraint, making it most challenging for those formulating the JCAP, to narrow down the necessary and sufficient goals (and corresponding LOAs). Further, but related to the above point, there exists well-established precise and focused microeconomic and development theory to understand how to overcome the tradables constraint; therefore, firmer conclusions can be drawn about the necessity and sufficiency of specific goals (and related LOAs). For instance, small farm staple agriculture, and wholesale and retail trade, while they might be useful economic tools to promoting economic growth, would not fit within a tradables objective, and therefore were not included. On the security side, while development practices and theories are still less well established than in the economic field, best practices and lessons learned in the region do provide roadmaps for specific security sector fields (e.g. fighting organized crime as related to cocaine trafficking, or preventing juvenile delinquency). Within each of those fields, it would be possible to narrow down necessary and sufficient goals. However, given that the Constraints Analysis does not propose specific field(s) within the security constraint, at the JCAP-level, the goals that were aligned to the constraints were apparently not selected based on an explicit combination of such existing theoretical and practical insights, to ensure that the constraint can be relieved. Rather, tackling the security constraint is based on a broad approach (including extradition, police reform, justice reform, prison reform, municipal prevention policies, among others), Partnership for Growth El Salvador Mid-Term Evaluation Report 40 with the expectation that together all the 14 goals can lead to overcoming the security constraint. 24 Although the 14 security goals are divided in two sub-constraints (institutional strengthening and prevention), these categories are again very general, and at least at the LOA level, there are several activities that do not clearly align to this differentiation either. The evaluation team’s analysis of the 4 security goals selected for in depth review revealed that these goals are passing meaningful implementation milestones, and many of their associated LOAs appear to be producing promising results. However, given the broad ‘security’ approach, it would be speculative to attempt to define which goals are necessary or not, while it is self-evident that none is sufficient in and by itself to overcome the constraints. 25 Within the tradables constraint, the situation is precisely that, with the exception of Goal 1, all the Goals are necessary, and a lack of progress on Goal 5, for instance, has limited success in relieving the constraint. 7.1 Findings Finding 1: A formal theory of change linking LOAs, goals, and constraints was not developed. Based on the review of PFG program documentation, the evaluation team concluded that generally PFG strategies for both the security and tradables constraints, as described in the goals and LOAs of the JCAP, reflect a loosely defined “theory of change.” The 14 crime and insecurity constraint goals, are divided into two sub-constraints—institutional strengthening (Goals 1-7) and crime and violence prevention (Goals 8-14)—that outline a broad approach to counter a wide range of challenges, including impunity, prison overcrowding, gangs and organized crime, extortion, and corruption. The theory of change implicit in PFG postulates that a broad approach, which includes police reform, prison reform, justice sector strengthening, extradition, improvement of educational opportunities, municipal prevention policies, and legal reforms, stands the best chance against a multi-causal, interwoven, and country-wide problem of crime and insecurity. However, the links between each LOA and each goal, and then between the subconstraints and constraints have not been spelled out. In other words, no narrative is available that explains why specific LOAs were chosen and not other ones, making it challenging to identify how goallevel commitments within the security constraint are supposed to adequately lead to achieving desired constraint-level outcomes. The low productivity of tradables constraint presents an even more challenging situation. The strategies being used to relieve this constraint are based on a standard microeconomic and economic development theory in which an improved investment climate, supported by improved infrastructure and human capital, leads to increased growth in the tradables sector. Improved technology in medium-sized domestic firms enables such firms to participate in this increased growth. The entire strategy is to create a virtuous cycle in which increased foreign investment generated by an improved investment climate creates opportunities for growth for domestic firms and workers (and non-incidentally, increased GOES tax revenues). Without explicitly stated theoretical interconnections, however, the JCAP does not clarify the interdependencies between LOAs and goals in relieving the constraint. Specifically, very real progress has 24 Partnership for Growth: El Salvador-United States: Joint Country Action Plan: 2011-2015. (2011, November) .http://photos.state.gov/libraries/elsavador/92891/octubre2011/Joint_Country_Action_Plan.pdf 25 This is discussed in more detail under Country Specific Question 3. Partnership for Growth El Salvador Mid-Term Evaluation Report 41 been made at the LOA and goal-levels under Goals 2, 3, 4, and 6. However, without substantial progress under Goal 5, and specifically on the investment climate LOAs under Goal 5, there is little theoretical reason to believe that progress will be made in increasing the productivity of tradables. Finding 2: The JCAP is considered a central guiding tool, although it is also considered overly ambitious. The JCAP—with its 20 goals and 153 LOAs—is the PFG document that comes closest to providing structure to the implied theories of change for the El Salvador PFG initiative. When interviewees were asked if the JCAP is performing its central task of guiding the PFG and whether or not they use it within their tasks, the majority (85 percent) of interviewed USG and GOES officials in El Salvador considered the JCAP essential to their work, using expressions like “the JCAP guides all our work,” “the JCAP is all we do,” and “the JCAP is our central tool.” Despite the centrality of the JCAP, 14 of the 20 Goal Leads interviewed argued that the goal-level commitments set out in the JCAP are overly ambitious for a 5-year program. Thus, even if goal-level commitments are capable of achieving constraint-level objectives and outcomes, the assumed length of time it would take to reach these goals was too short to realistically meet this target. Additionally, these Goal Leads observed that the consideration of external factors that could affect PFG goal and constraint attainment, including political factors such as a “gang truce”, is beyond the scope of the JCAP’s theory of change, making it difficult to attain constraint-level outcomes. Finding 3: The designers of the JCAP were faced with a variety of on-going initiatives, many of which were relatively aligned with the two constraints; thus the initial selection of activities and LOAs were based on pragmatic decisions, and not on a theoretical framework. Subsequent project and program designs and implementation at the LOA-level have contributed to tightening the focus on each constraint. Responses for this question varied within the different constraints; therefore, findings are provided by constraint. Specifics within the Security Constraint: Through the Constraints Analysis process, crime and insecurity were determined to be the constraints most affecting economic growth in El Salvador; however, based on collected data, the selection of the 14 associated goals was somewhat arbitrary. The selection process was, in fact, the product of pragmatic considerations, rather than based on a rigorous theoretical approach. For example, pre-PFG, USAID was already implementing various large justice and security sector reform programs under multi-year contracts. Interrupting them would generate not only significant direct costs but also would disrupt bilateral relations of agencies and communities invested in these programs. All of the interviewees who participated in the JCAP process expressed that the resulting selection of goals was, to some extent arbitrary, but they nevertheless, credited the broad categorization with addressing the main crime and security challenges in the country. While the JCAP divides the 14 security goals 26 into seven institutional-strengthening goals and seven crime and violence prevention goals, the evaluation team found that leadership and Goal Leads did not 26 The Crime and Insecurity constraint currently has 13 goals, after goals 1 and 2 were consolidated. Partnership for Growth El Salvador Mid-Term Evaluation Report 42 agree that this was the most coherent packaging of goals. Approximately half of the Goal Leads thought that it made more sense to categorize the 14 security goals along sector and agency-based lines such as: • • • • • • two goals related to the direct impact of crime on business activity (3 and 4); three goals (1&2, 6, and 9) related to justice sector reform; four goals (8, 10, 11, and partially 9) related to targeting high-crime communities and at-risk youth; two goals (12 and 13) related to improving prisons; two goals (5 and 14) related to anti-organized crime legislative and policy measures; and one goal (7) related to promoting a national dialogue on security and justice sector reform. However, others believed that such a narrow sector-based categorization would lead to a silo approach and therefore diminish opportunities for WGA, which has proven effective. That said the current categorization into seven institutional strengthening goals and seven prevention goals has had no managerial or operational consequences, as no sub-group specific coordination or reporting is taking place for either. Specifics within the Tradables Constraint: The JCAP specifies six goals under the tradables constraint, with the constraint-level outcomes aimed at achieving higher, sustained, and more inclusive economic growth in line with other high-performing emerging economies. 27 The goals reflect the standard economic development categories of infrastructure, human capital, technology, and the policy environment (with increased government revenue being necessary to fund infrastructure and education). Interviewees who participated in the Constraints Analysis stated that these four categories were considered equally restrictive impediments to economic growth. While basic trade and economic development theory provided some rigor to the tradables constraint’s implicit theory of change, it was obscured by a desire to incorporate existing projects. For example, the contribution of the regional Scholarship for Education and Economic Development (SEED) training program under Goal 3 is likely limited at best to twenty students, according to the most recent scorecard, sent to the US for general two-year degree training. Finding 4: The ability to remove constraints and attain goals is subject to the influences of a broad range of factors beyond the domain of PFG, including the politics of the host country. The relationship of goals to constraints is affected by factors outside the JCAP’s implicit theory of change. Political conflicts within El Salvador, for example, have limited the PFG’s ability to achieve the constraint-level objectives and outcomes. On both the security and tradables sides, USG and GOES PFG staff who worked on specific goals generally believed that certain GOES policies and actions have limited PFG’s effectiveness in achieving the constraint-level objectives and outcomes. The lack of progress in improving the investment climate under Tradables Goal 5, and the absence of active cooperation between public and private sector members envisioned under Tradables Goal 1, for instance, were seen as major obstacles to making progress on the tradables constraint. Of the private sector interviewees and USG interviewees in El Salvador who deal primarily with the tradables constraint, a total of 8 out of 11 interviewees volunteered that the political conflict between the previous government and the private sector was primarily responsible for the lack of investment and growth in the tradables sector. 27 Partnership for Growth: El Salvador 2011-2015. Fact Sheet http://www.state.gov/p/wha/rls/fs/2011/176636.htm Partnership for Growth El Salvador Mid-Term Evaluation Report 43 Similarly, on the security side, policy disagreements have translated into implementation delays. Two of the five GOES agencies that comprise the overarching coordinating entity for the justice administration (UTE) have put collaboration with the UTE on hold since mid-2013, thereby hindering activity progress. Likewise, the high turnover at the ministerial level—three ministers of Justice and three general directors of the National Police during this past government—also generated uncertainty, although most of the independent security experts contended that the GOES was not the only one to blame. About 60 percent of PFG leaders, Goal Leads, and experts asserted that, while the JCAP is a bilateral agreement between the executive branches of the two governments, goal achievement does not depend solely on their performance; it is also dependent on external factors such as the Salvadoran Legislative Assembly approving laws. Therefore, these interviewees argued that the goal targets for the first five years (as defined in the JCAP) should be less ambitious and more concrete, measurable, and not entirely dependent on the actions of the two executive branches. Improved frameworks for new task forces or prosecutorial and judicial protection would be more realistic benchmarks and, therefore, a better indicator of the adequacy of goals than the national impunity rate to which many other factors contribute. Many contended that more realistic and precisely formulated goals and LOAs would have properly facilitated the tracking of goal-level outcomes and would have allowed at least a correlative association with constraint-level indicators. While USG leadership involved with the JCAP formulation explained that they were careful not to include “non-achievable” (and thus irrelevant) goals in the JCAP, about half of the Goal Leads, both USG and GOES, asserted that non-achievable goals were included and that external factors, including, but not limited to GOES political will, continue to obstruct goal achievement. The Civil Asset Forfeiture Law is an example frequently mentioned by those working on the security constraint. While there is a purely technical part to this legislation on which the PFG was able to make progress, the law needed approval of the Salvadoran legislature in which different interests collided and delayed passage. Another external factor that was often mentioned was the “gang truce” of 2012. This truce was external to PFG, but the effects of its aftermath have (and could continue to) affect goal- and constraint-level PFG outcomes and indicators, including local and national homicide rates and security perceptions. In fact, the 2012 reduction in homicides was attributed by many to the truce, and the subsequent 2013-2014 increase in homicides to the end of the truce process. None of these changes in the first constraint-level indicator had any relation to PFG activities. Finally, there are external developments in the Central American region, Latin America, and the world economy that can affect the success of initiatives like PFG. While there have been few such developments since the inauguration of PFG, they cannot be ruled out in the future. Finding 5: The JCAP did not plan for the potential interactions among goals. The JCAP’s implicit theory of change did not take into account that the impact of the individual goals (and the LOAs within goals) on constraint-level indicators are not separate, but rather interact with each other; the expected benefits from some goals depend on the effective implementation of others. Lack of progress in particular goals (or LOAs) can prevent other goals (and LOAs) from making their expected contribution to growth. A clear example is Tradables Goal 5 (and the closely related Goal 1), which aims to improve the investment climate. If the investment climate does not improve, it will likely mean that workers trained under Goal 3 may not find jobs, and the improved infrastructure under Goal 2 may not Partnership for Growth El Salvador Mid-Term Evaluation Report 44 find users. On the crime and security side, a clear example of the connection between goals is that progress under Goals 1-2 (a strengthened justice administration), Goal 8 (at-risk youth) and Goal 10 (education) will most likely impact the results of Goal 11 (prevention in key municipalities). Finding 6: The JCAP M&E Addendum’s call for the formation of goal-level implementation teams to create and implement goal-level work plans was not fully realized. In addition to outlining the indicators for each constraint, the M&E Addendum to the JCAP required that “the governments of El Salvador and the United States establish implementation teams for each LOA to coordinate the JCAP process and ensure communication. The teams are to meet regularly to develop [various tasks]. 28” Further, the M&E Addendum states that each implementation team will develop “work plans” for each LOA. The evaluation team understands that the “implementation teams” were meant to be operational for each goal; that each implementation team would be led by the corresponding Goal Leads; and that each team would be integrated by the implementing agencies in charge of each LOA of the respective goal. The evaluation team considers that each of these teams would produce a work plan, and regularly meet (e.g. once monthly), to follow up on the work plan. On the security side, the evaluation team found that after the development of the JCAP, no guidance—such as terms of reference, training, or systematic oversight—on how to create implementation teams, work plans, or other components of the M&E Addendum was provided. As a result, not all goals and their LOAs have teams and plans. Generally, when USAID is the lead agency, which is the case for 60 percent of the LOAs, these plans take the form of contracts, statements of work or terms of reference, and work plans between USAID and US firms or Salvadoran non-governmental organizations (NGOs)—a practice that existed prior to PFG. The situation is slightly different in cases where other USG agencies are implementing LOAs. In such situations, the LOA objectives are most often integrated into the agencies’ strategic plans with no specific work plans for PFG projects produced at the LOA level. This lack of management within USG and guidance for the creation of systematic LOA work plans or implementation teams for all the 14 security goals is mirrored within GOES. Also, as many LOAs are “paired” with USG LOAs, GOES agencies seem to “wait” for USG guidance and do not take the lead. Some goals are on target (as detailed in Country Specific Question 3); however, without much coordination among goals spearheaded by PFG implementation teams and through PFG-specific work plans, it will be challenging to appropriately track whether goallevel commitments can attain constraint-level outcomes. 28 M&E Addendum to the JCAP, pg. 1 Partnership for Growth El Salvador Mid-Term Evaluation Report 45 7.2 Conclusions and Lessons Learned Given the absence of explicit theories of change, it is not possible to provide a theoretical assessment of the ability of goals selected in the JCAP to affect the desired constraint-level outcomes. The JCAP aligns twenty goals with the two constraints to growth, but it provides no explicit rationale for why these particular goals were chosen and how they are aligned. Without such a detailed rationale, it is speculative to argue if the goal-level commitments set forth in the JCAP alone are capable of achieving the constraintlevel objectives and outcomes. Because the constraints and goals are subject to a broad range of political and external factors, attempts to evaluate the appropriateness of goals would be speculative; reaching constraint-level objectives and outcomes depend not exclusively on goal-level performance. A goal may be entirely appropriate when first conceived, but world markets, policies, and societies are not static. Even with an explicit and foresighted theory of change, it is unlikely that every goal would remain ideal indefinitely. The theories of change implied in the JCAP— one for tradables, and another for security are considered to be interdependent. Goal-level outputs cannot lead to constraint-level outcomes in isolation because they are interdependent. The fact that bilateral goal-level implementation teams and goal-based work plans were not systematically created makes it more difficult to determine how and to what extent goals are relevant to constraint-level objectives. Uniform guidance through PFG-specific bilateral implementation teams for each goal was not provided and they did not develop goal-level work plans. The evaluation team concludes that the creation of goal-level implementation teams and development of goal-based work plans would have spearheaded the development of direct and continuous connections between LOAs and goals, allowing progress toward outcome achievement to be assessed. 7.3 Course Corrections and Recommendations Consider reviewing and revising (as applicable per goal) certain aspects of the JCAP, geared towards better specifying whether goal-level commitments can attain constraint-level outcomes. Ideally, this would mean that before the mid-term revision of the JCAP, a robust theory of change 29 constructed with technical rigor approaching that of the Constraints Analysis would be formulated. While this is an ideal process, it would take too much time and effort and may mean pausing implementation because of a new focus of design. Given these realities, the evaluation team recommends, instead, conducting a mapping exercise, led by an internal bilateral management team, of all the LOAs, goals, and constraint. This mapping exercise would provide an opportunity to review and reword applicable LOA indicators within the JCAP, with the main objective of aligning LOAs better to goals and goals better to the constraints. This exercise could be done through a workshop of GOES and USG constraints leaders and Goal Leads. The benefit of undergoing this exercise would be that the PFG initiative could collaboratively identify which indicators are unlikely to be reached within the 5-year period, and then 29 A robust theory of change is one that can hold up even under moderate deviations from the baseline state of affairs. In other words, the connections between goals and constraints should accommodate, or at least not be overly susceptible to, exogenous changes in markets or politics within expected ranges. Partnership for Growth El Salvador Mid-Term Evaluation Report 46 realign resources towards more attainable goals. This exercise should be conducted with the larger focus of identifying how impact would be measured eventually at the end of the PFG initiative. As discussed previously, this process should also focus on identifying how goals inter-relate and ensuring that implementers are better aware of how to work together to attain overall constraint outcomes through inter-goal coordination, where applicable. If needed, the PFG management team could engage an independent consultant from El Salvador who is well versed in this discipline and could provide an outsider’s perspective to the process. This mapping exercise could increase staff workload because of the time needed for planning and implementation. However, it would be very beneficial to the PFG El Salvador initiative’s ability to properly track success and identify challenges within implementation in order to make course corrections in a timely manner, and work towards being able to respond to this evaluation question at end line. Finally, given that the JCAP was meant to be a living document and that there is a new GOES administration, the mid-term review of the JCAP represents an opportunity for the El Salvador PFG initiative to properly align goals (and LOAs) to constraints and finalize M&E processes for effective programing. This bilateral process will also further foster the advantages of WGA. Specific to the tradables constraint, revisions should be based on a recognition of the fundamental importance of an adequate investment climate. In order to achieve success under the tradables constraint, a revised JCAP must reflect the active commitment of the GOES to the implementation of the investment climate (LOAs under Goal 5, for instance), and reflect the active political commitment of both governments to the successful implementation of those LOAs. Similar to the tradables constraint, most security constraint goals and LOAs are currently active. The team’s recommendations, therefore, include (re)defining indicators for LOAs and, when necessary, rewording their corresponding sections within the JCAP in order to ensure that the indicators are all attainable and can ultimately lead to the realization of constraint-level outcomes. At mid-term, systematically create, for each goal, implementation teams and work plans, as required by the M&E Addendum. The evaluation team recommends that goal-level implementation teams are created for each goal, as stipulated in the M&E Addendum. These implementation teams should be bilateral, led by the corresponding Goal Leads, and should meet regularly (e.g. once monthly). The goal-level implementation team’s main task would be to develop the goal-based work plans. Going forward, these plans would systematically guide goal-level implementation for both governments and all implementers involved. In developing (and/or finalizing) these work plans, it is particularly important to include indicators. That can appropriately assess how PFG activities are tracked outside of other agency objectives. Singling out PFG-specific indicators as much as possible will significantly help efforts to assess the initiative’s effectiveness, and eventually, its impact. Once these work plans are created, the implementation teams would have the job of ensuring that implementers are following the work plans. The six monthly scorecard meetings would be a good avenue for reviewing progress on all work plans, which would lead to eventually determining whether goal-level commitments can lead to constraints-level outcomes. Partnership for Growth El Salvador Mid-Term Evaluation Report 47 8. COUNTRY-SPECIFIC QUESTION TWO: THE PFG MODEL PLACES EMPHASIS ON EVIDENCE-BASED DECISION MAKING AND FACTBASED MONITORING. IS QUANTITATIVE AND OBJECTIVELY VERIFIABLE INFORMATION BEING USED TO MANAGE JCAP IMPLEMENTATION IN ORDER TO ACHIEVE AND MEASURE RESULTS? Overall, the El Salvador PFG initiative is to be commended for completing the M&E Addendum in a timely manner and as a guiding document for monitoring and evaluation (M&E) processes. Further, since the inception of PFG, the scorecard process was established through a bilateral collaboration. Five scorecards have been developed and made publicly available since 30, promoting transparency in tracking progress of the PFG process. Developing M&E guidance documents and maintaining a system of tracking provides a solid foundation for ensuring that quantifiable and verifiable information is available and progress is tracked. That said, the use of quantitative and objectively verifiable information to manage JCAP implementation differs among goals, as operations and agencies involved under a specific goal in general follow their own forms of management, reporting, and M&E. Beyond the scorecard process, M&E is a somewhat heterogeneous process within the PFG, and interviewees expressed concerns that the scorecard process may be somewhat subjective. Generally, across the two constraints and their corresponding goals, USG has a more systematized M&E process than GOES; however, even within USG, the level of rigor varies among agencies, and thus among and within goals when various LOAs are led by different agencies. Within GOES, most goal-leading and implementing agencies have either not implemented M&E or have only recently initiated M&E processes to meet donor requirements. Although interviewees considered the scorecard process somewhat subjective, there seems to be increased awareness of the importance of a more systematic use of goal-level indicators and M&E in general. The evaluation team used a variety of data sources to determine if quantitative and objectively verifiable information was being used to manage JCAP implementation. First, as part of its desk review, the team attempted to identify, collect, and review M&E documentation produced by USG and GOES agencies for each goal. The amount of available, relevant information differed across goals and was sparse or completely absent for certain goals. Interviews with leadership, Goal Leads, and LOA implementers from both the USG and GOES were a second source of information, which added significant data. The desk/ document review and interviews were supplemented by responses to two questions in the online survey: “In your opinion, are the appropriate indicators being used to allow for transparent, accountable, and factbased monitoring of the PFG?” and “Can you provide some examples of alternative indicators to allow for transparent, accountable, fact-based monitoring?” To answer Country-Specific Questions 2 and 3 subsequently, as required by the evaluation SOW, the evaluation team selected seven goals for in-depth evaluation, which provided a fourth source of information. As explained in the Methodology section, the goal selection process identified goals that 30 http://sansalvador.usembassy.gov/partnership-growth.html Partnership for Growth El Salvador Mid-Term Evaluation Report 48 were most representative of the PFG initiative. (See the Addendum for detailed information on the seven case studies). 8.1 Findings The findings for this second country-specific evaluation question are based on the information gathered from the four sources detailed above and are relevant for all goals. However, illustrations and examples provided in this section are mainly related to the seven goals that the evaluation team studied in depth. Finding 1: The nature of goal-level indicators is quite different for each of the two constraints. The different designs of the security and tradables constraints and their respective goal-level indicators have both strengths and weaknesses. 31 For the tradables constraint, PFG decided to use externally sourced indicators: four indicators are from the Central Reserve Bank of El Salvador, four from the World Economic Forum, and four from similar sources. All 12 goal-level indicators (two for each of the six goals) are well defined and sufficiently specific. (Refer to Annex 6 for list of indicators for constraints and goals). Using indicators from reputable external sources has its benefits: these sources generally incorporate transparent, well-defined, and internationally recognized measures of outcomes; they are publicly available; and they are cost effective. In most cases, the measures are collected at an international level, allowing PFG partners at the constraint level to place El Salvador’s performance in the context of its neighbors’ performance. However, using these external sources also has some drawbacks. Quite frequently, data cannot be consistently collected yearly for every country (let alone in real time), so the time lapse between data collection and reporting is often more than a year. Therefore, these data can only be used retrospectively. At the mid-term point of PFG, these data are best used as baseline measures. The nature of the M&E indicators under the security constraint is different and somewhat more complicated. The security constraint has 21 goal-level indicators (7 goals have two indicators each, and 7 other goals have one indicator each). Of these 21 goal-level indicators, 5 are products of annual public perception surveys conducted since 2012 by Centroamericana José Simeón Cañas University (UCA). While the UCA survey is academically produced, it is contracted by the USG for PFG purposes, and therefore cannot be considered a completely PFG-independent source. The remaining 16 goal-level indicators are generated internally to PFG, mainly as outcome measures associated with specific USG-led or GOES-led LOAs. For example, Security Goal 4 (ensure that the labor force is protected while transiting to and from work) is monitored by one of each of these two types of indicators: the “public perception of safety on public transport routes” 32 is an indicator produced annually with the UCA public perceptions surveys; the “number of reported crimes on public transportation” is an internal indicator and a product of the work of the public transportation task force. While not fully independent, the evaluation team considers the five UCA perception indicators used by PFG to be of high quality. The UCA public perception surveys are professionally conducted by Salvadoran data collectors. Additionally, and unlike indicators derived from externally sourced data 31 32 Refer to Annex 7 for full list of indicators for constraints and goals. Both Goal 4 indicators are cited from the Monitoring and Evaluation Addendum to the JCAP, p.4 Partnership for Growth El Salvador Mid-Term Evaluation Report 49 collections, survey items can be changed to meet specific, evolving PFG monitoring requirements. 33 However, not being able to use the UCA indicator to compare El Salvador’s results to other countries in the region, and the inevitable time-lag between data collection and reporting, presents challenges. The evaluation team also found that the 16 internally generated indicators under the security constraint are conceptually well aligned with their corresponding goals. However, they embody all of the usual M&E challenges: availability of data sources, appropriateness of data definition, and problems of data quality. Further, variations in established M&E methods, practices, and geographic coverage among lead agencies come into play as well. When indicators are being tracked under USAID projects, using rigorous USAID practices and methodologies, they are clearly defined, and data sources are well specified, in accordance with standard USAID M&E requirements. In cases where data were being collected by other USG or GOES agencies, definitions and sources were often less clear. In addition, a number of internally generated indicators have not yet been reported due to the timing of project startup, e.g., the definitions and procedures for collecting data to measure the number of reported crimes on public transportation (Goal 4) are still being developed. Finding 2: The six constraint-level indicators were mostly understood as long-term objectives, with limited relevance to performance at mid-term. The three constraint-level indicators for the tradables constraint—GDP growth rate, foreign direct investment as a percentage of GDP, and exports as a percentage of GDP—were considered by those interviewed to be well defined and sufficiently specific. However, the indicators for the security constraint—homicide rate, national perception of security, and prosecutions and convictions as a percentage of violent crimes reported,—were often considered overly general, e.g. lacking territorialization. The evaluation team also found that a majority of USG and GOES security Goal Leads were unaware of the three long-term overall indicators for their constraint and had to be prompted to discuss them. Goal Leads from both constraints stated that they were never briefed about how their particular goals and LOAs would affect the constraint-level indicators, information that would be useful to appropriately aligning indicators and monitoring to gauge progress for the overall PFG initiative. This lack of information about the relation between goal-level indicators and the constraint level indicators was especially prevalent on the security side. Once Goal Leads became aware, through prompting by the evaluation team, of the constraint-level indicators, they confirmed that these indicators were not systematically included in the scorecard report or discussed during scorecard meetings. 33 Decisions to add, delete, or change survey items should obviously be made with great care as this can impact the comparability of items across years. Partnership for Growth El Salvador Mid-Term Evaluation Report 50 Finding 3: Monitoring goal progress using the PFG scorecards is viewed as a negotiated and only partly fact-based process. The scorecard procedure is outlined in the M&E Addendum and the evaluation team found that both governments have made very explicit commitments to the scorecard process. Overall, respondents found the scorecards useful, and none of the respondents characterized producing the scorecards as an unneeded procedure or suggested discarding it. However, respondents also saw the scorecard process to be negotiated. For instance, part of the negotiated process, especially for goals associated with the tradables constraint, was to define which of the many well-specified and high-quality LOA indicators would be fed into goal monitoring and what weight they would be given. The results of scorecards for a number of goals under the security constraint also went through this selection process, especially when the goals were USAID-led and backed by indicator-rich M&E plans typically developed by USAID contractors. However, for some other goals on the security side (in particular those without USAID M&E plans), the scorecard process was based less on sound indicators—sometimes also because designated activities had not yet begun—and more on qualitative milestones. Milestone performance also feeds into the scorecards, and many Goal Leads see them as “indicators” of progress, even though they recognize that the process of determining where a project sits on its path to implementation is a subjective one. The evaluation team found that fact-based data and political interests were combined in the final scorecard score through a somewhat arbitrary process. For example, among the six tradables goals, only one, Goal 5 (support for Foreign Direct Investment) is rated as “behind schedule” on the fifth scorecard. However, the evaluation team found that only tradables Goal 4 (raise tax revenue to 16 percent of GDP by 2015) could be said to be “on-track” based on the goal-level indicators. The situation is somewhat similar for the security constraint. Overall, the USAID-led goals are in a better position, given that they have detailed M&E plans with well-established indicators to feed into the scorecards. However, a lack of clarity about how LOA performance, milestone results and performance as related to the two goal-level indicators are supposed to be weighted and then feed into the scorecard leaves significant room for political negotiation rather than evidence-based scoring. For example, on the fifth scorecard, 12 goals were reported as “on track,” and only Goals 12 and 13 (both related to prison reform) were reported as “behind schedule.” However, Goal 4 (security on public transportation) – as an example - demonstrated no improvement on one goal-level indicator (public perceptions of security on public transport) and was unable to report on the other (the number of reported crimes on public transport), as that indicator has yet to be gathered. Also, indicators at the LOA level were still only partly established for Goal 4. At the same time, however, Goal 4 did achieve significant results on LOA-level milestones in the 6-month period between the fourth (Nov 2013) and the fifth (May 2014) scorecards. Therefore, both the USG and GOES agencies believed that political will and commitment had significantly improved. Goal leading agencies therefore jointly decided that Goal 4 merited an “on track” score. Similar observations were made for other security goals. The document review further revealed that the monitoring process is essentially performed at the LOAlevel and that no rigorous M&E procedure was put in place to relate the LOA-level indicators to the goallevel indicators within the scoring process. The evaluation team also found that indicators, results, and milestones reported in the scorecards can easily become influenced by factors that should be irrelevant Partnership for Growth El Salvador Mid-Term Evaluation Report 51 when results data are being collected, such as the need for officials to report positive performance to their superiors. All respondents who had experienced a “behind schedule” score or were aware of the impact on others who had received one remarked that obtaining a score other than green (“on track”) served as an “action-forcing event.” Finding 4: PFG stakeholders felt that, in some instances, better indicators could have been selected to monitor PFG, but they did not readily provide alternatives. Respondents to the online survey provided a mixed response when asked if the best indicators were being used to track PFG progress. Among USG leadership, 50 percent indicated that “some of the best indicators are being used,’’ while 22 percent stated that the indicators being used were not the best available (see figure 8.1). Among non-leadership in USG, 4 percent stated that the best indicators are being used, 52 percent stated that some of the best indicators are used, and 16 percent expressed that the best indicators are not being used to allow for transparent, accountable, and fact-based monitoring of PFG. Figure 8.1: Are the appropriate indicators being used to allow for transparent, accountable, and fact-based monitoring of the PFG? (Based on survey responses only) The findings among GOES PFG staff were different: Sixty percent of leadership stated that some of the best indicators were being used, while 40 percent stated that the best were being used. None of the leadership staff within GOES who responded to the survey indicated that the current indicators were not the best. For non-GOES leadership staff, however, 14 percent expressed that only some of the best indicators were being used, and 19 percent thought that the best indicators were being used. However, 29 percent believed that the best available indicators were not being used, while 9 percent were unsure (See figure 8.2 below). Partnership for Growth El Salvador Mid-Term Evaluation Report 52 Figure 8.2: Are the appropriate indicators being used to allow for transparent, accountable, and fact-based monitoring of the PFG? (Based on survey responses only) Overall, only 8 of the 108 survey respondents provided suggestions for alternative indicators and few of these suggestions were clearly defined. A typical example of a suggestion provided was “some sort of security or crime index.” By contrast, 91 of the survey takers provided written comments on the advantages of PFG, representing an unusually high item response for an open-ended question. This finding suggests that survey respondents did not seem as engaged with the specifics of M&E as they are with “bigger picture” concepts. Finding 5: Data gathered from the semi-structured interviews, the document review, and the goal reviews indicate that the most effective M&E processes at the LOA level were performed under goals led by USAID. 34 During the interviews 100 percent of USAID implementers stated that they gathered and used objective results indicators in the normal course of their project implementation in accordance with standard USAID policy. A review of LOA documentation, as part of the in-depth review of the seven selected goals, found this to be the case for every USAID-managed LOA. (Refer to tables 8.1 and 8.2 below for a detailed overview of findings that demonstrate whether quantifiable and evidence-based M&E information is being gathered at the LOA-level). However, in the semi-structured interview process, in response to the question asked about use of indicators, only 21 of all 88 GOES and USG interviewees (24 percent) indicated that objective M&E measures or indicators were being used either in their own activities or more generally in PFG, outside of the scorecard process. This is consistent with our Finding 4 (see above) stating that objective M&E at the LOA-level did not appear to be at the top of most interviewees’ lists of concerns. 34 Note that MCC has also been cited as a partner that practices robust M&E activities. However, the Second MCC Compact is on hold, and as such, no MCC activities were underway during the evaluation. Partnership for Growth El Salvador Mid-Term Evaluation Report 53 A somewhat separate, but important point to make about the adequacy of M&E under the PFG is that the JCAP lists a total of 153 LOAs, which are partly GOES, partly USG responsibility. This may create the impression that separate indicators should exist for each of these LOAs. However, in several cases (and in particular when USAID-led), USG and GOES LOAs are “paired.” For example, under Tradables Goal 6, GOES LOA 6.4 (create an integrated system to serve SMEs seeking to export) is in practice paired with USG LOA 6.7 (support the establishment of small-business development centers throughout the country). This means that the same objectively verifiable indicator can sometimes be used to measure the performance of more than one LOA. A significant proportion of USG LOAs are managed by USAID, and a significant number of these are paired with GOES LOAs. To use the same example: tradables Goal 6 has 11 LOAs, three of which were implemented by the USAID SME Development Program. Hence, for Goal 6, there are quantitative and objectively verifiable indicators available for a total of 6 (and part of a seventh) out of 11 LOAs, objectively verifiable indicators can often be used to measure the performance of more than one LOA. More in general, because of this situation, the proportion of LOAs across PFG that use quantitative and objectively verifiable indicators was actually quite large. The evaluation team also found, based on the document review in particular, that among non-USAID agencies, nearly all the LOA indicators were input indicators rather than outcome indicators. While input indicators are objectively verifiable (e.g., number of people trained, pieces of equipment delivered), they are not indicators of outcomes (e.g., number of people who changed their practice based on training received), although they are sometimes mistaken for them. On the other hand, PFG related USAID documents reviewed by the evaluation team demonstrated that, while input measures were invariably included (since these were used for internal project management purposes), all documents included output or results measures as well. The evaluation team also learned that USG and GOES agencies managed their LOAs differently, thereby making it difficult to attain comparable M&E outputs. Among GOES agencies, the level of M&E varied widely. For instance, one ministry representative described a system that appeared to approach USAID/MCC standards (although the team did not obtain documentation that could verify the quality of the reporting system). Other agencies described systems that measured inputs only. Among USG agencies, there is consensus that USAID, more than any other agency, has implemented a well-defined, long-established M&E process for their goals, and that this is reflected in the quality of indicators. A number of indicators included in USAID’s goal-related M&E plans appear explicitly as “PFG indicators,” while others do not but are still used to monitor operational performance of PFG-related activities. Further, USAID routinely performs data-quality assessments for its implementers, carries out regular portfolio reviews, and has others tools in place to ensure ongoing M&E. Even within USAID, interviewees suggested that USAID had increased its M&E effort under PFG, bringing “more scrutiny and more depth” to monitoring than before. Finding 6: The M&E process for PFG at the goal and the LOA levels has evolved and has become more robust over time. Ninety percent of the Goal Leads of both governments active during the first and/or second years of the PFG (including individuals who no longer serve as Goal Leads) stated during the interviews that the scorecard process was initially somewhat of a “trial-and-error” system. Not only did many LOAs initially lack indicators or not track indicators, there was reluctance on both sides to be critical of their Partnership for Growth El Salvador Mid-Term Evaluation Report 54 counterpart’s performance, given the high-level political capital invested in the program. However, with time, the process became more systematic. The first scorecard, in particular, was mainly a USG product (produced in English and shared with PFG staff in Washington, D.C., before it was shared with GOES) meant to demonstrate that the PFG had indeed begun and that projects were underway, but the development of the scorecard had little relation to initial operations on the ground. The drafting of subsequent scorecard transformed into a more collaborative process, with Goal Leads frequently producing the first draft for their respective goals in Spanish. The draft was then reviewed by their agencies (mainly on the USG side). The evaluation team also found that the scorecard process progressively evolved into a forward-looking, milestones-setting process. 35 Fifty percent of the Goal Leads stated during interviews that the current scorecard process was sufficiently objective as a self-assessment process for both governments. Leadership interviewees indicated that the political negotiations that are part of the scorecard process are appropriate given the high-level political capital invested on both sides and that they do not threaten the rigor of the M&E process at the operational level, i.e., at the LOA level. Further, a number of GOES agencies that have been cooperating with USAID’s implementing firms on the USAID-led goals expressed their appreciation for the M&E rigor USAID brings to the process and the opportunity to learn from USAID’s lead. Such observations also extended to non-USAID goals. For example, the GOES agencies involved with security Goal 12 (reduce prison overcrowding) stressed the importance of the new prisoner classification system INL is helping to create within the National Directorate of Prisons and each prison facility. This new system will permit the use of much more detailed indicators to perform M&E in this field. While historically the only indicator available on overcrowding was the number of prisoners versus the number of beds per prison, the new prisoner classification system will allow different types of prisoners to be distinguished, and thus make it possible to measure how reform related to reducing prison overcrowding impacts different categories of prisoners. 8.2 Conclusions and Lessons Learned Quantitative and objectively verifiable information is not being universally used to manage JCAP implementation. The majority of LOAs were managed with quantitative and objectively verifiable information using USAID M&E procedures, as they were led by USAID or were GOES LOAs “paired” with USG. Moreover, the scorecard process has evolved, with more recent scorecards incorporating explicit, forward -looking targets that can be evaluated at the LOA level, even though those indicators sometimes measure inputs only. Confusion about what the scorecard scores mean. Interview responses from GOES and USG participants about the scorecard process reflect confusion regarding whether scores represent progress toward the goal-level indicators or measure progress on the various LOAs. Ideally, they should represent both. In practice, external events can affect the achievement of goal-level objectives. Alternatively some critical LOA may have been overlooked so that the totality of the LOAs did not, in fact, add up to 35 Milestones were set per semester and sometimes per year, depending mainly on specific dynamics under each goal Partnership for Growth El Salvador Mid-Term Evaluation Report 55 progress on the goal. Such discrepancies are important information that senior officials of the two governments need to assess. The nature of the goal- and constraint-level indicators tends to differ under the two constraints. Tradables indicators are more likely to be externally generated, i.e., by organizations not explicitly partnering under the PFG. This approach is cost effective and facilitates cross-country comparisons, but suffers from lags in reporting and a lack of control—and thus flexibility—over what is measured. Security indicators are more likely to be generated during implementation, and have thus often been less readily available at start-up than the tradables indicators have been. Many PFG stakeholders do not believe the scorecard process is an objective, fact-based monitoring process, but still consider the process useful. The PFG is a bilateral effort, and the scorecard process plays an important role in reaffirming this. While respondents felt the scorecard process was subjective and did not exclusively rely on objective outcome data, all appreciated the process and considered it an important tool. Also, receiving a less than an “on track” score is considered highly undesirable. Avoiding such a rating provides an incentive to perform. The systemic problem with the scorecard process, then, is not the grading system (“on track” or “behind schedule”); rather, it is that the evaluation process often ends at the LOA-level. The rating is often an aggregation of the LOA data. It is widely believed that at least some M&E indicators could be improved, but alternate, improved measures were not easily provided. Few interviewees and only one in five survey respondents felt that the indicators currently being used were as good as potential alternative better indicators. Only a few people, though, could do more than broadly sketch what a superior indicator might look like. The most precise suggestions were provided during the semi structured interviews on the security side. For USG LOAs not managed by USAID and MCC, as well as for GOES LOAs not “paired” with a USAID- or MCC-led LOAs, indicators based upon outcomes are not routinely collected. Generally, agencies other than USAID and MCC use process rather than outcome data as LOA indicators. Process data (inputs) are important, as without them, it is difficult to know if outcomes are produced efficiently. However, without objective outcome data, all that can be known is that resources were expended, and outcomes must be assumed. The M&E processes at the goal and LOA levels have grown more rigorous over time. Initially, many non-USAID and non-MCC LOAs lacked indicators. This was partly due to necessity: a number of such LOAs pre-dated PFG (such programs are sometimes referred to as “legacies”), and existing contracts may not have supplied indicators specific to the PFG LOA. Other activities were brand new, and individuals working on such new LOAs may not have known what M&E indicators would be most appropriate. As PFG progressed, however, more indicators were developed for such LOAs, often by “trial and error.” Partnership for Growth El Salvador Mid-Term Evaluation Report 56 8.3 Course Corrections and Recommendations Require and develop more robust M&E systems for each LOA, and institute training, as needed, to promote systematic tracking. As the two governments revisit the PFG at mid-term and also given the existence of a new GOES administration, the evaluation team recommends that they should aim to incorporate a USAID/MCC-type methodology for M&E activities for each LOA. For each goal, this process could best be led by a goal-level implementation team and formalized in goal-level work plans – as the evaluation team suggests to create in Country Specific Question 1. As indicated in the findings, many GOES LOAs are currently “paired” with USAID projects and so are already indirectly measured by USAID. Going forward, these informal arrangements should be formalized by the goal-level implementation teams and work plans, and the GOES partners brought more fully into the M&E process. Have USAID and MCC train and guide all goal-level implementation teams on this process. Given that USAID and MCC have more established M&E procedures, the evaluation team suggests that a possible way to incorporate more standardized M&E into PFG would be to have these entities facilitate the M&E process and conduct trainings, as needed, with all Goal Leads and LOA implementers. The evaluation team does note that leveraging the M&E procedures being used by USAID and MCC should be undertaken in a collaborative manner where all PFG partners discuss which aspects of these processes are most useful to the PFG initiative. This will ensure that the new procedures are relevant and that there is buy-in from all stakeholders for their use. Provide clear guidance on the role of the constraints-level indicators. This should help to reduce the current confusion among Goal Leads and implementers about the role and meaning of the six constraintlevel indicators (three per goal). Guidance is needed on how LOA level indicators, goal level indicators, and scorecards relate to the constraint-level indicators. Having all PFG partners and goal level implementation teams understand these links would facilitate the achievement of desired PFG outcomes. Bring more rigor to the scorecard process. The scorecard preparation and review has been an evolving process and the evaluation team would recommend that the USG and the GOES continue to experiment with ways to make it a more useful management tool. To facilitate a swift process of enhancing the objectivity of the scorecard process, the evaluation team recommends that attention be given to systematizing LOA indicators, which will lead to providing more evidence-based information for the scorecard process. New or modified indicators should be proposed by each goal level implementation team for consideration by the senior levels of the two governments. In recent reviews, PFG LOA teams have made commitments (or projections) of accomplishments to be made over the subsequent six-month review period, also known as ‘milestones’. The team suggests that that practice continues and that projections also be made for each LOA for the period after that, that is, for a total of 12 months. This would provide a clearer trajectory over which to evaluate PFG progress. The evaluation team also recommends that each implementation team discuss progress expected to be made, and progress actually made, over the same two periods with respect to the goal-level measures. This would help to improve on the current practice of ‘adding up’ of LOA-level accomplishments rather than measuring progress toward the goals. Having the public report card similarly discuss progress that has been made toward the goals would also re-enforce public understanding of PFG as a strategy toward agreed objectives and not simply as a set of agreed, bilateral activities. The evaluation team suggests that preparing the scorecard will be a task of the goal level implementation teams in full (to be formed, as suggested in other Partnership for Growth El Salvador Mid-Term Evaluation Report 57 recommendations). This will also help to link the scorecard to the goal level work plans (to be formulated by the goal level implementation teams, as suggested in other recommendations). Transform the twice yearly scorecard presentation process into a learning and innovative event. The evaluation team suggests to restructure the twice yearly scorecard meetings into a day-long workshop of learning and innovation for the 20 bilateral implementation teams (as to be created), instead of the current model of a goal leads only event. In addition to short presentations per goal (as currently offered), we recommend to include comprehensive progress presentations on each of the constraints as a whole, and to focus during the rest of the day on specific themes (e.g. M&E, branding, lessons learned etc.). Such events would contribute to building synergies among teams, goals and LOAs, provide opportunities for awareness creation among implementers about their position within the PFG process, and contribute more in general to team building and enlarging shared ownership of PFG, as it will bring the implementers more systematically on board. Actively engage all Goal Leads and implementing agencies in improving LOA-level outcome indicators. In line with making the scorecard process more rigorous, and establishing implementation teams and goal-level work plans for the next 2.5 years of PFG, identifying and refining M&E indicators should become an ongoing process and concern. The evaluation team recommends that at goal level, the input of LOA implementers be regularly sought as part of the recommended implementation teams. At the same time, those involved with the daily implementation of LOA should be made aware of how their achievements can positively affect the goals they are meant to achieve and what resources will be required to objectively and quantifiably measure key outcomes. Such processes across all goals would help to guarantee that the evidence of a progress generated at the operational level is fed into the scorecard discussions. Increase public knowledge of PFG monitoring findings as well discussions on progress. Though the scorecards are available publicly on the USG and GOES websites and often announced via press conference and/or social media, the evaluation team recommends that a concerted effort be made to increase knowledge of scorecard findings by making them more accessible to LOA implementers and other key stakeholders. Further, it recommends that PFG stimulate constructive but critical academic debate about the PFG progress by organizing workshops or learning sessions upon completion of the semi-annual review of scorecards to gain civil society and private sector input regularly on the progress and relevance of the PFG initiative in El Salvador. Such activities should be organized by the bilateral PFG management team. Partnership for Growth El Salvador Mid-Term Evaluation Report 58 Table 8.1: Summary of Availability of Quantifiable and Objectively Verifiable Information. Crime and Insecurity (Selected Goals Only) GOAL 1 & 2 LOAs GOES LOA1: Implement the necessary changes in practices, policies, regulations, and applicable laws as identified throughout the project. GOES LOA 2: Identify and make available appropriate staff to be trained. GOES LOA 3: Emphasize continuous vetting as a fundamental principle of the Security Plan of the Security Cabinet of the Government of El Salvador. GOES LOA 4: Fully participate in programs to support the justice sector. GOES LOA 5: Fully participate in analysis of the current codes. USG LOA 1: Provide technical assistance to improve the management and investigative capacity of the Attorney General (AGO), Public Defender (PGR), and National Civilian Police (PNC), forensic services, judges and court personnel. USG LOA 2: Support the development of effective case management models; improving police/prosecutor coordination; providing better and more equitable access to justice; re-engineering processes and change management procedures in the common crime unit; establishing career paths and leadership development within the Judicial sector; increasing accountability and transparency in the sector by enhancing judicial oversight and investigative capabilities; supporting civil society; and strengthening crime observatories. USG LOA 3: Provide assistance to promote increased coordination between justice sector actors and institutions; improve management and administration of justice sector institutions; and increase effectiveness of criminal investigation. USG LOA 4: Provide technical support in the areas of the pre-trial elements listed above as well as plea-bargaining (agreements with the public prosecutor’s office). USG LOA 5: Provide technical assistance to conduct an in-depth code analysis, draft legislative fixes, and develop strategies to enhance criminal justice efficiency and effectiveness relating to criminal procedure, an evidence code, and internationally accepted law enforcement tools; implementing such legislation, procedures, and strategies. GOAL 4 LOAs GOES LOA 1: Maintain a permanent exchange regarding the employment status of USG-vetted or USG-trained and advised personnel in accordance with Salvadoran procedures and regulations. GOES LOA 2: Improve supervision and control in accordance with the system of indicators of the Security Cabinet. GOES LOA 3: In collaboration with the USG, review the current regulations of the public transportation sector and identify areas to improve transparency and accountability. GOES LOA 4: Implement necessary changes in practices and regulations and encourage the approval of the necessary legislation. GOES LOA 5: Provide personnel, equipment, and facilities to support this program. Partnership for Growth El Salvador Mid-Term Evaluation Report Quantifiable and Objectively Verifiable Information YES [Scorecards, JSSP PMEP August 9, 2013] YES [PMP Narrative Report (Oct 2013)] NO [Cannot match indicators to LOA] YES [PMP Narrative Report (Oct 2013) Final] NO [Cannot match indicators to LOA] YES [Checci Quarterly Reports; Scorecards, JSSP PMEP August 9, 2013; Indicators: #1, #2, #3, #8] YES [Indicators are defined in the JSSP M&E Plan under the USAID contract with Checci, JSSP PMEP August 9, 2013; Indicators: #10, #22, #2, #6, #7, #9, #20, #21, #18, #19, #25] YES [Indicators are defined in the JSSP M&E Plan under the USAID contract with Checci, JSSP PMEP August 9, 2013; Indicators: #2, #10, #12] YES [Indicators are defined in the JSSP M&E Plan under the USAID contract with Checci, JSSP PMEP August 9, 2013; Indicators: #9, #6] YES [Indicators are defined in the JSSP M&E Plan under the USAID contract with Checci, JSSP PMEP August 9, 2013; Technical assistance provided to UTE addresses these areas ] Quantifiable and Objectively Verifiable Information Not provided Not provided YES [Scorecards, Copia de plan borrador mod proteccion al transporte public] PARTLY [Scorecards; legislative measures] YES [Scorecards; PNC internal information, Copia de plan 59 USG LOA 1: Provide technical assistance, training, and mentorship for the vetted units to combat crimes involving public transit, and facilitate cooperation between GOES and private sector. USG LOA 2: Provide technical assistance to help increase the transparency and accountability of the public transport system. GOAL 11 LOAs GOES LOA 1: Identify dedicated staff to develop and support municipal crime prevention councils. GOES LOA 2: Identify dedicated funding to implement crime prevention plans at municipal level. GOES LOA 3: Promote the decentralization of authority and responsibility for preventing crime to the municipalities. GOES LOA 4: Improved coordination among line ministries with a stake, resources and mandate for addressing key risk factors (lack of access to education, and employment opportunities). GOES LOA 5: Strengthen the cooperation between the government security organizations and said communities. GOES LOA 6: Implement vocational training for at-risk youth. USG LOA 1: Provide technical assistance to promote and facilitate components 2 and 5 of the strategy. USG LOA 2: Explore the provision of loan guarantees to facilitate financing of public goods at municipal level (i.e. sanitation, infrastructure) USG LOA 3: Region-wide study of the issue of violence and security as it impacts community groups in the hemisphere to bring the community-level experience from the region to bear in El Salvador. USG LOA 2: Explore ways to work with U.S. state/local institutions, the Governments of Mexico, Colombia, and U.S. interagency actors. USG LOA 3: Provide technical support in parole systems and prisoner classification systems. GOAL 12 LOAs GOES LOA 1: Improve the prisoner classification system. GOES LOA 2: Build a new prison facility and three prison farms, and implement an aggressive reintegration program. USG LOA 1: Provide technical assistance in the management of prisons/corrections, including mentoring and limited training in order to develop prison/correction officer train-the-trainer program, and develop and implement prison classification system. USG LOA 2: Explore ways to work with U.S. state/local institutions, the Governments of Mexico, Colombia, and U.S. interagency actors. USG LOA 3: Provide technical support in parole systems and prisoner classification systems. Partnership for Growth El Salvador Mid-Term Evaluation Report YES [DOJ internal reporting; scorecards, Copia de plan borrador mod proteccion al transporte public, Scorecards: 1&3] YES [USAID Casals Transparency Program’s M&E Plan; quarterly reporting by Casals; Scorecards: 1, 2, 3, Copia de plan borrador mod proteccion al transporte public] Quantifiable and Objectively Verifiable Information NO [Scorecard #1, 3, 4 ] NO [Scorecard #2, 3] PARTLY [legislative and policy initiatives in process or completed] Not provided YES [PNC information on Community Policing] NO [Scorecards #3, 4] YES [Crime and Violence Prevention Program’s M&E Plan; Creative Associates’ quarterly reporting] YES [Crime and Violence Prevention Program’s M&E Plan; Creative Associates’ quarterly reporting] Not Provided YES [INL internal reporting] YES [INL internal reporting] Quantifiable and Objectively Verifiable Information YES [Scorecards 1-5 ; INL internal reporting; and Directorate of Prisons information, Prison Classification Event-Unclassified Decision Memorandum , Mariona Study PARTLY [Scorecard 5, 3, 2, 1, INL internal reporting; Directorate of Prisons internal reporting] PARTLY [Scorecard 3, 1; INL internal reporting, Prison Classification Event-Unclassified] PARTLY [INL internal reporting] PARTLY [Scorecards 1-5; INL internal reporting] 60 Table 8.2: Summary of Availability of Quantifiable and Objectively Verifiable Information. Low Productivity and Tradables. (Selected Goals’ Only) GOAL 3 LOAs Quantifiable and Objectively Verifiable Information GOES LOA 3.1:Develop an education and employment plan for youth and YES [Year Three Work Plan (Oct 2011-Sept 2012) women Year Four Work Plan (Oct 2012-Now 2013)] GOES LOA 3.2 Establish training programs in English and ICT for YES. [Indirectly through USAID Improving Access to Employment Program, employed population and for youth Quarterly Reports and DOS reports, Scorecard #2, 4, 5] GOES LOA 3.3:Commit to making transformational reforms necessary to YES [Results should be available through quarterly reports from USAID Higher improve the quality of El Salvador's educational system Education Program Year Three Work Plan (Oct 2011-Sept 2012)] GOES LOA 3.4:Create a Talent Network of Salvadorans living abroad YES [Year Three Work Plan (Oct 2011-Sept 2012) Year Four Work Plan (Oct 2012-Now 2013)] GOES LOA 3.5:Support youth insertion in the labor market YES [Results can be inferred from USAID Improving Access to Employment Project, quarterly reports Year Three Work Plan (Oct 2011-Sept 2012), Year Four Work Plan (Oct 2012-Now 2013), Scorecard #2, 4, 5] GOES LOA 3.6:Improve basic computer training programs for entry level YES [Results can be inferred from USAID Improving Access to Employment positions and for employees Project quarterly reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #2] GOES LOA 3.7:Improve English for the workplace programs YES [Results can be inferred from USAID Improving Access to Employment Project quarterly reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #4, 5] GOES LOA 3.8:Strenghten and enhance productive diversification through YES [Indirect information will be available from USAID Higher Education scientific and technological diversification Program, 2nd and 3rd Semi Annual reports] GOES LOA 3.9:Focus on technological development in priority tradables YES [Results can be inferred from USAID Improving Access to Employment sectors Project quarterly reports, Year Three Work Plan (Oct 2011-Sept 2012)] GOES LOA 3.10:Design and create a program for the transfer and YES [Results can be inferred from USAID Improving Access to Employment assimilation of ICT Project quarterly reports and the Higher Education Program quarterly reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #2] GOES LOA 3.11:Stremgthen INSAFORP Partnership for Growth El Salvador Mid-Term Evaluation Report YES [Results can be inferred from USAID Improving Access to Employment Project quarterly reports, Scorecard #5] 61 GOES LOA 3.12:Improve capacity to collect, analyze, use, and disseminate YES [Results can be inferred from USAID Improving Access to Employment labor market data Project quarterly reports and US DOL TA reports, Year Four Work Plan (Oct 2012-Now 2013), Scorecard #2] GOES LOA 3.13:Use better data and analysis to project labor market YES [Results can be inferred from USAID Improving Access to Employment demand Project quarterly reports and US DOL TA reports, Scorecard #2] GOES LOA 3.14: Develop youth scholarship program YES [Results can be inferred through USAID reports on SEED Program] USG LOA 3.15: Support teacher training in English YES [USAID Improving Access to Employment Program, Quarterly Reports and DOS reports, Scorecard #2, 3, 4, 5] USG LOA 3.16: Provide technical support to work skills YES [USAID Improving Access to Employment Program, Quarterly Reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #5] USG LOA 3.17: Develop alliances, provide technical skills and language YES [USAID Improving Access to Employment Program, Quarterly Reports, Year training Three Work Plan (Oct 2011-Sept 2012), Year Four Work Plan (Oct 2012-Now 2013) Scorecard #1, 2, 3, 4, 5] USG LOA 3.18: Technical assistance to improve labor market information YES [U.S. DOL report dated, Oct 15, 2012 and USAID Improving Access to systems Employment Program Quarterly Reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #3] USG LOA 3.19: Help GOES build capacity to conduct surveys to match YES [U.S. DOL report dated, Oct 15, 2012 and USAID Improving Access to employer needs and skills Employment Program Quarterly Reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #1] USG LOA 3.20: Help GOES improve services for job seekers YES [USAID Improving Access to Employment Program, Quarterly Reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #3] USG LOA 3.21: Improve tertiary training and higher education quality YES [PPT presentation for USAID Higher Education Program, Year Three Work Plan (Oct 2011-Sept 2012)] USG LOA 3.22: TA to focus on innovation and technological development PARTLY [PPT presentation for USAID Higher Education Program] in priority tradables sectors USG LOA 3.23: Support development of a youth scholarship program YES [USAID SEED project monitoring reports] USG LOA 3.25:Launch government, business, labor dialogue on YES [Year Three Work Plan (Oct 2011-Sept 2012), productivity and competitiveness Scorecard #1] GOAL 5 LOAs Quantifiable and Objectively Verifiable Information GOES LOA 5.1: Strengthen institutional capacity to conduct market studies Not provided GOES LOA 5.2: Encourage investments from Salvadorans in the U.S. Partnership for Growth El Salvador Mid-Term Evaluation Report Not provided [Scorecard #3, 4] 62 GOES LOA 5.3: Strengthen Ministry of Economy and PROESA YES [PROESA Benchmarking Final Report] GOES LOA 5.4: Implement the Integrated Investor Attention System Not provided GOES LOA 5.5: Implement a strategy to remove barriers to FDI YES [FDI FUSADES DRAFT REPORT] GOES LOA 5.6: Improve plant inspection by Ministry of Agriculture Not provided GOES LOA 5.7: Strengthen PROESA’s institutional capacity to identify and YES [MCC funded diagnostic, legislation clarifying legal status of PROESA prioritize strategic sectors for the promotion and attraction of investments passed in April 2014, PROESA Benchmarking Final Report, Scorecard #1, 2, 4, 5] and for the development of an investment promotion strategy. GOES LOA 5.8: Design and implement a country image strategy based on YES [PROESA country image strategy with support of the IDB, PROESA studies of the perceptions of the business sector in the United States and Benchmarking Final Report, Scorecard #1] other countries of interest, in order to boost investment and trade. USG LOA 5.9: Offer institutional strengthening of PROESA in trade and YES [MCC funded diagnostic, legislation clarifying legal status of PROESA investment facilitation. passed in April 2014, PROESA Benchmarking Final Report, Scorecard #2, 3] LOA 5.10: Support improvements in the local business climate through the YES [Quarterly reporting of the USAID Municipal Competitiveness Project July 1Municipal Competitiveness Project; provide technical assistance to Sept 2013 (Annex B), July 1- Sept 2011 (Section X), July 1- Sept 2012 (Annex A) municipal councils. Scorecard #3] USG LOA 5.11: Partner with the IFC to use Doing Business Indicators as a PARTLY [Action plan completed by IFC and presented to Growth Council, diagnostic for the Growth Council. Scorecard #5] USG LOA 5.12: Undertake a review of the laws and processes in El YES [GOES and MCC commissioned FUSADES study of barriers to FDI, Salvador that facilitate investment in order to create an incentives system FUSADES DRAFT REPORT] tailored to different types of investors in key sectors. USG LOA 5.13: Support GOES in the design and implementation of a PARTLY [PROESA country image strategy with the support of the IDB, PROESA country image strategy based on studies of the perceptions of the business Benchmarking Final Report] sector in the United States and other countries of interest, in order to boost investment and trade. GOAL 6 LOAs Quantifiable and Objectively Verifiable Information GOES LOA 6.1: Strengthen the technical capacity of PROESA legal status. YES [Indirectly through annual reports of USAID SME Development Program (Chemonics)] GOES LOA 6.2: Strengthen the Ministry of Economy and PROESA to Not provided [some information on assistance to SMEs provided indirectly through continue developing the production and export strategy which improves annual reports of USAID SME Development Program (Chemonics)] innovation and quality GOES LOA 6.3: Work with USG to facilitate the export process Partnership for Growth El Salvador Mid-Term Evaluation Report YES [Indirectly through annual reports of USAID SME Development Program (Chemonics), Scorecard #3] 63 GOES LOA 6.4: Create an integrated system to serve SMEs seeking to YES [Indirectly through annual reports of USAID SME Development Program export (Chemonics), Scorecard #3, 4] GOES LOA 6.5: Strengthen and increase diversification through innovation Not provided USG LOA 6.6: Provide TA to the Ministry of Economy and PROESA to YES [Annual reports USAID SME Development Program (Chemonics), Scorecard serve SMEs seeking to export #1] USG LOA 6.7: Support the establishment of small business development YES [Annual reports USAID SME Development Program (Chemonics), Scorecard centers throughout the country #1, 3, 4, 5] USG LOA 6.8: Increase the internationalization of Salvadoran firms through YES [Annual reports USAID SME Development Program (Chemonics), Scorecard guided processes or mentoring #4] USG LOA 6.9: Explore ways to facilitate the speed of exports from El PARTLY [Assistance provided to firms measures number of firms assisted but not Salvador to the U.S. and reduce the rejection rate at the U.S. border their success in meeting requirements, Scorecard #1] USG LOA 6.10: Support the increased participation of Salvadorans abroad NO [IAF grant to FUPEC foundation does not appear to measure objective results, in the Salvadoran economy Scorecard #5] USG LOA 6.11: Promote opportunities for bi-national business alliances in NO [El Salvador Investment Challenge Facility anticipated as part of MCC second the tradables sector compact ] Partnership for Growth El Salvador Mid-Term Evaluation Report 64 9. COUNTRY-SPECIFIC QUESTION 3: AT THE MID-TERM, ARE THE PERFORMANCES OF THE SELECTED PFG INTERVENTIONS ON TARGET AND CREATING THE NECESSARY OUTPUTS TO ACHIEVE THE DESIRED OUTCOMES? As discussed in the response to the previous evaluation question, Country Specific Question 2—which addresses whether quantifiable and objective information exists to track the PFG initiatives—respondents expressed a desire for better indicators and an initiative-wide understanding of what should be tracked and how. This question— Country Specific Question 3— focuses on a related matter, regarding whether selected PFG interventions are on target. The evaluation team sought answers to this question through case studies of seven selected goals, which are detailed in the Addendum. Documented data sources for these case studies include a review of M&E documentation, such as the M&E plans as submitted by implementers; 36 and the semi-annual scorecards. To gather further information, the evaluation team interviewed not only all Goal Leads of the seven selected goals but also various implementers and independent experts. To provide context, the team conducted a limited amount of site visits to observe operational activities. The JCAP M&E Addendum was developed to provide guidelines for monitoring PFG interventions. According to the M&E Addendum: To ensure effective implementation and that the desired outcomes are achieved, the JCAP will be subject to a rigorous, transparent monitoring and evaluation process, that will include the significant participation of civil society and the private sector. Any changes to this M&E addendum (including revisions to reporting requirements or indicators) are to be documented during the high-level annual review rather than revising this addendum. 37 Further, the M&E Addendum requires that monitoring occur at three levels: constraint, goal, and LOA levels. Therefore, the evaluation team should have been able to determine easily whether interventions were on target at mid-term. Not surprisingly, the reality on the ground proved to be rather complex. The semi-annual scorecard is supposed to be the main tool for tracking goal-level progress. According to the M&E Addendum, “The scorecard is to be accompanied by a description that provides a justification for each score assigned. This justification is to include the results of the goal indicators included in this addendum along with other relevant supporting information and data (which may include results of monitoring and evaluation conducted independently by each government on individual LOAs).” As a result, as previously discussed, the current scorecard process is managed by the Goal Leads and does report goal-level progress. 36 Note that the evaluation team did not receive M&E plans from all selected goals, although this request was made prior to and during the site visits. This does not mean that the plans did not exist, but that the team was not able to review them. 37 PFG El Salvador-United States JCAP M&E Addendum, pg. 1 Partnership for Growth El Salvador Mid-Term Evaluation Report For tracking LOA-level progress, the M&E Addendum affirms the following: the respective implementation teams should coordinate work plans for each LOA which, in turn, will be developed by the responsible implementing agency in coordination with the overall team. These work plans are to contain timelines, performance indicators and targets for individual LOAs and will constitute inputs of the monitoring and evaluation process. Implementation teams will exchange regular updates on work plan performance. Results from the work plans will inform development of the semi-annual scorecard. The evaluation team interpreted the wording in the Addendum to mean that an implementation team would formulate LOA-level implementation plans to track each LOA’s progress, which cumulatively (for all LOAs under one goal), and in conjunction with the high-level, goal-level indicators themselves, would be used to evaluate goal-level progress. Additionally, the evaluation team understood the Addendum wording to indicate that the implementation teams would be composed of not only Goal Leads (as previously stated in Country Specific Question 1), but also representatives from the implementing agencies of each LOA under the particular goal. Such a team would therefore meet regularly (e.g. monthly), to monitor progress on each LOA and introduce course corrections when needed to stay on track. Also, the work plans would be technical documents that would define a more detailed road map than what is indicated in the JCAP for each goal, define indicators per LOA, and explain how these indicators would feed into the scorecards. Given these parameters, the evaluation team asked the following main sub-questions to leadership, Goal Leads, and implementers of USG and GOES teams in El Salvador during interviews: • • • • Is your goal(s) on target (or behind schedule)? Which M&E mechanisms are used to evaluate if goal(s) are on target (or behind schedule), beyond the scorecards? Have there been any major changes to how the PFG approach is implemented, specific to your goal(s)? If yes, what are they? And why have they been instituted? In what way do you coordinate with LOA(s) implementers within your goal to ensure that the performance of your goal is on target? Have implementation teams and work plans as required by the M&E addendum been developed and are these fostering monitoring activities? Partnership for Growth El Salvador Mid-Term Evaluation Report 9.1 Findings Finding 1: Responses varied concerning whether PFG interventions are on track at mid-term, with majority responses stating that PFG is on track. The PFG initiative was launched in 2011 and therefore mid-term would be slated at 2.5 years after initiation. Overall, when asked whether PFG interventions are on track at mid-term, there were three groups of responses as indicated in figure 9.1– a majority of respondents, 38 percent from both USG and GOES believe that PFG interventions are on target at mid-term. Figure 9.1: Responses about whether or not PFG Selected Interventions are on Target (Based on survey responses only) Figure 9.2: Interviewee Responses concerning the status of PFG at Mid-Term On Target: • The PFG is generally on target; however, concerns exist with a transition to a new government and the unknowns that come with having new GOES staff. It could be challenging to re-establish relationships that have already been built. • PFG is doing really well on the working level. The political will, coordination and communication piece, that is what has not quite caught on yet, moving from year 2-3. Not on Target: • Not yet, I don't think. I also think it is too soon to expect any evidence that it's on target; But the question is what the target is? What is supposed to be achieved after 5 years? • Challenges exist, which are slowing down progress, including bureaucratic inefficiency, inconsistency of judicial decisions and in general comparatively low quality of public administration performance. Mixed Impressions: • Cannot say yes or no, based on the implementation, because there is no source data that can say is valid to support either claim • PFG cannot only be judged by projects status as it is a large-scale government-to-government initiative. • PFG did not come with a manual. Therefore it took time to properly plan out how the initiative should be implemented. Therefore, simply concentrating on interventions takes away from identifying the broader potential positive influence of the PFG initiative as a whole that should be highlighted in this mid-term evaluation. • It’s difficult to say what’s on track because we don’t have beginning baseline, specific benchmarks; it’s not written in JCAP, so we cannot say. • It took about a year after PFG was officially launched to complete negotiations for the start of the PFG initiative between the two governments prior to beginning the implementation of projects (except for projects that were already underway prior to PFG). Partnership for Growth El Salvador Mid-Term Evaluation Report There also seemed to be general confusion concerning what “on target” means at mid-term, particularly about whether such an assessment should be based on overall constraint-level indicators, goal-level indicators, LOA-level indicators, or scorecards and their milestones. Finding 2: The seven selected goals are largely on target according to semi-annual scorecards. The fifth semi-annual scorecard findings (May 2014) indicated that the majority of the seven selected goals were on target. This is true for four out of five security goals (80 percent), and two out of three tradables goals (66 percent). Table 9.1 below provides a comparative summary of all the semi-annual scorecards issued until mid-term for the selected security and tradables constraint goals. Table 9.1: PFG - Comparative overview of semi-annual scorecards issued since the start of PFG: November 2011– May 2014 Constraint 1: Crime and Insecurity 1. Strengthen justice sector institutions 2. Improve criminal justice procedures 4. Reduce impact of crime on commuters/public transportation 11. Prevent crime & violence in key municipalities and support reforms Scorecard 1 May 2012 Scorecard 2 Nov 2012 Scorecard 3 May 2013 Scorecard 4 Nov 2013 Scorecard 5 May 2014 On Track On Track On Track On Track On Track On Track On Track On Track On Track Behind Schedule On Track On Track On Track On Track On track On Track On Track 12. Reduce overcrowding in prisons On Track On Track On Track On Track Constraint 2: Low Productivity in the Tradables Sector 3. Strengthen labor force to match labor market demand 5. Support a strategy for attracting and promoting foreign direct investment 6. Surmount low productivity in tradables Scorecard 1 May 2012 Scorecard 2 Nov 2012 Scorecard 3 May 2013 Scorecard 4 Nov 2013 Behind Schedule Scorecard 5 May 2014 On Track On Track On Track On Track On Track On Track Behind Schedule Behind Schedule Behind Schedule Behind Schedule On Track On Track On Track On Track On Track Further details on finding 2 regarding the Security Constraint LOAs’ status, based on scorecard and other M&E documentation review. Based on the scorecard review findings, for the security side, while Goal 4 was scored “behind schedule” on scorecard 3 (May 2013), the goal was back “on track” on subsequent scorecards. The document review and other sources show, as further detailed in Table 9.2 below indicate that five of the seven goal 4 LOAs are on target, one is behind, and insufficient information was gathered to provide a valid judgment about one. As already explained in the CSQ2 chapter, recent progress on goal 4 is the product of increased bilateral coordination, planning, and teamwork that has led to operational progress and concrete results in terms of scorecard milestones. Progress on goal 4 is commendable as this goal was initiated after the initiation of PFG, and the fact that it took nearly two years (until August 2013) to initiate the central activity of goal 4. Until then, USG and Salvadoran counterparts on Goal 4 had not been to agree on priorities and sequencing of steps, explaining the “behind schedule” score of May 2013. Partnership for Growth El Salvador Mid-Term Evaluation Report Since August 2013, proactive stands by DOJ OPDAT (a resident legal advisor placed in El Salvador since March 2012) and ICITAP (Program Manager placed in El Salvador since June 2013), broke through resistance on both sides and led to a constructive partnership with the vice-minister of transportation and the creation of the Task Force. Since then, counterpart agencies have met on the second Wednesday of every month to discuss cases of investigation, criminal organizations, and corruption within the public transportation system. The evaluation team interviewed representatives the GOES and USG agencies active under this goal, and all expressed satisfaction with the operational progress made, as reflected by the “on track” score on the May 2014 scorecard. Goal 4 is now often cited as a best practice example of planned interagency cooperation. Contrary to Goal 4, Security Goal 1&2 have always scored “on track,” which is commendable given that this goal has gone through various implementation challenges. Most LOAs were at risk of coming to a standstill in 2012-2013 when USAID had to award a new 5-year implementation contract. The fact that the contract was eventually awarded in 2014 to the same entity that had implemented the prior contract certainly contributed to continuity and diminished the risk of LOA standstills. At mid-term the evaluation team found that six of the ten LOAs for goals 1&2 were on target (as indicated in Table 9.4 below), three were partly on target, and the remaining one was too vaguely worded in the JCAP to be assessed. GOES implementers interviewed during the site visits all considered their LOAs on target; further all USAID implemented projects related to this goal are on target. Security Goal 11 has similar findings to goal 4, with the USAID-led LOA implementation being on target. According to the interviews, the removal and replacement of the initial principal GOES counterpart official was an obstacle, but as goal 11 has a wide range of strong counterparts (in particular mayors), program progress was not seriously at risk. This goal has a 5-year M&E implementation plan which was made available to the evaluation team. The plan fully accounts for PFG objectives, developed indicators for each of the LOA (among others), and includes logical relation between LOA-level indicators and goallevel progress tracking. Based on the document review, the team found that nine of the eleven LOAs under this goal are on target. Insufficient information was gathered to provide a valid score for the remaining two. Currently, the only security constraint goal that is behind schedule on the May 2014 scorecard is Goal 12. The team’s document review confirmed this, as it indicated that two of the five LOAs were on target, two others were partially on target, and for one, insufficient evidence was gathered to arrive at a valid score. Further details on findings 2 regarding the Tradables Constraint LOAs’ status based on scorecard and other M&E documentation review. As indicated in scorecard findings, as well as the evaluation team’s review, the LOAs for Tradables Goals 3 and 6 have been successful in achieving their individual targets. Goal 3 was successful in improving workforce training and labor market information systems. The individual interventions in Goal 3 are well established and efforts to improve human capital and its utilization in El Salvador, as reflected in USAID project M&E documents, are working as expected. Similarly, for Goal 6, although the methodologies for technology transfer are less well established worldwide, the LOA M&E documents indicate that the individual LOAs are proceeding as expected. The unknown in the case of Goal 6 is whether firms will be able to implement successfully the technical assistance provided, given the overall investment climate. The M&E documentation does not yet provide clear indicators of this. Partnership for Growth El Salvador Mid-Term Evaluation Report Tradables Goal 5 is the only goal that has been ‘behind schedule’, a status which has been consistent since the inception of PFG. At mid-term, after reviewing documents and additional information gathered from interviews and site visits, the team determined that at most 5 of the 13 LOAs are on target, while the others were suffering delays/facing other obstacles, or insufficient information was gathered to arrive at a valid score (see table 9.4 below). Tradables Goal 5 is the most troubled of the tradables goals, and arguably, the most important. The measurements that make up the goal 5 goal-level indicators are, for the most part, measures that are firmly within the control of GOES itself. However, as the scorecard rankings show, El Salvador has made essentially no progress in improving its investment policy climate since PFG began (as measured by the goal-level indicators of Goal 5). No matter how much progress El Salvador makes in improving infrastructure, human capital, and technology (goals 2, 3, and 6), it is unlikely to achieve significant improvement in its production of tradables unless significant progress is made in improving the investment climate. The effects of the six goals on the tradables constraint are not separate and additive. If the investment climate is not significantly improved, trainees under goal 3 will not find jobs, infrastructure built under goal 2 will not have users, and technology introduced under goal 6 will not be utilized. Further, from the evaluation team’s review, the challenges faced with goal 5 can be viewed as consisting of two parts: policy reform (e.g., the “doing business indicators”) and investment promotion. LOAs 5.5, 5.6, 5.10, 5.11, and 5.12 relate to policy reforms or institutional development designed to improve the investment climate (in the case of 5.10, at the municipal level). LOAs 5.1 – 5.4, 5.7, 5.8, 5. and 5.13 relate specifically to investment promotion (see table 9.4 for more details). For the second part to be successful, the first must be effective. It is unlikely that Goal 5 will succeed in promoting foreign investment unless the required policy and institutional reforms are in place. As the table for Goal 5 clearly indicates, these reforms are not in place. Overall, LOA activities, particularly those on the tradables side, are generally on track, as the scorecards indicate, as demonstrated by the in depth review of M&E and other project documentation, reviewed for the selected goals and detailed in Table 9.3 for the security constraint and Table 9.4 for the tradables constraint. When additional M&E data beyond the scorecards were not available, the evaluation team reviewed rankings only based on the score cards and supporting documentation and the results of interviews. M&E data from implementers, as well as interviews with Goal Leads and implementers indicate that most of the selected interventions are on target. 38 Finding 3: No goal-level implementation teams and limited work plans as described in the M&E Addendum to the JCAP were identified. The evaluation team found that for the 7 selected tradables and security goals, goal-level implementation teams as described in the M&E Addendum had not be developed, although a variety of implementation teams do exist at LOA level within some of these goals. As a result, there is less uniformity in the M&E models used to track goal progress than one might expect based on the guidelines provided in the M&E addendum. 38 It is important to note that some LOAs have only recently begun implementation; others have completed their life cycle. Partnership for Growth El Salvador Mid-Term Evaluation Report Strictly speaking, goal level-work plans should have been formulated by the implementation teams. Given that the evaluation team found (as discussed previously) that no implementation teams were formed, logically, no goal-level work plans (as promised in the M&E Addendum) could have been created. However, the evaluation team did find that work plans exist for LOAs that are led by USAID implementers (as this is a contractual requirement), as indicated in the Country-Specific 2 chapter. The existing work plans did not always clearly and separately define PFG indicators and how to measure them. Partnership for Growth El Salvador Mid-Term Evaluation Report Table 9.2: Status of Selected Crime and Insecurity Constraint LOAs at Mid-Term (Selected Goals Only) GOAL 1 & 2 LOAs GOES LOA1: Implement the necessary changes in practices, policies, regulations, and applicable laws as identified throughout the project. GOES LOA 2: Identify and make available appropriate staff to be trained. GOES LOA 3: Emphasize continuous vetting as a fundamental principle of the Security Plan of the Security Cabinet of the Government of El Salvador. GOES LOA 4: Fully participate in programs to support the justice sector. GOES LOA 5: Fully participate in analysis of the current codes. USG LOA 1: Provide technical assistance to improve the management and investigative capacity of the Attorney General, Public Defender, and National Civilian Police, forensic services, judges, and court personnel. USG LOA 2: Support the development of effective case management models; improve police/prosecutor coordination; provide better and more equitable access to justice; re-engineer processes and change management procedures in the common crime unit; establish career paths and leadership development within the Judicial sector; USG LOA 3: Provide assistance to promote increased coordination between justice sector actors and institutions; improve management and administration of justice sector institutions; and increase effectiveness of criminal investigation. USG LOA 4: Provide technical support in the areas of the pre-trial elements listed above as well as pleabargaining (agreements with the public prosecutor’s office). USG LOA 5: Provide technical assistance to conduct an in-depth code analysis, draft legislative fixes, and develop strategies to enhance criminal justice efficiency and effectiveness relating to criminal procedure, an evidence code, and internationally accepted law enforcement tools; implement such legislation, procedures GOAL 4 LOAs GOES LOA 1: Maintain a permanent exchange regarding the employment status of USG-vetted or USGtrained and advised personnel in accordance with Salvadoran procedures and regulations. GOES LOA 2: Improve supervision and control in accordance with the system of indicators of the Security Cabinet. GOES LOA 3: In collaboration with the USG, review the current regulations of the public transportation sector and identify areas to improve transparency and accountability. GOES LOA 4: Implement necessary changes in practices and regulations and encourage the approval of the Partnership for Growth El Salvador Mid-Term Evaluation Report LOA (intervention) on Target YES (Scorecards) PARTLY (Site visit showed that GOES is only providing 50 percent of the agreed upon prosecutors for the Rapid Solution Units.) N/A PARTLY (Interviews with GOES and USG officials and implementers; site visits) PARTLY (Interviews with GOES and USG officials and implementers; site visits) YES (Scorecards; Checci reporting) YES (Checci Quarterly Reporting; scorecards) YES (Checci Quarterly Reporting; scorecards) YES (Checci Quarterly Reporting; scorecards) YES (Checci Quarterly Reporting; scorecards) LOA (intervention) on Target YES N/A YES (Scorecards) Behind Schedule 72 GOAL 1 & 2 LOAs GOES LOA1: Implement the necessary changes in practices, policies, regulations, and applicable laws as identified throughout the project. GOES LOA 2: Identify and make available appropriate staff to be trained. GOES LOA 3: Emphasize continuous vetting as a fundamental principle of the Security Plan of the Security Cabinet of the Government of El Salvador. necessary legislation. GOES LOA 5: Provide personnel, equipment, and facilities to support this program. USG LOA 1: Provide technical assistance, training, and mentorship for the vetted units to combat crimes involving public transit and facilitate cooperation between GOES and the private sector. USG LOA 2: Provide technical assistance to help increase the transparency and accountability of the public transport system. GOAL 11 LOAs GOES LOA 1: Identify dedicated staff to develop and support municipal crime prevention councils. GOES LOA 2: Identify dedicated funding to implement crime prevention plans at municipal level. GOES LOA 3: Promote the decentralization of authority and responsibility for preventing crime to the municipalities. GOES LOA 4: Improve coordination among line ministries with a stake, resources, and mandate for addressing key risk factors (lack of access to education, and employment opportunities). GOES LOA 5: Strengthen the cooperation between the government security organizations and communities. GOES LOA 6: Implement vocational training for at-risk youth. USG LOA 1: Provide technical assistance to promote and facilitate components 2 and 5 of the strategy. USG LOA 2: Explore the provision of loan guarantees to facilitate financing of public goods at the municipal level (i.e. sanitation, infrastructure) USG LOA 3: Region-wide study of the issue of violence and security as it impacts community groups in the hemisphere to bring the community-level experience from the region to bear in El Salvador. USG LOA 2: Explore ways to work with U.S. state/local institutions, the Governments of Mexico, Colombia, and U.S. interagency actors. USG LOA 3: Provide technical support in parole systems and prisoner classification systems. GOAL 12 LOAs Partnership for Growth El Salvador Mid-Term Evaluation Report LOA (intervention) on Target YES (Scorecards) PARTLY (Site visit showed that GOES is only providing 50 percent of the agreed upon prosecutors for the Rapid Solution Units.) N/A (see Scorecard May 2014). YES (Scorecard May 2014) YES (DOJ internal reporting; scorecards) YES (quarterly Casals reports; scorecards) LOA (intervention) on Target YES (interviews in El Salvador performed by Optimal) YES (interviews in El Salvador performed by Optimal) YES (Scorecards 1-5; interviews in El Salvador) YES (Scorecard May 2014; interviews by Optimal in the field) YES (Scorecard May 2014; field interviews) YES (Scorecards) YES (Scorecard May 2014; Creative Associates’ quarterly reporting) YES (Scorecard May 2014; Creative Associates’ quarterly reporting) Unknown N/A. YES (Scorecard May 2014) LOA (intervention) on Target 73 GOAL 1 & 2 LOAs GOES LOA1: Implement the necessary changes in practices, policies, regulations, and applicable laws as identified throughout the project. GOES LOA 2: Identify and make available appropriate staff to be trained. GOES LOA 3: Emphasize continuous vetting as a fundamental principle of the Security Plan of the Security Cabinet of the Government of El Salvador. GOES LOA 1: Improve the prisoner classification system. GOES LOA 2: Build a new prison facility and three prison farms, and implement an aggressive reintegration program. USG LOA 1: Provide technical assistance in the management of prisons/corrections, including mentoring and limited training in order to develop a prison/correction officer train-the-trainer program, and develop and implement prison classification system. USG LOA 2: Explore ways to work with U.S. state/local institutions and the governments of Mexico, Colombia, and U.S. interagency actors. USG LOA 3: Provide technical support in parole systems and prisoner classification systems. LOA (intervention) on Target YES (Scorecards) PARTLY (Site visit showed that GOES is only providing 50 percent of the agreed upon prosecutors for the Rapid Solution Units.) N/A PARTLY (Scorecard May 2014) PARTLY (Scorecard May 2014) YES (Scorecard May 2014; INL internal reporting) N/A. YES (Scorecard May 2014) Table 9.3: Status of Selected Low Productivity and Tradables Constraint LOAs at Mid-Term (Selected Goals Only) GOAL 3 LOAs LOA (intervention) on Target GOES LOA 3.1: Develop an education and employment plan for youth and women Unclear, Scorecard four states that the Ministry did not formally launch the plan, and scorecard five is silent. GOES LOA 3.2: Establish training programs in English and Information and Communication Technology (ICT) for employed population and for youth 50 students are enrolled under the SUPERATE program for IT and English skills. Seven alliances were signed this period, including a broad alliance with three firms anchored in the airport and 17 surrounding municipalities (Los Nonualcos), as well as another one in the microfinance sector with three institutions to train youth for employment. GOES LOA 3.3: Commit to making transformational reforms necessary to improve the quality of El Salvador's educational system Unclear GOES LOA 3.4: Create a Talent Network of Salvadorans living abroad USAID reports it is having conversations with the recently established El Salvador Global network composed of 21 talented Salvadorans living abroad. Partnership for Growth El Salvador Mid-Term Evaluation Report 74 GOAL 3 LOAs LOA (intervention) on Target GOES LOA 3.5: Support youth insertion in the labor market The USG provided general employment skills training for 631 at-risk youths, and 62 percent have found employment to date. In addition, 12 vocational orientation workshops were held to help 1,401 disadvantaged high school students understand specific job and training opportunities. 150 students started their first year of the English Access Micro-scholarship program, and 190 continued through the second year of the program. GOES LOA 3.6: Improve basic computer training programs for entry-level positions and for employees 50 students are enrolled under the SUPERATE program for IT and English skills. Seven alliances were signed this period, including a broad alliance with three firms anchored in the airport and 17 surrounding municipalities (Los Nonualcos), as well as another one in the microfinance sector with three institutions to train youth for employment. GOES LOA 3.7: Improve English for the workplace programs 50 students are enrolled under the SUPERATE program for IT and English skills. GOES LOA 3.8: Strengthen and enhance productive diversification Oct. 2012—Science Fair on renewable energy and robotics; Feb. 2013—Conference on through scientific and technological diversification Climate Change in El Salvador; Feb. 2013—Earth Science workshops; March 2013—Women in Science workshops GOES LOA 3.9: Focus on technological development in priority tradables sectors The USG provided general employment skills training for 631 at-risk youth, and 62 percent found employment to date. In addition, 12 vocational orientation workshops were held to help 1,401 disadvantaged high school students understand specific job and training opportunities. 150 students started their first year of an English Access Micro-scholarship program, and 190 continued through the second year of the program. GOES LOA 3.10: Design and create a program for the transfer and assimilation of ICT INSAFORP produced a study on future professional training in ICT, and trained 3,642 participants GOES LOA 3.11: Strengthen INSAFORP Technical assistance provided GOES LOA 3.12: Improve capacity to collect, analyze, use, and disseminate labor market data Technical assistance has been provided to the Ministry of Labor for preparing an operations manual, identify technical qualifications for hiring new staff, and for developing bulletins. GOES LOA 3.13: Use better data and analysis to project labor market demand Technical assistance has been provided to the Ministry of Labor on preparing an operations manual, desired technical qualifications for hiring of new staff, and for developing bulletins GOES LOA 3.14: Develop youth scholarship program 25 students given scholarships in most recent 6-month period USG LOA 3.15: Support teacher training in English Four modules for the National English for Work Program have been designed between USG and INSAFORP. 151 English teachers were trained through different modalities, including 21 trained in Costa Rica. Partnership for Growth El Salvador Mid-Term Evaluation Report 75 GOAL 3 LOAs LOA (intervention) on Target USG LOA 3.16: Provide technical support to work skills The USG provided general employment skills training for 631 at-risk youth, and 62 percent have found employment to date. In addition, 12 vocational orientation workshops were held to help 1,401 disadvantaged high school students understand specific job and training opportunities. 150 students started their first year of the English Access Micro-scholarship program, and 190 continued through the second year of the program USG LOA 3.17: Develop alliances, provide technical skills and language training 50 students are enrolled under the SUPERATE program for IT and English skills. Seven alliances were signed this period including a broad alliance with three firms anchored in the airport and 17 surrounding municipalities (Los Nonualcos), as well as another one in the microfinance sector with three institutions to train youth for employment. USG LOA 3.18: Provide technical assistance to improve labor market information systems Technical assistance has been provided to the Ministry of Labor with preparing an operations manual, identify technical qualifications for hiring of new staff, and for developing bulletins USG LOA 3.19: Help GOES build capacity to conduct surveys to match employer needs and skills Technical assistance has been provided to the Ministry of Labor with preparing an operations manual, desired technical qualifications for hiring of new staff, and for developing bulletins USG LOA 3.20: Help GOES improve services for job seekers The vocational orientation website elijomifuturo.com for youths was developed in Nov., and 8,712 different young people have used it. The USG and Ministry of Labor implemented four job fairs in San Miguel, Ciudad Arce, Ilopango, and Lourdes in which 1,631 job seekers, principally youths, and 110 businesses participated. At least 438 youths found jobs. USG LOA 3.21: Improve tertiary training and higher education quality He USAID Higher Education project was approved and procurement process has started. USG LOA 3.22: Provide technical assistance to focus on innovation INSAFORP produced a study on future professional training in ICT and trained 3,642 participants and technological development in priority tradables sectors USG LOA 3.23: Support development of a youth scholarship program 25 students were given scholarships in most recent 6-month period USG LOA 3.24: Support implementation of talent network of Salvadorans abroad USAID is having conversations with the recently established El Salvador Global network composed of 21 talented Salvadorans living abroad. USG LOA 3.25: Launch government, business, labor dialogue on productivity and competitiveness Not implemented GOAL 5 LOAs LOA (intervention) on Target GOES LOA 5.1: Strengthen the institutional capacity to conduct market studies Unclear Partnership for Growth El Salvador Mid-Term Evaluation Report 76 GOAL 3 LOAs LOA (intervention) on Target GOES LOA 5.2: Encourage investments from Salvadorans residing According to interviewees, the GOES and the IDB are developing a project to increase investments of Salvadorans living abroad in the U.S. GOES LOA 5.3: Strengthen the Ministry of Economy and PROESA Legislation to strengthen PROESA was passed by the Legislative Assembly in April. GOES LOA 5.4: Implement the Integrated Investor Attention System Unclear GOES LOA 5.5: Develop and implement a strategy to remove barriers to FDI Study carried out, but recommendations not implemented according to interviewees GOES LOA 5.6: Improve plant inspection by Ministry of Agriculture Unclear GOES LOA 5.7: Strengthen PROESA See LOA 5.3 GOES LOA 5.8: Design and implement a country image strategy Development of an image strategy has begun USG LOA 5.9: Strengthen PROESA Delays in naming permanent director of PROESA and clarifying its legal status have delayed results USG LOA 5.10: Improve local business climate through the Municipal Competitiveness Project A total of 40 municipal ones-stop windows operating in 33 municipalities as of November 2013. Also, IFC has launched a municipal-level Doing Business Assessment. USG LOA 5.11: Partner with IFC to improve Doing Business Indicators through the Growth Council The IFC continues to work with the Growth Council to identify reforms to improve the business climate USG LOA 5.12: Undertake a review of laws and processes in El Salvador that facilitate investment in order to create an incentive system tailored to different types of investors in key sectors Study carried out, but recommendations not implemented according to interviewees USG LOA 5.13: Support GOES in the creation of a country image strategy Delayed GOAL 6 LOAs LOA (intervention) on Target GOES LOA 6.1: Strengthen the technical capacity of PROESA’s legal status. Strengthening of PROESA has been delayed by lack of a permanent director and delays in clarifying its legal status. However, substantial progress has been made in training and technical assistance through Chemonics. Partnership for Growth El Salvador Mid-Term Evaluation Report 77 GOAL 3 LOAs LOA (intervention) on Target GOES LOA 6.2: Strengthen the Ministry of Economy and PROESA MINEC's Productive Development Fund reported grants of $1.9 million to 111 SMEs to continue developing the production and export strategy which generating 90 jobs in the most recent 6-month scorecard. MINEC approved an additional $6.4 improves innovation and quality million in funding to institutions providing business development services in latest six-month reporting period GOES LOA 6.3: Work with USG to facilitate the export process USAID project has provided assistance to firms to guide them through the export process. Little progress has been made in streamlining the process GOES LOA 6.4: Create an integrated system to serve SMEs Firms assisted by the project increased exports by $7.96 million, most from SMEs. seeking to export GOES LOA 6.5: Strengthen and increase diversification through MINEC and private sector have agreed on six priority sectors. innovation USG LOA 6.6: Provide TA to the Ministry of Economy and Firms assisted by the project increased exports by $7.96 million, most from SMEs. PROESA to serve SMEs seeking to export USG LOA 6.7: Support the establishment of small business 12 Small Business Development Centers established. The 1,969 firms assisted to date have development centers throughout the country produced sales of $11.3 million, a 143 percent increased, and generated 3,430 jobs USG LOA 6.8: Increase the internationalization of Salvadoran firms The USG provided international business development assistance to 483 businesses. These firms increased domestic sales and exports by $6.3 million, generating 760 jobs. through guided processes or mentoring USG LOA 6.9: Explore ways to facilitate the speed of exports from 29 SMEs received TA in food safety and best agricultural practices as of the May 2013 El Salvador to the U.S. and reduce the rejection rate at the U.S. scorecard border USG LOA 6.10: Support the increased participation of Salvadorans A $255,000 grant was approved by USG for the FUPEC Foundation to work with Salvadoran abroad in the Salvadoran economy hometown associations in the U.S., extending assistance to microbusinesses and grassroots associations in El Salvador. USG LOA 6.11: Promote opportunities for bi-national business alliances in the tradables sector Partnership for Growth El Salvador Mid-Term Evaluation Report None as of yet 78 9.2 Conclusions and Lessons Learned As the first Partnership for Growth agreement, the El Salvador PFG was clearly subject to challenges in both design and implementation. In some cases, these problems were recognized early on and attempts were made to rectify them. The Monitoring and Evaluation Addendum is an example of such an attempt. In other cases, such as the absence of any consistent process to evaluate whether goal-level indicators are being met, these problems continue to be an obstacle to successful implementation. While it is too late to reinitiate the El Salvador PFG, one clear conclusion for any future PFG or PFGlike program is that discussions about selection of goals and LOAs must include results-level indicators and targets from the beginning. This way, future programs can establish realistic constraint and goallevel targets that can be achieved within the program's timeframe. A second conclusion is that a periodic review process, such as the El Salvador scorecard process, is not addressing in full goal-level and constraint-level indicators, as well as any discrepancies between the sum of LOA indicators and goal-level indicators. This is analogous to the project review process, where input indicators and results indicators are compared and any discrepancy is addressed and corrected. The evaluation team found that implementation teams and work plans were not developed for all goals. Therefore, LOA-level progress tracking (except for USAID-led goals) is being done in a less rigorous manner than promised in the M&E Addendum. Given the former, and the findings under Country-Specific Question 2, the team concludes that M&E practices mainly reflect lead agencies’ preestablished M&E procedures rather than PFG-specific procedures, except for in the scorecard process. 9.3 Recommendations and Course Corrections Leverage the change in GOES administration at midterm to conduct a PFG goal-by-goal review. Since the newly initiated GOES Administration was not a party to the current PFG JCAP, the evaluation team recommends that the USG El Salvador team should use the opportunity of having a ‘new’ partner (given that many GOES counterparts will likely change due to the new government) to undertake a thorough review of the constraints, goals, and LOAs with a view to consolidating them, improving M&E systems, with well specified results-level indicators for LOAs that lack them, and agree on a system of more frequent (beyond semi-annual review) periodic review (preferably quarterly) of goal and LOA-level indicators with a view to making needed course corrections as applicable at mid-term which would ensure that the PFG initiative as whole can achieve its stated outcomes by endline. This review could be facilitated by the recommended bilateral PFG management team. Establish implementation teams and require them to develop LOA work plans. The evaluation team recommends that PFG establishes implementation teams per goal, as foreseen by the M&E addendum, and provide a protocol for their tasks and responsibilities. Such teams would include Goal Leads and main LOA implementers from both governments, and should meet e.g. monthly, to monitor Partnership for Growth El Salvador Mid-Term Evaluation Report 79 progress and make necessary course corrections per LOA to guarantee being on target. These goal level implementation teams would be guided and supported by the PFG bilateral management team. Similarly, goal level work plans for each goal should be developed to guide the remaining 2.5 years of implementation. These work plans should be dedicated to specifics PFG indicators per LOA. The work plans should also follow the overall theory of change developed (recommended in previous chapters) to include definitions of LOA specific indicators and how they feed into goal-level indicators which foster a fact-based inclusion into the scorecards, promoting more rigor in the findings that are established for PFG’s progress concerning project implementation. Partnership for Growth El Salvador Mid-Term Evaluation Report 80 10. OVERALL SUMMARY AND CONCLUSION The preceding sections of this report address the specific questions posed by the scope of work. This brief summary provides a more general and “impressionistic” assessment of the El Salvador Partnership for Growth Program. While the preceding sections contain a number of points which can be taken as critical of PFG, and made many suggestions for improvement, the evaluation team's overall conclusion is that the PFG initiative in El Salvador has been a remarkably successful development activity. Partnership: The most notably fact about PFG is that it is clearly a true partnership. Although a number of points of conflict have been noted in this evaluation and points of conflict and disagreement have been documented, the fact remains that at each stage, the USG and the GOES have each been represented at the table as equal participants and points of conflict have been faced and addressed. It has to be said that this is novel in development practice. The Whole of Government Approach: The degree to which the various USG agencies have worked together in the design and implementation of PFG is both notable and unprecedented in the experience of the evaluators. This is particularly notable in the security area where the USG agencies involved do not typically work closely together and where inter-agency conflict and suspicion are more typical that what the team has observed in El Salvador. Quantitative and Objective Monitoring and Evaluation: Although the evaluation has noted the shortcomings in the M&E process, and suggested ways in which it should be improved, most, of the LOAs are subject, directly or indirectly, to quantitative and objectively verifiable indicators. The team has suggested ways in which these monitoring standards can be applied to the remaining LOAs, and more systematically overall. Breaking New Ground on Security: While the evaluation has noted some shortcomings in the design and articulation of the Security Constraint, the team notes that the comprehensive approach chosen to improve security sector governance in El Salvador represents a promising way forward. The team hopes that its suggestions will be taken in the spirit of positive suggestions based on the team's experience in other countries. Fine-tuning of an Innovative Process: Overall, the evaluation team hopes that its recommendations will be taken in the spirit of constructive suggestions and enable the Mission and the GOES to improve this innovative and somewhat untested process. Partnership for Growth El Salvador Mid-Term Evaluation Report 81 11. ADDENDUM – CASE STUDIES OF 7 SELECTED GOALS As has been described in previous sections of the PFG El Salvador Mid-Term Evaluation Final Report, as part of the evaluation planning process, the team conducted a goal selection process to identify PFG goals that represent the larger PFG initiative for an in depth study. The goal selection process is found in a separate Evaluability and Goal Selection Report. For the low productivity in tradables constraint, goals 3, 5, and 6 were chosen, mainly based on criteria of subconstraint and WGA/ interagency representativeness. Under the crime and insecurity constraint, goals 1-2, 4, 11, and 12 were chosen, similarly based on their subconstraint and WGA/interagency representativeness, but in addition on criteria of representativeness across lead (GOES and USG) agencies, participation in multi-goal programs, diversity of implementing partners, and preference for PFG (“non-legacy”) programs. This Addendum provides the detailed findings for all the seven case studies. 11.1 Crime and Insecurity Constraint (4 Case Studies) Security Goals 1&2 Sub-Constraint: Institutional Strengthening Institutional Strengthening, Strengthen Justice Sector Institutions and Improve Criminal Justice Procedures Goal objective (as stated in JCAP): “Professionalize justice sector institutions and improve criminal justice practices and procedures to make them more effective in combating crime and insecurity in El Salvador, as well as enhance the public perception of these government institutions.” USG lead agency: USAID (D&G; Justice Sector Strengthening Program JSSP); co-lead: INL USG implementer: Checci and Company Consulting, Inc. GOES lead agency: Technical Executive Unit UTE of the Justice Sector and Justice Sector Coordinating Committee; co-lead: PNC. GOES implementers: Various GOES agencies: PNC, Ministry of Justice and Public Security, UTE, Attorney General’s Office, Public Defender’s Office, Salvadorian Institute for Women’s Development (ISDEMU), Forensic Medical Institute of the Supreme Court, National Judicial Council, judges, and court personnel. LOAs summary (for details, see JCAP): the LOAs’ focus is, among others, to enhance, modernize and professionalize specific institutional units in the Justice sector through training, changes in regulations and applicable laws, and strengthening of designated disciplinary units and the Human Rights Ombudsman’s Office. The GOES and the USG each committed to five LOAs, but the USG’s wording is more precise and detailed in terms of specific contributions. M&E Indicators: (i) Public satisfaction with the performance of justice and security institutions; (ii) Number of criminal cases resolved through conviction or alternative dispute processes. Scorecards: May 2012: on track; Nov. 2012: on track; May 2013: on track; Nov. 2013: on track; May 2013: on track. Partnership for Growth El Salvador Mid-Term Evaluation Report 82 Summary on Implementation USG. Among the 14 security goals, only Goals 1&2 (initially Goals 1 and 2) as well as Goal 9 are supported on the USG side by USAID’s Justice Sector Strengthening Program (JSSP), a USAID Rule of Law (ROL) program. From 2008 to 2018, USAID has contracted implementation of its JSSP under two subsequent 5-year contracts with the American firm Checci. PFG was created and the JCAP signed while the first JSSP contract (2008-2012) was entering its last year; also, objectives and LOAs of Goals 1&2, as well as Goal 9 in the JCAP were formulated in line with the already existing JSSP objectives and ongoing activities. Thus, initially, the initiation of PFG did not lead to much change in terms of operational activities, reporting, or monitoring and evaluations (M&E) for USAID’s JSSP or Checci. Indicators already existing were maintained, but renamed as “PFG Goal 1 indicator” or “PFG Goal 2 indicator.” INL was put in charge of one LOA, but without coordination with Checci. The SOW for the second JSSP implementation contract (2013-2018) is to a large extent a follow-up to the prior one building on results, impacts, and lessons learned. The contract is still a JSSP contract (a contract to implement USAID’s JSSP program). However, part of the JSSP program coincides with the objectives and LOAs of Goals 1&2 and Goal 9. Also, activities and indicators are systematically referenced in the JSSP work plan and M&E plan, noting their relation to the respective goals. In addition, coordination between USAID and INL was enhanced for this LOA. GOES. GOES leadership for Goals 1&2 was initially split between the PNC for Goal 1 and the UTE for Goal 2. Attributing leadership responsibility for Goal 1 to the PNC seems to reflect the frequently mentioned “rushed” character of the JCAP because the PNC had in reality very limited formal responsibilities for the LOAs under Goal 1. On the USG side, Goal 1&2 was led through one and the same agency (USAID’s JSSP program). In 2012, GOES and USG officials decided to fold Goals 1 and 2 into a single goal, 1&2, with UTE as the GOES lead-agency. This change is reflected in the scorecards; starting with scorecard 3 (November – May 2013), the goals are scored as one. At mid-term, these were the only goals of the ES PFG that had been folded together. Documents reviewed: Documents reviewed for this case study included the October 2013 JSSP Annual Report, the JSSP Fact Sheet, August 2013 Performance Monitoring and Evaluation Plan, the Performance Monitoring Plan, a spreadsheet of the JSSP Work Plan, the JSSP Work Plan Narrative, and five scorecards. Supplemental documents included a report of the number of criminal cases resolved through conviction or alternative process (See Annex 5 for the reference list for selected goals). Observational field study: Goals 1&2 include a wide range of interventions and activities. Given time limitations in the field, Optimal (in coordination with GOES and USG goal leads) - selected three of the many activities for site visits and on-site interviewing: 39 • The Rapid Solutions Unit (Unidad de Soluciones Tempranas – UST) in the town of Mejicanos. • The Victim’s Assistance Center (Unidad Institucional de Atención a Mujeres en Situación de Violencia) at the police station in the town of Apopa. • The Judicial School (Escuela Judicial), part of the National Judiciary School (Consejo Nacional de Judicatura), in San Salvador. 39 In particular with the outgoing and incoming Checci COP, various members of the Checci team, the director of the technical planning justice sector (UTE), the former PNC lead of Goal 2, officials of the Rape Center, and officials of the National Judiciary School. Partnership for Growth El Salvador Mid-Term Evaluation Report 83 The team also interviewed active and former USAID and GOES goal leads and co-goal leads; as well as the former and active Checci COP and various other members of the Checci team. Findings based on third-party data: Two M&E indicators defined to measure performance on Goal 1&2 are (i) Public satisfaction with the performance of justice and security institutions; (ii) Number of criminal cases resolved through conviction or alternative dispute processes. LAPOP’s 2012 data indicates that Salvadorans expressed a somewhat higher degree of trust in the National Police (PNC) and their justice system than did other Central Americans (See figure 11.1 and table 11.1 below). Furthermore, the LAPOP’s data suggest that between 2010 (before PFG started in fall 2011) and 2012 (Year 1 of PFG), Salvadoran trust in PNC had increased from 40 to 49 percent (a 22 percent increase), a figure that is about twice as high as public trust in the police in Guatemala and Honduras (which saw a dramatic decline in trust), as is seen on the same figure 11.1 and table 11.1 below. In Central America, only Guatemala and Panama scored higher levels of trust in their police than El Salvador. Nevertheless, the evaluation team considers that attributing the lack of increase of public trust in the Justice System, as well as the 8 percent increase of public trust in the police to Goal 1&2 is somewhat problematic, since PFG implementation was only in Year 1. On the other hand, USAID’s pre-existing Justice Sector Strengthening Program (JSSP, 2008-2013) was folded into Goal 1&2, and the JSSP program may certainly have contributed to these results. However, external factors may have come into play as well, and stress the difficulties to attribute changes in justice and security sector indicators to specific PFG interventions. In particular, the significant decrease in homicides and crime during 2012 may have positively influenced public trust in the police, even though the decrease was not necessarily the product of improved policing, but e.g. also of the gang truce celebrated at that moment. PNC itself mainly credited the increase in public trust in its agency to implementation of Community Policing, which has been in effect since 2010 with support of USAID, and with support of PFG (Goals 9 and 11) since November 2011. Partnership for Growth El Salvador Mid-Term Evaluation Report 84 Figure 11.1: Public satisfaction with the performance of judicial system and National Police in El Salvador and other countries in Central America in 2010 and 2012 National Police Judicial System 0% El Salvador Belize Costa Rica Guatemala Honduras Mexico Nicaragua Panama El Salvador Belize Costa Rica Guatemala Honduras Mexico Nicaragua Panama 10% 20% 50% 60% 42.1% 42.1% 37.6% 53.2% 44.6% 33.6% 30.0% 30.7% 45.0% 22.1% 40.4% 39.6% 32.5% 47.2% 46.1% 34.7% 40.2% 48.7% 30.2% 44.5% 37.8% 34.8% 21.4% 23.2% 47.4% 18.9% 25.8% 30.2% 47.6% 56.0% 52.0% 58.2% 2010 30% 40% 2012 Partnership for Growth El Salvador Mid-Term Evaluation Report 85 Table 11.1: Public satisfaction with the performance of the judicial system and the National Police in El Salvador and other countries in Central America in 2010 and 2012 40 El Salvador Belize Costa Rica Guatemala Judicial System Honduras Mexico Nicaragua Panama El Salvador Belize Costa Rica Guatemala National Police Honduras Mexico Nicaragua Panama Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI Percent 95% CI 2010 42.1% (38.6% - 45.7%) 37.6% (33.9% - 41.3%) 44.6% (41% - 48.3%) 30.0% (27.3% - 32.8%) 45.0% (42.3% - 47.7%) 40.4% (37.3% - 43.4%) 32.5% (29.3% - 35.7%) 46.1% (40.1% - 52%) 40.2% (37.5% – 43%) 30.2% (26.6% - 33.8%) 37.8% (33.8% - 41.7%) 21.4% (18.7% - 24.1%) 47.4% (43.1% - 51.7%) 25.8% (22.9% - 28.6%) 47.6% (43.8% - 51.4%) 52.0% (43.9% - 60.1%) 2012 42.1% (39.3% - 45%) 53.2% (50.1% - 56.3%) 33.6% (30.7% - 36.5%) 30.7% (27.2% - 34.2%) 22.1% (19.7% - 25.4%) 39.6% (35.6% - 43.6%) 47.2% (44.3% - 50.5%) 34.7% (30.8% - 38%) 48.7% (46.1% - 51.2%) 44.5% (41.7% - 47.2%) 34.8% (32.3% - 37.4%) 23.2% (20.1% - 26.3%) 18.9% (15.9% - 21.4%) 30.2% (26.5% - 33.9%) 56.0% (52.8% - 59.2%) 58.2% (52.4% - 62.9%) External factors – in particular a new rise in homicides and crime since 2012 – seem to be responsible as well for the more recent decrease in public trust in the Justice and Security sector. The Instituto Universitario de Opinión Publica (IUDOP’s) of the Universidad Centroamericana (UCA) reported that both households and enterprises were less satisfied with PNC in 2013 than in 2012 (see figure 11.2 and table 11.2 below): 52 percent in 2012, versus 46 percent in 2013. Likewise, while in 2012, about 39 percent of small businessmen were satisfied with PNC performance, in 2013 this number decreased to 35 percent. This finding seems to be at odds with PNC’s argument that the implementation of Community Policing is, in and by itself leading to an increase of public trust in the PNC, given positive findings in selected towns or wards where community policing roll-out is already taking place. (USAID conducts assessments in each community municipality on crime statistics and trust in the police before the intervention and every year thereafter.) Attributing increases in public trust in security 40 Source: Perception of Security and Confidence in Public Institutions, Universidad Centroamericana Jose Simeon Cañas (2013) Partnership for Growth El Salvador Mid-Term Evaluation Report 86 and justice agencies to specific PFG activities may imply that when public trust for some reason decreases, that the same activity or agency will now be held accountable for the negative result. It points to the wider challenge of how to best select the quantitative and objectively verifiable information to be used to manage JCAP implementation in order to achieve and measure results (CSQ2) and to know if a goal is on target and creating the necessary outputs to achieve the desired goals (CSQ3). Figure 11.2: Percent of Salvadorans who are satisfied with the performance of the National Police, households vs. microenterprises, 2012 and 2013 41 60% 52.2% 45.9% 50% 39.2% 40% 35.2% 30% 20% 10% 0% 2012 2013 2012 Household 2013 Microenterprise El Salvador Table 11.2: Percent of Salvadorans who are satisfied with the performance of police, households vs. microenterprises, 2012 and 2013 42 Household Enterprise Percent 95% CI Percent 95% CI 2012 52.2% (50.2 - 54.2) 39.2% (35 - 43.4) 2013 45.9% (43.9 - 47.9) 35.2% (31 - 39.3) As for public satisfaction with the performance of the justice administration, Salvadorans did not show an increase in trust between 2010 and 2012 (see figure 11.2 and table 11.2 above): the level remained the same at 42 percent (better than other countries in the region, except Belize and Nicaragua). Since 2012, changes in trust in judges and courts – which we can take as a proxy for the justice administration overall – also did not significantly change. Household satisfaction in this area decreased by 2.7 percent, from 29.3 percent in 2012 to 26.6 percent in 2013 (see figure 11.3 and table 11.3 below). Satisfaction among small businesses remained low and stable (19.4 percent in 2012 and 19.3 percent in 2013). Public trust in the justice administration thus seems 41 42 Source: Perception of Security and Confidence in Public Institutions, Universidad Centroamericana Jose Simeón Cañas (2013) Source: Perception of Security and Confidence in Public Institutions, Universidad Centroamericana José Simeón Cañas (2013) Partnership for Growth El Salvador Mid-Term Evaluation Report 87 to be less impacted by external factors (as the new crime wave) than public trust in the Police. That said, trust is not improving and no positive impact of Goal 1&2 can be observed at mid-term. Figure 11.3: Percent of Salvadorans who are satisfied with the performance of judges and the court, households vs. microenterprises, 2012 and 2013 43 30% 29.3% 26.6% 25% 20% 19.4% 19.3% 2012 2013 15% 10% 5% 0% 2012 2013 Household Microenterprise El Salvador Table 11.3: Percent of Salvadorans who are satisfied with the performance of judges and the court, households vs. microenterprises, 2012 and 2013 44 Household Enterprise Percent 95% CI Percent 95% CI 2012 29.3% (27.5% - 31.2%) 19.4% (16% - 22.8%) 2013 26.6% (24.8% - 28.4%) 19.3% (15.9% - 22.8%) The second M&E indicator for Goal 1&2 is the number of criminal cases resolved through conviction or alternative dispute processes. Although data for criminal cases resolved through alternative dispute processes were not publically available, UN-CTS data between 2010 and 2012 were available for the number of convictions per 100,000 adults (see next figure 11.4, below). In El Salvador, the number of convictions remained practically constant over time, and low compared to other countries in the region: 125 per 100,000 in 2010 (before PFG), and 123 per 100,000 in 2012 (after PFG initiation). While this kind of quantitative and objectively verifiable information is available to manage JCAP implementation in order to achieve and measure results (Country-Specific Question 2), at least in 2012 (year 1 of the PFG), this indicator did not permit to establish if the goal was on target and creating the necessary outputs to achieve the desired goals (Country-Specific Question 3). (Data for 2013 were not yet available). 43 44 Source: Perception of Security and Confidence in Public Institutions, Universidad Centroamericana Jose Simeon Cañas (2013) Source: Perception of Security and Confidence in Public Institutions, Universidad Centroamericana José Simeón Cañas (2013) Partnership for Growth El Salvador Mid-Term Evaluation Report 88 Figure 11.4: Convictions per 100,000 Adults (2010, 2011, 2012) 45 Rate per 100,000 adults 350 308 295 300 238 213 186 250 200 150 250 235 188 197 139 125 125 123 107 100 50 0 El Salvador Costa Rica Honduras 2010 2011 Mexico Nicaragua Panama 2012 Responses to country-specific questions Country-Specific Question 2: Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? The evaluation team found that in addition to the third-party perception study data (see above), quantitative and objectively verifiable information for Goal 1-2 has been systematically collected by Checci, with technical assistance from USAID, under both implementation contracts (2008-2012; 2013-2018), by following procedures laid out in full detail in the M&E Plan of the USAID-Checci contract for the JSSP program. The 2013-2018 M&E plan includes 27 indicators. Of these, four indicators are explicitly referenced in the plan as PFG indicators. Checci and USAID use these indicators to steer JSSP implementation and to make course corrections when necessary. On the GOES side, no additional indicators are systematically gathered for M&E purposes for Goals 1-2. Rather, Goal 1-2 lead agencies, UTE, and the Justice Sector Coordinating Committee receive and use the indicators produced through Checci on behalf of USAID.UTE did create an inter-institutional committee to manage statistics and promote the use of statistics as a fundamental tool for decision making in Year 1 of PFG, but has encountered challenges with implementation. The Justice Sector Coordinating Commission did conduct an impact study in early 2012 on the new Criminal Procedures Code, which identified areas where implementation of the Code should be improved. The Checci-produced indicators also feed into the scorecard process. USAID and UTE representatives (without participation of Checci) define the contents of the scorecard by evaluating progress on the LOAs by 6-month milestones and in terms of the two overall goal specific indicators (see above). Goal 1-2 have always scored “on track” on the scorecards, including on their last one (May 2014). 45 Source: United Nations Office on Drugs and Crime, Convictions per 100,000 adults (aged 18 or older), (2012) Partnership for Growth El Salvador Mid-Term Evaluation Report 89 Country-Specific Question 3: At the midterm, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? The interventions for Goals 1-2 are on-target in terms of the JSSP 2013-2018 work plan, formulated by USAID and its implementer Checci. No other work plan for Goals 1-2 exists than the JSSP 2013-2018 plan and its related M&E Plan. Compared to pre-PFG reporting, Checci considered that M&E requirements under PFG had become more exact, that site visits to monitor project progress by the USAID superior had increased, that the GOES had increased its commitment to comply with agreed upon objectives, activities and goals, and that midlevel personnel at the GOES and USG were doing a better job in leading and coordinating LOAs. Checci also considered that these changes contributed to being on target. However, Checci was not sure if these changes were the result of the creation of PFG only, or if they also related to USG paying increased attention to M&E rigor, and USAID in particular further enhancing its M&E worldwide. To gather further information, the evaluation team also performed observations and further interviewing for three of the Goal 1-2 activities through site visits: • • • Visit of the Rapid Solutions Unit (Unidad de Soluciones Tempranas – UST) in the town of Mejicanos. Since 2010, 16 UST were created with USG support; three more will be created under PFG. Among the direct beneficiaries of UST are the detainees now processed in a timelier manner by the justice administration. Other beneficiaries are UST staff who receives training and other professional development support, and the prison system, as alternative dispute processes should help reduce overcrowding. UST tracks the numbers of beneficiaries, including by gender. Observed weakness: GOES has partly disregarded its commitments under PFG by reducing the assigned prosecutors from 12 at inauguration to six now at the visited UST. The team learned from interviews that similar challenges are experienced at other UST. This critical factor, however, is not an explicit indicator of performance under Goal 1-2, and is not mentioned on the scorecard (e.g. of May 2014). Visit to the Victim’s Assistance Center (Unidad Institucional de Atención a Mujeres en Situación de Violencia) at the police station in the town of Apopa. Created in 2013, this Center is coordinated through the PNC Office of Citizen Complaints, works through formalized protocols, and is creates a support network for the two existing Centers and additional 13 to be created until 2018 under the USAID-Checci contract. Direct beneficiaries are the women and their children attended to at the Victim’s Assistance Center, the staff that receives training, and the PNC, as it improves its services. The Center tracks all its activities, producing objective and verifiable quantitative and qualitative indicators for M&E purposes. Observed strength: As a result of PFG bringing many agencies together, the Victim’s Assistance Center has been able to strengthen its interagency coordination and cooperation with the health, justice, and education services, contributing to its efficiency and sustainability. Visit to the Judicial School (Escuela Judicial), part of the National Judiciary School (Consejo Nacional de Judicatura), in San Salvador. The Judicial School was created in the 1990s as part of USG-supported efforts to strengthen the independence of the justice sector following the signing of the peace accords. Under Goal 1-2, PFG supports concrete activities, but also a diagnosis of the Judicial School’s current situation and a next phase strategic plan. Observed weakness: a lack of awareness among the officials Partnership for Growth El Salvador Mid-Term Evaluation Report 90 of the Judicial School about the other activities under Goal 1-2, somewhat illustrative of the silo like implementation of the many activities included under Goal 1-2. GOES implementers of Goal 1-2 interviewed during the site visits indicated that their LOAs are on target. The evaluation team confirmed this through a document review. Currently (see table 11.4 below), the evaluation team found that six of the 10 LOAs for Goal 1-2 are on-target, three are partly on target and the remaining one is too vaguely worded in the JCAP to be able to judge. GOES implementers interviewed during three site visits considered that their LOAs were on target, and the evaluation team found that performance on each of these three activities were also on target in terms of their USAID contract requirements. The evaluation team found that although the implementers directly involved with the three visited activities considered that their respective activity was on target, and while it’s evident that e.g. the Victim’s Assistance Center is providing better services to victims of gender violence than existed before, and the Rapid Response Unit is solving cases effectively, the implementers were unable to answer the question to what degree their activity was creating the necessary outputs to achieve the desired outcomes at goal level, as they considered that so many more activities are being performed by other agencies under Goal 1-2. A large majority of the officials and implementers involved with Goal 1-2 in general considered that it was too early to evaluate whether their activities would be able to contribute to lowering the three overall indicators (national homicide rate, national security perception, and national impunity rate). However, some suggested that a narrowed-down geographical version of these three indicators, i.e. measuring them for the communities of their interventions (e.g. there where special programs as the Unidad de Soluciones Tempranas (UST) and The Women at risk Assistance Centers were put into place), might increase their usefulness as indicators to measure desired outcomes for Goal 1-2. Conclusions The evaluation team found that USG and, to a lesser extent, GOES, is gathering and using verifiable information under Goal 1-2 to manage JCAP implementation to achieve and measure results (see table 11.4 below). All activities are part of USAID’s JSSP program. As USAID has contracted implementation of this program with Checci, conforming with its SOW, Checci has formulated and is implementing a rigorous M&E plan. As Checci operates the JSSP under a detailed 5-year work plan, no separate bilateral goal-level work plan was produced for Goal 1-2. While this is the case for all the goals led by USAID, the evaluation team finds that it is contrary to the spirit of what was announced in the JCAP M&E addendum. This discrepancy implies that the current implementation team, work plan and the M&E efforts for Goal 1-2 continue to be heavily concentrated on the USAID & Checci side. Overall, the evaluation team found – after document review, interviews, and site visits - that performance of Goal 1-2 is on target. This is also reflected by continuous “on track” scoring on the five scorecards issued until now. However, after analyzing the information currently available for the two goal-level indicators, and listening to Goal 1-2 leads, implementers, some beneficiaries and independent experts, we found that at mid-term it was difficult to say if the activities in place are creating the necessary outputs to achieve the desired outcomes. The team found that external factors (as crime spikes) intervene, that it’s too early to know, and that goal-level and constraint-level indicators are many times considered too general, and therefore too ambitious. Partnership for Growth El Salvador Mid-Term Evaluation Report 91 Course Corrections and Recommendations Course corrections and recommendations listed here focus in particular on findings as related to CountrySpecific Questions 2 and 3. However, the evaluation team also includes some more general recommendations for Goal 1-2, when considered relevant for possible mid-term course corrections. Revise the wording of Goal 1-2 LOAs in the JCAP. GOES leadership in particular, and some on the USG side, considered that a rewrite of Goal 1&2 in the JCAP may permit more precise and systematic LOAs. The evaluation team recommends that the LOA section in the JCAP for the last two years of the PFG is revised, as it will be useful to more clearly define the implementation and monitoring processes in order to attain expected outcomes. Consider adding geographic focus to the goal-level and constraint-level indicators. As part of the JSSP program activities are both geographically focused – although it also includes countrywide components – and by subject manner (i.e. a few specific crimes to reduce impunity are selected) the evaluation team recommends adding geographic focus to some of the indicators or targeted surveys used for Goal 1-2. Create a bilateral goal-level implementation team. Although Checci has formed work teams for specific LOAs, an overall implementation team specific to Goal 1-2 – as required by the JCAP M&E Addendum - was not formed. Such a team would increase inclusiveness, and possibly commitment on the GOES side towards implementation. The team should include representatives of both GOES and USG implementers for each LOA. Create a bilateral goal-level work plan. Goal 1-2 is essentially a segment of USAID’s Justice Sector Strengthening Program. While USG/USAID considers that a work plan exists (the JSSP work plan), the GOES leadership has a different take on the situation: they consider that the existing “work plan” is the SOW for Checci under its contract with USAID and not the Goal 1-2 work plan as required by the JCAP M&E Addendum. A work plan for the last two years of Goal 1-2, written by the lead agencies and implementers from both sides (USG and GOES), would contribute to making Goal 1-2 a less heavily USAID-steered undertaking. Take concrete steps to enhance inclusiveness. The GOES implementers the team spoke to during the site visits were aware of being part of PFG, but did not know specifics. The USG implementer (Checci) had a clear understanding of what PFG implies for Goal 1-2, but not for other goals. Neither the GOES implementers interviewed nor the Checci team members interviewed had ever seen a scorecard. While Checci considers it good standard procedure that USAID represents them in meetings with GOES, they feel they have not been invited to be an explicit part of PFG, except to the extent that PFG has added certain contractual obligations to their own contract and SOW, mainly in terms of M&E and reporting. Not only the implementing firm, but also most GOES officials involved with Goal 1-2, explained that they would favor a more inclusive approach and would like to be more explicitly part of PFG. In addition to creating a goal-level work plan and installing a bilateral goal-level implementation team, simple additional steps could increase inclusiveness: e.g. guarantee systematic distribution of scorecards and other relevant PFG reports and documents among all stakeholders (including at the municipal level) of Goals 1-2; create a Goal 1-2 webpage (as part of a PFG/APC website) where all relevant info for Goal 1-2 can be found; create a yearly or semi-yearly “Goal 1-2 immersion or workshop day,” where all stakeholders involved (including those at municipal level), will be briefed on the Partnership for Growth El Salvador Mid-Term Evaluation Report 92 progress of Goal 1-2; create a monthly Goal 1-2 coordinating meeting, led by the bi-lateral goal-leading agencies with participation of the POC’s of each LOA and specific projects; produce lessons learned (including obstacles, challenges, moments of crisis) to share among the Goal 1-2 stakeholders. Improve public communication. The team found that public awareness and visibility of the activities and results undertaken under Goal 1-2 among beneficiaries (at least the ones the team spoke to) is too limited. This contributes to the difficulty for PFG implementers to perceive how their specific activities contribute to achieving the overall objectives and outcomes. Processes, results, and lessons learned are rarely documented in a way that is more accessible to the public at large and tend to stay buried in internal USG and GOES reporting. The evaluation team recommends improving public communication around Goal 1-2. The following could be considered: (i) a stand-alone PFG website must be created through a bilateral effort. Such a website should include all PFG related documents, including the CA, the JCAP, and goal-by-goal information and reporting. Such a site should include links to the USG Embassy PFG and the GOES Presidential Office PFG website, but be a stand-alone page led by the bilateral PFG management and coordinating team. (ii) The PFG coordinating team (and bilateral goal teams) could also consider engaging NGOs or research centers to document processes and lessons learned under the PFG in a more systematic way and put them up for debate. Partnership for Growth El Salvador Mid-Term Evaluation Report 93 Table 11.4: Summary Status of Goal 1-2 at Mid-term GOAL 1- 2 LOAs GOES LOA 1: Implement the necessary changes in practices, policies, regulations, and applicable laws as identified throughout the project. GOES LOA 2: Identify and make available appropriate staff to be trained. Quantifiable and Objectively Verifiable Information YES (Scorecards, JSSP PMEP August 9, 2013) YES [PMP Narrative Report (Oct 2013)] GOES LOA 3: Emphasize continuous vetting as a fundamental principle of the Security Plan of the Security Cabinet of the Government of El Salvador. GOES LOA 4: Fully participate in programs to support the justice sector. NO [Cannot match indicators to LOA] YES (PMP Narrative Report (Oct 2013) Final) GOES LOA 5: Fully participate in analysis of the current codes. NO (Cannot match indicators to LOA) USG LOA 1: Provide technical assistance to improve the management and investigative capacity of the Attorney General (AGO), Public Defender (PGR), and National Civilian Police (PNC), forensic services, judges and court personnel. USG LOA 2: Support the development of effective case management models; improving police/prosecutor coordination; providing better and more equitable access to justice; re-engineering processes and change management procedures in the common crime unit; establishing career paths and leadership development within the Judicial sector; increasing accountability and transparency in the sector by enhancing judicial oversight and investigative capabilities; supporting civil society; and strengthening crime observatories. USG LOA 3: Provide assistance to promote increased coordination between justice sector actors and institutions; improve management and administration of justice sector institutions; and increase effectiveness of criminal investigation. YES [Checci Quarterly Reports; Scorecards, JSSP PMEP August 9, 2013; Inidcators: #1, #2, #3, #8] YES (Indicators are defined in the JSSP M&E Plan under the USAID contract with Checci, JSSP PMEP August 9, 2013; Indicators: #10, #22, #2, #6, #7, #9, #20, #21, #18, #19, #25) USG LOA 4: Provide technical support in the areas of the pre-trial elements listed above as well as plea-bargaining (agreements with the public prosecutor’s office). Partnership for Growth El Salvador Mid-Term Evaluation Report YES (Indicators are defined in the JSSP M&E Plan under the USAID contract with Checci, JSSP PMEP August 9, 2013; Indicators: #2, #10, #12) YES (Indicators are defined in the JSSP M&E Plan under the USAID contract with Checci, JSSP PMEP August 9, 2013; Indicators: #9, #6) LOA (intervention) on Target YES (Scorecards) PARTLY (Site visit showed that GOES is only providing 50 percent of the agreed upon prosecutors for the Rapid Solution Units.) N/A PARTLY (Interviews with GOES and USG officials and implementers; site visits) PARTLY (Interviews with GOES and USG officials and implementers; site visits) YES (Scorecards; Checci reporting) YES (Checci Quarterly Reporting; scorecards) YES (Checci Quarterly Reporting; scorecards) YES (Checci Quarterly Reporting; scorecards) 94 GOAL 1- 2 LOAs USG LOA 5: Provide technical assistance to conduct an in-depth code analysis, draft legislative fixes, and develop strategies to enhance criminal justice efficiency and effectiveness relating to criminal procedure, an evidence code, and internationally accepted law enforcement tools; implementing such legislation, procedures, and strategies. Partnership for Growth El Salvador Mid-Term Evaluation Report Quantifiable and Objectively Verifiable Information YES (Indicators are defined in the JSSP M&E Plan under the USAID contract with Checci, JSSP PMEP August 9, 2013; Technical assistance provided to UTE addresses these areas) LOA (intervention) on Target YES (Checci Quarterly Reporting; scorecards) 95 Security Goal 4 Sub-Constraint: Institutional Strengthening Security in El Salvador’s public transportation Goal objective (as stated in JCAP): “Facilitate economic growth by ensuring that El Salvador’s labor force is protected from crime while transiting to and from work, and ensuring that the public transportation service providers serving the labor force are protected from crime.” USG lead agency: DOJ (including ICITAP and OPDAT); co-leads: INL; USAID (Casals) USG implementers: USAID implements through Casals; the other agencies do so directly. GOES lead agency: PNC & VMT GOES implementers: Ministry of Justice and Public Security; Vice-Ministry of Transportation; ViceMinistry of Transparency; Ministry of Public Works; others involved: ASI and private businesses. LOA Implementer summary (for details see JCAP): The goal has two main parts: implementation of a vetted special unit, or task force, of police and prosecutors, and the use of vetted courts; and identifying and promoting legislation to provide deterrent sentences in this field, since the transportation law is antiquated. This translates into five LOA for GOES, and two for USG: providing technical assistance to (A) the vetted units and (B) to the public transportation system to help increase transparency and accountability. M&E Indicators: Number of reported crimes in public transportation; public perception of safety on public transportation routes. Scorecards: May 2012: on track; Nov. 2012: on track; May 2013: behind schedule; Nov. 2013: on track; May 2014: on track Summary on Implementation Goal 4 is led by DOJ’s International Criminal Investigative Training Assistance Program (ICITAP) and Office of Overseas Prosecutorial Development, Assistance & Training (OPDAT), but INL and USAID’s Transparency Program also contribute. Goal 4 is a completely new measure, since prior to PFG, no comparable activities were undertaken by GOES with or without USG assistance. In March 2012, USG placed a full-time representative in El Salvador as Resident Legal Advisor (RLA) to assist with the implementation of task forces to combat public transportation and crime. The ICITAP in-country representative (Program Manager) arrived in June 2013. The goal refers to security as a hindrance to economic development, since it tries to increase security in public transportation used by the work force (and in particular the poor) to travel from home to work. After signing the JCAP in November 2011, it took nearly two years (until August 2013) to jump-start a central activity of Goal 4 – the Anti-Extortion in Public Transportation Task Force, as Salvadoran and US counterparts were unable to agree on priorities and sequencing of steps. In the meantime, Casals started the goal component that was attributed to USAID’s Transparency Program, which covers Goal 6 under its USAID Transparency contract, with its main counterpart being the Asociación de Industriales de El Salvador (ASI). Documents reviewed: The evaluation team reviewed the Transparency and Governance project fact sheet, in addition to a document stating the number of public and closed hearings held in January 2014, Partnership for Growth El Salvador Mid-Term Evaluation Report 96 and five scorecards. The team also reviewed a Public Transportation protection plan and the Acta Asocio document. Observational studies: For an in depth study of Goal 4, the evaluation team investigated the following two activities through interviews and a site visit for the first mentioned activity. • • A training session and diploma event coordinated by ICITAP for the Transit Crimes Task Force and Business Crimes Task Force integrated by criminal investigation, police, and special prosecutors. USG lead agency: DOJ/ ICITAP and OPTAD; GOES lead agency: PNC and Attorney General. The creation (currently in the development phase) of a network of Human Resources departments of 500 businesses where workers can register a complaint at work about insecurity encountered during their public transit to or from work, and which are then aggregated by the Anti-Corruption Citizen Observatory at Asociacion de Industriales de El Salvador (ASI). This system will provide workers with a faster and anonymous, less retaliation prone system to denounce crime in public transportation, and allow businesses and the ASI to bring cases to the attention of police and the justice administration. USG lead agency: USAID (under a contract with Casals); ES counterpart: Asociacion de Industriales de El Salvador (ASI) Findings based on Third Party Data The first M&E indicator for this goal is the number of reported crimes in public transportation. The IUDOP’s 2013 study collected data on the number of respondents that have witnessed a robbery, assault or murder while riding the bus or minibus in the past 12 months. The study registered a small, statistically insignificant decrease in the percentage of respondents that witnessed a crime in public transportation in 2013 versus 2012 (see figure 11.5 and table 11.5). Partnership for Growth El Salvador Mid-Term Evaluation Report 97 Figure 11.5: Number of reported crimes in public transportation in El Salvador (2012-2013) 46 40% 30% 29.2% 27.0% 20% 10% 0% 2012 2013 Table 11.5: Number of reported crimes in public transportation in El Salvador (2012-2013) 47 2012 2013 Percent 29.2% 27.0% 95% CI (27.1% - 31.4%) (24.8% - 29.1%) This same study also collected data on the public’s perception of safety on public transport routes (Figure 11.6). Between 2012 and 2013, the percentage of respondents that felt safe decreased significantly from 32 percent to 26 percent. It is not unlikely that the decrease is related to the increase in murders and other crimes over the same period of time. 46 Source: Perception of Security and Confidence in Public Institutions, Universidad Centroamericana Jose Simeon Cañas, Perception of Security and Confidence in Public Institutions (Baseline and Second Measurement reports), (2013) 47 Source: Universidad Centroamericana Jose Simeon Cañas, Perception of Security and Confidence in Public Institutions (Baseline and Second Measurement reports), (2013) Partnership for Growth El Salvador Mid-Term Evaluation Report 98 Figure 11.6: Public perception of safety on public transport routes in El Salvador (2012 – 2013) 48 40% 33.8% 36.2% 37.8% 33.9% 30% 21.9% 19.8% 20% 10.4% 10% 6.2% 0% Not at all safe A little safe 2012 Somewhat safe Very safe 2013 Responses to Country-Specific Questions Country-Specific Question 2: Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? Under the activities implemented by DOJ and INL, in particular the Task Force (since August 2013), quantitative and objectively verifiable information is being used to track progress and manage implementation. A central indicator is related to training activities. The agencies partnering on the Task Force proceed based on their work plan and six-month milestones. The ASI related complaint network is implemented under a USAID contract with Casals, which has to comply with a rigorous SOW-related M&E plan. The only LOA under this goal being implemented by USAID aims to create the workers complaint system in 500 businesses, in alliance with the ASI. Quantitative and objectively verifiable information for this LOA will be systematically collected by Casals, with technical assistance from USAID, and follows procedures laid out in full detail in the M&E Plan on its contract. These indicators will be used by Casals, USAID, and ASI to steer implementation and to make course corrections when necessary. On the GOES side, no additional indicators are being systematically gathered for M&E purposes. Rather, the public and private counterpart agencies involved – PNC, Attorney General, ASI, among others – work with the metrics gathered by the agencies involved by USG. Activities, mainly in terms of results, feed into the scorecard process. DOJ and PNC representatives, with input from USAID (but without participation of Casals and ASI), define the contents of the scorecard by auto-evaluating progress on the LOAs and by relating to the six-month milestones. They also try to score progress on the scorecard with two overall goal specific indicators (see above). Goal 4 scored always “on 48 Source: Universidad Centroamericana Jose Simeon Cañas, Perception of Security and Confidence in Public Institutions (Baseline and Second Measurement reports), (2013). Partnership for Growth El Salvador Mid-Term Evaluation Report 99 track” (“green” on the scorecard), except over the November 2012–May 2013 period, essentially because no progress had been made on creating the joint Task Force. Country-Specific Question 3: At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? At mid-term, performances of activities seemed to be on target in the sense that they contribute to creating the expected PFG outputs and outcomes. Since August 2013, Proactive stands by DOJ (a resident legal advisor placed in El Salvador since March 2012) along with ICITAP’s Program Manager (placed in El Salvador since June 2013), breaking through resistance on both sides, created a partnership with the viceminister of transportation, and jump-started the Task Force. Since then, counterpart agencies meet every second Wednesday of every month to discuss cases of investigation, criminal organizations, and corruption within the public transportation system. They formulated and are implementing a work plan for the Task Force, including regular training and guidance for operational work. As a result, there was significant progress. The Public Transportation Task Force started investigating public transportation crime cases and will become a formal unit in the PNC. All police and prosecutors in the unit have been vetted. A procedures manual for the Task Force was drafted and a training plan was initiated to improve investigation and interview techniques, analysis and surveillance, and personal defense. In addition, the GOES continues to improve the transparency and accountability of the Vice Ministry of Transportation (VMT). The ASI-related complaint system continues to grow: 23 sectors of industry have signed on to and adopted the Code of Ethics for the Industrial Sector and, through its corresponding agencies, a total of 500 businesses are foreseen to sign on and commit to the Code during 2014. The Public Observatory for the Integrated Mass Transportation System (SITRAMSS) and the House of Transparency (Casa de Transparencia) are also operational. Goal 4 is now sometimes cited as a best practice example in terms of planned interagency cooperation. However, given that the Task Force in particular has only very recently come to life, it is still too early to measure its impact on the two overall indicators established for this goal: (i) Number of reported crimes against small and medium businesses; (ii) Perception of small and medium-sized business community on the effect of police and prosecutorial actions on crime against their businesses. Its recent implementation explains why these indicators are still not mentioned on the scorecards, including in May 2014. In 2012, before the creation of the Task Force, a public survey was conducted to measure perception of safety on public transportation routes and to create a baseline. On a scale of 0 to 100, the survey indicated a safety perception of 36. Also, the lead agencies and implementers on both sides are not convinced that their activities, and Goal 4 in general, can significantly contribute to achieving, in PFG’s two remaining years, the desired constraint-level objectives and outcomes, as measured by the three long-term indicators (homicide rate, security perception, impunity rate). They all considered that a mid-term evaluation is too premature to judge. Some of the stakeholders involved with Goal 4 suggested that the three long-term indicators are too ambitious and difficult to impact in only five years. An example mentioned of an indicator that might have been easier or more likely to produce movement within a five-year period was the extortion victimization rate among small and medium businesses. Partnership for Growth El Salvador Mid-Term Evaluation Report 100 Conclusions After a late start, Goal 4 is now making fast progress due to hands-on coordination and team work among involved USG and GOES agencies. Quantitative and objectively verifiable information about progress under the various LOAs is being gathered to manage implementation to achieve and measure results, although in a less rigorous manner than when it is contracted out to third parties, as USAID does under its Goal 4 LOAs and other goals it leads or is involved with. Under Goal 4, the LOAs are defined more in terms of concrete tasks (create, train, and vet a joint task force; implement changes in practices and regulations; promote interagency coordination, provide supervision and control etc.) than in terms of process. Therefore, indicators of progress are more results oriented (amount of training hours delivered, amount of police officials vetted, number of changes made in regulations), and less impact oriented, e.g. number of arrests and proper adjudication of cases. Goal 4 is also not performed (except for one activity) by an implementer who, under its contract, is beholden to a strict M&E plan. DOJ, ICITAP, OPDAT, and others active under Goal 4 have different M&E traditions than USAID. PFG, and in particular its Whole of Government Approach, implies that all become more aware of these different approaches to implementation and M&E. Given that no further guidance or training was provided on how to bring more uniformity to the M&E process beyond the JCAP M&E Addendum, it is not surprising that each lead agency has maintained its own approaches and practices. Course Corrections and Recommendations JCAP rewording of Goal 4 at mid-term. Goal 4 has made significant progress, and the LOA indicators will soon be accomplished, but a mid-term rewording may help more clearly define next steps. Lessons learned from Goal 4. Since Year 2 of the PFG, Goal 4 has been a good example of a goal being driven by bilateral inter-agency coordination. Documentation and sharing of lessons learned about the processes that promoted cooperation (and any challenges faced) would be helpful to other goals. Partnership for Growth El Salvador Mid-Term Evaluation Report 101 Table 11.6: Summary Status of Goal 4 at Mid-term GOAL 4 LOAs GOES LOA 1: Maintain a permanent exchange regarding the employment status of USG-vetted or USG-trained and advised personnel in accordance with Salvadoran procedures and regulations. GOES LOA 2: Improve supervision and control in accordance with the system of indicators of the Security Cabinet. GOES LOA 3: In collaboration with the USG, review the current regulations of the public transportation sector and identify areas to improve transparency and accountability. GOES LOA 4: Implement necessary changes in practices and regulations and encourage the approval of the necessary legislation. GOES LOA 5: Provide personnel, equipment, and facilities to support this program. USG LOA 1: Provide technical assistance, training, and mentorship for the vetted units to combat crimes involving public transit, and facilitate cooperation between GOES and private sector. USG LOA 2: Provide technical assistance to help increase the transparency and accountability of the public transport system. Partnership for Growth El Salvador Mid-Term Evaluation Report Quantifiable and Objectively Verifiable Information Not provided LOA (intervention) on Target Not provided N/A YES (Scorecards, Copia de plan borrador mod proteccion al transporte public) YES (Scorecards) PARTLY measures) Behind Schedule (see Scorecard May 2014). (Scorecards; legislative YES (Scorecards; PNC internal information, Copia de plan borrador mod proteccion al transporte public, Scorecards: 1&3) YES (DOJ internal reporting; scorecards, Copia de plan borrador mod proteccion al transporte public, Scorecards: 1&3] YES (USAID Casals Transparency Program’s M&E Plan; quarterly reporting by Casals; Scorecards: 1, 2, 3, Copia de plan borrador mod proteccion al transporte public) YES YES (Scorecard May 2014) YES (DOJ internal reporting; scorecards) YES (quarterly Casals reports; scorecards) 102 Security Goal 11: Sub-Constraint: Crime and Violence Prevention Municipal Crime Prevention Goal objective (as stated in JCAP): “Prevent crime and violence in key municipalities of El Salvador and support reforms, as outlined in Components 2 (Social Prevention of Violence and Crime) and 5 (Institutional and Legal Reform) of the National Policy for Justice, Public Safety and Violence Prevention.” USG lead agency: USAID (D&G); co lead: INL USAID implementer: Creative Associates Inc. (Creative) GOES lead agency: Ministry of Justice and Public Security (National Prevention Office PREPAZ); GOES implementer: PREPAZ, municipal governments. M&E indicators: (i) Number of Municipalities with Crime Prevention Councils; (ii) Incidence of Selected Violent Crimes Reported in Key Municipalities. LOAs summary (for details see JCAP: promote and help build municipal and community-level capacity and resources for municipal government crime prevention plans and initiatives, supported by publicprivate sector alliances, and in line with the National Policy for Justice, Public Safety and Violence Prevention. GOES is taking on six LOAs, and USG two (which are implemented by Creative Associates Inc., as part of a 5-year contract for USAID’s Community-Based Crime and Violence Prevention project. Scorecards: May 2012: on track; Nov. 2012: on track; May 2013: on track; Nov. 2013: on track; May 2014: on track Summary of Implementation Goal 11 was selected for case study by Optimal because it brings most explicitly sub-national authorities into play, in particular municipal governments, and may thus illustrate how PFG and WGA operates at the territorial level. USAID has been involved with municipal prevention programs in El Salvador since 2006. In 2009, USAID awarded a 5-year contract for municipal prevention work to Creative Associates, Inc. Activities under that contract were mostly rolled into Goal 11 when the JCAP was written in 2011. Creative was then also awarded the new USAID Crime and Violence Prevention Program contract (20142018). Under the new contract, Creative is working with two groups of selected high-risk municipalities: (1) municipalities that receive technical assistance, training and funding for the implementation of prevention projects at the community level, and (2) municipalities that receive only technical assistance and training. There is a common part to municipal prevention policies, such as municipal prevention committees, and crime observatories, but each municipality defines the specific ingredients of its prevention agenda. Documents reviewed: The evaluation team reviewed two CVPP quarterly reports, the CVPP M&E Plan, in addition to the program’s year one work plan and five scorecards. A supplemental document reviewed is an article on the national dialogue for security. Annex 7 has a full list of referenced documents. Partnership for Growth El Salvador Mid-Term Evaluation Report 103 Observational study: For the case study, the evaluation team visited and interviewed the Mayor, the Municipal Violence Prevention Council, the youth philharmonic orchestra and music school, and a youth outreach center in Chalchuapa, as well as the Crime Prevention Observatory in Santa Ana. Findings based on Third-party Data The first M&E indicator for this goal was the number of municipalities with crime prevention councils. Data for this indicator come from Creative as part of their USAID Community-Based Crime and Violence Prevention project. Between 2008 (prior to PFG) and 2012 (after PFG’s start in November 2011), the number of municipalities where crime prevention councils were being created increased 1.5 times, from four to ten yearly (Figure 11.7). This progress can to a large extent be directly attributed to PFG, although certain other donors also contributed to this result. Figure 11.7: Number of municipalities with crime prevention councils (2008-2012) 49 12 Numer of municipalities 10 10 9 9 6 4 4 2008 2009 3 0 2010 2011 2012 The second M&E indicator was the incidence of selected violent crimes reported in key municipalities. USAID, Checci, and PNC monitor the homicide and other crime rates in the affected municipalities, and when possible, narrowed down for specific communities (neighborhoods or wards) of intervention. The evaluation team considers that attributing short-term changes in homicide and other crime rates directly to Goal 11 activities is problematic (although somewhat less so when the rates are strictly narrowed down to specific neighborhoods or wards of intervention). This challenge invoked what happened in Medellin, Colombia: while the municipal administration (20042007) was already widely credited with dramatically reducing murder rates through comprehensive prevention policies, between 2007 and 2009, due to vendettas related control over the cocaine trade and other organized crime, homicides in Medellin tripled from 804 to 2.187 cases, including in city areas heavily intervened by reform-oriented programs and considered exemplary of the turnaround. Only now (2014) has Medellin been able to reverse the situation and is again approaching the rates observed in 2007. Currently based on the Medellin experience, most independent analysts and experts agree that in 49 Sources: RTI. Community-Based Crime and Violence Prevention Project. Final Report (April 2013), Number of Municipalities with Crime Prevention Councils that are involved in US program Partnership for Growth El Salvador Mid-Term Evaluation Report 104 contexts of conflict and/or heavy organized crime and homicide rate, tendencies are less predictable than in more “normal” contexts. The evaluation team found strong evidence and general consensus among independent experts interviewed in El Salvador, but also among most of the USG agencies interviewed, that in El Salvador, external factors – such as truce dynamics, organized crime dynamics, and gang dynamics – directly and significantly influence homicide and other crime rates. This is one reason why it continues to be problematic to attribute short-term tendencies in homicide and crime risks to PFG activities, let alone to a single agency or goal. Another reason is that even in specific communities of intervention, other PFG goals (as well as non-PFG activities) may impact the homicide and other crime rates. Thus, the Vanderbilt Impact Evaluation Study in El Salvador (2012) seems neglect the potential influence of external factors when attributing 40 percent fewer reported occurrences of murders (2011-2012) than would be expected if USAID’s programs would not have been administered. Responses to Country-Specific Questions Country-Specific Question 2: Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? The evaluation team’s document review confirms that the current USAID-Creative SOW for Goal 11 has the most detailed and precise M&E plan of all USAID programs in El Salvador. Quantitative and objectively verifiable information is constantly being used to manage LOAs of this goal (see table 11.7 below). This information also feeds into the scorecards. Given that implementation of Goal 11 is backed by a detailed work plan (as developed by Creative under USAID guidance), it is not in need to define additional milestones, as goals without detailed work plans tend to do for the scorecard process. Goal 11 aligns closely with the National Violence Prevention Strategy (ENPV, for its acronym) led by the PREPAZ office of the Ministry of Justice and Public Security. ENPV formulation profited from technical assistance from USAID, through Creative as its implementer, and other bi-lateral and multi-lateral assistance. The new USAID-Creative contract (2013-2018) also includes strengthening of a GOES system of indicators for municipal crime prevention policies by PREPAZ. PREPAZ has already been trained by USAID/Creative to enhance its capacity to run this indicator system. Thus, while currently USAIDCreative is still the main producer of indicators, this role is meant to be taken on, over time, by PREPAZ. Country-Specific Question 3: At the midterm, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? Performances at mid-term are on target. Based on the document review the evaluation team found (see table 11.8 below) that currently nine of the 10 LOAs are on target. Not enough information was gathered to provide a valid score for the other one. Even though the goal confronted certain obstacles that could have hindered performance – the first USAID implementation contract being closed, and a new one being attributed; and the sudden 2013 removal of the involved and experienced director of PREPAZ, the GOES lead agency – these obstacles were overcome, in part thanks to the large network of GOES agencies – both national and local - directly Partnership for Growth El Salvador Mid-Term Evaluation Report 105 involved in and committed to advancing this goal. At the national level, Goal 11 commitments strongly line up with the National Violence Prevention Strategy (ENPV), which permits additional leverage to the already strong PFG leverage to overcome obstacles and move activities forwarded when needed. At the local level, participating high-risk municipalities are selected, among other criteria, on their political buyin and community interest in Goal 11 objectives, which guarantees local political commitment to reach targets and create the outputs necessary to achieve the desired outcomes. The evaluation team also found, through site visits and in-site interviewing, that the local-level counterparts and implementers are convinced that their activities can contribute to lowering the homicide rate and improving the security perception, at least in their own communities and municipalities. Conclusion Quantitative and objectively verifiable information is used under Goal 11 to manage its implementation in order to achieve and measure results. At the mid-term, performance of most Goal 11 LOAs is on-target. Stakeholders involved strongly believe that this goal and the municipal-level activities it promotes will help create the outputs to achieve the desired outcomes at constraint level. Course Corrections and Recommendations The evaluation team has no specific suggestions for course corrections or other recommendations in relation to the issues involved under Country-Specific Questions 2 and 3 given that the evaluation team found no problems or challenges for Goal 11 at mid-term. The case study approach to Goal 11 did yield some additional findings, based on which it formulates the following more general recommendation considered relevant for possible mid-term course corrections. El Salvador is increasingly accumulating lessons learned around municipal prevention programs, which have been running for more than five years, with support from USAID. As the evaluation team’s visits to various activities showed – a Municipal Violence Prevention Council, a youth philharmonic orchestra and music school, as well as a youth outreach center and a Crime Prevention Observatory– many of these have over time consolidated into self-sustained projects, and much can be learned from these processes for other municipalities involved with PFG. A bilateral effort to produce lessons learned could best take the form of an external research project, hosted by a local university. Such a project could not only produce case-studies, but public debate and an information campaign around activities implemented, documenting potential actions gathered by interested parties for moving forward. Partnership for Growth El Salvador Mid-Term Evaluation Report 106 Table 11.7: Summary Status of Goal 11 at Mid-term GOAL 11 LOAs GOES LOA 1: Identify dedicated staff to develop and support municipal crime prevention councils. GOES LOA 2: Identify dedicated funding to implement crime prevention plans at municipal level. GOES LOA 3: Promote the decentralization of authority and responsibility for preventing crime to the municipalities. GOES LOA 4: Improved coordination among line ministries with a stake, resources and mandate for addressing key risk factors (lack of access to education, and employment opportunities). GOES LOA 5: Strengthen the cooperation between the government security organizations and said communities. GOES LOA 6: Implement vocational training for at-risk youth. USG LOA 1: Provide technical assistance to promote and facilitate components 2 and 5 of the strategy. USG LOA 3: Region-wide study of the issue of violence and security as it impacts community groups in the hemisphere to bring the community-level experience from the region to bear in El Salvador. USG LOA 2: Explore ways to work with U.S. state/local institutions, the Governments of Mexico, Colombia, and U.S. interagency actors. USG LOA 3: Provide technical support in parole systems and prisoner classification systems. Partnership for Growth El Salvador Mid-Term Evaluation Report Quantifiable and Objectively Verifiable Information NO (Scorecard #1, 3, 4 ) NO (Scorecard #2, 3) PARTLY (legislative and policy initiatives in process or completed) Not provided LOA (intervention) on Target YES (interviews in El Salvador performed by Optimal) YES (interviews in El Salvador performed by Optimal) YES (Scorecards 1-5; interviews in El Salvador) YES (Scorecard May 2014; interviews by Optimal in the field) YES (PNC information on Community Policing) NO (Scorecards #3, 4) YES (Crime and Violence Prevention Program’s M&E Plan; Creative Associates’ quarterly reporting) YES (Study completed by Vanderbilt/CARSI per August 2014, to be published in October 2014) YES (Scorecard May 2014; field interviews) YES (Scorecards) YES (Scorecard May 2014; Creative Associates’ quarterly reporting) YES (INL internal reporting) N/A. YES (INL internal reporting) YES (Scorecard May 2014) YES (Written comments on a draft of this report, by USAID El Salvador) 107 Security Goal 12 Sub-Constraint: Crime and Violence Prevention Reduce Overcrowding in Prisons Goal objective (as stated in JCAP): “Reduce overcrowding in prisons, thereby allowing the Salvadoran prison system to safely, securely, and humanely manage an increasing population.” USG lead agency: INL USG implementer: INL GOES lead agency: PNC (National Direction of the Prison Directorate (DGCP). GOES implementer: PNC M&E indicator: Prison Population as a Percentage of Prison Capacity LOAs under goal 12: Improving the prisoner classification system, expediting the review process to extend privileges and liberties as established by law, separate prisoners by various categories and implement aggressive reintegration (including by building a new prison facility and three prison farms, and by running the Yo Cambio program, an alternative sentencing program); improve the Penitentiary School; create and implement prevention and substance abuse treatment programs; partner with private sector to assist ex-offenders; and monitor sentence compliance of individuals convicted of organized crime. Scorecards: May 2012: on track; November 2012: on track; May 2013: on track; November 2013: on track; May 2014: behind schedule. Summary of implementation Goal 12 is led by INL and the GOES Prison Directorate. Together they also lead goal 13 (Enhance the security of prisons). In addition, INL, since 2013, is engaged in overcoming overcrowding and other problems in temporary holding cells (bartolinas, mainly located at police stations), an activity not included in the JCAP, but strongly related to Goal 12. Documents reviewed: The evaluation team reviewed two presentations on crime and prison statistics in addition to a summary of a prison classification presentation event, a document of prison data for Mariona prison and five scorecards. Observational field study: For the case study, Optimal received a guided visit by the INL-ES prison advisor and a local police chief to a bartolina under construction; in addition to interviews with INL-ES officials and the GOES Director of the Prison Directorate (DGCP), it also performed an interview with the INL-ES prison advisor. Findings based on Third-Party Data The only goal level indicator established for goal 12 is “Prison Population as a Percentage of Prison Capacity”. When measured by the country’s prison capacity and prison population, prison overcrowding is a serious issue in El Salvador. The ratio of the country’s prison population over prison capacity per 100,000 people is the highest in Central America, with about 300 percent overcrowding. Overcrowding in El Salvador has continued to increase from 2010-2013 (see Figure 11.8, below). Although prison capacity has remained essentially the same, prison population has increased by about 9%. In addition, based on information obtained from interviews as more precise data are not available, the bartolinas also seem to have increased their population between 2010 and 2013. Partnership for Growth El Salvador Mid-Term Evaluation Report 108 Figure 11.8: CTS and ICPS Prison population and prison capacity* for El Salvador and other Central American Countries 2010-2013 50 Panama Nicaragua Mexico Honduras Guatemala Costa Rica Belize El Salvador 0 100 Rate per 100,000 people 200 300 400 500 131 130 133 Prison Capacity 391 422 424 Prison Population Prison Capacity 316 Prison Population 476 495 183 194 204 Prison Capacity 259 Prison Population 314 Prison Capacity 77 Prison Population 105 114 Prison Capacity 159 153 160 160 164 155 Prison Population Prison Capacity 197 Prison Population 211 76 Prison Capacity Prison Population 153 Prison Capacity 219 349 Prison Population 383 2010 2011 2012 2013 A baseline to measure progress on prison overcrowding has thus been established for Goal 12. 50 Source: United Nations Office on Drugs and Crime (Prison population), International Centre for Prison Studies (Prison capacity). Data from 2013 are not from the end of the year; Specific months vary between countries. Partnership for Growth El Salvador Mid-Term Evaluation Report 109 Responses to country-specific questions Country-Specific Question 2: Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? The team found evidence that quantifiable and objectively verifiable information is available for all five LOAs (see table 11.8 below). As shown above, Goal 12 also has a baseline on overcrowding in the prison system (as of May 2014, it was still at 317 percent). Goal 12 activities are directly implemented by INL and the Prison Directorate (DGCP). Thus, there are no consultant firms with extensive contractual M&E plans active under Goal 12. The information used to manage goal implementation is mainly based on milestones defined bi-laterally for each 6-month PFG scorecard period. INL reports for PFG activities follow INL standard procedures of reporting. For Goal 12, the reports include quantitative information on program results as number of personnel trained, type and amount of equipment provided, progress with procurement, number of participants in the Yo Cambio program, and number of former participants of the Yo Cambio program. Country-Specific Question 3: At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? Following standard procedures, INL-ES has produced a Strategic Country Plan, which includes USG commitments under Goal 12, as well as under other goals INL is involved with in addition to non-PFG activities. INL considers this plan as the equivalent of the work plan for Goal 12. On its side, the GOES Directorate of Prisons considers that its own Strategic Plan, which also includes the objectives and activities of PFG Goals 12 and 13 (but is not limited to these), doubles as a work plan for these goals. Goal 12 thus seems to have two work plans (the INL Strategic Country Plan and the Directorate of Prison Services strategic plan). Nevertheless, a specific goal work plan, as promised by the JCAP M&E addendum, does not exist. Based on their own respective plans, both USG and GOES agencies involved in Goal 12 consider that LOAs are on target at mid-term. However, over the last 6-month period, the milestones established in November 2013 were not fully met by May 2014, which is the reason why the goal received a “behind schedule” grade on the fifth scorecard. Based on the document review, interviews and data gathered during the field visits, the evaluation team found that two of the five LOAs are on target, two others are partly on target, and for one not enough evidence was gathered to arrive at a valid score (see table 11.8 below). Among all the 14 crime and insecurity constraint goals, Goal 12 was the only one that did not obtain an “on track” on the fifth scorecard (May 2014). All interview respondents involved with Goal 12 considered that on the long term, prison reform, and in particular the commitments made under Goal 12, will contribute both directly and indirectly to improving the security situation, and therefore PFG’s ability to attain its three long term indicators (decrease homicide rate, increase security perception and decrease impunity rate). The evaluation team found that opinions about the Bartolinas initiative – as explained, closely related to Goal 12, but formally not a part of PFG - are more divided. After first realizing a diagnostic of the bartolinas Partnership for Growth El Salvador Mid-Term Evaluation Report 110 problem, INL-ES formulated and is now implementing a 2-year construction plan to overcome the bartolina crisis. However, in part because it is not part of the PFG, the INL initiative does not seem to profit from the same kind of unanimous bi-lateral support as the activities that are included in the JCAP. Some officials (of both governments) argued that the bartolinas should not be reformed, but abolished, as officially no temporary holding centers beyond the needs of an up to 72-hour temporary holding facility should exist at Police Stations. Others argued that the current humanitarian crisis at the bartolinas is extremely serious, and that decreasing the prison system overpopulation will take time, so that rapidly improving the bartolinas and its processing facilities is the reasonable thing to do at this point. Inside the PNC in particular, but also in other GOES agencies, there is no universal agreement on the USG approach. Parts of PNC expressed strong disapproval, mainly based on the belief that PNC will end up administrating a jail system, while formally Salvadoran law forbids longer than 72hour stays at a temporary holding cell. Others argued that improving bartolinas now does not exclude that they will indeed be able (although not on the short term) to comply with the maximum 72-hour stay. Conclusions Quantifiable and objectively verifiable information is available for each of the five LOAs of Goal 12 (see table 11.8 below) and a baseline on prison overcrowding (overall, and per prison) has been established since 2010 (as of May 2014, still at 317 percent). Among all the 14 crime and insecurity constraint goals, Goal 12 was the only one that did not obtain an “on track” score on the May 2014 scorecard. Not all the milestones established for the previous six months period were met. Based on its own review of data sources, the evaluation team found that only two of the five LOA were on target. At the same time, however, both GOES and USG officials interviewed considered that in terms of their respective agency’s work plan that important progress was being made on Goal 12, including during the last 6-month period. Goal 13, which is strongly related to Goal 12 and led by the same agencies, did also obtain a “behind schedule” score. Both GOES and USG officials interviewed also considered that on the long term the commitments made under Goal 12 will contribute both directly and indirectly to improving the security situation in the country. Course Corrections and Recommendations Course corrections and recommendations listed here focus in particular on findings as related to CSQ2 and CSQ3. However, the evaluation team also includes some more general recommendations for Goal 12, when considered relevant for possible mid-term course corrections. Consider combining Goals 12 and 13. GOES and USG officials involved with Goal 12 were equally split on combining Goals 12 and 13 into 12&13 (as was earlier done with Goals 1& 2) to help bring more weight to the issue of prison reform. Combining them would also be a good opportunity to better include the ongoing bartolinas reform project in the PFG, and a good opportunity for USAID and INL to continue increased collaboration. Create a goal-level work plan. Formulating an explicit bi-lateral work plan for the next two years for Goal 12 (or for Goal 12&13) – as promised by the JCAP M&E addendum - may contribute to enhancing commitments from both governments with implementation. Partnership for Growth El Salvador Mid-Term Evaluation Report 111 Create a goal-level implementation team. Although informal coordination towards implementation among USG and GOES agencies is frequent and permanent around Goal 12, establishing a formal implementation team for Goal 12 (or Goal 12&13) will improve management, increase inclusiveness – while Goal 12 is currently still overly led by INL, and commitment towards implementation. Table 11.8: Summary Status of Goal 12 at Mid-Term GOAL 12 LOAs GOES LOA 1: Improve the prisoner classification system. GOES LOA 2: Build a new prison facility and three prison farms, and implement an aggressive reintegration program. USG LOA 1: Provide technical assistance in the management of prisons/corrections, including mentoring and limited training in order to develop prison/correction officer train-the-trainer program, and develop and implement prison classification system. USG LOA 2: Explore ways to work with U.S. state/local institutions, the Governments of Mexico, Colombia, and U.S. interagency actors. USG LOA 3: Provide technical support in parole systems and prisoner classification systems. Quantifiable and Objectively Verifiable Information YES (Scorecards 1-5 ; INL internal reporting; and Directorate of Prisons information, Prison Classification EventUnclassified Decision Memorandum , Mariona Study) PARTLY (Scorecard 5, 3, 2, 1, INL internal reporting; Directorate of Prisons internal reporting) PARTLY (Scorecard 3, 1; INL internal reporting, Prison Classification EventUnclassified) LOA (intervention) on Target PARTLY (INL internal reporting) N/A. PARTLY (Scorecards 1-5; INL internal reporting) YES (Scorecard May 2014) Partnership for Growth El Salvador Mid-Term Evaluation Report PARTLY (Scorecard May 2014) PARTLY (Scorecard May 2014) YES (Scorecard May 2014; INL internal reporting) 112 11.2 Low Productivity in the Tradables Sector (3 Case Studies) Tradables Goal 3 Sub-Constraint: Human Capital Strengthen Labor Force to Match Labor Market Demand (Human Capital) Goal objective (as stated in JCAP): “Improve the quality of the education system in order to create a more highly qualified and technologically skilled labor force. The Governments of El Salvador and the United States will join forces to help ensure that education of the labor supply matches labor market demand, and they pledge to carry out the actions outlined below. The success of these actions will facilitate a cumulative investment in human capital that will boost the productivity of labor and of the tradables sectors.” USG lead agency: USAID/DOL/DOS USG implementer: CARANA, PAS, DOL GOES lead agency: INSAFORP GOES implementers: INSAFORP, Labor Ministry, Ministry of Education LOAs: A total of 25 (14 GOES and 11 USG). M&E Indicators: (i) Global Competitiveness Index: Higher Education and Training; (ii) Global Competitiveness Index: Labor Market Efficiency Scorecards: May 2012: on track; Nov. 2012: on track; May 2013: on track; Nov. 2013: on track. Summary of implementation Goal 3 is led by USAID, but the Department of State and the Department of Labor also contribute. Most of the lines of action of this goal were legacy programs that started prior to PFG, such as the USAID Improving Access to Employment Project and the LAC SEED Scholarship Program. The goal clearly relates to low productivity as a hindrance for economic development as it tries to boost the qualification of El Salvador’s labor force. After signing the JCAP in November 2011, it took two and a half years to launch the USAID Higher Education Assistance Project — LOAs 21 and 22, due to delays in the project design and contracting. Documents reviewed: The evaluation team reviewed seven Quarterly reports, Year 3 & 4 Work plan, Report on Ministry of Labor’s Statistics program, Summary of PFG Technical Assistance to Ministry of Labor, the Goal 3 Working Group Document, the Labor Force Statistics document, a final report of the Improving Access to Employment Program, the Higher Education Program Presentation, report on Salvadoran Organizations aimed to Promote Labor Force, and a letter to Roberto Antonio Morales, and five scorecards. (See Annex 7for all documents reviewed for this goal). Observational studies: In order to prepare the case study for goal 3, the evaluation team conducted interviews with tradables goal leads, and LOA implementing partners. However, no observational studies were conducted for this goal. Findings based on third-party data Out of 148 countries ranked in terms of the quality of higher education by business leaders between 2010 and 2013, El Salvador’s position showed little change over time, from 101st position in 2010 to 100th in 2013. In Partnership for Growth El Salvador Mid-Term Evaluation Report 113 2013, Salvadoran higher education was ranked as being of better quality than its three neighbors (Guatemala, Honduras and Nicaragua). Figure 11.9 illustrates higher education and training data. Figure 11.9: Global competitiveness index: higher education and training Out of 148 countries ranked in terms of the flexibility to shift workers from one economic activity to another between 2010 and 2013, El Salvador’s position declined over time, from 88th position in 2010 to 121th in 2013. In 2013, the Salvadoran labor marker efficiency was ranked lower than only one other Central American country (Honduras). Figure 11.10: Global competitiveness index: labor market efficiency Country-Specific Question 3 asks whether the goals identified (and the underlying LOAs) are sufficient to achieve the desired results. One key factor in the case of tradables, and this is true for the other tradable goals as well, is that the rest of the world, and the rest of the Central American region, are not standing still. Even though El Salvador has made progress in improving labor market efficiency and labor market quality, as measured through the scorecard process, the table above suggests that it has lost ground to several of its Central American neighbors. Partnership for Growth El Salvador Mid-Term Evaluation Report 114 Findings based on Interviews The interviews suggest that Goal 3 is among the most complex of the goals of PFG, involving 25 lines of action distributed among numerous GOES and USG agencies. At the same time, Goal 3 LOAs are largely traditional technical assistance and training activities with a strong emphasis on vocational training, particularly in technical areas. Technical assistance is also provided by USAID and the U.S. Department of Labor in the areas of gathering and analysis of labor market information. Interviewees indicated that these were largely “legacy” activities, but also very important for the achievement of Constraint 2 results. Some progress has been made on the first of the goal-level indicators, the Global Competitiveness Index, Sub-Index for Higher Education and Training, but El Salvador has lost ground on the second, the Sub-Index on Labor Market Efficiency. Moreover, the lack of progress on tradables Goal 5 raised concerns among some interviewees about whether jobs would be available for trainees when the program reaches its conclusion. Responses to Country-Specific Questions Country-Specific Question 2: Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? USG uses quantitative and objectively verifiable information to manage the implementation of the lines of action in order to achieve and measure results. The following are some of the milestones set for the period of May-Nov 2013: The English for Work – INSAFORP - modules will have been tested; at least 47 English teachers will be trained; and at least 300 additional youths trained for immediate employment opportunities. However, at this early stage of implementation, many indicators are still input and process measures, rather than results indicators (e.g. number of trainees placed in employment). GOES also uses quantitative and objective indicators, although these tend to be indicators of the process type (e.g. publication of two Ministry of Labor quarterly bulletins and the conduct of the establishment). However, for some lines of action, like the insertion of youth into the labor force, quantitative and objectively verifiable results indicators are used. In 2013, the GOES measured a result equal to 86 percent of the target, short of the target but still indicating respectable progress toward the LOA. The fact that they can demonstrate this shortcoming indicates both that quantitative results indicators are available and that they are being utilized in PFG management Country-Specific Question 3: At the mid-term, are the performances of the selected PFG interventions ontarget and creating the necessary outputs to achieve the desired outcomes? At mid-term, some of the lines of action are on-target; others are not. The following are score examples for a sample of GOES LOAs. a) Training programs in the English language and information and communications technologies (ICT) for the employed population and for youth: On-target. b) Strengthen and enhance productive diversification through scientific and technological innovation: Behind schedule. c) Improve capacity to collect, analyze, use, and disseminate labor market data in order to correct mismatches between labor supply and demand: Behind schedule. The following are examples of scorings for a sample of USG LOAs: d) Support teacher training in English: On-target. e) In consultation with the GOES, develop alliances, provide technical skills and language training Partnership for Growth El Salvador Mid-Term Evaluation Report 115 assistance and develop alliances to improve access to employment for youth in El Salvador’s tradable sector: Behind-schedule. Conclusions The individual lines of action of Goal three are on track, with two exceptions. However, as measured by the goal-level indicators, it has made little progress in improving the overall productivity of the Salvadoran labor force. Moreover, as suggested above, although El Salvador has made progress in improving its workforce quality and labor market efficiency, so have its regional neighbors. It has lost ground relative to its regional competitors. Much comment was made about the unwieldy nature of Goal 3 with its 25 LOAs, which made it difficult to understand its overall strategy, and gave no indication of priority among its various activities. What is desperately needed for Goal 3 is a strategy, reflecting an informed view of the future human capital needs of an outward-oriented, successful Salvadoran economy, as well as the human and financial resources that will be available to meet those needs. Inevitably, difficult choices will have to be made, primarily by the GOES, but by the USG as well, in terms of meeting immediate needs as well as longer terms needs (what we have characterized elsewhere as the trade-off between “wholesale” and “retail” activities). Course Corrections and Recommendations Goal 3 needs focus and prioritization. The team was not able, in the short time available, to evaluate the many individual LOAs to determine which should be expanded, which should be re-focused, and which should be reduced or eliminated altogether. Such an assessment should be a high priority in any refocusing of PFG. Goal 3, at present, is a mixture of “wholesale” and “retail” activities – retail activities, training of individual students in specific subjects, plays an important role in “ground truthing” an economic growth strategy. However, their direct contribution to economic growth, even in a small country like El Salvador, is inevitably small. If, however, Goal 3 is to play a significant role in measurably increasing El Salvador's production of highvalue tradables, the strategy should focus on improving institutions and systems, as the new Higher Education Project does and as Goal 3’s assistance to the Ministry of Labor on labor force information systems has done. Only in this way can Goal 3 play a significant role in alleviating the tradables constraint in the medium term. Country Specific Question 3 asks whether the Goals as currently constituted will achieve the objectives of PFG. The analysis with specific regard to Goal 3 is not currently conclusive. However, a reoriented Goal 3 informed by a rigorous strategy can play a critical supporting role in achieving those goals. It must be emphasized that for Goal 3 to play a significant role in increasing the productivity of tradables, there must be significant progress in improving the investment climate under Goal 5. Partnership for Growth El Salvador Mid-Term Evaluation Report 116 Table 11.9: Summary Status of Tradables Goal 3 at Mid-term GOAL 3 LOAs GOES LOA 3.1:Develop an education employment plan for youth and women Quantifiable and Objectively Verifiable Information and YES [Year Three Work Plan (Oct 2011-Sept 2012) Year Four Work Plan (Oct 2012-Now 2013)] LOA (intervention) on Target Unclear, Scorecard four states that the Ministry did not formally launch the plan, and scorecard five is silent. GOES LOA 3.2 Establish training programs in YES (Indirectly through USAID Improving Access to 50 students are enrolled under the SUPERATE program English and ICT for employed population and for Employment Program, Quarterly Reports and DOS for IT and English skills. Seven alliances were signed youth reports, Scorecard #2, 4, 5) this period, including a broad alliance with three firms anchored in the airport and 17 surrounding municipalities (Los Nonualcos), as well as another one in the microfinance sector with three institutions to train youth for employment. GOES LOA 3.3:Commit to making YES [Results should be available through quarterly Unclear transformational reforms necessary to improve the reports from USAID Higher Education Program Year quality of El Salvador's educational system Three Work Plan (Oct 2011-Sept 2012)] GOES LOA 3.4:Create a Talent Network of YES [Year Three Work Plan (Oct 2011-Sept 2012) Salvadorans living abroad Year Four Work Plan (Oct 2012-Now 2013)] USAID reports it is having conversations with the recently established El Salvador Global network composed of 21 talented Salvadorans living abroad. GOES LOA 3.5:Support youth insertion in the labor YES (Results can be inferred from USAID Improving market Access to Employment Project, quarterly reports Year Three Work Plan (Oct 2011-Sept 2012), Year Four Work Plan (Oct 2012-Now 2013), Scorecard #2, 4, 5) The USG provided general employment skills training for 631 at-risk youths, and 62 percent have found employment to date. In addition, 12 vocational orientation workshops were held to help 1,401 disadvantaged high school students understand specific job and training opportunities. 150 students started their first year of the English Access Micro-scholarship program, and 190 continued through the second year of the program. GOES LOA 3.6:Improve basic computer training YES (Results can be inferred from USAID Improving 50 students are enrolled under the SUPERATE program programs for entry level positions and for Access to Employment Project quarterly reports, Year for IT and English skills. Seven alliances were signed employees Three Work Plan (Oct 2011-Sept 2012), Scorecard #2) this period, including a broad alliance with three firms anchored in the airport and 17 surrounding municipalities (Los Nonualcos), as well as another one in the microfinance sector with three institutions to train youth for employment. GOES LOA 3.7:Improve English for the workplace YES (Results can be inferred from USAID Improving 50 students are enrolled under the SUPERATE program programs Access to Employment Project quarterly reports, Year for IT and English skills. Three Work Plan (Oct 2011-Sept 2012), Scorecard #4, 5) Partnership for Growth El Salvador Mid-Term Evaluation Report 117 GOAL 3 LOAs Quantifiable and Objectively Verifiable Information LOA (intervention) on Target GOES LOA 3.8:Strenghten and enhance YES (Indirect information will be available from Oct. 2012—Science Fair on renewable energy and productive diversification through scientific and USAID Higher Education Program, 2nd and 3rd Semi robotics; Feb. 2013—Conference on Climate Change in technological diversification El Salvador; Feb. 2013—Earth Science workshops; Annual reports) March 2013—Women in Science workshops GOES LOA 3.9:Focus on technological YES [Results can be inferred from USAID Improving The USG provided general employment skills training development in priority tradables sectors Access to Employment Project quarterly reports, Year for 631 at-risk youth, and 62 percent found employment Three Work Plan (Oct 2011-Sept 2012)] to date. In addition, 12 vocational orientation workshops were held to help 1,401 disadvantaged high school students understand specific job and training opportunities. 150 students started their first year of an English Access Micro-scholarship program, and 190 continued through the second year of the program. GOES LOA 3.10:Design and create a program for YES (Results can be inferred from USAID Improving INSAFORP the transfer and assimilation of ICT Access to Employment Project quarterly reports and the produced a study on future professional training in ICT, Higher Education Program quarterly reports, Year Three and trained 3,642 participants Work Plan (Oct 2011-Sept 2012), Scorecard #2) GOES LOA 3.11:Stremgthen INSAFORP YES (Results can be inferred from USAID Improving Technical assistance provided Access to Employment Project quarterly reports, Scorecard #5) GOES LOA 3.12:Improve capacity to collect, YES (Results can be inferred from USAID Improving analyze, use, and disseminate labor market data Access to Employment Project quarterly reports and US DOL TA reports, Year Four Work Plan (Oct 20122013), Scorecard #2) Technical assistance has been provided to the Ministry of Labor for preparing an operations manual, identify technical qualifications for hiring new staff, and for developing bulletins. GOES LOA 3.13:Use better data and analysis to YES (Results can be inferred from USAID Improving Technical assistance has been provided to the Ministry Access to Employment Project quarterly reports and US of Labor on preparing an operations manual, desired project labor market demand DOL TA reports, Scorecard #2) technical qualifications for hiring of new staff, and for developing bulletins GOES LOA 3.14: Develop youth scholarship YES (Results can be inferred through USAID reports on 25 students given scholarships in most recent 6-month program SEED Program) period USG LOA 3.15: Support teacher training in English YES (USAID Improving Access to Employment Four modules for the National English for Work Program, Quarterly Reports and DOS reports, Scorecard Program have been designed between USG and #2, 3, 4, 5) INSAFORP. 151 English teachers were trained through different modalities, including 21 trained in Costa Rica. Partnership for Growth El Salvador Mid-Term Evaluation Report 118 GOAL 3 LOAs Quantifiable and Objectively Verifiable Information LOA (intervention) on Target USG LOA 3.16: Provide technical support to work YES (USAID Improving Access to Employment The USG provided general employment skills training skills Program, Quarterly Reports, Year Three Work Plan (Oct for 631 at-risk youth, and 62 percent have found 2011-Sept 2012), Scorecard #5) employment to date. In addition, 12 vocational orientation workshops were held to help 1,401 disadvantaged high school students understand specific job and training opportunities. 150 students started their first year of the English Access Micro-scholarship program, and 190 continued through the second year of the program USG LOA 3.17: Develop alliances, provide YES (USAID Improving Access to Employment technical skills and language training Program, Quarterly Reports, Year Three Work Plan (Oct 2011-Sept 2012), Year Four Work Plan (Oct 2012-Now 2013) Scorecard #1, 2, 3, 4, 5) 50 students are enrolled under the SUPERATE program for IT and English skills. Seven alliances were signed this period including a broad alliance with three firms anchored in the airport and 17 surrounding municipalities (Los Nonualcos), as well as another one in the microfinance sector with three institutions to train youth for employment. USG LOA 3.18: Technical assistance to improve YES (U.S. DOL report dated, Oct 15, 2012 and USAID labor market information systems Improving Access to Employment Program Quarterly Reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #3) Technical assistance has been provided to the Ministry of Labor with preparing an operations manual, identify technical qualifications for hiring of new staff, and for developing bulletins USG LOA 3.19: Help GOES build capacity to YES [U.S. DOL report dated, Oct 15, 2012 and USAID conduct surveys to match employer needs and skills Improving Access to Employment Program Quarterly Reports, Year Three Work Plan (Oct 2011-Sept 2012), Scorecard #1] Technical assistance has been provided to the Ministry of Labor with preparing an operations manual, desired technical qualifications for hiring of new staff, and for developing bulletins USG LOA 3.20: Help GOES improve services for YES (USAID Improving Access to Employment The vocational orientation website elijomifuturo.com for job seekers Program, Quarterly Reports, Year Three Work Plan (Oct youths was developed in Nov., and 8,712 different young people have used it. The USG and Ministry of 2011-Sept 2012), Scorecard #3) Labor implemented four job fairs in San Miguel, Ciudad Arce, Ilopango, and Lourdes in which 1,631 job seekers, principally youths, and 110 businesses participated. At least 438 youths found jobs. USG LOA 3.21: Improve tertiary training and YES [PPT presentation for USAID Higher Education He USAID Higher Education project was approved and higher education quality Program, Year Three Work Plan (Oct 2011-Sept 2012)] procurement process has started. Partnership for Growth El Salvador Mid-Term Evaluation Report 119 GOAL 3 LOAs Quantifiable and Objectively Verifiable Information USG LOA 3.22: TA to focus on innovation and PARTLY (PPT presentation technological development in priority tradables Education Program) sectors USAID Higher INSAFORP produced a study on future professional training in ICT and trained 3,642 participants USG LOA 3.23: Support development of a youth YES (USAID SEED project monitoring reports) scholarship program 25 students were given scholarships in most recent 6month period USG LOA 3.25:Launch government, business, YES (Year 3 Work Plan (Oct 2011-Sept 2012), labor dialogue on productivity and competitiveness Scorecard #1) Not implemented Partnership for Growth El Salvador Mid-Term Evaluation Report for LOA (intervention) on Target 120 Tradables Goal 5 Sub-Constraint: Investment Climate Attract Foreign Direct Investment Goal objective (as stated in JCAP): “Support a strategy for attracting and promoting FDI and making El Salvador a more attractive place for foreign investment. The measures described are aimed at streamlining the establishment of operations for potential investors and simultaneously focusing on and scaling up efforts to promote and attract investments.” USG lead agency: USAID/MCC USG implementer: IAF/COMMERCE/MCC/RTI International GOES lead agency: PROESA GOES implementers: PROESA, Ministry of Agriculture and Livestock, Municipal Development Councils, The Growth Council LOAs: A total of 13 (8 GOES and 5 USG). M&E Indicators: (i) Foreign Direct Investment as Percent of GDP; (ii) FutureBrand Country Brand Ranking Scorecards: May 2012: on track; Nov. 2012: behind schedule; May 2013: behind schedule; Nov. 2013: behind schedule. Summary of implementation Goal 5 is led by USAID and MCC. Most of the USG LOAs are legacy programs that started prior to PFG, such as the Municipal Competitiveness Project, implemented by RTI International. This goal clearly addresses the causes of the relatively low level of foreign direct investment, a factor that is considered indispensable for the increase in productivity and inclusive economic growth. The lines of action are supposed to solve this problem by creating a business environment in El Salvador that will stimulate such investments. There have been few delays on the USG side, but many project recommendations to improve the investment climate have gone unimplemented by GOES. Documents reviewed: The evaluation team reviewed three quarterly reports, a PROESA Benchmarking Final Report, an Asociación Nacional de la Empresa Privada (ANEP) Encuentro Nacional de la Empresa Privada (ENADE) Yearly Forum Report, the FUSADES Draft Report, Productive Investment Determinants and their relationship to Effective Demand Report, Fundemos and Research Center of Public Opinion Study – 2013 Report, Foreign Direct Investment 2002-2012 spreadsheet, Fiscal Transparency Portal document, five scorecards, and a document of Goal 5 objective and LOAs. See Annex VII for the list of documents reviewed for the assessment of this goal. Observational Studies: In order to prepare the case study for Goal 5, the evaluation team conducted interviews with tradables goal leads, and LOA implementing partners. However, no observational studies were conducted for this goal. Findings based on third-party data Foreign Direct Investment (FDI) as a percentage of GDP was tracked for all Central American countries between 2010 and 2012, except Belize since data for 2012 were not available. FDI growth in El Salvador displays a positive trend, but from a very low base. From 2011 to 2012, the growth rate of FDI in El Salvador increased by 28.8 percentage points (from 23.6 percent growth in 2010-2011 to 52.4 percent growth in 2011Partnership for Growth El Salvador Mid-Term Evaluation Report 121 2012). However, as a percentage of GDP, El Salvador consistently lags behind the rest of Central American. In 2013, El Salvador’s FDI per capita was less than a third of that of the rest of Central America. Figure 11.11: Foreign direct investment as percent of GDP Between 2010 and 2012, the Central American countries were also ranked according to the value of the “brands” of each. El Salvador’s position showed only a small fluctuation during this period, from 105th position in 2010 to 107th in 2012. It is important to note that the number of countries included each year in the FutureBrand index varied over time. During this period, the other Central American countries were consistently valued above El Salvador. Figure 11.12: FutureBrand country brand ranking Goal 5 essentially consists of two parts, measures to improve the investment climate and measures to improve foreign investors’ perceptions of that climate. Figure 11.12 illustrates the problem that El Salvador faces, and thus directly addresses Country-Specific Question 3. In terms of foreign investors' perception of the investment climate, El Salvador ranks dead last among the seven Central American countries. Until that perception (and the reality it reflects) changes, PFG will not succeed. Partnership for Growth El Salvador Mid-Term Evaluation Report 122 Findings based on interviews Of the goals under the tradables constraint, Goal 5 is the farthest from being on-target. Interviewees from the private sector almost unanimously attributed this lack of progress to a hostile attitude on the part of the government toward the private sector. Whatever the rights and wrongs of this matter, recent scorecards have consistently indicated that this goal is “behind schedule.” Despite this setback, some progress has been achieved, especially at the municipal level. Interviewees cited progress with the Municipal Competitiveness Project at the local level. The recently enacted law clarifying the legal status and powers of PROESA was also cited by both Salvadoran and USG interviewees as the basis for hope of progress in the near future. Interviewees in other goals, most notably Goal 3, expressed concern that delays in moving forward on Goal 5 could mean that workforce trainees under Goal 3 might be unable to find employment. Responses to country-specific questions Country-Specific Question 2: Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? USG agencies rely on quantitative data to monitor the outcomes of the lines of action and assess whether the programs’ milestones are met. For GOES agencies, outcomes were occasionally monitored along with program milestones; however, milestones were rarely used to measure whether or not the programs were on track. Country-Specific Question 3: At the mid-term, are the performances of the selected PFG interventions ontarget and creating the necessary outputs to achieve the desired outcomes? Goal 5, and specifically the policy reform LOAs of Goal 5, are the key to the success or failure of PFG with respect to the tradables constraint. Unless the investment climate improves, and improves substantially, the labor force entrants trained under Goal 3 will not find jobs, the infrastructure built under Goal 2 will not have commercial users, and the technology transferred under Goal 6 will not be employed. As pointed out above, LOAs under Goal 5 fall into two categories, measures to improve the investment climate (LOAs 5.5, 5.6, 5.10, 5.11, and 5.12) and measures to promote investment (LOAs 5.1, 5.2, 5.3, 5.4, 5.7, 5.8, 5.9, and 5.13). The problem is that essentially no progress has been made in the five policy reform LOAs. With absence of such progress, the investment promotion LOAs have no story to tell and there has been, and will not be, any increase in foreign investment. Conclusions Tradables Goal 5 is the most troubled of the tradables goals and arguably (along with its closely allied Goal 1) the most important. The reality is, as the scorecard rankings show, that El Salvador has made essentially no progress in improving its investment climate, as measured by the goal-level indicators of Goals 1 and 5, since PFG began. However, much progress El Salvador makes in improving infrastructure, human capital, and technology (Goals 2, 3, and 6), it is unlikely to achieve significant improvement in its production of tradables unless it takes measures to improve the investment climate. With a new government, El Salvador has an opportunity to make a fresh start. The measurements that make up Goal 5 and Goal 1 goal-level indicators are, for the most part, measures that are firmly within the control of GOES itself. To the extent that outside assistance is required, both the IFC and the USG are entirely able to assist, an opportunity that should not be missed. Partnership for Growth El Salvador Mid-Term Evaluation Report 123 Course corrections and recommendations For better or worse, policy reform is a task for the GOES alone, although donors can be a critical source of advice and implementation assistance. The most urgent task facing the new GOES if it is to increase economic growth is the reform of the investment climate along the lines measured by the IFC’s “Doing Business” indicators and similar measures. The fact that PFG is already in existence and the measures needed already identified is a valuable asset that the new GOES should immediately take advantage of. The items measured by “Doing Business” are simply a prompt and reliable functioning of government as it impacts investors. There is not more critical opportunity facing the two governments than this one. Partnership for Growth El Salvador Mid-Term Evaluation Report 124 Table 11.10: Summary Status of Tradables Goal 5 at Mid-term GOAL 5 LOAs Quantifiable and Objectively Verifiable Information LOA (intervention) on Target GOES LOA 5.1: Strengthen the institutional Not provided capacity to conduct market studies Unclear GOES LOA 5.2: Encourage investments from Not provided (Scorecard #3, 4) Salvadorans residing in the U.S. According to interviewees, the GOES and the IDB are developing a project to increase investments of Salvadorans living abroad GOES LOA 5.3: Strengthen Ministry of Economy YES (PROESA Benchmarking Final Report) and PROESA Legislation to strengthen PROESA was passed by the Legislative Assembly in April. GOES LOA 5.4: Implement the Integrated Investor Not provided Attention System Unclear GOES LOA 5.5: Develop and implement a strategy YES (FDI FUSADES Draft Report) to remove barriers to FDI Study carried out, but recommendations implemented according to interviewees GOES LOA 5.6: Improve plant inspection by Not provided Ministry of Agriculture Unclear not GOES LOA 5.7: Strengthen PROESA’s YES (MCC funded diagnostic, legislation clarifying See LOA 5.3 institutional capacity to identify and prioritize legal status of PROESA passed in April 2014, PROESA strategic sectors for the promotion and attraction of Benchmarking Final Report, Scorecard #1, 2, 4, 5) investments and for the development of an investment promotion strategy. GOES LOA 5.8: Design and implement a country YES (PROESA country image strategy with support of Development of an image strategy has begun image strategy based on studies of the perceptions the IDB, PROESA Benchmarking Final Report, of the business sector in the United States and other Scorecard #1) countries of interest, in order to boost investment and trade. USG LOA 5.9: Offer institutional strengthening of YES (MCC funded diagnostic, legislation clarifying Delays in naming permanent director of PROESA and PROESA in trade and investment facilitation. legal status of PROESA passed in April 2014, PROESA clarifying its legal status have delayed results Benchmarking Final Report, Scorecard #2, 3) LOA 5.10: Support improvements in the local business climate through the Municipal Competitiveness Project; provide technical assistance to municipal councils. YES (Quarterly reporting of the USAID Municipal A total of 40 municipal ones-stop windows operating in Competitiveness Project July 1-Sept 2013 (Annex B), 33 municipalities as of November 2013. Also, IFC has July 1- Sept 2011 (Section X), July 1- Sept 2012 (Annex launched a municipal-level Doing Business Assessment. A) (Scorecard #3) Partnership for Growth El Salvador Mid-Term Evaluation Report 125 GOAL 5 LOAs Quantifiable and Objectively Verifiable Information LOA (intervention) on Target USG LOA 5.11: Partner with the IFC to use Doing PARTLY (Action plan completed by IFC and presented The IFC continues to work with the Growth Council to Business Indicators as a diagnostic for the Growth to Growth Council, Scorecard #5) identify reforms to improve the business climate Council. USG LOA 5.12: Undertake a review of the laws YES (GOES and MCC commissioned FUSADES study Study carried out, but recommendations and processes in El Salvador that facilitate of barriers to FDI, FUSADES Draft Report) implemented according to interviewees investment in order to create an incentives system tailored to different types of investors in key sectors. USG LOA 5.13: Support GOES in the design and PARTLY (PROESA country image strategy with the Delayed implementation of a country image strategy based support of the IDB, PROESA Benchmarking Final on studies of the perceptions of the business sector Report) in the United States and other countries of interest, in order to boost investment and trade. Partnership for Growth El Salvador Mid-Term Evaluation Report 126 not Tradables Goal 6 Sub-Constraint: Innovation and Internationalization Surmount Low Productivity in Tradables (Strengthen Institutions for Internationalization) Goal objective (as stated in JCAP): “Surmount low productivity of tradables by transforming factors of production of the tradables sector through the implementation of strategies to improve innovation and quality, and a focus on the international market. As a result of PFG, Salvadoran firms will be more prepared to confront global markets and compete successfully and the necessary institutions will be prepared to support them.” USG lead agency: USAID/MCC/IAF/DOC USG implementer: IAF/COMMERCE/MCC/ Chemonics International GOES lead agency: MINEC GOES implementers: MINEC/PEDEMYPE/CONAMYPE/FONDEPRO LOAs: A total of 11 (5 GOES and 6 USG). M&E Indicators: (i) Global Competitiveness Index: Business Sophistication; (ii) Number of Businesses Exporting over $500,000 Scorecards: May 2012: on track; Nov. 2012: on track; May 2013: on track; Nov. 2013: on track. Summary on implementation Goal 6 is led by USAID and MCC, but IAF and DOC also contribute. This goal did not include projects that were started prior to PFG. The chief USAID implementing activity, the SME Development Program implemented by Chemonics, was inaugurated on October 3, 2011 and essentially coincided with the inauguration of PFG. However, its design and contracting predated PFG and USAID indicates that the project is being refocused and retargeted to bring the project into closer alignment with PFG. The goal relates to low productivity as a hindrance for economic development as it tries to increase El Salvador’s competitiveness both in the internal market and in export markets by improving technology, enabling potential export firms to meet export quality standards, especially sanitary and phytosanitary standards for food products, and enabling SMEs to meet procedural and documentation requirements of destination markets. After signing the JCAP in November 2011, the major activities of goal 6 began to be implemented almost immediately under the SME Development Program Mentioned above. Some activities involving PROESA were delayed due to delays in clarifying the legal status of PROESA and delays in naming a permanent director. Documents reviewed: The evaluation team reviewed four quarterly reports for the SME Development Program and supplemental documents that included El Salvador Advances in Information Access Report, the Productive Development Fund Presentation, the MINEC- Report of Work, the Integral System for the Promotion of Production Report, the Accountability CONAMYPE June 2012 and May 2013 Report, and Five Scorecards. See Annex 7 for the list of documents reviewed for the assessment of this goal. Observational Studies: In order to prepare the case study for Goal 6, the evaluation team conducted interviews and a site visit to the following activity: • SBDC (CDMYPE) at Universidad Francisco Gavidia. Partnership for Growth El Salvador Mid-Term Evaluation Report 127 Findings based on Third Party Data Between 2010 and 2013, a total of 148 countries were ranked in terms of the quality of the country’s overall business networks and its individual firms’ strategies. El Salvador’s position showed a small improvement during this period, from 68th position in 2010 to 60th in 2013. This ranking suggests that most of the improvement was accomplished between 2012 and 2013, when El Salvador gained 22 positions. The other Central American countries showed less fluctuation during this period than El Salvador. In 2013, the Salvadoran business environment was ranked as more sophisticated than two of its neighbors (Honduras and Nicaragua), but less sophisticated than the remaining three Central American countries. Thus, third-party data suggests that Salvadoran firms are relatively well-positioned to identify and adopt technology and procedures to take advantage of new opportunities. Thus, in response to Country-Specific Question 3, the answer is that, given a more favorable policy environment, the Salvadoran private sector would, in all likelihood, respond favorably in terms of adopting new technology and new procedures under Goal 6. Figure 11.13: Global competitiveness index: business sophistication Number of Businesses Exporting over $500,000 Publically available data was not found for this indicator. Findings based on interviews. The interviews suggest that despite being classified as “on track” the accomplishments to date, in terms of aggregate exports and employment is limited, as indicated in the Goal level indicators. Interviewees expressed the view that a strong foundation has been established in terms of training and institutional development but that the lack of overall success in expanding exports is due in part to weak export markets, both in the United States and in the Central American region. Responses to Country-Specific Questions Country-Specific Question 2: Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? The USG agencies rely on quantitative data to monitor the outcomes of the lines of action and assess whether the programs’ targets for program indicators are met. These data were not found for the GOES agencies, where outcomes were only seldom monitored and program indicators were rarely used to measure whether the programs were on track. Partnership for Growth El Salvador Mid-Term Evaluation Report 128 Nonetheless, since seven of the eleven LOAs are implemented by or paired with activities of the USAID SME Development Project implemented by Chemonics, the answer to Country-Specific Question 2 is, for the most part yes. The answer for the other four LOAs is unclear. Country-Specific Question 3: At the mid-term, are the performances of the selected PFG interventions ontarget and creating the necessary outputs to achieve the desired outcomes? The scorecard process has judged the individual LOAs of Goal 6 to be on-target, with the exception of specific LOAs related to PROESA, which have been delayed because of delays in naming a permanent head for PROESA and clarifying its legal status. A review of the M&E documentation supports the view that this is correct. In terms of Country-Specific Question 3, the views of interviewees as well as third-party data discussed above, strongly support the proposition that, given a more favorable policy environment, firms receiving assistance under Goal 6, mainly small and medium enterprises, would be well-positioned to respond in terms of increased production, employment, and exports. Conclusions The 11 LOAs of Goal 6 appear to be relatively well conceptualized and targeted to the objectives of a) Improving the technological sophistication of producers of promising small and medium-sized Salvadoran exporters. b) Improving the ability of SME exporters to meet quality-control requirements. c) Improving the ability of current and potential SME exporters to meet documentation requirements of destination markets. Course Corrections and Recommendations As indicated above USAID is currently undertaking modifications and course corrections to more tight align its SME project with PFG Goal 6. With these adjustments, the LOAs under this goal appear well-conceptualized and tightly focused and achieving expected results. The focus on SMEs (especially medium enterprises) seems correct. Larger firms are likely to have the contacts and resources to obtain relevant technological information on their own. Smaller firms are most likely to participate in international markets as suppliers to larger domestic firms and the problems of quality control and documentation fall on the direct exporter. Medium-sized firms have the potential to export but, lacking the experience and contacts of larger firms, can benefit from focused donor assistance. The team therefore recommends an informal joint review of the overall strategy of Goal 6, taking into account the adjustments to the USAID SME project, with a view to confirming the overall approach, the distribution of resources among Goal 6 initiatives, and the distribution of resources and activities between “retail” (that is firm-level assistance) and “wholesale” institutional development activities. Partnership for Growth El Salvador Mid-Term Evaluation Report 129 Table 11.11: Summary Status of Tradables Goal 6 at Mid-term GOAL 6 LOAs Quantifiable and Objectively Verifiable Information LOA (intervention) on Target GOES LOA 6.1: Strengthen the technical YES (Indirectly through annual reports of USAID Strengthening of PROESA has been delayed by lack of a permanent director and delays in clarifying its legal status. capacity of PROESA legal status. SME Development Program (Chemonics)) However, substantial progress has been made in training and technical assistance through Chemonics. GOES LOA 6.2: Strengthen the Ministry Not provided (some information on assistance to MINEC's Productive Development Fund reported grants of of Economy and PROESA to continue SMEs provided indirectly through annual reports of $1.9 million to 111 SMEs generating 90 jobs in the most developing the production and export USAID SME Development Program (Chemonics)) recent 6-month scorecard. MINEC approved an additional strategy which improves innovation and $6.4 million in funding to institutions providing business quality development services in latest six-month reporting period GOES LOA 6.3: Work with USG to YES (Indirectly through annual reports of USAID USAID project has provided assistance to firms to guide facilitate the export process SME Development Program (Chemonics), Scorecard them through the export process. Little progress has been made in streamlining the process #3] GOES LOA 6.4: Create an integrated YES [Indirectly through annual reports of USAID Firms assisted by the project increased exports by $7.96 system to serve SMEs seeking to export SME Development Program (Chemonics), Scorecard million, most from SMEs. #3, 4) GOES LOA 6.5: Strengthen and increase Not provided diversification through innovation MINEC and private sector have agreed on six priority sectors. USG LOA 6.6: Provide TA to the YES (Annual reports USAID SME Development Firms assisted by the project increased exports by $7.96 Ministry of Economy and PROESA to Program (Chemonics), Scorecard #1) million, most from SMEs. serve SMEs seeking to export USG LOA 6.7: Support the establishment YES [Annual reports USAID SME Development 12 Small Business Development Centers established. The of small business development centers Program (Chemonics), Scorecard #1, 3, 4, 5) 1,969 firms assisted to date have produced sales of $11.3 throughout the country million, a 143 percent increased, and generated 3,430 jobs USG LOA 6.8: Increase the YES (Annual reports USAID SME Development The USG provided international business development internationalization of Salvadoran firms Program (Chemonics), Scorecard #4] assistance to 483 businesses. These firms increased domestic through guided processes or mentoring sales and exports by $6.3 million, generating 760 jobs. Partnership for Growth El Salvador Mid-Term Evaluation Report 130 GOAL 6 LOAs Quantifiable and Objectively Verifiable Information LOA (intervention) on Target USG LOA 6.9: Explore ways to facilitate PARTLY (Assistance provided to firms measures 29 SMEs received TA in food safety and best agricultural the speed of exports from El Salvador to number of firms assisted but not their success in practices as of the May 2013 scorecard the U.S. and reduce the rejection rate at meeting requirements, Scorecard #1) the U.S. border USG LOA 6.10: Support the increased NO (IAF grant to FUPEC foundation does not appear A $255,000 grant was approved by USG for the FUPEC participation of Salvadorans abroad in the to measure objective results, Scorecard #5) Foundation to work with Salvadoran hometown associations Salvadoran economy in the U.S., extending assistance to microbusinesses and grassroots associations in El Salvador. USG LOA 6.11: Promote opportunities NO (El Salvador Investment Challenge Facility None yet for bi-national business alliances in the anticipated as part of MCC second compact) tradables sector Partnership for Growth El Salvador Mid-Term Evaluation Report 131 ANNEXES ANNEX 1 - EVALUATION STATEMENT OF WORK STATEMENT OF WORK PARTNERSHIP FOR GROWTH (PFG) MID-TERM EVALUATION: EL SALVADOR AND THE PHILIPPINES I. BACKGROUND INFORMATION Partnership for Growth (PFG) is a set of bilateral partnerships between the United States and a select group of four countries (El Salvador, Ghana, the Philippines, and Tanzania) to accelerate and sustain broad-based economic growth by putting into practice the principles of President Obama’s September 2010 Presidential Policy Directive on Global Development. It involves rigorous joint analysis of constraints to growth, the development of joint action plans to address these constraints, and high-level mutual accountability for implementation. One of PFG’s signature objectives is to engage governments, the private sector and civil society with a broad range of tools to unlock new sources of investment, including domestic resources and foreign direct investment. By improving coordination, leveraging private investment, and focusing political commitment throughout both governments, the Partnership for Growth enables partners to achieve better development results. Core principles of the Partnership for Growth include: • • • • • • • Country ownership and partnership; High-level political leadership and commitment to development progress; Rigorous, evidence-based joint analysis on constraints to growth conducted by integrated teams of U.S. Government and PFG country officials; Joint decision-making on where to focus and prioritize resources; Use of a broad range of tools, including catalytic policy change, institutional reform, aid, diplomatic engagement, and other ‘non-assistance’ policy tools; Leveraging the whole of the US government; 51 Transparency, mutual accountability and fact-based monitoring and evaluation. The PFG process consists of several steps, including: 51 Examples of how whole of government is expressed and applied include http://www.whitehouse.gov/blog/2013/01/07/whole-government-commitment-inclusive-entrepreneurial-growth and http://www.state.gov/documents/organization/153139.pdf. Partnership for Growth El Salvador Mid-Term Evaluation Report • • • • • • Agreement to initiate PFG with selected partner countries; Joint analysis on constraints to growth, followed by broad consultation, dialogue on the findings; Development of joint country action plans (JCAPs) that outline potential tools, reforms, technical assistance and resources that can be applied over the next five years to address priority constraints to growth; Implementation of priority initiatives by USG agencies and partner governments; Regular monitoring and evaluation, which includes semi-annual scorecards Transparency and consultation with private and public sectors. Documentation on PFG design, goals, objectives and accomplishments can be found at http://www.state.gov/e/eb/ifd/odf/pfg/countries/index.htm and http://www.mcc.gov/pages/activities/activity-two/partnership-for-growth. II. CONTEXT An important aspect to bear in mind at all times is that PFG is a bilateral partnership at the country level. The evaluation will be conducted by an external evaluator agreed upon by the U.S. government (USG) and, for evaluations in their countries, the governments of El Salvador and the Philippines. II.a Timing Considerations This evaluation will only focus on El Salvador and the Philippines, as they are the first PFG countries due for a mid-term evaluation in 2013 and 2014. It will span the U.S. government and national (i.e. Salvadorian and Filipino) government leadership, implementing agencies, activities, strategies, stakeholders and audiences (both public and private). El Salvador is scheduled to hold national Presidential elections in February 2014; the evaluation team must therefore consider the time limitations on national government staff and personnel leading up to this date and efficiently and effectively plan its engagement with these stakeholders. The fourth PFG-El Salvador scorecard (see section II.d “Existing Documents and Data Sources” for scorecard details) is scheduled for release in late November 2013, which requires extensive consultations and negotiations between the two government teams. There may be overlap between the mid-term evaluation field visit and efforts to complete the scorecard. In addition, the country-specific results of the evaluation will serve as a tool for transitioning into the new administration in El Salvador. By June 30, 2013, the current Philippines administration would have completed the first half of its sixyear term. It is presently reviewing priorities articulated in the Philippine Development Plan. The administration will likely focus on efforts that will better ensure inclusive growth, increased employment, and policy continuity into the successor administration. A PFG evaluation in spring 2014 will benefit from the GPH’s review and revalidation of its priorities. II.b Target Areas and Groups No single criterion was used to identify target populations for PFG activities. Some PFG activities are national in scope, and others target specific sub-populations, regions and sectors. Partnership for Growth El Salvador Mid-Term Evaluation Report II.c Results Frameworks and Intended Results The El Salvador and the Philippines PFG efforts have tailored unique results framework developed in response to the constraints to growth analysis. Following is the logical framework, reflected in detail in the Joint Country Action Plan (JCAP). The frameworks reflect only the constraints and the goals necessary to alleviate or address the constraints. More information on the agreed lines of action to achieve the goals can be found in each country’s JCAP. Constraint = a binding constraint to growth, identified explicitly in the constraints to growth analysis 52 Goal = a necessary objective to alleviate and address the constraint, identified in the Joint Country Action Plan Line of Action = a programmatic response, by one or both governments identified in the Joint Country Action Plan. A line of action may be a project or a policy change or any other discrete intervention at the implementation level. The government responsible for executing the line of action is clearly identified in the JCAP. For the Philippines, the lines of action are identified in “Section B: Summary Matrix of PFG Activities” starting on page 15 of the JCAP. For El Salvador, the lines of action are identified under the “What the GOES / USG intends to [do]” bulleted lists under each Goal description, starting on page 8 of the JCAP. LOA may in many cases be synonymous with project or activity. 52 A Constraints Analysis is a study based on the growth diagnostic approach originally developed by Haussmann, Rodrik, and Velasco (2005) and since elaborated by others, including the United States Millennium Challenge Corporation (MCC). Growth diagnostics seeks to identify, for a particular country at a particular point in time, the principal barriers – the “binding constraints” – to that country achieving and sustaining faster economic growth. It starts with the premise that those constraints affect growth by preventing private investment and entrepreneurship from reaching the levels they would attain in the absence of those constraints. Partnership for Growth El Salvador Mid-Term Evaluation Report El Salvador Results Framework - Adapted from the Joint Country Action Plan - LOA = Line of Action Partnership for Growth El Salvador Mid-Term Evaluation Report 135 PHILIPPINES Results Framework Adapted from the Joint Country Action Plan BROAD-BASED AND INCLUSIVE GROWTH ACCELERATED AND SUSTAINED REGULATORY QUALITY IMPROVED Trade and Investment Facilitated Regulatory Bottlenecks, Entry Barriers and Discriminatory Provisions to Investments Reduced RULE OF LAW AND ANTICORRUPTION MEASURES STRENGTHENED Efforts to Improve Judicial Efficiency Supported Fiscal Space by Expanding the Tax Base Increased Anti-Corruption Institutions Strengthened Fiscal Space by Minimizing Revenue Loss Increased Contract Enforcement Strengthened Competition Policy Improved Intellectual Property Rights Enforcement Strengthened Philippine Participation in Regional and International Trade Arrangements Enhanced Integrity- and ConfidenceEnhancing Measures Supported Human Capacity for Economic Growth Enhanced FISCAL PERFORMANCE IMPROVED Fiscal Space by Improving Expenditures Management Increased Corporate Governance Strengthened Opportunities for Corruption Reduced Ethical Standards Enforced Accountability Measures Strengthened Partnership for Growth El Salvador Mid-Term Evaluation Report 136 II.d Existing Documents and Data Sources A wide range of documentation is publicly available on PFG, including semi-annual “scorecards” of progress made on JCAP implementation. Scorecards: As described in the El Salvador M&E Addendum, once PFG implementation began the governments of El Salvador and the United States decided to publish semi-annual scorecards to periodically report to the public on progress towards achievement of PFG goals. The two governments use the descriptors “Ahead of Schedule,” “On Track,” “Behind Schedule,” or “Completed”, to characterize progress on each goal. Scores reflect the consensus view of the two governments. The score for each goal is accompanied by a description that provides a justification for each score assigned. This justification includes the results of the goal indicators included in the M&E addendum along with other relevant supporting information and data (which may include results of monitoring and evaluation conducted independently by each government on individual LOAs). The goal indicators are designed to reflect the shared purposes of the governments of El Salvador and the United States. If data for goal indicators is not available in a particular reporting period, progress will be reported in the following period. The Philippines’ PFG performance is tracked through indicators that are already collected and which are mostly publicly available and verifiable, as follow: • • • • Overall: GDP Growth, Exports Growth, Positive Credit Ratings, Index of Economic Freedom, Employment Growth, Non-Metro Manila Regions Share in GDP Enhanced Domestic Investment Climate: Foreign direct investment (FDI) to GDP ratio, Capital formation to GDP ratio, IMD Competitiveness Ranking, World Economic Forum Competitiveness Ranking, Global Enabling Trade Index Ranking, Doing Business Ranking, Government Effectiveness Indicator (World Governance indicators), Land Rights and Access, Regulatory Quality Indicator (World Governance Indicators) Rule of Law and Anti-Corruption: Corruption Perceptions Index, Control of Corruption Ranking, Rule of Law index – Regulatory Enforcement, Rule of Law Index – Absence of Corruption, Court Congestion Indicator Fiscal Space: Tax Effort, Infrastructure Expenditures to GDP ratio, Education Expenditures to GDP ratio, National Government Deficit to GDP ratio, Open Budget Ranking The above indicators will be supplemented by activity level output and outcome indicators for specific programs The following sites archive multiple reference documents for El Salvador and the Philippines: El Salvador: http://sansalvador.usembassy.gov/partnership-growth.html http://tecnica.presidencia.gob.sv/temas/iniciativa-asocio-para-el-crecimiento.html Additional baseline data El Salvador: Not as much crime and citizen security data is available as information on productivity and investment, so the USG and GOES are making efforts to obtain more information for PFG monitoring purposes as well as to inform public policy and donor interventions outside of PFG. A recent extensive baseline study on public perceptions on security and crime was completed by USAID/El Salvador, available publicly at: http://pdf.usaid.gov/pdf_docs/PNAEA859.pdf. Partnership for Growth El Salvador Mid-Term Evaluation Report 137 This information should be used by the evaluation team both as background to understand the development context in the country, and to inform the responses to the evaluation questions. The scorecards are made public in English on the US Embassy in El Salvador website, and in Spanish on the Executive Secretary to the President of El Salvador website. To date, three semi-annual scorecards have been released. The Philippines: http://manila.usembassy.gov/partnership_for_growth.html http://www.iro.ph/index.php http://www.neda.gov.ph/PDP/rm/pdprm2011-2016.pdf http://222.127.10.196/national.html In addition, the Philippines PFG team prepares updates that can be made available to the evaluation team. III. EVALUATION RATIONALE III.a Evaluation Purpose The evaluation will serve two purposes. As a result, there are two sets of evaluation questions. Purpose 1: The first purpose is to evaluate whether the PFG process demonstrates improvements over prePFG assistance approaches. In particular, the evaluation will examine the extent to which the PFG’s whole-of-government and constraints analysis approach led to a change in the manner of USG delivery of development assistance and whether these changes demonstrated improvements in terms of operational efficiency, selection, coordination, design and management of development interventions, and ultimately increased the probability and effectiveness of assistance efforts in achieving verifiable results. The findings and conclusions of this part of the mid-term evaluation will help decision makers determine whether PFG indicates an improved model for providing assistance and whether it portends a higher probability of achieving desired development results. Furthermore, it will inform governments in their work with all donors. Purpose 2: The second purpose is to: 1) evaluate whether PFG efforts have been developed in such a way as to allow for the eventual determination of their impact on addressing the identified constraints and desired outcomes; and, 2) to evaluate the performance of certain initiatives to date to determine whether or not they are moving in the right direction, are considered necessary and sufficient to achieve PFG goals, and are contributing to national interests through the integration and coordination of work done by both governments. The findings and conclusions of the country-specific portion of the mid-term evaluation are of particular relevance and will provide tangible input to the national government and USG entities for identifying obstacles and optimizing PFG implementation in the field, allowing for country program course corrections where feasible and needed in order to enhance the likelihood of achieving sustainable, cost-effective and measurable results. In fulfilling this second purpose, the expectation is to conduct an assessment of the evaluability of the PFG JCAPs (i.e. in other words, assess the extent to which the current PFG programs, as designed and implemented, are evaluable and can or will demonstrate , in verifiable terms, the results they intend to deliver) and evaluations of performance to date. The contractor first will be asked to conduct a preliminary evaluability assessment of each country’s PFG JCAP (this assessment links to question “1” of the country-specific questions in Section IV.a). The Partnership for Growth El Salvador Mid-Term Evaluation Report 138 preliminary evaluability assessment should use the available program information to assess the following components of each goal and its corresponding LOAs (or LOA equivalent): • problem diagnostic and baseline situation assessment; • causal logic of activities, objectives, and outcomes; • intended beneficiaries; and • data availability. The preliminary evaluability assessment will be used to identify goals and LOAs that are ready or amenable for further in-depth “second-tier” review during the mid-term evaluation, taking into consideration Post and host country recommendations, when it is technically feasible and maintains the integrity of the evaluative methodology. At a minimum, at least two goals (one per constraint) that are amenable will then be reviewed to determine whether: • The indicators selected to measure their progress cover the overall logic of the PFG interventions; • There are any major gaps in data collection and analysis that could prevent the interagency partners and joint steering committees from: o Adequately managing implementation towards expected results; o Evaluating the effectiveness of PFG. This second tier evaluability assessment links to question “2” of the country-specific questions. (See section IV.a for details) The goals and LOAs selected for the second tier assessment also will form the “sample” of LOAs or projects that will be evaluated to determine performance at the mid-term. This performance evaluation links to question “3” of the country-specific questions. (See section IV.a for details) The current scope is only for the El Salvador and Philippines mid-term evaluations. It is expected that the other PFG countries will undertake mid-term PFG evaluations at a later date. Similarly, a final evaluation of PFG and PFG efforts in each country is anticipated. While not covered under this SOW, data captured may be employed in the eventual final evaluations and provide the foundation for making conclusions at that time. III.b Audience and Intended Uses The mid-term evaluation will be made available on-line to the public. There are many audiences for the mid-term evaluation, including: Implementers • The national government Minister of Foreign Relations (or the equivalent) and PFG Coordinating Committee in, • The U.S. Ambassadors and Country Teams, • The White House and participating U.S. Agency PFG Coordinators and country desk officers in Washington, DC, • Relevant agencies/organizations implementing JCAP activities; Stakeholders • Citizens of El Salvador and the Philippines, • Civil society representatives and organizations, in the U.S., El Salvador, the Philippines; • Private sector commercial companies and organizations, in the U.S., El Salvador and the Philippines, Partnership for Growth El Salvador Mid-Term Evaluation Report 139 • • IV. Diaspora communities residing in the United States, and The international donor community interested in lessons learned from applying the 2005 Paris Declaration on Aid Effectiveness and the 2008 Accra Agenda for Action-assistance 53 EVALUATION DESIGN AND METHODOLOGY The Contractor should propose the most rigorous evaluation methodology feasible and cost-effective given the learning potential and scope of the study. To the greatest extent possible, the Contractor shall analyze and collect quantitative data. IV.a Evaluation Questions The evaluation questions address issues of common concern for all PFG countries (the cross-cutting questions), as well as country specific questions tailored to each country’s individual situation. Most of the cross-cutting evaluation questions will focus on organizational management structures which are common across all four countries. Country-specific evaluation questions are more appropriate to testing the theory of change at the technical level and will be used to make country-specific recommendations in the final Mid-Term Evaluation Report. Mid-Term Cross-Cutting Evaluation Questions: 1. What are the advantages and/or disadvantages of the PFG* whole of government approach to development assistance? The intent of this question is to assess the extent to which the PFG efforts intended changes in development assistance have or have not materialized. The whole of government approach is relevant to identifying areas for assistance, selecting interventions, and determining implementation coordination. The question is relevant both to national government agencies and institutions, and U.S. government agencies and institutions overseas and in Washington DC. * Explanation of “Whole of Government”: In large bilateral efforts such as Partnership for Growth, many different governmental agencies and ministries are involved in both governments in different dimensions of the larger effort. Within the U.S. government, the term “whole of government” reflects efforts to align each agency’s activities to achieve a common objective. Footnote 1 provides resources for further explanation. 2. To what extent has Partnership for Growth affected the workload on national government and U.S. government staff, as compared to the workload created by traditional forms of development assistance delivery? 3. What contribution has non-assistance 54 made to the PFG process and how can it be utilized moving forward? 53 For further information on the 2005 Paris Declaration on Aid Effectiveness and the 2008 Accra Agenda for Action-assistance please visit http://www.mcc.gov/pages/activities/activity-two/aid-effectiveness: http://pdf.usaid.gov/pdf_docs/pdacq942.pdf Partnership for Growth El Salvador Mid-Term Evaluation Report 140 Mid-Term Country-Specific Evaluation Questions: Country-specific questions look directly at the efforts unique to a PFG country. In this portion of the evaluation, evaluators are expected to assess the evaluability of the PFG effort in each country and, to the extent possible, determine progress to date in a select amount of initiatives in PFG framework. El Salvador 1. The constraints analysis does not identify remedies to address the binding constraints to growth. For each of the constraints, are the goal-level commitments set forth in the JCAP alone capable of achieving the constraints-level objectives and outcomes? (See Section III.a “Evaluation Purpose” for details on expectations related to this question.) 2. The PFG model places an emphasis on evidence-based decision making and fact-based monitoring. Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? (See Section III.a “Evaluation Purpose” for details on expectations related to this question.) 3. At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? The Philippines 1. The constraints analysis does not identify remedies to address the binding constraints to growth. For each of the constraints, are the goal-level commitments set forth in the JCAP alone capable of achieving the constraints-level objectives and outcomes? (See Section III.a “Evaluation Purpose” for details on expectations related to this question.) 2. The PFG model places an emphasis on evidence-based decision making and fact-based monitoring. Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? (See Section III.a “Evaluation Purpose” for details on expectations related to this question.) 54 PFG calls upon the US Government (USG) and partner countries to be more comprehensive and creative in our development work – to reach beyond aid to all the instruments that both governments can bring to bear to connect and amplify the impact of current investments and unlock growth potential. USG commitments under PFG are comprised of both assistance and non-assistance tools that, undertaken in close coordination with partner countries, will maximize our impact and success. In addition to those actions already identified by the interagency and partner countries, additional non-assistance activities should be considered over the life of PFG for a sustained and focused effort. Non-assistance options provide a venue for demonstrating United States support to partner countries and the Partnership for Growth. Options are intended to fully leverage the United States’ unique convening authority, NGOs, professional organizations and academic institutions, donor groups, regional banks, and diaspora communities, and policy options for development results. Partnership for Growth El Salvador Mid-Term Evaluation Report 141 3. At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? IV.b Evaluation Design The evaluation will be a performance evaluation, but should highlight the results of any impact or other rigorous analyses done separately on PFG goals or lines of action (LOAs) at the country level. A performance evaluation should include descriptive questions. The mid-term evaluation will include but not be limited to semi-structured interviews, focus groups of stakeholders, and documentation reviews. Where feasible and appropriate, efforts should be made to incorporate quantitative data collection or analysis to measure program performance. The evaluator is expected to incorporate input from a reasonable range of civil society and the private sector. Offerors are encouraged to propose cost effective approaches to the evaluation. Additionally, for addressing country-specific questions, the contract may propose various methodologies to create a representative sample of the larger effort (for example, selecting to analyze only certain LOAs or goals, based on the level of foreign assistance investment they’re receiving) to ensure the scope of the evaluation is manageable and cost-effective while retaining its ability to provide a general assessment of the PFG effort and provide actionable recommendations for the Steering Committees, partner governments and US interagency going forward. At a minimum, at least two goals (one per constraint) that are amenable to an in-depth second-tier review will be selected for this purpose. Cost-benefit and cost-effectiveness analysis will not be utilized in the mid-term evaluations. IV.c Evaluation Points of Contact The COR for this evaluation will be the primary POC for the cross-cutting and both country-specific evaluations. The COR will be located in Washington. He or she will have responsibility for representing the evaluation and its progress to the larger USG PFG group. Each country will establish a POC team, consisting of one USG POC in Washington, one USG POC in the field, and one national government POC. The POC teams for each country will be responsible for communications with the COR. The USG-POC in Washington DC will help the evaluation teams liaise with all relevant stakeholders within the US inter-agency community at headquarters. The USG-POC based in the partner country, either within the U.S. Embassy or in another U.S. Agency there, will help the evaluation team reach all relevant USG stakeholders in country. The national government POC will help the evaluation team reach all relevant stakeholders within the country. IV.d Planning for Data Collection Within the first 6 months of PFG implementation, a USG Goal Lead was named in El Salvador for each of the twenty goals. Goal leads are responsible for coordinating and consolidating line of action monitoring information that feeds into the semi-annual PFG scorecard. In general, the technical focus of each Goal determines which US Agency will be selected to act as Goal Lead and shepherd information collection among all USG agencies with lines of action under that goal. Goal Leads have been named from the Department of State, USAID, Treasury, Department of Commerce, Department of Justice, and MCC. Partnership for Growth El Salvador Mid-Term Evaluation Report 142 Likewise, the GOES has named Goal Leads on their side to consolidate information on GOES progress related to their corresponding lines of action and to coordinate with the corresponding USG Goal Leads. Finally, several US agencies that do not have permanent representation in the country are implementing lines of action under PFG in El Salvador. The US Labor Department and Inter-American Foundation are two examples. Twenty 'mirror' Washington USG Goal Leads have been named to assist with the consolidation of information and field queries from Post that need input or guidance from Washington, including progress reports from the non-presence agencies. For the Philippines, the GPH and the USG have stood up a Steering Committee within the first six months of JCAP approval to set the policy directions on the areas for PFG support, approve the general plans of action of the PFG Technical Sub-Committees on Regulatory Quality, Rule of Law and Anti-Corruption, and Fiscal Space, and oversee overall progress, among other functions. The three sub-committees provide advisory technical support to implement program goals and objectives and coordinate with implementing agencies involved in program activities. These sub-committees also assess/conduct analysis of sector performance and overall performance of programs and how these contribute to PFG goals; review overall PFG progress for reporting to the Steering Committee; and conduct periodic technical discussions and exchange views on sector issues affecting progress. One week of field work in Washington DC is estimated in the present scope of work so the evaluation team can meet with the USG Washington-based Goal Leads and other Washington-based PFG stakeholders. The field work in Washington DC should take place before the field work in country. In addition to the monitoring data on program activities normally collected by U.S. government and national government agencies during the course of implementation, PFG’s emphasis on shared responsibility with the national government and public transparency has resulted in an additional layer of periodic monitoring data that will be available to the evaluation team, such as the scorecards (see description above) and other host country data systems. The three evaluation POCs identified in section IV.c will provide the evaluation team with access to all existing PFG program monitoring data. The format, frequency and type of monitoring data collected by the GOES and GPH may be significantly different from the formats and types used by the U.S. government. The evaluation team will process the information and identify information gaps and data quality concerns in an inception report, to guide additional data collection required as part of the evaluation. Once the gaps in monitoring information are identified, the evaluation team will fill out the “Pre-Field Visit Data Needs and Analytical Guide” below and discuss the recommended approach with the COR to negotiate a final guide to be used once the team is in country. Partnership for Growth El Salvador Mid-Term Evaluation Report 143 Template 1: Pre-Field Visit Data Needs and Analytical Guide Cross-Cutting Questions Evaluation Questions Type of Answer Needed (e.g. descriptive, normative, causeeffect) Data Collection Method(s) Gender Disaggregation of Data, where Possible Sampling Selection Criteria or Data Analysis Method(s) or Data Analysis Method(s) 1. What are the advantages and/or disadvantages of the PFG whole-ofgovernment approach to development assistance? 2. To what extent has Partnership for Growth affected the workload on national government and U.S. government staff, as compared to the workload created by traditional forms of development assistance delivery? 3. What contribution has non-assistance made to the PFG process and how can it be utilized moving forward? Template 2: El Salvador Country-Specific Questions Evaluation Questions Type of Answer Needed (e.g. descriptive, normative, causeeffect) Data Collection Method(s) Gender Disaggregation of Data, where Possible Sampling Selection Criteria 1. The constraints analysis does not identify remedies to address the binding constraints to growth. For each of the Partnership for Growth El Salvador Mid-Term Evaluation Report 144 constraints, are the goal-level commitments set forth in the JCAP alone capable of achieving the constraints-level objectives and outcomes? 2. The PFG model places an emphasis on evidence-based decision making and fact-based monitoring. Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? 3. At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? Template 3: The Philippines Country-Specific Questions Evaluation Questions Type of Answer Needed (e.g. descriptive, normative, causeeffect) Data Collection Method(s) Gender Disaggregation of Data, where Possible Sampling Selection Criteria or Data Analysis Method(s) 1. The constraints analysis does not identify remedies to address the binding constraints to growth. For each of the constraints, are the goal-level commitments set forth in the JCAP alone capable of achieving the constraints-level objectives and outcomes? Partnership for Growth El Salvador Mid-Term Evaluation Report 145 2. The PFG model places an emphasis on evidence-based decision making and fact-based monitoring. Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? 3. At the mid-term, are the performances of the selected PFG interventions on target and creating the necessary outputs to achieve the desired outcomes? V. EVALUATION PRODUCTS The set of evaluation milestones/products required are detailed below: 1. [Written Document ] Work Plan – Due to the COR within the first 5 business days after start of the evaluation. The work plan will detail the Evaluation Team’s schedule in weekly blocks of time for the various tasks and deliverables, including desk review, inception report development, evaluation design, interviews (in the U.S., El Salvador, and the Philippines), data collection, data analysis and preparation of initial evaluation results, report writing, briefings and presentations. 2. [Written Document] Inception Report and Preliminary Evaluability Assessment– Due to the COR within 3 weeks after the start of the evaluation. The inception report (see section IV.d) is a desk review of all existing documentation and monitoring data relevant to the specific PFG evaluation in question. The PFG evaluation places added emphasis on the inception report to ensure that all available monitoring and program data has been received, read and analyzed by the evaluation team prior to approval of field work. A useful template and guide for the inception report is provided by the UNODC at http://www.unodc.org/documents/evaluation/IEUwebsite/Chapter_4_C.pdf. The current scope adopts the UNDP’s definition of an inception report: “Evaluation inception report—An inception report should be prepared by the evaluators before going into the full-fledged evaluation exercise. It should detail the evaluators’ understanding of what is being evaluated and why, showing how each evaluation question will be answered by way Partnership for Growth El Salvador Mid-Term Evaluation Report 146 of: proposed methods; proposed sources of data; and data collection procedures. The inception report should include a proposed schedule of tasks, activities and deliverables, designating a team member with the lead responsibility for each task or product. The inception report provides the programme unit and the evaluators with an opportunity to verify that they share the same understanding about the evaluation and clarify any misunderstanding at the outset.” (source: http://web.undp.org/evaluation/handbook/Annex3.html) The preliminary evaluability assessment should use the available program information to assess the ability of the JCAP projects to demonstrate in measurable terms the results they intend to deliver (See section III.a “Evaluation Purpose” for details regarding the expectations and scope of the preliminary evaluability assessment.) The contractor should propose a methodology for sampling LOA for review in order to ensure that the scope of the evaluation and field work is manageable and cost-effective while retaining its ability to provide a general assessment of the PFG effort and provide actionable recommendations for the Steering Committees, partner governments and US interagency going forward, and will take into account Post and host country recommendations 3. Updated Methodology and Evaluation Plan. Once a final sampling strategy has been decided, the detailed evaluation methodology should be updated based on the preliminary review of all available JCAP and PFG data. The updated methodology should include a Pre-Field Visit Data Needs and Analytical Guide Report, which includes information on data gaps, sampling strategy, pre-tested interview questionnaires and data collection timeline. See section IV.d for the basic template, which can be adapted to country-specific needs with COR agreement. 4. End of Field Visit Debrief Debrief to national and U.S. governments in El Salvador and the Philippines, including Washington, DC PFG staff via teleconference. This will be delivered prior to departing the country while there to conduct the field visit. This presentation will update the team on the status of evaluation progress, identify any outstanding data or information, and describe any preliminary evaluation findings to date. 5. Draft Evaluation Reports (See Deliverable Six for types and quantities of reports) Draft reports will be provided for all final reports outlined in deliverable seven. Draft reports “a” and “b,” as described in deliverable six, are due to the COR within four weeks after the end of the field visit. Draft report “c” is due two weeks after the presentation (deliverable seven) of report “b.” The evaluating findings shall be treated as an independent assessment and opinion of the contractor. USAID, GOP and GOES stakeholders will review the draft evaluation report with the expressed and sole objective of reviewing the factual accuracy of any information contained therein and to indicate areas where further clarification are warranted. The contractor should address these concerns prior to submitting a final report. The evaluation reports should include but may not be limited to the following elements: Partnership for Growth El Salvador Mid-Term Evaluation Report 147 1. 2. 3. 4. Executive Summary Objectives of the evaluation, including evaluation questions Methodology used and limitations of study Results of analysis, assessment of performance against evaluation questions, and to what extent results can be attributed to the actual interventions 5. Key lessons learned, recommendations, and course corrections for PFG implementation. 6. Final Mid-Term Evaluation Reports The contractor will provide three Mid-Term Evaluation Reports: a. A mid-term evaluation report covering the country-specific and cross-cutting findings, lessons learned, recommendations, and course corrections for PFG implementation in El Salvador; b. A mid-term evaluation report covering the country-specific and cross-cutting findings, lessons learned, recommendations, and course corrections for implementation in the Philippines; c. And, a report that compares and contrasts the findings, conclusions and recommendations from the cross-cutting questions in El Salvador and the Philippines, to identify commonalities in the PFG process regardless of the location of its implementation. Report “c,” in the list above will be issued after the completion of both the El Salvador and the Philippines fieldwork. Reports will be due to the COR within 1 week of receiving COR written feedback on the draft evaluation report (see schedule below for total estimated time line). Reports must adhere to the evaluation report requirements outlined USAID’s ADS chapter 203.3.1.8. The evaluator will provide a Spanish translation of the executive summary for report a. in the list above as well as submit a final report in Spanish. 7. Evaluation Report Presentations: d. Due within 1 week after the delivery of the accepted final report “a” under deliverable six, the proposer will deliver an in-person presentation in Washington, DC, to Washington-based USG PFG staff and to national and U.S. government personnel in El Salvador, who will participate via video or teleconference. e. Due within 1 week after the delivery of the accepted final report “b” under deliverable six, the proposer will deliver an in-person presentation in Washington, DC, to Washington-based USG PFG staff and to national and U.S. government personnel in the Philippines, who will participate via video or teleconference. VI. TEAM COMPOSITION For the life of the contract, the team will contain two permanent staff members, the Evaluation Team Leader and the Data Methods Specialist. For each country, two additional evaluation specialists with sector-specific experience will be added for those portions of the contract. For example, the El Salvador evaluation will require an evaluation specialist with experience in citizen security and an evaluation Partnership for Growth El Salvador Mid-Term Evaluation Report 148 specialist who has a background in economic development and trade. The Philippines will require an evaluation specialist with a background in economic development, trade and public finance, and another with a background in rule of law and anti-corruption. The Offeror is encouraged to consider the inclusion of country nationals or regional country nationals to the evaluation team. In addition to their core technical specialties, country nationals are instrumental in ground-truthing information analyzed during the evaluation and helping the rest of the team see the larger picture and put things in perspective. The Offeror must verify the availability of any personnel working on the evaluation for more than 60 days. Please include letters of availability for all applicable personnel when submitting the proposal. Submissions not including letters of availability will not be considered for the award. The permanent staff members, as well as the specialists required for the El Salvador portion of the contract, must be able to communicate in Spanish—allowing them to analyze documents in Spanish and to conduct interviews and hold conversations in Spanish. Evaluation Team Leader a) An advanced degree in Economics, Business Administration, Statistics, Economic Development, or a related field; b) At least 15 years professional experience in evaluation, including in overseas settings; c) Experience managing teams and working with USG and international governments; d) Proven ability to design and implement quantitative and qualitative research instruments and methodologies; e) Ability to communicate in Spanish and English. Data Methods Specialist: a) An advanced degree in social science, statistics or mathematics; b) At least 7 years technical experience with qualitative and quantitative study design, questionnaire development, data collection, quality control, coding and analysis; c) Ability to design, manage, and implement qualitative and quantitative field-based data collection for evaluations; d) Proven competency in the use of data management software for evaluation; e) Ability to communicate in Spanish and English. The following are suggested specialists required for each country: El Salvador Citizen Security Evaluation Specialist a) An advanced degree in Economics, Business Administration, Statistics, Economic Development, Law, Criminology or a related field; b) At least 7 years professional experience in evaluation, including in overseas settings; c) At least 5 years of experience in the fields of promoting citizen security, protecting at-risk youth, and/or crime reduction; d) Proven ability to implement quantitative and qualitative evaluation instruments and methodologies; e) Ability to communicate in Spanish. Partnership for Growth El Salvador Mid-Term Evaluation Report 149 Productivity and Tradables Evaluation Specialist a) An advanced degree in Economics, Business Administration, Statistics, Economic Development, or a related field; b) At least 7 years professional experience in evaluation, including in overseas settings; c) At least 5 years of experience in the fields of economic development, trade, and/or business development; d) Proven ability to implement quantitative and qualitative evaluation instruments and methodologies; e) Ability to communicate in Spanish. The Philippines Productivity, Tradables and Public Finance Evaluation Specialist a) An advanced degree in Economics, Business Administration, Statistics, Economic Development, or a related field; b) At least 7 years professional experience in evaluation, including in overseas settings; c) At least 7 years of experience in the fields of economic development, trade, and public finance; d) Proven ability to implement quantitative and qualitative evaluation instruments and methodologies; Rule of Law and Anti-Corruption Evaluation Specialist a) A law degree plus an advanced degree in economics, business administration or public policy b) At least 7 years professional experience in evaluation, including in overseas settings; c) At least 7 years of prior technical experience with a focus on rule of law, anti-corruption enforcement and corruption prevention e) Proven ability to implement quantitative and qualitative evaluation instruments and methodologies; VII. EVALUATION MANAGEMENT A) Logistics The various POCs listed above will provide logistical support in terms of providing the team with the necessary U.S. and host-country contacts, contact information and required background information. Other logistics required for the execution of the evaluations will be the responsibility of the contractor. See section IV.d for additional information. B) Scheduling The contract is expected to begin in September 2013 and run until July 2014. The El Salvador portion of the evaluation will precede the Philippines portion, while the cross-cutting elements will spread across both. The El Salvador field work must take place prior to January 1, 2014. An evaluation schedule follows: Partnership for Growth El Salvador Mid-Term Evaluation Report 150 SEE ACCOMPANYING SPREADSHEET C) Level of Effort The USG has anticipated that the evaluation will require 626 personnel days to complete. An USG estimate of the time LOE by personnel is provided below, but the level and type of staff and their days is at the discretion of the Proposer. TASKS (DAYS) Team Leader SME (1) RS SME (2) GM DMS HW Work Plan – 3 1 8 2 16 3 2.Inception Report/ Preliminary Evaluability Assessment 10 5 4. Interviews in DC 3 5. Field Work 18 6. Field Visit Debrief 2 7. Draft Evaluation Report 20 8. Final midterm Evaluation Report 5 9. Cross Cutting report 5 10. Evaluation Report Presentation 2 Per Country Totals 73 10 80 4 32 4 32 36 288 4 32 30 240 4 32 4 32 4 32 102 12 3.Updated Methodology /Evaluation Plan. 5 40 2 16 2 16 18 144 2 16 15 120 2 16 2 16 4 32 53 Additional Days for the Philippines 2 2 4 2 Additional Cross-Country Report 10 0 0 10 TOTAL LOE 158 108 208 152 3 3 18 2 20 5 3 2 71 SME = Subject Matter Expert DMS = Data Methods Specialist Partnership for Growth El Salvador Mid-Term Evaluation Report 151 ANNEX 2 - WORK PLAN No. 1 2 3 LEAP 1400-319: Midterm Performance Evaluation of Partnership for Growth (PFG) Project EL SALVADOR Deliverable Due Date Status/ No. of Days Work Plan December 31, 2013 PFG Review December 31 - January 2, 2014 LEAP Team revises with PFG comments January 2 - 6, 2014 PFG Review Final Version and Approves January 6 - 10, 2014 LEAP Team Submit Final January 15. 2014 Inception Report February 3, 2014 Completed PFG Review February 3 - 10, 2014 5 days LEAP Team revises with PFG comments February 11 - 18, 2014 6 days PFG Review Final Version and Approves February 19 - 25, 2014 6 days LEAP Team Submit Final- March 3, 2014 3 days Evaluability Assessment February 27, 2014 February 28 - March 4, 2014 Completed 6 days March 5 - 13, 2014 6 days March 14 - 18, 2014 3 days March 19 - 21, 2014 3 days March 24 - 27, 2014 4 days March 27 - 28, 2013 2 days March 31 - April 9, 2014 8 days April 10 - 18, 2014 7 days Updated Methodology and Evaluation Plan March 19, 2014 Completed LEAP Team submits Updated Methodology and Evaluation Plan March 19, 2014 PFG Review LEAP Team revises with PFG comments, submits Evaluability Assessment Criteria component PFG Reviews Evaluability Assessment Criteria component LEAP Team revises Full Evaluability Assessment with PFG comments and submits PFG Reviews Final Evaluability Assessment Version LEAP Team incorporates comments and resubmits PFG FINAL Review LEAP Team revises and submits FINAL 4 Completed PFG Review March 20 - 24, 2014 3 days March 27, 2014 March 31 - April 1, 2014 3 days 2 days PFG Review April 2 - 7, 2014 6 days LEAP Team revises and submits FINAL Data Collection Instruments (Interview Guides and Online Survey) April 8 - 18, 2014 9 days March 10, 2014 Completed LEAP Team Submits Final UMEP LEAP Team revises and re-submits LEAP Team submits first installment of data collection instruments March 10, 2014 Partnership for Growth El Salvador Mid-Term Evaluation Report 152 No. LEAP 1400-319: Midterm Performance Evaluation of Partnership for Growth (PFG) Project EL SALVADOR Deliverable Due Date Status/ No. of Days PFG reviews first installment of data collection instruments March 11 - 17, 2014 5 days LEAP Team revises and submits all data collection instruments March 18 - 19, 2014 2 days Pre-testing of data collection instruments March 19 - 24, 2014 4 days PFG Reviews Final data collection instruments March 20 - 24, 2014 3 days LEAP Team revises and submits Final data collection instruments March 25 - 26, 2014 2 days PFG Review March 26 - 27, 2014 2 days LEAP Team revises and submits FINAL interview protocols March 27 - 28, 2014 2 days March 31, 2014 Completed El Salvador Field Work 1 March 31 – April 11, 2014 10 days In Brief with USG PFG Team El Salvador Field Work 2 March 31 – April 2, 2014 April 22 – May 1, 2014 10 days Field Visit 5 End of Field Visit Debrief April 24 , 2014 Completed 6 Evaluation Report May 28, 2014 In Progress POC PFG Review May 29 – June 3, 2014 5 days June 4 - 17, 2014 10 days June 18 - June 27, 2014 8 days June 30 – July 7, 2014 6 days PFG Review July 8 – 16, 2014 7 days LEAP Team revises with PFG comments LEAP Team continues more in depth revision with PFG comments July 17 – 23, 2014 5 days July 24 – 29, 2014 4 days PFG Review LEAP Team revises with PFG comments for final approval July 30 – August 8, 2014 5 days August 11 – 15, 2014 8 days Travel for Report Presentation in El Salvador August 18 – 22, 2014 5 days TBD (Based on availability) Not Started LEAP Team revises with PFG comments PFG Review LEAP Team revises with PFG comments 7 Evaluation Report Presentations Partnership for Growth El Salvador Mid-Term Evaluation Report 153 ANNEX 3 - ADVANTAGES AND DISADVANTAGES FOR DATA COLLECTION METHODS FOR THIS EVALUATION The tables below provide brief descriptions of the pros and cons of different data collection types (e.g. survey, interviews), tool structure (e.g. semi-structured) and method of delivery (e.g. email). Type Survey Individual Interview Group Interview Observation Advantages Disadvantages • Standardization; • Easy to do with a large group; • Ease of administration; • Suitability to tabulation and statistical analysis; • Sample can be used to provide much information about a population; • Can be sensitivity to subgroup differences; • Can be used to record behaviors as well as opinions, attitudes, beliefs and attributes; • Can be combined with other methods, i.e., observation or case study. • Allows for clarification; • Able to gather in-depth information and to pursue hunches; • Can tailor the line of discussion to the individual; • Easier to reach those who are considered unreachable (the poor, homeless, high status, mobile, etc.); • May be easier to reach specific individuals (i.e., community leaders, etc.); • More personalized approach; • Easier to ask open-ended questions, use probes and pick up on nonverbal cues; • Longer interviews are sometimes tolerated. Particularly with in-home interviews that have been arranged in advance, people may be willing to talk longer face-to-face than to someone on the phone. • Can increase the sample size substantially; • Provide some quality controls on data collection can focus on the most important topics and issues in the program; • Can assess the extent to which there is relatively consistent, shared views among participants. • Setting is natural; • Can generates relevant, quantifiable data; • Most useful for studying a “small unit” such as a classroom, Extension Council, etc. • May requires a separate data-entry step; • May require a data cleaning step; • Can be more time-consuming compared with less formal methods; • Respondents might misinterpret questions, depending on how questions are designed and asked; • Tendency for scope of data to be limited (no follow-up questions); • Samples must be carefully selected to ensure statistical meaning. • Reactive effect: interviewer’s presence and characteristics may bias results; • Cost more per interview than other research methods. This is particularly true of in-home interviews, where travel time is a major factor; • Requires strong interviewing skills; • Slowest method of data collection and analysis; • Responses may be less honest and thoughtful; • Interviewer should go to location of respondent; • Respondents who prefer anonymity may be inhibited by personal approach; • May reach only a smaller sample; • Difficult to analyze and quantify results. Partnership for Growth El Salvador Mid-Term Evaluation Report • Less questions can be asked than in an individual interview; • Require considerable group process skill; • Conflicts may arise; • Status differences may become a factor; • More difficult to guarantee confidentiality. • Requires skilled data collector; • The evaluator has less control over the situation in a natural environment; • Hawthorne effect (awareness of being observed might affect behavior); • If observer chooses to be involved in the activity, he/she may lose objectivity • Not realistic for use with large groups 154 Advantages Type Archival (data already collected by agency) Structure Structured Semistructured Unstructured • Low cost; • Unobtrusive; • Can be highly accurate; • Often good to moderate validity; • Usually allows for historical comparisons/trend analysis; • Often allows for comparisons with larger populations. • May be difficult to access local data; • Often out of date; • When rules for record-keeping are changed, makes trend analysis difficult or invalid; • Need to learn how records were compiled to assess validity; • May not be data on knowledge, attitudes, and opinions; • May not provide a complete picture. Advantages Disadvantages • Emphasizes reliability (how accurately different respondents' answers can be compared); • Can reach a large sample; • A representative sample is possible and results can be used to make statements; • Questions are structured and asked in the same way so that respondents’ answers can be more easily compared. • Provides valuable information from context of participants (and stakeholder) experiences; • Use of pre-determined questions provides uniformity. • Respondents are forced to choose between the alternatives provided by the interviewer; • It can be difficult to obtain reliable data on opinions, attitudes and values; • Interviewer has to stick to the agreed questions even though interesting lines of enquiry might emerge in an interview; • More time-consuming than postal or online questionnaires. • Can be time consuming to collect and analyze data; • Requires some level of training or practice in order to prevent interviewer suggesting answers. • Interviewer bias is unavoidable. To some extent the interviewer will affect the responses of the interviewee; • Can develop in all sorts of directions. This makes comparison between data from different interviews difficult; • Respondents are likely to present themselves in a favorable light, exaggerating aspects of their behavior which they see as socially desirable and minimizing aspects seen as undesirable;. • Unstructured interviews can take up a great amount of time and cost for the interviews to take place. • Respondents may be more likely to discuss sensitive and painful experiences if they feel the interviewer is sympathetic and understanding; • If respondent feels at ease with the interviewer, they might open up and be very honest; • They give respondents time and opportunities to develop their answers; • Gives the interviewer more chance to pursue a topic, to explore with any further questions, and ask the respondent to qualify and develop their answers. Advantages Delivery Email Disadvantages • Can gather responses within a short timeframe; • Economy of scale (large samples do not cost much more than smaller ones, except for any cost associated with acquiring the sample); • Files with pictures, videos and audio can be attached; • It allows the respondent to answer at their leisure. For this reason, they are not considered intrusive; Partnership for Growth El Salvador Mid-Term Evaluation Report Disadvantages • List of emails must be already owned or purchased; • Respondents might pass questions along to someone else to answer. Limited capacity to enforce recruitment criteria; • Respondent might dislike unsolicited email even more than unsolicited regular mail; • Email surveys may not be used to generalize findings to whole populations as people who have email are different from those who do not; 155 Advantages Delivery Internetbased Postal Telephone Face-toface • Records can be easily compiled into a database. • Can gather responses within a short timeframe; • Economy of scale (large samples do not cost much more than smaller ones, except for any cost associated with acquiring the sample); • Pictures, videos and audio files can be included; • Can use complex question skipping logic, randomizations and other features not possible with paper questionnaires or most email surveys; • Respondents might be more honest answering to questions about sensitive topics when giving answers to a computer instead of a person or writing them down on paper; • Complete record of session instantly available. • Can include pictures; • Allows the respondent to answer at their leisure. For this reason, they are not considered intrusive. • Can be administered with respondent in hard to reach areas with limited internet access; • Relative anonymity may result in frank discussion of sensitive issues. • Can include pictures, videos and audio; • Can assess body language; • Can use complex question skipping logic; • Have participants’ undivided attention. Partnership for Growth El Salvador Mid-Term Evaluation Report Disadvantages • Email surveys cannot automatically skip questions or randomize question or answer choice order or use other automatic techniques that can enhance surveys the way on-line webbased surveys can; • Can’t assess body language or tone of voice; • May require a data cleaning step. • Current use of the Internet is growing but far from universal. Internet surveys may not reflect the population as a whole; • Respondents may quit a long internet-based survey on the middle more easily than they would if talking to a good interviewer; • Respond may respond to an email invitation to take a Web-based survey and then not answer the survey; • Respondents might pass questions along to someone else to answer. Limited capacity to enforce recruitment criteria; • Can’t assess body language or tone of voice. • Requires the name and address of the respondents; • Long duration of time. It might take several weeks after mailing out of questionnaires before a minimum numbers of responses are returned. • Respondents might pass questions along to someone else to answer. Limited capacity to enforce recruitment criteria; • Can’t assess body language or tone of voice; • May requires a separate data-entry step; • May require a data cleaning step. • Can’t assess non-verbal reactions; • More difficult to get reactions to visuals; • Noise interference from callers’ environments might distract respondents; • May need to train data collector • May requires a separate data-entry step; • May require a data cleaning step. • Responders lose anonymity; • Logistical challenges and travel expenses; • May need to train data collector; • May requires a separate data-entry step; • May require a data cleaning step. 156 ANNEX 4 - DATA COLLECTION INSTRUMENTS 4.1 Stakeholder Types for Interview Guides 1-5 This annex presents the five types of stakeholders and interview guides developed (Section 4.1); a full copy of each of the five interview guides (Section 4.2); and a full copy of the online survey can be found in Section 4.3. Leadership – Interview Guide 1 Respondent Type: The targeted respondents for this semi-structured Interview Guide # 1 are current and former USG and GOES officials who have held or hold leadership positions within the PFG initiative, particularly POCs and others at their level who will be more informed of the implementation of PFG. Architect – Interview Guide 2 Respondent Type: The targeted respondents for this semi-structured Interview Guide # 2 are stakeholders directly involved with the design and planning of the El Salvador PFG, or whose inputs were sought after for these processes. The stakeholders mainly include USG and GOES officials (current and former), members of the Growth Council, representatives from private sector and other civil society organizations, as well as independent experts. Goal Lead – Interview Guide 3 Respondent Type: The targeted respondents for this semi-structured Interview Guide # 3 are active and former GOES and USG Goal Leads who have directly worked on respective goal(s). LOA Implementer – Interview Guide 4 Respondent Type: The targeted respondents for this semi-structured Interview Guide # 4 are the implementers of all or selected lines of action (LOA) under the selected goals. The interview will be performed primarily with the chiefs of party, directors, and/or coordinators or their representatives. Independent Expert – Interview Guide 5 Respondent Type: The targeted respondents for this semi-structured Interview Guide # 5 are independent experts. Experts include academics, subject matter experts, journalists and others who contribute to public debate on the PFG in general or specific areas of the PFG, but are not responsible for directing or implementing components of the PFG. The guide includes cross-cutting questions (to gauge changes in the operational efficiency, selection, coordination, design, and management of development interventions under the PFG strategy as compared to previous / other approaches) as well as goal and LOA level related questions (the latter in particular will be contextualized by the SME interviewer for the specific area of expertise of the interviewee at hand). Partnership for Growth El Salvador Mid-Term Evaluation Report 157 4.2 Semi-structured Interview Guides 1-5 Interview Guide 1 - PFG LEADERSHIP Background Information: The Partnership for Growth (PFG) aims to achieve accelerated, sustained, broad-based economic growth in partner countries, including El Salvador and the Philippines, through bilateral agreements between the United States Government (USG) and the partnering countries’ national governments. Using principles set forth in President Obama’s September 2010 Presidential Policy Directive on Global Development, the PFG requires rigorous, joint analyses of countries’ individual constraints to growth in order to develop joint action plans to address the most pressing of these constraints and to establish high-level mutual accountability for the goals and lines of action (LOAs) selected to alleviate them. This interview guide was designed to collect information on cross-cutting questions about the program. Respondent Type: The targeted respondents for this semi-structured Interview Guide # 1 are current and former USG and GOES officials who have held or hold leadership positions within the PFG initiative, particularly POCs and others at their level who will be more informed of the implementation of PFG. Central Focus of Questions: This guide includes (i) cross-cutting questions on the PFG Whole of Government Approach (WGA), changes in operational efficiency and work load, as well as on nonassistance; and (ii) El Salvador specific questions on the remedial capacities of the JCAP, on M&E issues, and on the mid-term performance of selected goals as related to the desired outcomes. The interviewer will note that the term Whole of Government Approach is not known to all parties, especially in El Salvador. Be prepared to probe with the terms inter-agency cooperation or inter-agency coordination. Methodology. Semi-structured interview. Approximately 1 hour. INTERVIEW QUESTIONS A. CROSS-CUTTING Advantages and disadvantages of the PFG approach in general 1. Based on your role as a Goal-Lead within the PFG initiative has the PFG approach resulted in changes in the way responsibilities and leaderships are shared or exercised within or among the USG agencies directly involved in your goal? If so, how? What are the principal advantages and disadvantages of these changes? 2. Has the PFG resulted in changes in the implementation coordination process between USG and GOES agencies? If yes, how? What are the main advantages and disadvantages of these changes? 3. Has the PFG process resulted in changes in levels or forms of funding allocation for the PFG goal and LOAs you are leading? Please explain. ___________________________________________________________________________________ Partnership for Growth El Salvador Mid-Term Evaluation Report 158 The following questions seek responses concerning the PFG process – in particular the Constraint Analysis (CA) and the Whole of Government Approach (WGA) – and whether these new approached have demonstrated improvements over pre-PFG assistance approaches. [Information obtained within this section will feed into CCQ1 (advantages/disadvantages of the PFG approach to development assistance), CCQ2 (PFG impact on workload, and CCQ3 (on the role of “non-assistance”). Regarding the WGA (to USG ONLY) The Whole of Government Approach (WGA) is relevant to identifying areas for assistance, selecting interventions, and determining implementation coordination. The approach reflects efforts to align each agency’s activities to achieve a common objective. 4. In your opinion, has the WGA led to change in the way the USG delivers development assistance in El Salvador? What kind of change? Please provide specific example(s). 5. In your opinion, compared to previous forms of development assistance, has the WGA approach in El Salvador led to: a. Change(s) in which areas for assistance and development initiatives are selected? (Please explain and/or provide example(s)) b. Change(s) in design of development initiatives? (Please explain and/or provide an example(s)) c. Change(s) in management of development initiatives? (Please explain and/or provide an example(s)) d. Change(s) in coordination of development initiatives? (Please explain and/or provide an example(s)) e. Change(s) in operational efficiency? (Please explain and/or provide example(s)) f. Change(s) in workload? (Please explain and/or provide example(s)) Regarding the WGA (to GOES ONLY): The Whole of Government Approach (WGA) is relevant to identifying areas for assistance, selecting interventions, and determining implementation coordination. The approach reflects efforts to align each agency’s activities to achieve a common objective, hence promoting inter-agency coordination and collaboration NOTE FOR THE INTERVIEWER: Interviewers should prompt with “interagency efforts/collaboration” since WGA is not a widely used term. 6. How is the WGA being implemented within the El Salvadoran Government? 7. In your opinion, compared to traditional forms of development assistance, has the WGA led to: a. Change in design of development initiatives? (Please explain and/or provide an example) b. Change in coordination of development initiatives? (Please explain and/or provide an example) c. Change in operational efficiency? (Please explain and/or provide example) Partnership for Growth El Salvador Mid-Term Evaluation Report 159 d. Change in workload? (Please explain and/or provide example) ________________________________________________________________________ On non-assistance (Both USG and GOES) “Non-assistance” tools include diplomatic engagement, convening authority, and other forms of nonmonetized assistance to engage both governmental and non-governmental stakeholders in support of catalytic policy change and development priorities. 8. What contribution has non-assistance made to the PFG process in El Salvador? Please provide specific examples. 9. How can non-assistance (within the context of El Salvador) be best utilized moving forward? EL SALVADOR - SPECIFIC QUESTIONS The following questions seek responses concerning whether the PFG has been developed in such a way to allow for the eventual determination of their impact on addressing the identified constraints and desired outcomes. This focuses on the Joint Country Action Plan (JCAP), implementation teams, and their work plans. [Information obtained will feed into CSQ2 (if JCAP goal level commitments are capable of achieving the constraint level objectives and goals]. Joint Country Action Plan (JCAP) As you know, the Constraints Analysis (CA) was centered on identifying the central binding constraints to growth, but did not identify remedies to address these. To address these remedies, the JCAP was produced. 10. What indications do you have that the JCAP is performing its central task of guiding the PFG to perform and move in the right direction? 11. To what extent are the goal level commitments set forth in the JCAP capable of achieving the constraints-level objectives and outcomes? 12. Are the goals and LOAs in the JCAP well defined remedies to overcome the constraints? 13. Were there additional goals and LOAs that you think should have been included in the Initiative that do not already exist? If yes, please list and explain. _____________________________________________________________________________________ Implementation Teams The M&E Addendum to the JCAP announced that “The governments of El Salvador and the United States have established implementation teams (e.g. Goal Leads, counterparts, Working Groups, Constraint Leads) that include representatives of each agency responsible for executing a LOA to Partnership for Growth El Salvador Mid-Term Evaluation Report 160 coordinate the JCAP process and ensure communication. The teams are to meet regularly to develop [various tasks].” 14. How and when were the implementation teams formed? 15. How do they operate? 16. How have they performed on the [various tasks] they are charged with? Work Plans The M&E Addendum to the JCAP also explains that each implementation team will develop “work plans” for each LOA. 17. To your knowledge, have the work plans referenced been developed? 18. When were the work plans developed (provide number that exist, if able)? Are they produced within a certain frequency? And are they uniform across participating agencies? 19. If they were not developed: why not? 20. What mechanism is used to measure performance of a specific LOA if no work plan was developed? ___________________________________________________________________________________ Evidence-Based Decision Making and Fact-Based Monitoring The following questions are in reference to PFG’s overarching goal of promoting evidence-based decision making and fact-based monitoring. [Information obtained will feed into CSQ2 (PFG emphasis on quantitative and objectively verifiable evidence feeding into decision making and fact-based monitoring]. 21. The PFG model places specific emphasis on [1] evidence-based decision making and [2] fact-based monitoring. a. Is quantitative and objectively verifiable information being used to manage JCAP implementation in order to achieve and measure results? Please explain how. b. How is evidence-based decision making part of managing PFG? (Please illustrate and/or provide an example). c. How is “fact-based monitoring” designed and managed under PFG? (Please illustrate and/or provide an example). 22. As stated in the M&E Addendum, the PFG is producing semi-annual scorecards, which will track LOA indicators. Partnership for Growth El Salvador Mid-Term Evaluation Report 161 a. How is consensus reached in determining which indicators are included within each goal, given that various agencies and two governments are involved in decision making? b. Can you identify a specific case in which consensus was lacking? Why was there a lack of consensus and how was an agreement attained? c. How are LOA-level indicators taken into account when preparing the scorecards? d. From your point of view, how do you assess the role of the PFG scorecards in monitoring performance and making sure the necessary outputs are produced to achieve the desired PFG objectives? e. Given that the PFG includes 153 LOAs, how do you identify under-performing LOAs, and what systems are used to assess their impact on outcomes? 23. The M&E Addendum also states that high level representatives of both governments will perform a yearly “general review of JCAP implementation” (each November from 2012 to 2016). a. What format did the yearly November reviews of 2012 and 2013 take? What information was reviewed? Who participated? b. Which indicators were reviewed to gauge progress towards successfully addressing the two constraints? How were the LOA level indicators taken into account for the 2012 and 2013 November reviews? c. Was there a common methodology for the 2013 and 2012 reviews? If changes were made to the review methodology, what were the changes and why were they made? d. Did the conclusions of the review lead to specific actions (e.g. to overcome an obstacle identified during the review)? If “yes”, what were these actions and how have they been enacted? 24. The M&E Addendum states that progress on the security constraint would be gauged in particular through 3 indicators (national homicide rate; public security perception; and prosecutions and convictions as a percentage of reported violent crimes). a. Was a written review of these indicators produced for the November 2012 and November 2013 reviews? If not, how was the review of these indicators performed? 25. Likewise, the M&E addendum states that progress on the tradables constraint would be gauged in particular through 3 indicators (per capita GDP growth; exports as a percentage of GDP; foreign direct investment as a percentage of GDP). a. Was a written review of these indicators produced for the November 2012 and November 2013 reviews? If not, how was the review of these indicators performed? ____________________________________________________________________________ Partnership for Growth El Salvador Mid-Term Evaluation Report 162 Being on target, course-corrections, and moving forward [Information obtained will feed into CSQ3 (if selected interventions are on target and creating the necessary outputs to achieve the desired outcomes)]. 26. Today, at the mid-term of implementation of the PFG approach, what evidence exists to demonstrate whether the overall El Salvador PFG performance is on target and creating the necessary outputs to achieve the desired outcomes? 27. For each of the two constraints, and for each of the selected goals [interviewer should name the goal that is applicable to the respondent, if respondent does not work with a specific goal, ask question in general], are the various interventions GOES and USG committed to in the JCAP on target? Provide examples. If not on target, can you share reasons why they are behind? 28. In practice, under each constraint, and for the selected goals [interviewer should name the goal that is applicable to the respondent, if respondent does not work with a specific goal, ask question in general], which M&E mechanisms are used to evaluate if interventions are on target or below target? 29. To what extent are the results of not only the goal level outputs, but the LOA level outputs (as committed to by USG or GOES under the JCAP) subject to periodic discussion among the PFG partners? Partnership for Growth El Salvador Mid-Term Evaluation Report 163 Interview Guide 2 – PFG ARCHITECTS Respondent Type: The targeted respondents for this semi-structured Interview Guide # 2 are stakeholders directly involved with the design and planning of the El Salvador PFG, or whose inputs were sought after for these processes. The stakeholders mainly include USG and GOES officials (current and former), members of the Growth Council, representatives from private sector and other civil society organizations, as well as independent experts. Central Focus of Questions: The focus of the interview questions are on the PFG overall objectives, the role that Whole of Government Approach (WGA); Constraint Analysis (CA), Joint County Action Plan (JCAP), non-assistance, monitoring and evaluation (M&E), and yearly performance reviews, play in the PFG performance. Methodology: Semi-structured interview. Approximately 1 hour. INTERVIEW QUESTIONS On the WGA: The Whole of Government Approach (WGA) is relevant to identifying areas for assistance, selecting interventions, and determining implementation coordination. The approach reflects efforts to align the activities of each agency in order to achieve a common objective” 1. Have any changes been realized with how the design of development assistance initiatives (particularly in El Salvador) has been approached as result of the initiation of the WGA approach? a. If yes, what are the changes? b. Please cite specific examples 2. Have there been distinctive differences between the PFG approach and other economic-growth development approaches? a. Please cite examples ___________________________________________________________________________________ On the JCAP The El Salvador CA was centered on identifying the central binding constraints to growth, but not on identifying remedies to address these. To address these remedies, the JCAP was produced. Is the JCAP fulfilling its role? 3. Are there any indication that the JCAP is leading towards the achievement of constraints-level objectives and outcomes? 4. Is there any evidence that the goal-level commitments set forth in the JCAP have been effective in achieving the constraints-level objectives and outcomes? _____________________________________________________________________________________ On non-assistance Partnership for Growth El Salvador Mid-Term Evaluation Report 164 “Non-assistance” tools include diplomatic engagement, convening authority, and other forms of nonmonetized assistance to engage both governmental and non-governmental stakeholders in support of catalytic policy change and development priorities. 5. What contribution (if any) has non-assistance made to the PFG process, in relation to El Salvador? 6. How can non-assistance (within the context of El Salvador) be best utilized moving forward? 7. How do you think PFG can best measure “non-assistance” and its contribution to reaching its overall objectives for the PFG initiative in El Salvador? _____________________________________________________________________________________ On evidence-based decision making and fact-based monitoring The PFG places specific emphasis on evidence-based decision making, fact-based monitoring, and quantitative verifiable information. 8. With the initiation of the PFG, have changes been realized in terms of improving monitoring systems? 9. How was evidence-based decision making designed for the PFG initiative? What mechanisms were included in the design to inform its appropriate implementation? Please illustrate and/or provide an example. 10. How is “fact-based monitoring” designed and managed under PFG? What mechanisms were included in the design to inform its appropriate implementation? (Please illustrate and/or provide an example). The M&E Addendum also states that high level representatives of both governments will perform a yearly “general review of JCAP implementation” (each November from 2012 to 2016). 11. What was envisioned to be the outcome of these yearly meetings? Please provide specific examples. 12. Do you know if these meetings have occurred and have the proposed outcomes been realized? _____________________________________________________________________________________ On the PFG Main Constraints Progress on the security constraint is gauged in particular by 3 indicators (national HOM rate; public security perception; prosecutions and convictions as a percentage (%) of reported violent crimes). 13. What was the rationale for choosing these three indicators among others? 14. Why were 3 indicators chosen, and not more / less? Partnership for Growth El Salvador Mid-Term Evaluation Report 165 Progress on the tradables constraint is being gauged in particular by 3 indicators (per capita GDP growth; exports as a percentage of GDP; foreign direct investment as a percentage of GDP). 15. What was the rationale for choosing these three indicators among others? 16. Why were 3 indicators chosen, and not more / less? _____________________________________________________________________________________ On the PFG at Mid-Term 17. Today, at mid-term, is there any evidence that the overall ES PFG performance is on target and creating the necessary outputs to achieve the desired outcomes? The CA identifies challenges including bureaucratic inefficiency, inconsistency of judicial decisions and in general comparatively low quality of public administration performance. 18. What risk do these pose for PFG performance, if any? 19. If there are risks, what mechanisms is the PFG using to diminish these risks? Partnership for Growth El Salvador Mid-Term Evaluation Report 166 Interview Guide 3 - PFG GOAL LEADS Respondent Type: The targeted respondents for this semi-structured Interview Guide # 3 are active and former GOES and USG Goal Leads who have directly worked on respective goal(s). Central Focus of Questions: This guide includes cross-cutting questions to gauge changes in the operational efficiency, selection, coordination, design, and management of development interventions under the PFG strategy as compared to previous / other approaches. Methodology: Semi-structured interview. Approximately 1 hour. Overall Note to Interviewer: Some of the Goal Leads are LOA Implementers, therefore there will be the need to ensure that repetitive questions are not asked. The evaluation coordination team will ensure that the appropriate guide is provided to the interviewer. INTERVIEW QUESTIONS Advantages and disadvantages of the PFG approach in general 1. Based on your role as a Goal-Lead within the PFG initiative, has the PFG approach resulted in changes in the way responsibilities and leaderships are shared or exercised within or among the USG agencies directly involved in your goal? If so, how? What are the principal advantages and disadvantages of these changes? 2. Has the PFG resulted in changes in the implementation coordination process between USG and GOES agencies? If yes, how? What are the main advantages and disadvantages of these changes? 3. Has the PFG process resulted in changes in levels or forms of funding allocation for the PFG goal and LOAs you are leading? Please explain. ___________________________________________________________________________________ The following questions seek responses concerning how the Whole of Government Approach (WGA) and Joint Country Action Plans (JCAP) have demonstrated improvements (or not) over pre-PFG assistance approaches Regarding the WGA: The Whole of Government Approach (WGA) is relevant to identifying areas for assistance, selecting interventions, and determining implementation coordination. The approach reflects efforts to align each agency’s activities to achieve a common objective, hence promoting inter-agency coordination and collaboration [NOTE FOR THE INTERVIEWER: Interviewers should prompt with "interagency efforts/collaboration" since WGA is not a widely used term.] 4. Are you aware of the WGA as described? 5. In your role as a Goal Lead, have you experienced how the WGA is being implemented within the PFG initiative? Please provide specific examples. Partnership for Growth El Salvador Mid-Term Evaluation Report 167 [NOTE FOR INTERVIEWER: If the interviewee is not aware of the WGA under his/her goal: skip c) and d)] 6. In your opinion, compared to traditional forms of development assistance, has the WGA led to: a. Change(s) in the design of development initiatives? (Please explain and/or provide an example) b. Change(s) in the coordination of development initiatives? (Please explain and/or provide an example) c. Change(s) in operational efficiency? (Please explain and/or provide example) d. Change(s) in workload? (Please explain and/or provide example) 7. Has the WGA impacted the performance of the activities you are directing as a Goal Lead? If yes, please explain and provide examples. ___________________________________________________________________________________ Changes in development approach due the introduction of the JCAP: As you know, while the Constraints Analysis identified the central binding constraints to growth, the Joint Country Action Plan (JCAP) defines the remedies to address these. 8. As a Goal Lead, how do you relate to the other JCAP goals, goal leaders, and the JCAP in general? 9. As a Goal Lead, do you consider that the JCAP is performing its central role in guiding the PFG to perform and move in the right direction? 10. For your goal, does the JCAP provide sufficient guidance on performance benchmarks for the LOA? ___________________________________________________________________________________ On non-assistance as an inherent part of the PFG: “Non-assistance” tools include diplomatic engagement, convening authority, and other forms of nonmonetized assistance to engage both governmental and non-governmental stakeholders in support of catalytic policy change and development priorities. 11. What role is non-assistance playing under your goal? Please provide an example. 12. Please provide examples of specific cases, e.g. of enhanced goodwill, access, receptivity, collaboration or additional or different resources (non-monetized ones, local level ones, etc.) 13. Are there any requirements/ instructions received from PFG leadership in identifying or documenting "non-assistance”? Please provide specific examples. 14. Are you able to measure progress on “non-assistance”, if yes, what types of indicators are typically used? Partnership for Growth El Salvador Mid-Term Evaluation Report 168 15. How can non-assistance be best utilized under your goal going forward? _____________________________________________________________________________________ On Workplans: The JCAP M&E addendum explains that each implementation team will develop a “workplan” for each LOA. 16. For each of the LOAs under your goal, was a workplan developed? If no, why not? If yes, who developed the workplan? 17. For each LOA under your goal, does the workplan include any of the following to promoted measurability: a. Indicators; baselines; benchmarks. b. Were these developed by the implementation team? If not by the implementation team, how were they established? 18. How do you measure performance of a specific LOA if no workplan was developed? __________________________________________________________________________________ On evidence-based decision making and fact based-monitoring: As you know, the PFG model places specific emphasis on [1] evidence-based decision making and [2] fact-based monitoring. 19. For your goal, how do you use quantitative and objectively verifiable information to manage implementation in order to achieve and measure results? 20. For your goal, please provide examples of evidence-based decision making? What role (if any) does “quantitative verifiable information” play in this decision making? 21. Please explain how “fact-based monitoring” is an integral part of your goal implementation? What role (if any) does “quantitative verifiable information” have in this? 22. Is there a specific M&E plan for your goal as a whole? Is there an M&E plan for each of the LOAs under your goal? When and how were these formulated? 23. Do you use a PFG-issued or a goal-specific M&E indicator system? How do the LOA feed into this? How does your system feed into the PFG system in general? The PFG produces semi-annual scorecards per goal. LOA level work plan generated indicators are meant to feed into these: 24. For your goal, how is consensus reached on the scorecard, given that various agencies and two governments are involved? If there was lack of consensus, how was it overcome? Partnership for Growth El Salvador Mid-Term Evaluation Report 169 25. For you goal, please explain how the LOA level (work plan) indicators fed into the scorecards? Progress on the security constraint is gauged in particular from 3 indicators (national HOM rate; public security perception; prosecutions and convictions as a percentage (%) of reported violent crimes). [Note to Interviewer: only ask for goal leads working in the crime and insecurity constraint; Note that you might not receive very much input on these, and should skip over quickly, if you do not] 26. To what extent do these three indicators reflect performance under your goal? 27. If any, which other indicator would you like to see included, as related to your own goal? Progress on the tradables constraint is gauged in particular by 3 indicators (per capita GDP growth; exports as a percentage of GDP; foreign direct investment as a percentage of GDP). [Note to Interviewer: only ask for goal leads working in the low productivity and tradables constraint; Note that you might not receive very much input on these, and should skip over quickly, if you do not] 28. To what extent do these three indicators reflect performance under your goal? 29. If any, which other indicator would you like to see included, as related to your own goal? On being on-target and creating the necessary outputs to achieve the desired outcomes: 30. At mid-term of PFG implementation, is there any evidence that the overall ES PFG performance is on target and creating the necessary outputs to achieve the desired outcomes? Please provide specific example(s). 31. Is your goal(s) on target (or behind target)? Which M&E mechanisms are used to evaluate if goal(s) are on target (or behind target), beyond the scorecards? 32. Have there been any major changes to how the PFG approach is implemented, specific to your goal(s)? If yes, what are they? And why have they been instituted? 33. Please provide examples of successes made and challenges faced with implementing your goal(s). 34. In what way do you coordinate with LOA(s) implementers within your goal to ensure that the performance of your goal is on target? Partnership for Growth El Salvador Mid-Term Evaluation Report 170 Interview Guide No. 4 - LOA IMPLEMENTERS (only for selected goals) Respondent Type: The targeted respondents for this semi-structured Interview Guide # 4 are the implementers of all or selected lines of action (LOA) under the selected goals. The interview will be performed primarily with the chiefs of party, directors, and/or coordinators or their representatives. Central Focus of Questions: The guide includes questions regarding the performance of PFG, the monitoring of activities, evidence based decision making, non-assistance, and beneficiaries. Methodology: Semi-structured interview with COP / director / coordinator. Given the detailed questioning, the COP will most likely be joined by team members, or part of the interview will be realized with team members directly. About 1 hour (per LOA team). Overall Note to Interviewer: Some of the Goal Leads are LOA Implementers, therefore there will be the need to ensure that repetitive questions are not asked. The evaluation coordination team will ensure that the appropriate guide is provided to the interviewer. INTERVIEW QUESTIONS The following questions seek responses concerning whether the PFG has been developed in such a way as to allow for the eventual determination of their impact on addressing the identified constraints and desired outcomes. This focuses on the JCAP, implementation teams, and their workplans. On the JCAP 1. To what extent are the activities you implement guided by the goal-level commitments set forth in the JCAP? 2. What indications do you have that the LOA(s) you and your team are implementing, contribute to the corresponding goal as established in the JCAP? On implementation teams The M&E addendum to the JCAP explains that implementation teams would be formed, to include representatives of each agency responsible for executing an LOA. These implementation teams would help coordinate the JCAP process and ensure communication. The implementation teams would meet regularly to develop various tasks. [These tasks are specified in the M&E addendum]. 3. For your LOA, does such an implementation team exist? [Noted to Interviewer: If NO, please go on to 2.3] 4. If yes, when and how was it formed? Who is included in the implementation team? 5. For your LOA, how has the implementation team been operating and what is their performance? On Workplans: Partnership for Growth El Salvador Mid-Term Evaluation Report 171 The JCAP M&E addendum explains that each implementation team will develop a workplan for each LOA. 6. For your LOA, when and how was a workplan developed? What role has the implementation team played in this? 7. For your LOA, does the workplan include any of the following to promoted measurability: a. Indicators; baselines; benchmarks. b. Were these developed by the implementation team? If not by the implementation team, how were they established? 8. If there are no workplans for your LOA, how is progress measured? ________________________________________________________________________________________ The following questions seek responses concerning evidence based decision-making and fact based-monitoring. As you know, the PFG model places specific emphasis on [1] evidence based decision making and [2] fact-based monitoring. 9. Does your LOA have a specific M&E plan? When and how were these formulated? 10. How is progress measured for your LOA? Do you have set indicators for measuring progress? Please explain and provide specific examples. 11. Have any changes been made to your LOA targets, if yes, what are these and why were the changes made? The PFG is producing semi-annual scorecards per goal. LOA-level workplan-generated indicators are meant to feed into these. 12. How do your activities feed into the scorecards? The PDF performs a yearly “general review of JCAP implementation” (November 2012, 2013) 13. Are you aware of these annual meetings? Did you participate in the 2012 and/or 2013 reviews? 14. Were indicators for your LOA included in these annual reviews? If yes, what role did you play? 15. Did the conclusions of the review induce changes for your LOA? Please explain. _____________________________________________________________________________________ The following questions seek responses concerning beneficiaries: Partnership for Growth El Salvador Mid-Term Evaluation Report 172 16. How are the beneficiaries of your activity defined? 17. How do you monitor and evaluate impact among them? 18. Is the monitoring strategy defined generally by a PFG methodology or does each LOA have its own specific methodology? 19. Is gender equality among beneficiaries considered as a measure? If yes, how are you working towards attaining this measure? And how is gender equality measured with the LOA? Partnership for Growth El Salvador Mid-Term Evaluation Report 173 Interview Guide No. 5 – INDEPENDENT EXPERTS Respondent Type: The targeted respondents for this semi-structured Interview Guide # 5 are independent experts. Experts include academics, subject matter experts, journalists and others who contribute to public debate on the PFG in general or specific areas of the PFG, but are not responsible for directing or implementing components of the PFG. The guide includes cross-cutting questions (to gauge changes in the operational efficiency, selection, coordination, design, and management of development interventions under the PFG strategy as compared to previous / other approaches) as well as goal and LOA level related questions (the latter in particular will be contextualized by the SME interviewer for the specific area of expertise of the interviewee at hand). Methodology: Semi-structured interview with independent experts, approximately 1 hour. INTERVIEW QUESTIONS Advantages and disadvantages of the PFG-WGA approach: The Whole of Government Approach (WGA) is relevant to identifying areas for assistance, selecting interventions, and determining implementation coordination. The approach reflects efforts to align each agency’s activities to achieve a common objective [Note to Interviewer: it is likely that the independent experts will not know too much about WGA, so when analyzing be careful to denote if there was confusion with the response.] 1. From your point of view, has the PFG WGA approach in El Salvador led to change coordination between the GOES and the USG on selecting, planning and implementing growth-oriented development programs? If yes, what changes in leadership, coordination and distribution of responsibilities have you observe? Please provide specific examples. 2. What are the principal advantages and disadvantages of the PFG WGA approach to development? The role of “non-assistance” under the PFG: “Non-assistance” tools include diplomatic engagement, convening authority, and other forms of nonmonetized assistance to engage both governmental and non-governmental stakeholders in support of catalytic policy change and development priorities. 3. In your opinion (if you are aware of this concept), what contribution has non-assistance made to the PFG process in El Salvador? Please provide specific examples. 4. How do you think non-assistance has contributed to the PFG initiative in El Salvador? 5. How do you think PFG can best measure “non-assistance” and its contribution to reaching its overall objectives for the PFG initiative in El Salvador? Constraint selection and performance Partnership for Growth El Salvador Mid-Term Evaluation Report 174 6. Are you know, the PFG initiative in El Salvador selected two constraints – crime/ insecurity and low productivity in tradables constraints. Within these constraints, 20 goals and 153 LOAs have been created to address the constraints. Based on your observations of the PFG initiative in El Salvador, what is your opinion on the effectiveness of the program in responding to these constraints? Please provide specific examples. 7. Would you have chosen other goals or LOAs to address these constraints? If yes, please mention and explain why. The use of quantitative, objectively and verifiable information to achieve and measure results: 8. How do the PFG performance indicators and its M&E methodology compare to practices used in pre-PFG approaches? 9. In your opinion or within your expertise, how relevant, objective and verifiable are the quantitative indicators the PFG is using? Please provide specific examples if known. On JCAP Goal-Level Commitments As you know, the Constraints Analysis (CA) was centered on identifying the central binding constraints to growth, but did not identify remedies to address these. To address these remedies, the JCAP was produced. 10. In your opinion, to what extent are the goal-level commitments set forth in the JCAP capable of achieving the constraints-level objectives and outcomes? 11. Do you think the goals and LOAs in the JCAP are well defined remedies to overcome the constraints? Please explain why. 12. Were there additional goals and LOAs that you think should have been included in the Initiative that do not already exist? If yes, please list and explain. On being on target and creating the necessary outputs to achieve the desired outcomes: 13. From your point of view, how do you assess the role of the PFG scorecards in monitoring performance and making sure the necessary outputs are produced to achieve the desired PFG objectives? 14. Which indicators or measuring instruments other than the scorecards are best suited to clarify the relation between development activities, goal-level commitments and constraint level objectives and outcomes? Partnership for Growth El Salvador Mid-Term Evaluation Report 175 4.3 Confidential Online Survey Partnership for Growth Government Agency Survey Page 1 of 14 Dear respondent, You are receiving this questionnaire because of your general knowledge and/or affiliation with the Partnership for Growth program (PFG) between the Governments of El Salvador and the United States of America, and the Philippines and the United States. This is a confidential survey and your identity will be known only to the evaluation team and will not be shared. All survey responses are treated by Optimal Solutions Group, LLC in strict confidentiality. Individual responses will not be reported or made public, except to the extent required by law. This is to ensure that your responses can be as frank as possible, without concern for the possible sensitivities of any other parties. It is a brief questionnaire that should take less than 20 minutes to complete. Your participation is absolutely voluntary. If you wish not to answer a question, simply skip it and move to the next one. By participating in this survey you are giving your informed consent. The confidential information you provide will be invaluable to the successful conduct of the PFG evaluation. Please complete the survey no later than May 12, 2014. If you have any questions or issues please contact Optimal at [email protected]. Thank you in advance for your cooperation. Partnership for Growth Government Agency Survey Page 2 of 14 The Partnership for Growth (PFG), was initiated in 2011 through bilateral agreements between the United States Government (USG) and partnering countries’ national governments (El Salvador and the Philippines) with the aim of achieving accelerated, sustained, broad-based economic growth in partner countries. The PFG requires the identification of countries’ constraints to growth in order to develop a joint plan to address the most pressing of these constraints. It also requires transparency, mutual accountability and fact-based monitoring and evaluation. The following questions request information on the PFG’s ability to meet these goals. Partnership for Growth El Salvador Mid-Term Evaluation Report 176 This portion of the survey asks questions concerning your assignment and workload on PFG. Q1. To the best of your recollection, when did you begin work on PFG? More than 2 years ago Between 1 and 2 years ago One year ago or less Q2. For your agency/ institution, did you have a role in the planning and development of PFG? Yes No Partnership for Growth El Salvador Mid-Term Evaluation Report 177 Partnership for Growth Government Agency Survey Agency Survey Page 3 of 14 Q3. For approximately how many weeks were you involved in the planning and development of PFG? (please enter a non-negative, numeric value only) Q4. During the PFG planning and development stages, approximately how many hours per week, on average, did you dedicate to these tasks? Zero 1 to 5 hours 6 to 10 hours 11 to 15 hours 16 to 20 hours More than 20 hours per week Partnership for Growth El Salvador Mid-Term Evaluation Report 178 Partnership for Growth Government Agency Survey Agency Survey Page 4 of 14 Q5. What is or was your specific PFG assignment? (Select All that Apply) Leadership Goal lead Project management Project implementation Monitoring and Evaluation Other Q6. Do you have experience planning, implementing or monitoring development projects outside of PFG? Yes No Partnership for Growth El Salvador Mid-Term Evaluation Report 179 Partnership for Growth Government Agency Survey Agency Survey Page 5 of 14 Q7. As a result of your involvement with PFG, has/did your workload Increased significantly Increased somewhat Stayed about the same Decreased somewhat Decreased significantly Q8. On average, about how much time per week do/did your PFG responsibilities require? Zero 1 to 5 hours 6 to 10 hours 11 to 15 hours 16 to 20 hours More than 20 hours per week Partnership for Growth El Salvador Mid-Term Evaluation Report 180 Partnership for Growth Government Agency Survey Agency Survey Page 6 of 14 Q9. As a result of your involvement with PFG, for each of the tasks in the table, has/did your workload… Stay Increase Increase about Decrease Decrease significantly somewhat the somewhat significantly same PFG task coordination with colleagues within my government PFG task coordination with colleagues in other (partner) governments Monitoring progress (indicators, site visits, milestones) of PFG tasks Communicating on PFG with my superiors and senior leadership in my government Managing PFG activities Designing and/or procuring PFG activities Other administrative tasks Partnership for Growth El Salvador Mid-Term Evaluation Report 181 Partnership for Growth Government Agency Survey Agency Survey Page 7 of 14 Q10. On average, about how many hours per week are/were dedicated to the PFG tasks in the table? (please enter non-negative, numeric values only) Average Hours per Week PFG task coordination with colleagues within my government PFG task coordination with colleagues in other (partner) governments Monitoring progress (indicators, site visits, milestones) of PFG tasks Communicating on PFG with my superiors and senior leadership in my government Managing PFG activities Designing and/or procuring PFG activities Other administrative tasks Partnership for Growth El Salvador Mid-Term Evaluation Report 182 Partnership for Growth Government Agency Survey Agency Survey Page 8 of 14 We would now like to ask you a few brief questions about your perceptions of the PFG approach. Q11. In your opinion, compared to other approaches to development assistance intended to affect economic growth, does the PFG represent A significant improvement An improvement No change A step backwards A significant step backwards Don’t know Q12. In your opinion, is PFG meeting its goal of advancing economic growth in El Salvador? Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Don’t know Partnership for Growth El Salvador Mid-Term Evaluation Report 183 Partnership for Growth Government Agency Survey Agency Survey Page 9 of 14 Q13. One of PFG’s goals is to employ “non-assistance” development tools. “Nonassistance” tools include diplomatic engagement, convening authority, and other forms of non-monetized assistance to engage both governmental and non-governmental stakeholders in support of catalytic policy change and development priorities. Have you seen nonassistance tools being used in the PFG activity you are or were involved with? Yes No Not sure Partnership for Growth Government Agency Survey Agency Survey Page 10 of 14 Q14. Can you briefly describe the non-assistance tools you have seen, and how they were used? Example no. 1: Example no. 2: Example no. 3: Partnership for Growth Government Agency Survey Agency Survey Partnership for Growth El Salvador Mid-Term Evaluation Report 184 Page 11 of 14 Q15. In your opinion, are the appropriate indicators being used to allow for transparent, accountable and fact-based monitoring of the PFG? The best available indicators are being used Some of the best available indicators are being used The best available indicators are not being used Not sure Partnership for Growth Government Agency Survey Agency Survey Page 12 of 14 Q16. Can you provide some examples of alternative indicators to allow for transparent, accountable, fact-based monitoring of the PFG? Example no. 1: Example no. 2: Example no. 3: Partnership for Growth El Salvador Mid-Term Evaluation Report 185 Partnership for Growth Government Agency Survey Agency Survey Page 13 of 14 Q18. In your opinion, what are the main strengths of PFG program? Q19. In your opinion, what are the main weaknesses of PFG program? Survey Completed Thank you for your valuable contribution to the PFG evaluation! Partnership for Growth El Salvador Mid-Term Evaluation Report 186 ANNEX 5 - TABLE OF PFG GOALS AND LINES OF ACTION 55 CRIME AND INSECURITY CONSTRAINT GOAL Goals 1 & 2: Professionalize justice sector institutions and improve criminal justice practices and procedures to make them more effective in combating crime and insecurity in El Salvador, as well as enhance the public perception of these government institutions. Goal 3: Reduce the impact of organized crime on small and medium businesses, potentially the most dynamic sector of the economy whose contribution to growth is key to the economic wellbeing of El Salvador. Goal 4: Facilitate economic growth by ensuring El Salvador’s labor force is protected from crime while 55 LOA GOES LOA1: Implement the necessary changes in practices, policies, regulations, and applicable laws as identified throughout the project. LOA 2: Identify and make available appropriate staff to be trained. LOA 3: Emphasize continuous vetting as a fundamental principle of the Security Plan of the Security Cabinet of the Government of El Salvador. LOA 4: Fully participate in programs to support the justice sector. LOA 5: Fully participate in analysis of the current codes. USG LOA 6: Provide technical assistance to improve the management and investigative capacity of the Attorney General (AGO), Public Defender (PGR), and National Civilian Police (PNC), forensic services, judges and court personnel. LOA 7: Support the development of effective case management models; improving police/prosecutor coordination; providing better and more equitable access to justice; re-engineering processes and change management procedures in the common crime unit; establishing career paths and leadership development within the Judicial sector; increasing accountability and transparency in the sector by enhancing judicial oversight and investigative capabilities; supporting civil society; and strengthening crime observatories. LOA 8: Provide assistance to promote increased coordination between justice sector actors and institutions; improve management and administration of justice sector institutions; and increase effectiveness of criminal investigation. LOA 9: Provide technical support in the areas of the pre-trial elements listed above as well as plea-bargaining (agreements with the public prosecutor’s office). LOA 10: Provide technical assistance to conduct an in-depth code analysis, draft legislative fixes, and develop strategies to enhance criminal justice efficiency and effectiveness relating to criminal procedure, an evidence code, and internationally accepted law enforcement tools; and implementing such legislation, procedures, and strategies. GOES LOA 1: Maintain a permanent exchange regarding the employment status of special units and personnel trained and advised in accordance with Salvadoran procedures and regulations. LOA 2: Improve supervision and control in accordance with the system of indicators of the Security Cabinet. LOA 3: Support the implementation of necessary changes in practices and regulations and promote the approval of the necessary legislation. LOA 4: Provide personnel, equipment, and facilities. USG LOA 5: Provide technical assistance, training, and mentorship for the units used to combat and prosecute crimes against businesses; and facilitate engagement between GOES and private sector. GOES LOA 1: Maintain a permanent exchange regarding the employment status of USG vetted or USG trained and advised personnel in accordance with Salvadoran procedures and regulations. Wording is taken directly from the Joint Country Action Plan (JCAP). Partnership for Growth El Salvador Mid-Term Evaluation Report 187 CRIME AND INSECURITY CONSTRAINT GOAL transiting to and from work, and ensuring that the public transportation service providers serving the labor force are protected from crime. Goal 5: Remove assets from criminal organizations and fund and support security programs through the use of seized property and assets confidence in the government. Goal 6: Professionalize El Salvador’s civil service and enhance public confidence in the government. Goal 7: Promote a national dialogue on actions to improve citizen security in El Salvador. Actively involve all sectors of national life, including the private sector, the media, nongovernmental organizations, churches, etc. in efforts to solve the problem of insecurity. LOA LOA 2: Improve supervision and control in accordance with the system of indicators of the Security Cabinet. LOA 3: In collaboration with the USG, review the current regulations of the public transportation sector, and identify areas to improve transparency and accountability. LOA 4: Implement necessary changes in practices and regulations and encourage the approval of the necessary legislation. LOA 5: Provide personnel, equipment, and facilities to support this program. USG LOA 6: Provide technical assistance, training, and mentorship for the vetted units to combat crimes involving public transit, and facilitate cooperation between GOES and private sector. LOA 7: Provide technical assistance to help increase the transparency and accountability of the public transport system. GOES LOA 1: Submit and encourage the approval of necessary legislation in support of this goal. USG LOA 2: Designate a Financial Crimes advisor in El Salvador on an intermittent basis to work with the GOES financial regulation authorities, the Attorney General’s Office; and the Financial Intelligence Unit. LOA 3: Provide technical assistance to conduct an analysis of current laws, draft and implement legislative fixes, and develop strategies to enhance criminal justice efficiency and effectiveness relating to money laundering and comprehensive asset forfeiture, including the appropriate management and disposal of forfeited assets. GOES LOA 1: Facilitate a discussion of the reforms with relevant sectors of the Salvadoran society through public and closed hearings. LOA 2: Provide the necessary resources to implement the new law. LOA 3: Select the members of the Access to Information Institute in a meritbased fashion, in accordance with established law and regulations. LOA 4: Provide adequate budgetary resources for the functioning of the Institute of Access to Information, and the Access to Information Units. LOA 5: Ensure merit-based hiring of personnel within the framework of the proposal to modernize the State, a plan under the responsibility of the Secretary for Strategic Affairs, through the Sub-secretary for Governability and Modernization. USG LOA 6: Provide technical assistance to promote and facilitate comprehensive civil service reform. LOA 7: Support GOES in increasing responsiveness to accountability towards its citizens through increasing transparency in government and ethical behavior by public officials. GOES LOA 1: Pursue the relations necessary to secure commitments at all levels in the fight against organized crime. LOA 2: Report periodically on progress in security plans and investments in security. USG LOA 3: Promote El Salvador’s positive national dialogue related to improving citizen security and understand the roles of all stakeholders in making a more secure society. Partnership for Growth El Salvador Mid-Term Evaluation Report 188 CRIME AND INSECURITY CONSTRAINT GOAL Goal 8: Assist at-risk youth between ages 16-25 through efforts to afford them economic opportunities and engage them in productive activities. Goal 9: Support the PNC to strengthen its service orientation as a means for violence prevention and effective crime control with a focus on building leadership skills within the police force and on improved relationships between police and communities. Goal 10: Improve educational opportunities for in school and out of school youth in targeted high risk municipalities with high crime rates. The USG is dedicated to supporting the Ministry of Education in implementation of their “Social Education Plan” and the GOES’ “Five Year Plan” by focusing efforts on the four areas of concern described in the plan. Goal 11: Prevent crime and violence in key municipalities of El Salvador and support reforms, as outlined in components 2 (Social Prevention of Violence and Crime) and 5 LOA GOES LOA 1: Find beneficial participants and make efforts to identify promising job and education opportunities for the graduates of the training program. The Ministry of Labor is dedicated to ensuring that their national policy for youth employment encourages the employment of these individuals, and that the National Commission of Micro and Small Enterprises takes steps to facilitate the pursuit of beneficial self-employment opportunities. LOA 2: Promote the establishment of a strategic partnership with the private sector to generate employment opportunities for at-risk youth and youth in the process of rehabilitation. USG LOA 3: Create a training program including vocational courses, remedial education, life skills and job skills training, community service, internships and job placement directed to at-risk youth. The US Government is dedicated to working with NGOs and local educational institutions in the identification of technical/vocational programs to be offered for at-risk youth. LOA 4: Engage the private sector and local academic institutions to offer greater workforce training and vocational programs to better align the labor force’s skills with current market demands. Engage with the private sector to promote greater involvement by the business community in educational programs that target at-risk youths. GOES LOA 1: Demonstrate political will at the senior and mid-levels of the police to adopt community policing practices. LOA 2: Continue to encourage implementation of the Ley de la Carrera Policial (Police Career Law). USG LOA 3: Targeted technical assistance to the PNC in adopting a forcewide community-based policing approach that is already being piloted in the country and championed by the PNC. LOA 4: Training venues for the aforementioned force-wide communitybased policing approach, sponsor regional officer exchanges to promote a regional approach to community-based policing and identification and implementation of best practices. GOES LOA 1: Ensure participation of teachers, administrators, students, and parents. Sustain the leadership and commitment of the Ministry of Education to implement the full-time inclusive schools strategy. USG LOA 2: Provide technical assistance; training for school principals, teachers, community, students in safe schools and full time inclusive schools; institutional strengthening of local implementing partners; sharing evidencebased practices in safe schools; monitoring and evaluation. GOES LOA 1: Identify dedicated staff to develop and support municipal crime prevention councils. LOA 2: Identify dedicated funding to implement crime prevention plans at municipal level. LOA 3: Promote the decentralization of authority and responsibility for preventing crime to the municipalities. Partnership for Growth El Salvador Mid-Term Evaluation Report 189 CRIME AND INSECURITY CONSTRAINT GOAL (Institutional and Legal Reform) of the National Policy for Justice, Public Safety and Violence Prevention. Goal 12: Reduce overcrowding in prisons, thereby allowing the Salvadoran prison system to safely, securely, and humanely manage an increasing population. Goal 13: Enhance the security of the prisons for their improvement as correctional facilities, prevent them from perpetuating and magnifying criminal activity in El Salvador, and help former offenders become full, contributing members of society. Goal 14: Promote the use of extraditions as a deterrent for crime and a means to reinforce national security. LOA LOA 4: Improved coordination among line ministries with a stake, resources and mandate for addressing key risk factors (lack of access to education, and employment opportunities). LOA 5: Strengthen the cooperation between the government security organizations and said communities. LOA 6: Implement vocational training for at-risk youth. USG LOA 7: Provide technical assistance to promote and facilitate components 2 and 5 of the strategy. LOA 8: Region-wide study of the issue of violence and security as it impacts community groups in the hemisphere to bring the community-level experience from the region to bear in El Salvador. GOES LOA 1: Improve the prisoner classification system. LOA 2: Build a new prison facility and three prison farms, and implement an aggressive reintegration program. USG LOA 3: Provide technical assistance in the management of prisons/corrections, including mentoring and limited training in order to develop prison/correction officer train-the-trainer program, and develop and implement prison classification system. LOA 4: Explore ways to work with U.S. state/local institutions, the Governments of Mexico, Colombia, and U.S. interagency actors. LOA 5: Provide technical support in parole systems and prisoner classification systems. GOES LOA 1: Improve the recruitment, training, management, and monitoring of prison personnel to ensure integrity and effectiveness within the prison system. LOA 2: Continue the “Zero Corruption” program. LOA 3: Strengthen the Penitentiary School. LOA 4: Strengthen the penitentiary information system. USG LOA 5: Continue to make available the equipment, training and expertise needed to enhance security in the prisons, to include recommendations on technology that will aid in security. GOES LOA 1: Declare that extradition is an important tool in the fight against violent crime, organized crime, and narcotics trafficking. LOA 2: Taking into account the outcomes of bilateral consultations, consider ways in which the bilateral extradition relationship can be enhanced. LOA 3: Examine existing extradition legislation, and consider legislative changes, with a view to enhancing the effectiveness and efficiency of the extradition process. LOA 4: If necessary, encourage subsequent consultations concerning the need for a new Extradition Treaty. USG LOA 5: Technical assistance to draft legislative fixes, develop strategies to enhance criminal justice efficiency and effectiveness relating to effective laws and procedures regarding extradition; and implementing such legislation, procedures, and strategies. Partnership for Growth El Salvador Mid-Term Evaluation Report 190 LOW PRODUCTIVITY IN TRADABLES CONSTRAINT Sub-constraint: Loosening Constraints on Capital Formation GOAL Goal 1: Facilitate the establishment of a Growth Council to promote an environment of trust and improve the business climate (as measured by the Doing Business indicators) and investments in activities or sectors regarded as strategic. The goal in this area is that, by the end of the PFG, the GOES and the private sector will have established a relationship based on trust, understanding and clarity, in which private investment can have the greatest impact, spurring inclusive economic growth and improving social conditions in El Salvador. LOA GOES LOA 1: Officially establish a Growth Council at the national level made up of five representatives from the business community and five GOES officials, which will consult with the President on a quarterly basis. The objective of the Growth Council is to remove bottlenecks to growth by facilitating a climate of trust, communication, and collaboration between the two sectors; to discuss challenges and develop options for improving productivity; to monitor, evaluate, and report on PFG progress. The Growth Council will publish annually the results of diagnostic work using a series of indicators on bottlenecks to private sector investment. LOA 2: Respond in writing to formal recommendations of the Growth Council within a reasonable time. LOA 3: Ensure that the Growth Council consults on a regular basis with El Salvador’s Economic and Social Council (CES), to better inform both bodies of the work and successes of each. LOA 4: Use good offices in support of the creation and functioning of municipal-level councils aimed at encouraging competitiveness and growth. LOA 5: Launch a national program of women producers and entrepreneurs. LOA 6: Improve the civil service through targeted civil service reform of those areas identified by the Growth Council as bottlenecks to competitiveness in tradables and continue to pursue legislation aimed at broad-based civil service reform. LOA 7: Pursue efforts to address shortcomings identified with the World Bank/IFC on the Doing Business Indicators. LOA 8: Continue to research the challenges of productivity of the tradables sector. USG LOA 9: Use its good offices to ensure that the relationships between all segments of the private sector, domestic and international, and the GOES transpire in a constructive and complimentary atmosphere. LOA 10: Provide technical assistance in establishing the Salvadoran Growth Council, municipal councils and share best practices from the U.S. Council on Competitiveness. LOA 11: Promote actions and measures to improve “doing business.” LOA 12: Support the GOES in identifying new strategies for improving the investment environment and developing operations in sectors and activities that have been identified as priority for the growth and development of the Salvadoran economy. LOA 13: Provide technical assistance in continued research into the challenges of productivity of the tradables sector Sub-constraint: Loosening Constraints on Capital Formation GOAL Goal 2: Reduce firms’ costs due to infrastructure to improve their competitiveness. LOA GOES LOA 1: Submit the draft Public-Private Partnership bill, work towards its enactment, and secure the institutional strengthening required for its implementation. LOA 2: Concession of the specialized multi-terminal Container Port, Stage 1, of the Central American Port of La Union. LOA 3: Support the development of areas adjacent to the La Union Port facilities LOA 4: Modernize and expand El Salvador’s International Airport and seek Partnership for Growth El Salvador Mid-Term Evaluation Report 191 LOW PRODUCTIVITY IN TRADABLES CONSTRAINT out the best public-private participation options. LOA 5: Promote a regulatory framework for renewable energy (MRER by its Spanish acronym). USG LOA 6: Provide technical support and assistance for public-private partnership legislation and project management. LOA 7: Strengthen the public-private partnership management capacity at PROESA. LOA 8: Provide technical assistance to support CEPA (Port Authority Executive Commission) in its effort to successfully complete a thirty-year operating concession of the Port of La Unión. LOA 9: Back the La Union port concession by supporting GOES efforts to attract world class private operators. LOA 10: Explore financing options for additional investments to upgrade the La Union port. LOA 11: Provide technical support for air freight and airport infrastructure development once the airport has been concessioned. LOA 12: Provide technical assistance in developing renewable energy LOA 13: Support knowledge transfer on capabilities for generating renewable energy. Sub-constraint: Loosening Constraints on Capital Formation GOAL Goal 3: Improve the quality of the education system in order to create a more highly qualified and technologically skilled labor force. The Governments of El Salvador and the United States will join forces to help ensure that education of the labor supply matches labor market demand, and they pledge to carry out the actions outlined below. The success of these actions will facilitate a cumulative investment in human capital that will boost the productivity of labor and of the tradables sectors. LOA GOES LOA 1: Develop an education and employment plan for youth and women. LOA 2: Commit to making transformational reforms necessary to improve the quality of El Salvador's educational system. LOA 3: Create a talent network of Salvadorans living abroad. LOA 4: Support youth insertion in the labor market. LOA 5: Improve basic computer training programs for entry level positions and for employees. LOA 6: Improve English for the workplace programs for entry level jobs and for employees. LOA 7: Strengthen and enhance productive diversification through scientific and technological innovation. LOA 8: Focus on innovation and technological development in priority tradables sectors for the Salvadoran economy (e.g.: agro-foods, marine resource development, and development of the coastal-marine belt). LOA 9: Design and create a program for the transfer and assimilation of ICT through public-private partnerships. LOA 10: Strengthen INSAFORP (Salvadoran Institute for Professional Training) to orient their training programs toward strategic activities and identify synergies between separate initiatives for vocational and technical training. LOA 11: Improve capacity to collect, analyze, use and disseminate labor market data in order to correct mismatches between labor supply and demand. LOA 12: Use better data and analysis to project labor market demand in key tradables sectors, and transform vocational training programs. LOA 13: Develop a youth scholarship program. USG LOA 14: Support teacher training in English. LOA 15: Provide technical support to build the work skills of young people ages 16 - 25. (Access to work program and training of at-risk youth). Partnership for Growth El Salvador Mid-Term Evaluation Report 192 LOW PRODUCTIVITY IN TRADABLES CONSTRAINT LOA 16: In consultation with the GOES, develop alliances, provide technical skills and language training assistance and develop alliances to improve access to employment for youth in El Salvador’s tradables sector. LOA 17: Technical assistance to improve labor market information systems to better match employers’ needs and employee skills. LOA 18: Help GOES build the capacity to conduct the surveys and analyze data needed to match labor supply with present and future labor market demand. LOA 19: Help GOES improve services for job seekers, including through career counseling, web-based labor market information tools, and partnerships between industry and educational institutions. LOA 20: Work with the GOES on efforts to improve tertiary training and higher educational quality linked to workforce needs, gaps, and opportunities. LOA 21: Technical assistance to focus on innovation and technological development in priority tradables sectors. LOA 22: Support the development of a youth scholarship program. LOA 23: Support implementation of a talent network with Salvadorans living abroad. LOA 24: Assist GOES in launching a dialogue between government, business and labor on strategies to encourage the creation of good jobs and improve productivity and competitiveness. GOES LOA 1: Strengthen institutional capacity to conduct market studies. LOA 2: Encourage investments from Salvadorans residing in the United States. LOA 3: Strengthen, finance and support the Ministry of the Economy and PROESA to implement the proposed measures. LOA 4: Continue ongoing efforts to develop and implement the Integrated Investor Attention System (SIAI). LOA 5: Develop and implement a strategy to address the causes and barriers that have kept investors from setting up operations in El Salvador. LOA 6: Strengthen high quality infrastructure in the inspection, certification and laboratory test services offered by the Plant Health Department (DGSV) under the Ministry of Agriculture and Livestock. LOA 7: Strengthen PROESA’s institutional capacity to identify and prioritize strategic sectors for the promotion and attraction of investments and for the development of an investment promotion strategy LOA 8: Design and implement a country image strategy based on studies of the perceptions of the business sector in the United States and other countries of interest, in order to boost investment and trade. USG LOA 9: Offer institutional strengthening of PROESA in trade and investment facilitation. LOA 10: Support improvements in the local business climate through the Municipal Competitiveness Project; provide technical assistance to municipal councils. LOA 11: Partner with the IFC to use Doing Business Indicators as a diagnostic for the Growth Council. LOA 12: Undertake a review of the laws and processes in El Salvador that facilitate investment in order to create an incentives system tailored to different types of investors in key sectors. LOA 13: Support GOES in the design and implementation of a country image strategy based on studies of the perceptions of the business sector in the United States and other countries of interest, in order to boost investment Partnership for Growth El Salvador Mid-Term Evaluation Report 193 LOW PRODUCTIVITY IN TRADABLES CONSTRAINT and trade. Goal 4: Raise (net) tax revenues to GOES LOA 1: Improve its information systems and taxpayer databases to 16 percent of GDP by 2015 and use reduce task evasion and avoidance. public resources efficiently and LOA 2: Improve transparency and efficiency in the use of public resources. transparently. These goals are also LOA 3: Improve legal and accounting systems and strengthen rule-based included in the implementation of government to be able to move toward results-based budgeting in several the fiscal pact, which is an integral sectors. part of the PQD priority areas, as USG LOA 4: Engage in programs to optimize processes, promote greater defined by the Economic and Social transparency, and ensure that taxpayer rolls are consistent and transparent. Cabinet. LOA 5: Provide technical assistance to improve tax administration and reduce evasion and avoidance. LOA 6: Provide technical assistance to develop a results-based budget with a multi-year perspective. LOA 7: Support to implement the Law on Access to Public Information. LOA 8: Provide technical assistance to improve customs administration. LOA 9: Offer programs to improve tax collection. LOA 10: Support improved tax collection at the municipal level through Domestic Finance for Development (DF4D). DF4D supports domestic revenue mobilization, transparency and anti-corruption, each activity reinforcing the other and putting developing countries on a stronger path towards sustainable and broad economic growth and opportunity. Sub-constraint: Loosening Constraints on Capital Formation GOAL Goal 5: Support a strategy for attracting and promoting FDI and making El Salvador a more attractive place for foreign investment. The measures described are aimed at streamlining the establishment of operations for potential investors and simultaneously focusing on and scaling up efforts to promote and attract investments. LOA GOES LOA 1: Strengthen institutional capacity to conduct market studies. LOA 2: Encourage investments from Salvadorans residing in the United States. LOA 3: Strengthen, finance and support the Ministry of the Economy and PROESA to implement the proposed measures. LOA 4: Continue ongoing efforts to develop and implement the Integrated Investor Attention System (SIAI). LOA 5: Develop and implement a strategy to address the causes and barriers that have kept investors from setting up operations in El Salvador. LOA 6: Strengthen high quality infrastructure in the inspection, certification and laboratory test services offered by the Plant Health Department (DGSV) under the Ministry of Agriculture and Livestock. LOA 7: Strengthen PROESA’s institutional capacity to identify and prioritize strategic sectors for the promotion and attraction of investments and for the development of an investment promotion strategy. LOA 8: Design and implement a country image strategy based on studies of the perceptions of the business sector in the United States and other countries of interest, in order to boost investment and trade. USG LOA 9: Offer institutional strengthening of PROESA in trade and investment facilitation. LOA 10: Support improvements in the local business climate through the Municipal Competitiveness Project; provide technical assistance to municipal councils. LOA 11: Partner with the IFC to use Doing Business Indicators as a diagnostic for the Growth Council. LOA 12: Undertake a review of the laws and processes in El Salvador that facilitate investment in order to create an incentives system tailored to different types of investors in key sectors. Partnership for Growth El Salvador Mid-Term Evaluation Report 194 LOW PRODUCTIVITY IN TRADABLES CONSTRAINT LOA 13: Support GOES in the design and implementation of a country image strategy based on studies of the perceptions of the business sector in the United States and other countries of interest, in order to boost investment and trade. Sub-constraint: Promoting Innovation and Internationalization GOAL LOA Goal 6: Surmount low productivity GOES LOA 1: Strengthen the technical capacity of PROESA. of tradables by transforming factors LOA 2: Strengthen the Ministry of Economy and PROESA to continue of production of the tradables sector developing the production and export strategy which improves innovation through the implementation of and quality. strategies to improve innovation LOA 3: Work with the USG to facilitate the export process. and quality, and a focus on the LOA 4: Create an integrated system to serve small and medium enterprises international market. As a result of seeking to export. PFG, Salvadoran firms will be more LOA 5: Strengthen and increase diversification through innovation. prepared to confront global markets USG LOA 6: Provide technical assistance for the Ministry of the Economy and compete successfully and the and PROESA to serve SMEs seeking to export. necessary institutions will be LOA 7: Support the establishment of small business development centers prepared to support them. throughout the country. LOA 8: Increase internationalization of Salvadoran firms through guided processes or “mentoring.” LOA 9: Explore ways to facilitate the speed of exports from El Salvador to the United States and reduce the rejection rate of Salvadoran exports at the United States border. LOA 10: Support the increased participation of the Salvadorans living abroad in the Salvadoran economy. LOA 11: Promote opportunities for bi-national business alliances in the tradables sector. Partnership for Growth El Salvador Mid-Term Evaluation Report 195 ANNEX 6 – PFG CONSTRAINTS AND GOAL-LEVEL INDICATORS CONSTRAINT-LEVEL INDICATORS Indicator National homicide rate Security Security Security Public perception of security Prosecutions and convictions as a percentage of violent crimes reported Indicator Per capita GDP growth Tradables Tradables Tradables Exports as a percentage of GDP Foreign Direct Investment as a percentage of GDP GOAL-LEVEL INDICATORS SECURITY CONSTRAINT Goal 1. Strengthen Justice Sector Institutions 2. Improve Criminal Justice Procedures 3. Reduce Impact of Crime on Businesses 4. Reduce Impact of Crime on Public Indicators Public Satisfaction with the Performance of Justice and Security Institutions (UCA Perception Survey) Number of Criminal Cases Resolved through Conviction or Alternative Dispute Processes Perception of Small and Medium-Sized Business Community of the Effect of Police and Prosecutorial Actions on Crime against their Businesses. Number of Reported Crimes against Small and Medium Businesses. Number of Reported Crimes in Public Transportation Partnership for Growth El Salvador Mid-Term Evaluation Report Source UCA Perception of Security & Confidence in Public Institutions Study (Baseline: March 2013) JSIP Project: w/ CHECCHI UCA Perception of Security & Confidence in Public Institutions Study (Baseline: March 2013) GOES -- PNC GOES -- PNC 196 SECURITY CONSTRAINT Goal Indicators Source Public Perception of Safety on Public Transport Routes UCA Perception of Security & Confidence in Public Institutions Study (Baseline: March 2013) Number of Asset Forfeiture Cases and Value of Assets Seized DOJ Public Confidence in Government Institutions. UCA Perception of Security & Confidence in Public Institutions Study (Baseline: March 2013) Number of Access to Information Law Requests Fulfilled Transparency & Governance Project w/ CASALS & Assoc. 7. Promote National Dialogue on Public Security Public Perception of National Consensus on Public Security UCA Perception of Security & Confidence in Public Institutions Study (Baseline: March 2013) 8. Assist At-Risk Youth with Economic Opportunities Number of At-Risk Youth who Find Employment or Create Businesses as a Result of Training Improving Access to Employment Program w/ CARANA - FUNDEMAS CRS PATI Number of Municipalities Implementing Community Policing Programs. Checchi, INL Incidence of Selected Violent Crimes Reported in Targeted Communities. Checchi, PNC Enrollment Rate in Secondary Schools in HighRisk Municipalities. Education for Children & Youth Project, FEDISAL data Transport 5. Remove Assets from Criminal Orgs 6. Strengthen Civil Service 9. Strengthen PNC 10. Improve Educational Opportunities 11. Prevent Crime & Violence in Key Municipalities 12. Reduce Overcrowding in Prisons 13. Enhance Security in Prisons Number of Municipalities Offering Education Programs for At-Risk Youth Number of Municipalities with Crime Prevention Councils. Incidence of Selected Violent Crimes Reported in Key Municipalities Prison Population as a Percentage of Prison Capacity (overcrowding rate) Number of Crimes Committed in Prisons. Partnership for Growth El Salvador Mid-Term Evaluation Report Education for Children & Youth Project, FEDISAL data CVPP II and SolucionES FEPADE INL INL 197 SECURITY CONSTRAINT Goal Indicators 14. Promote Use of Extraditions Source Number of Prisons with Special Intelligence Units Number of Extraditions Adjudicated by the Supreme Court. INL FBI TRADABLES CONSTRAINT Goal Indicators Source Frequency Calculation of indicator/ Website 1. Improve the institutional environment Ease of Doing Business Rank International Finance Corporation (World Bank) Annually http://www.doingbusiness.org/ Private sector perception of business climate FUSADES Quarterly Informe de Coyuntura Económica http://www.fusades.org/ Investment Central Bank (public and private) in infrastructure as percent of GDP* Annually (Construction/GDP)*100 http://www.bcr.gob.sv/bcrsite/? x21=31 Global Competitiveness Index: Infrastructure World Economic Forum Annually http://www.weforum.org/issues /global-competitiveness Global Competitiveness Index: Higher Education and Training World Economic Forum Annually http://www.weforum.org/issues /global-competitiveness Global Competitiveness Index: Labor World Economic Forum Annually http://www.weforum.org/issues /global-competitiveness 2. Investment in Infrastructure 3. Human Capital Partnership for Growth El Salvador Mid-Term Evaluation Report 198 Goal Indicators Source Frequency Calculation of indicator/ Website Percentage of Central Bank tax collected relative to GDP* Annually (Taxes Nonfinancial Public Sector/GDP)*100 http://www.bcr.gob.sv/bcrsite/? cdr=12&lang=es#var_anotacion _99 Open Budget Index Ranking International Budget Partnership Every two years http://internationalbudget.org/w hat-we-do/open-budget-survey/ Foreign Direct Investment (FDI) as Percent of GDP* Central Bank Annually (Direct Investment Balance/GDP)*100 http://www.bcr.gob.sv/bcrsite/? cdr=55&lang=es FutureBrand Country Brand Ranking FutureBrand Annually http://www.futurebrand.com/for esight/cbi Global Competitiveness Index: Business Sophistication World Economic Forum Annually http://www.weforum.org/issues /global-competitiveness Number of Business Exporting (sales over $500,000)** Central Bank Quarterly Informe de Comercio Exterior http://www.bcr.gob.sv Market Efficiency 4. Strengthening tax collection and transparency 5. Attracting Foreign Direct Investment 6. Strengthening Institutions and Business for Internationali zation Partnership for Growth El Salvador Mid-Term Evaluation Report 199 ANNEX 7 - LIST OF REFERENCE AND SOURCES – SELECTED GOALS Security References Casals and Associates, Inc. Transparency and Governance Project: Fact Sheet. October 2013. Checchi and Company Consulting, Inc. USAID Justice System Improvement Program Fact Sheet. October 2013. Checchi and Company Consulting, Inc. Justice System Improvement Program: Annual Report: March 4September 30, 2013. El Salvador. 18 October 2013. Checchi and Company Consulting, Inc. Justice System Improvement Program: Performance Monitoring and Evaluation Plan. El Salvador. 9 August 2013. Checchi and Company Consulting, Inc. Justice System Improvement Program: Life of Project Work Plan: March 2013-March 2018. El Salvador. May 2013. Checchi and Company Consulting, Inc. Justice System Improvement Program: Performance Monitoring Plan Report: FY 2013. El Salvador. 18 October 2013. Checchi and Company Consulting, Inc. Justice Sector Strengthening Project: Work Plan: March, 2008 Checchi and Company Consulting, Inc. Justice Sector Strengthening Project Indicators: Result Framework Analysis. Checchi and Company Consulting, Inc. Justice Sector Strengthening Project: Results Framework Analysis. Creative Associates International Inc. USAID Crime and Violence Prevention Project: Monitoring and Evaluation Plan March 2013-2018. October 2013. Creative Associates International Inc. USAID Crime and Violence Prevention Project: Quarterly Report April-June, 2013. July 15, 2013. Creative Associates International Inc. USAID Crime and Violence Prevention Project: Quarterly Report July-September 2013. October 2013. Creative Associates International Inc. USAID Crime and Violence Prevention Project: Work Plan 20132014. June 2013. Granados J. Personal correspondence. Notes on Milestone from Jenny. Zelaya, M. Personal correspondence. Number of Public and Closed Hearings. 24 January 2014. Ministry of Justice. Prison Statistics Presentation. 03 March 2014 Nelson Soaring Zablah “National Dialogue for Security (Gang Violence/Attacks on Police).” La Prensa Grafica. April 17, 2014. Partnership for Growth El Salvador Mid-Term Evaluation Report 200 Partnership Act. August 2014. Preliminary overview of Centro Penal Esperanza (Mariona) Prison Population Strengthening Plan for Goal 4. August 29, 2013. Unclassified Decision Memorandum: Presentations of Prisons Classification System Pilot Program. 26 April 2013 US Embassy El Salvador. JUPI 24 Hour Court. Pilot Program Proposal Presentation Partnership for Growth El Salvador Mid-Term Evaluation Report 201 Tradables References “Benchmarking Promotion Agencies Latin American Exports and Investments “. Institutional Strengthening in Investment and Export Promotion PROESA. February 2013. Brunn, B. USAID Improving Access to Employment Program, Work Plan: Year Three: FY 2012. August 2011. Brunn, B. USAID Improving Access to Employment Program, Work Plan: Year Four: FY 2013. November 2012. Brunn, B. USAID Improving Access to Employment Program, Quarterly Report April-June 2013 Brunn, B. USAID Improving Access to Employment Program, Quarterly Report April-June 2012 Brunn, B. USAID Improving Access to Employment Program, Quarterly Report January-March 2012 Brunn, B. USAID Improving Access to Employment Program, Quarterly Report January-March 2013 Brunn, B. USAID Improving Access to Employment Program, Quarterly Report July-September 2013 Brunn, B. USAID Improving Access to Employment Program, Quarterly Report July-September2012 Brunn, B. USAID Improving Access to Employment Program, Quarterly Report October-December 2012 Brunn, B. USAID Improving Access to Employment Program, Draft Final Report, December 2009November 2013. Central Reserve Bank of El Salvador. Productive Investment Determinants and their Relationship to the Effective Demand. Communications SAE. El Salvador Advances in Access to Information. December 11, 2013. Country Trade Brand Ranking. Bloom Consulting. 2013. Directory of Economics Units. 2011-2012 El Salvador Ministry of Economy. Accountability CONAMYPE. June 2012 – May 2013. El Salvador Ministry of Economy. Directory of Statistics and Census. El Salvador Ministry of Economy. Memory of Labor. June 2012 to May 2013. El Salvador Ministry of Economy. PDF Integral System Production Development. FDI Annual Values. Goal 5. 2002 – 2012. Granados J. Personal correspondence. Development Fund Productive Presentation. 23 Feb 2014. Partnership for Growth El Salvador Mid-Term Evaluation Report 202 Granados J. Personal correspondence. Number of Information requests and those fulfilled: Security Goal 6. January 16, 2014. Legislative Assembly of El Salvador. Center of Legislative Documentation. Special Law of PublicPrivate Partnerships. May 2013. Major Achievements of the DIGESTYC. June 12 – May 13 Morales, RA. Second Report on Technical Assistance Provided to El Salvador’s Labor Ministry Labor Force Statistics Program. 23 May 2013. Morales, Roberto, Summary Report of Labor Force Statistics Program Exchange between the US and El Salvador. 15 October 2012 Morales, Roberto, letter from R. Morales to Roberto D. Solano of the MTPS. US DoL – BLS. Washington DC. May 30, 2013. Najano, H. Personal correspondence letter from H. Najano to Kathleen Schalch. MTPS. 11 February 2013. Reed, María Elena. Summary of the Visit of Technical Assistance Provided to the Ministry of Labor of El Salvador, Under the Partnership for Growth: 11-15 June 2012. Department of Labor. June 22, 2012. Research Triangle Institute International. USAID Municipal Competitiveness Project (MCP): Quarterly Performance Report: July 1 – Sept. 30, 2012. North Carolina, USA. Oct. 31, 2012. Research Triangle Institute International. USAID Municipal Competitiveness Project (MCP): Quarterly Performance Report: July 1 – Sept. 30, 2013. North Carolina, USA. Sep. 30, 2012. Tradables Goal 3 Work Group Spreadsheet. 17 July 2013 Trigueros, Alvaro Arguello. FUSADES Final Report: Salvadoran Foundation for Economic and Social Development, Study for the Implementation of a System for Continuous Improvement and Administrative Simplification in El Salvador, 2013. USAID Office of Economic Growth, Higher Education Program, Presentation. USAID SME Development Program. Nov, 2013. USAID SME Development Program. First Annual Report. Jan, 2013. USAID SME Development Program. Second Annual Report. Jan, 2014. Partnership for Growth El Salvador Mid-Term Evaluation Report 203 ANNEX 8 - LIST OF REFERENCES AND SOURCES - GENERAL Advances in Access to Information. El Salvador. 11 December 2013. Aguilar, Jeannette. The Perception of Security and Confidence in Public Institutions: Baseline for the Partnership for Growth Joint Country Action Plan. University Public Opinion Institute (IUDOP), at Centroamericana José Simeón Cañas University (UCA). March 2013. Andrade, K. Personal correspondence. “Copy of Partnership for Growth Preliminary Report May-October 2013 without notes.” 8 October 2013. APP Commission. FOMILENIO II Presentation. 2014. Bolles, V. Monthly Summary. Office of Technical Assistance: GDIM. El Salvador. November 11-April 12. Carle, D. Statement of Senator Patrick Leahy on A Second Millennium Challenge Compact for El Salvador. 12 September 2013. < http://www.leahy.senate.gov/press/statement-of-senator-patrick-leahyon-a-second-millennium-challenge-compact-for-el-salvador>. Casals and Associates, Inc. USAID Transparency and Governance Project: Fact Sheet. October 2013. Center for Global Development. “Partnership for Growth: A New Model for USG Engagement on Development?” Panel discussion. 28 March 2012. Center of Development for Medium and Small Businesses – Francisco Gavidia University. Presentation for JICA Delegation Visit. Center of Development for Medium and Small Businesses Closing Report 2012. Center of Development for Medium and Small Businesses Closing Report 2013. Cerritos, M. and López, J. Open Budget Survey 2012 – El Salvador. 2012. <http://internationalbudget.org/wp-content/uploads/OBI2012-ElSalvadorCS-English.pdf>. Checchi and Company Consulting, Inc. Justice Sector Strengthening Project: Revised Performance Monitoring and Evaluation Plan. El Salvador. 9 August 2013. Chemonics International. USAID SME Development Program: First Annual Report October 2011September 2012. January 2013. Chemonics International. USAID SME Development Program: Second Annual Report October 2012September 2013. United Nations Surveys on Crime Trends and the Operations of Criminal Justice Systems (CTS), 20102013. The Council on Competitiveness . American Energy & Manufacturing Competitiveness Partnership: A Summary of Public Private Partnerships. Washington D.C. 2013. <http://www.compete.org/images/uploads/File/PDF%20Files/AEMC_Part_PPP_Summary_FINAL.pdf>. Partnership for Growth El Salvador Mid-Term Evaluation Report 204 Department of Labor. Report of Achievements. June 2012.Development Alternatives, Inc. El Salvador Fiscal Policy and Expenditure Management Program: Quarterly Report: June-August 2011. September 2011. Development Alternatives, Inc. El Salvador Fiscal Policy and Expenditure Management Program: Quarterly Report: March-May 2012. June 2012. Development Alternatives, Inc. El Salvador Fiscal Policy and Expenditure Management Program: Quarterly Report: June-August 2012. September 2012. Development Alternatives, Inc. El Salvador Fiscal Policy and Expenditure Management Program: Quarterly Report: September-November 2012. December 2012. Development Alternatives, Inc. El Salvador Fiscal Policy and Expenditure Management Program: Quarterly Report: December 2012-February 2013. March 2013. Development Alternatives, Inc. El Salvador Fiscal Policy and Expenditure Management Program: Quarterly Report: March-May 2013. June 2013. Dale, Helle C. “The White House Embraces Smart Power: Now What?” The Heritage Foundation. 7 October 2010. Development Alternatives, Inc. El Salvador Fiscal Policy and Expenditure Management Program: Quarterly Report: June-August 2013. September 2013. Directory of Economics Units. 2011-2012. Draft 10 Success Stories for Annual Report. September 2013. El Salvador Ministry of Finance. Basic Statistics on Public Finance, June 2013. 17 December 2013. <http://www.transparenciafiscal.gob.sv/portal/page/portal/PCC/EstadisticasFiscales/correccion_por_FOP_Y_FOSEDU_A_Estadisticas_Fiscales_14-08-2013-Rev.pdf>. El Salvador Ministry of Finance. Fiscal Transparency Portal. 13 January 2014. <http://www.transparenciafiscal.gob.sv/portal/page/portal/PTF/Gastos:GobiernoCentral>. El Salvador Ministry of Economics. April 2013 <http://www.digestyc.gob.sv/index.php/institucion/marco-institucional/logros.html>. El Salvador National Action Plan for Youth Employment. 2012-2024. FONDEPRO Results Presentation. 7 November 2013. Government of El Salvador. Five-Year Development Plan 2010-2014. 2010. Government of El Salvador. Five-Year Development Plan 2010-2014. Presentation. 2010. Granados, J. Personal correspondence. “Number and Type of National Dialogue Events 2011-2013.” 24 January 2014. Partnership for Growth El Salvador Mid-Term Evaluation Report 205 Hausman R, Rodrik D, Velasco A. “Growth Diagnostics.” John F. Kennedy School of Government, Harvard University. March 2005. Indicators Table to Scorecard IV for Midterm Evaluation. INL, El Salvador. Extinction of Authority. June 2013 Draft. Inter-American Foundation. 2013 Annual Report. 2013 Iris Foundation of Central America. Rule of Law Assistance Final Report. 2013. Jensen, Jeri. Toward a New Paradigm of Sustainable Development: Lessons from the Partnership for Growth. 13 September 2013. Mercado-Rios, J. Department of Justice, Office of Overseas Prosecutorial Development, Assistance and Training, Memorandum 7776. Millennium Challenge Corporation. “MCC’s Use of Constraints Analysis – Fact Sheet.” 24 June 2013. Ministry of Education. Joint Direction EITP: Integrated System Statistical Data. Mission Activity Tracker Reports. Organization for Economic Cooperation and Development. Whole of Government Approaches to Fragile States. 2006 Partnership for Growth. Strengthening Plan for Goal 4. 29 August 2013. Partnership for Growth Communications Strategy Presentation. 21 December 2012. Partnership for Growth. Crime and Security Lines of Action – Pre-Constraints Analysis. 2013. Partnership for Growth. Performance Data Tables Format – Indicator 2 M&E. 2014. Partnership for Growth. El Salvador Goal Lead List. 14 May 2014. Partnership for Growth: Community Policing Strategy. 5 March 2014. Partnership for Growth: El Salvador-United States: Mission Activity Tracker Reports El Salvador. November 1, 2011-December 1, 2013. Partnership for Growth: El Salvador-United States: Communication Strategy. 12 December 2012. Partnership for Growth: Indicators and Reports Session meeting. 19 September 2012. Partnership for Growth. Memory Aid. August 24, 2012.Partnership for Growth Bulletin: El Salvador – United States – Number 4. Technical Secretary of the President; Ministry of Safety & Justice; Ministry of Economy; Embassy of the US. El Salvador. December 2012. Partnership for Growth El Salvador Mid-Term Evaluation Report 206 Partnership for Growth Bulletin: El Salvador – United States – Number 5. Technical Secretary of the President; Ministry of Safety & Justice; Ministry of Economy; Embassy of the US. El Salvador. April 2013. Partnership for Growth Bulletin: El Salvador – United States – Number 7. Technical Secretary of the President; Ministry of Safety & Justice; Ministry of Economy; Embassy of the US. El Salvador. October 2013. Partnership for Growth HLR Agenda. 28 November 2012. Partnership for Growth Fair Photos. December 2013. Partnership for Growth: Higher Education (HE) Statement of Work Summary. 2014. Presentation on Considerations of US Government Milestones. Proposed Partnership for Growth Indicators. May 2014. Reed, ME. Summary of the Visit of the Technical Assistance provided by the Ministry of Labor of El Salvador, Under the Partnership for Growth: June 11 – 15, 2012. 22 June 2012. RTI. Community-Based Crime and Violence Prevention Project. Final Report, April 2013 Salazar, S. Agenda. May 6 – 17, 2013. El Salvador. Salazar, S. 2011 Consulting Report. 10-14 February 2014. Salazar, S. International Council Extinction of Authority. Social Investment Fund for Local Development. Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects. February 2014. Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: January-March 2013. USAID Temporary Income Support Program. April 2013 Salvadoran Foundation for Integral Education. Education for Children and Youth: Monitoring Indicator 2013. Salvadoran Foundation for Integral Education. Education for Children and Youth: updated Table approved 19 December 2013. Segovia, A. Partnership for Growth 3rd Semi-annual Monitoring and Evaluation Report: Highlighted Package of Laws. Technical Secretariat, Presidency of the Republic of El Salvador. 12 September 2013. <http://tecnica.presidencia.gob.sv/component/k2/item/834-en-el-tercer-reporte-semestral-de-seguimientoy-evaluaci%C3%B3n-del-asocio-para-el-crecimiento-destacan-el-paquete-de-leyes.html> Segovia, A. Signing of FOMILENIO II pending the Approval of APP Law. Technical Secretariat, Presidency of the Republic of El Salvador. 6 December 2013. <http://tecnica.presidencia.gob.sv/component/k2/item/889-firma-de-fomilenio-ii-en-espera-a-laaprobaci%C3%B3n-de-ley-app.html> Partnership for Growth El Salvador Mid-Term Evaluation Report 207 Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: April-June 2013.” USAID Temporary Income Support Program. July 2013. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: July-September 2013.” USAID Temporary Income Support Program. July 2013. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). First Progress Report 21 December 2010.” February 2011. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: January-March 2011. April 2011. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: April-June 2011.” July 2011. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: July-September 2011.” September 2011. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: October-December 2011.” January 2012. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: January-March 2012. April 2012. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: July-September 2012. Oct 2012. Social Investment Fund for Local Development. “Limited Scope Grant Agreement for Global Financial Crisis Mitigation Effects Project. Activity No. 519.0462. Economic Growth for the 21st Century (Grow 21). Progress Report: October-December 2012.” January 2013. USAID Action Plan: Program Temporary Income Support. 24 July 2012. USAID Addendum to the Plan of Action for the Extension and Expansion Program of Temporary Income Support. 7 March 2013. USAID. Country Development Cooperation Strategy Guidance, Version 3. 29 May 2013 USAID Successfully Closes its Improving Access to Employment Program. 7 November 2013. < http://sansalvador.usembassy.gov/news/2013/11/07.html> USAID. Understanding PFG within USAID. PowerPoint presentation. Partnership for Growth El Salvador Mid-Term Evaluation Report 208 USAID. Official List of all PFG Partners and Agencies. U.S. Department of Labor, International Labor Affairs Bureau. Eliminating Child Labor in El Salvador through Economic Empowerment and Social Inclusion. December 2010. U.S. Department of State. El Salvador 2012 Human Rights Report. 2012 U.S. Department of State. Partnership for Growth: El Salvador-United States, Joint Country Action Plan 2011-2015. November 2011. U.S. Department of State. Partnership for Growth: El Salvador-United States, Joint Country Action Plan Addendum. November 2011. U.S. Department of State. United States of America-El Salvador Partnership for Growth, Joint Statement of Principles. November 2011. White House, Office of the Press Secretary. Remarks by President Obama and President Funes of El Salvador in Joint Press Conference. 22 March 2011. White House, Office of the Press Secretary. Fact Sheet: U.S. Global Development Policy. 22 September 2010 The World Bank Group. Doing Business 2014 – Economy Profile: El Salvador. Washington DC. 2013. <http://www.doingbusiness.org/data/exploreeconomies/~/media/giawb/doing%20business/documents/pro files/country/SLV.pdf?ver=2> The World Bank Group. Doing Business: Measuring Business Regulations. 2014 Zelaya, M. Personal correspondence. “Security Goal 6 Information.” May 2014. Partnership for Growth El Salvador Mid-Term Evaluation Report 209