OpenEMIS Jordan Evaluation 2017

,
CONTENT
1. EXECUTIVE SUMMARY ............................................................................................ 3

2. EVALUATION PURPOSE & SCOPE .......................................................................... 5
3. EVALUATION METHODOLOGY ........................................................ ........................ 6
3.1. Overview of evaluation methodology ........ ............... ........ ............................................ 6
3.2. Evaluation matrix ............................... .................................... ................ ...................... 6
4. MAIN FINDINGS ....................................................................................................... 10

4.1 . Development objective : To improve education planning and management in Jordan 10
4.1 1 Assessment of Development Objective .................................................. 10
4.1.2. Assessment of Development Objective Indicator ............................................ 10
4.2. Immediate objective: To establish a government-owned, cost-effective Education
Decision Support System ................................ .... .................... ....... ....... ..................... ......... 11
4.2.1 Assessment of Immediate Objective ................................................................ 11
4.2.2 Assessment of Immediate Objective Indicator. ... . . .. ... .. ... ... ...... ......... .. 12
4.3. Outputs/deliverables .......... ........................ .................. ... ....... ................... .......... ...... 14
4.3.1. Overall assessment of Expected Results ......................................................... 14
4 3.2. Assessment of Expected Results Indicators .............................................. 15
4.4 . Year 1 Activities ........................... ........ ....... ....... ........ .... .......... ... ............... ... ......... ... 18
4.4.1. Overall assessment of Year 1 Activities. ..... ... . . . ................................ .... 18
4.4.2 Assessment of Year 1 indicators ......................... ......................................... 18
4 .5. Year 2 & 3 Activities ....................... .... ......... ... ............... ......... ........... ....... ..... .... ........ 21
4.5.1. Overall assessment of Years 2 & 3 activities .................................................. 21
4.5.2. Assessment of Years 2 & 3 indicators.. .. .. . ..... ... ..... .......... ... ........ 22
4.6. Year 3 & 4 Activities ...... ....... ..... ................ ....................................................... ......... 24
4.7. OECD DAC Criteria .................................... .......... .... ...... ........... ....................... ......... 29
4.7.1 Relevance........ ... ....... .... . . ......... ............................................... ........ 29
4.7.2. Efficiency. ........... ......... .............................. ..... .................. ... ......... .... 30
4 7.3. Effectiveness ..... ............ ................................................ ... ............. ....... 31
4.7.4. Impact ................................................. ... ............................... 33
4 7.5 Sustainability . . ... ... ... . .... ........ .... ... .. ............. ..... .. .. .. . . 33
4.8 . Gender ............. ...................... ....... ..... ... .... ........ .... .................. ...... ............................ 33
5. LESSONS LEARNED ............................................................................................... 34

6. RECOMMENDATIONS ............................................. ................................................ 36
7. ANNEX 1: LIST OF SECONDARY DATA ................................................................ 38
8. ANNEX 2: LIST OF INTERVIEWS WITH KEY PROJECT STAKEHOLDERS .......... 39
9. ANNEX 3: LIST OF FIELD OBSERVATION VISITS AND INTERVIEWS ................. 41
10. ANNEX 4: BENEFICIARY INTERVIEW GUIDELINES ....................... ...................... 42
11. ANNEX 5: OECD DAC CRITERIA INDICATIVE QUESTIONS ................................. 44
1. EXECUTIVE SUMMARY
The OpenEMIS project made a strong contribution to its overall development objective of improving
education planning and management in Jordan - both through the core OpenEMIS workstream and
through the additional technical advisory services rendered in the second half of the project. The project
also made a strong contribution to its immediate objective of establishing a government-owned, cost
effective Education Decision Support System, without fully realizing it in practice.
The OpenEMIS system provides a comprehensive set of tools for aggregating, analyzing, reporting on
and disseminating educational data in Jordan and supporting evidence-based policy-making at central,
regional and school levels. The OpenEMIS platform has been adapted to the Jordanian educational
system and full customized to meet the needs of MoE. It has been successfully rolled out across the
country, data collection processes are established, and numerous external sources attest to the
accuracy of the data. A variety of reporting and data visualization tools are available, and the system is
integrated with other public-sector databases. Yet while the core MoE team can manage some
components of OpenEMIS independently, MoE does not have the capacity to autonomously sustain
the use of the system . In particular, the core MoE EMIS team is not yet self-sufficient in fully exploiting
the system 's various reporting functionalities , so the extent to which the EMIS is being used to its full
potential for evidence-based policy-making in MoE is limited . However, significant progress was made
in accelerating capacity building and knowledge transfer in the latter stages of the project, and the
ground has also been prepared for focusing on these areas in Phase Two of the project.
The additional 'Complementary Technical Advisory Services' project component made a strong
contribution to improving education planning and management in Jordan by allowing MoE to request
support in specific priority areas . Capacity gaps also remain in certain areas addressed - notably on
the exploitation of the GIS platform to its full potential. But these can be addressed in the second phase
of the project, and the overall impact of the technical advisory services is nonetheless assessed to be
strong.
The project's contribution to improving education management and planning in Jordan should also be
seen within the wider education sector and national context. UNESCO is currently supporting the MoE
in developing a five-year Education Sector Plan (ESP), which should be finalized by the end of 2017.
With additional capacity building and support to MoE, the enhanced EMIS should strengthen the
implementation of the ESP by providing the basis for the effective monitoring of key indicators and the
assessment of education sector performance. More broadly, given the large investments being made
by international donors in Jordan's education sector (particularly in the areas of school-building and
budgetary support to bolster the country 's resilience to the impacts of the large influx of Syrian
refugees), MoE is under considerable pressure to provide data to justify funding requests and
demonstrate achievements . The OpenEMIS project-and particularly its aggregation and dissemination
of data on Syrian refugees in Jordanian schools - has made a strong contribution to allowing MoE to
meet these demands .
The OpenEMIS project was highly relevant to the beneficiaries ' needs and the project model was
contextually appropriate - although the time and resources needed to train key MoE stakeholders and
build capacity on the management and use of the system was underestimated . The project made a
strong effort to adapt and respond to changes in the internal and external context, some of which
negatively influenced the implementation of the core OpenEMIS elements of the project, resulting in
delays. The UNESCO core project team's resources appear to have been used efficiently, but there
were some inefficiencies in how CSF resources were deployed. The amount of work delivered under
the project is considerable, and as such cost ef{iciency is assessed to be good. UNESCO 's planning,
management and coordination of the project was adequate overall, ~!though certain areas could have
been improved - in particular the project's M&E strategy . Beneficiaries were overall satisfied with the
services delivered, but recognize the need for additional capacity building if the project's achievements
are to be sustainable.
The evaluation identified the following key lessons learned :
1. The design of the project underest imated the magnitude of implementing Open EM IS in a country
of Jordan's size and the corresponding capacity building needs of MoE .
2. A disconnect between UNESCO/MoE 's initial conception of Open EM IS and the subsequent vision
of other key MoE stakeholders for the system 's purpose and functionality delayed and complicated
OpenEMIS implementation .
3. A lack of clearly defined roles and responsibilities within MoE in relation to the EMIS negatively
impacted project implementation and sustainability .
4. The project adapted well to unanticipated external developments, but these had an unavoidable
negative impact on project implementation.
5. Top-level MoE engagement and support is crucial for the successful implementation and
sustainability of a large-scale technical assistance project such as OpenEMIS .
The following recommendations are made for any future iteration of the project:
1. Continue building QRC 's capacity to maintain and manage the core IT aspects of the Open EM IS
platform, but identify a more suitable MoE counterpart for strengthening the link between EMIS data
and its utilization for evidence -based policy formulation.
2. Incorporate broader institutional support for building a culture of evidence-based policy making in
MoE .
3. Conduct a comprehensive stakeholder consultation to ensure cross -MoE consensus on the primary
features and functionalities of the OpenEMIS platform, and clarify institutional roles and
responsibilities in relation to the EMIS.
4. Incorporate change management, commun ications and stakeholder management workstreams

into the core project management component.
5. Ensure the project is coordinated and aligned with other donor support to MoE relating to planning ,
M&E , research and evidence-based policy making .
2. EVALUATION PURPOSE & SCOPE
UNESCO Amman in November 2017 commissioned Altai Consulting to conduct an End-of-Project
Evaluation of the European Union-funded project 'Technical Assistance to Enhance Accessibility and
Use of the Jordanian MoE EMIS for Evidence-based Policy Formulation .' The project began in February
2014 and concluded on 30 November 2017 .
According to the Terms of Reference (ToR) , the main purpose of the evaluation is to assess the
performance (activities , outputs, outcomes) of the project and to generate lessons and
recommendations for UNESCO to improve planning, implementation , management, and monitoring
and evaluation of future similar interventions .
In addition , according to the ToR the evaluation should :
• Assess the achievements of the project outputs and target indicators towards the contribution
to the outcomes
• Assess the effectiveness and efficacy of the project in meeting the stipulated results
• Draw lessons for improving the design and management of similar projects in the future
including identifying issues encountered through implementation and identify the level of
engagement and role of the program stakeholders
• Formulate recommendations for strengthening accountability; including exit strategy and
sustainability mechanisms
• Consider the relevance , effectiveness, efficiency, impact and sustainability of the overall project
• Assess the extent to which gender considerations were mainstreamed in the project
3. EVALUATION METHODOLOGY
3.1. OVERVIEW OF EVALUATION METHODOLOGY
The methodology for th is final evaluation comprises:
• Review of secondary data . This includes project documentation (including progress reports) ;
EU, UNESCO and Government of Jordan strategic documentation; evaluations and
assessments already carried out ; and implementing partner surveys . A list of secondary data
reviewed as part of the evaluation is provided in Annex 1.
• Interviews with key project stakeholders and implementation and sub-contracted

partners. A list of interviewees is provided in Annex 2.
• Field observation visits and interviews with beneficiaries at project implementation sites
(selected Ministry of Education Field Directorates and schools) in Jordan. A list of the
Field Directorates and schools where field observation visits and interviews were conducted is
provided in Annex 3. The interview guidelines that were used to structure interv iews with
beneficiaries are provided in Annex 4.
3.2. EVALUATION MATRIX

The matrix below sets out the evaluation areas and corresponding indicators, sources of data, data
collection methods and data analysis techniques that form the basis of the evaluation methodology.
Table 1. Evaluation matrix

Evaluation area Indicators Sources of data Data collection Data analysis
methods technique
Development objective: To Indicator 1: Timely, • MoE (key • Qualitati ve • Qualitati ve

improve education planning relevant integrated project inter views with analysis of
and management in Jordan education data available counterparts , key MoE intervie w
on demand for decision Planning & stakeholders output
support Research • Qualitati ve • Assessment of
Department , interviews with secondary
QRC, ETC) other key data
• Beneficiaries stakeholders
(field (UNESCO , EU ,
directorates and partners)
schools) • Field observation
• Other key visits and
stakeholders interviews with
(UNESCO , EU , beneficiaries
partners) • Review of
• Secondary data secondary data
Immediate objective: To Indicator 2: Evidence of • MoE (key • Qualitative • Qua Iitative
establish a government- appropriate use of project interv iews with analysis of
owned , cost-effective education decision- counterparts , key MoE intervie w
Education Decision Support support system for Planning & stakeholders output
System improved planning and Research
management • Qualitative • Assessment of
Department , interviews with secondary
. QRC , ETC)
• Beneficiaries
other key
stakeholders
data
(field (UNESCO , EU,

visits and
l ,, • _i 1
• Other key interv iews with
stakeholders beneficiaries
(UNESCO , EU, • Revie w of
partners) secondary data
• Secondary data
Outputs/deliverables Indicator 3: OpenEMIS
• MoE (key • Qualitative • Qua Iitative
operational and owned by project interviews with analysis of
• Education analysis
the government
counterparts , key MoE interview
tools and products
developed to assist key Indicator 4: OpenEMIS Planning & stakeholders output
stakeholders in administration team with Research
• Qualitative • Assessment of
decision support with the capacity to sustain the Department , interviews with secondary
existing education data use of the system QRC , ETC) other key
. An operational
OpenEMIS with current
Indicator 5: Launch of an
• Beneficiaries
(field
stakeholders
(UNESCO , EU ,
data
integrated education
data on institutions, directorates and partners)
decision support system
students, teachers and schools) • Field observation
with data from OpenEMIS
.
staff and other relevant public- • Other key visits and
An integrated sector databases stakeholders intervie ws with
education decision (UNESCO , EU, beneficiaries
support system partners) • Review of
established and used • Secondary data secondary data
by education system in
the country
Year 1 Activities Indicator 6: OpenEMIS • MoE (key • Qualitati ve • Qualitative
loaded with existing project interviews with analysis of
education data
Indicator 7: OpenEMIS Planning & stakeholders output
used with other data Research • Qualitative • Assessment of
visualization and reporting Department, inter views with secondary
tools to generate key QRC , ETC)
other key data
planning and • Beneficiaries stakeholders
management reports (field (UNESCO , EU ,
Indicator 8: A team of
key of MoE officers
trained in use of • Other key visits and
education data for better stakeholders interviews with
planning and (UNESCO , EU, beneficiaries
management partners) • Rev iew of
Year 2 & 3 Activities Indicator 9: OpenEMIS • MoE (key • Qualitative • Qualitati ve
piloted in 50+ schools project interviews with analysis of
Indicator 10: OpenEMIS counterparts , key MoE interview
scaled up to cover all Planning & stakeholders output
schools , students , Research • Qualitative • Assessment of
teachers and staff Department , interviews with secondary
QRC , ETC) other key data
(field (UNESCO , EU,
stakeholders interviews with
(UNESCO , EU, beneficiaries
partners) • Revie w of
. • Secondary data secondary data
Year 3 & 4 Activities Indicator 11: OpenEMIS
integrated with other
• MoE (key
project
• Qualitative
interviews with
. Qualitati ve
analysis of
public databases to counterparts , key MoE interview
Planning & stakeholders output
establish an education
decision-support system
Research
Department ,
• Qualitative
interviews with
. Assessment
of
QRC , ETC) other key secondary
Indicator 12: Education
• Beneficiaries stakeholders data
simulation modelling
(field (UNESCO, EU,
capacity building and
training provided to
government officers schools) • Field observation
Indicator 13: Availability interviews with
stakeholders
of a reviewed National beneficiaries
(UNESCO , EU,
Teacher Professional
Standards
Indicato r 14 : Availability
of generic frameworks for
teacher professional
development and teacher
induction and in-service
programmes
Indicator 15: Availability

of an Education Sector
Financing report

of National Satellite
Account for Education

of a cost simulation model
Indicator 18 : Availability
of a GIS platform to map
education inequities ,
support decision-making
for rationalization of the
school network and
teacher allocation across
schools

of a workplan for
implementation of MoE
ICT strategy
Indicator 20 : Availability
of a MoE VE work plan for
implementation of the
decisions stemming from
the National Conference
on Human Resource
Development Strategy
Indicator 21: 80% of MoE

EMIS officers at central
and regional level
satisfactorily trained on
their EMIS tasks
OECD DAC Criteria Please refer to Annex 5:

• MoE (key • Qualitative
. Relevance
OECD DAC criteria
project interviews with
• Qualitative
analysis of
. Efficiency
indicative questions
. Effectiveness
.
Planning &
Research
stakeholders output
. Impact Department ,
• Qualitative
interviews with
• Assessment of
. Sustainability
QRC, ETC)
• Beneficiaries
other key
stakeholders
secondary
data
(field
directorates and (UNESCO , EU,
schools) partners)
• Other key • Field observation
stakeholders visits and
(UNESCO, EU, interviews with
partners) beneficiaries
• Secondary data • Review of
secondary data
Mainstreaming of gender Gender considerations • MoE (key • Qualitative • Qua Iitative
considerations were adequately project interviews with analysis of
assessed and counterparts , key MoE interview
mainstreamed in project Planning & stakeholders output
design and Research
implementation • Qualitative • Assessment of
Department, interviews with secondary
QRC, ETC) other key data
(field (UNESCO, EU ,
schools)
• Field observation
stakeholders intervie ws with
(UNESCO, EU, beneficiarie s
..
4. MAIN FINDINGS
4.1. DEVELOPMENT OBJECTIVE: TO IMPROVE EDUCATION PLANNING AND
MANAGEMENT IN JORDAN
4.1.1. ASSESSMENT OF DEVELOPMENT OBJECTIVE

The project has made a strong contribution to the development objective of improving
education planning and management in Jordan - both through the core OpenEMIS workstream
spanning the duration of the project and through the additional technical advisory services
rendered in Years 3 and 4 .
The enhanced EMIS system provides a comprehensive set of tools for aggregating, analyzing, reporting
on and disseminating educational data in Jordan and supporting evidence-based policy-making at
central and regional levels. Gaps exist in MoE's capacity to autonomously manage the system and
utilize it to its full potential. However , significant progress was made in accelerating capacity building
and knowledge transfer in the latter stages of the project, and the ground has also been prepared for
focusing on these areas in Phase Two of the project.
The additional 'Complementary Technical Advisory Services' project component also made a strong
contribution to improving education planning and management in Jordan by allowing MoE to request
support in specific priority areas - from teacher professional development and education sector
financing to developing a GIS platform and implementing the MoE ICT strategy . Some proposed areas
of support under this project component were prematurely cancelled and some consu ltancies were less
impactful than others. Capacity gaps also remain in certain areas addressed - notably on the
exploitation of the G IS platform to its full potential. But these can be addressed in the new project, and
the overall impact of the technical advisory services is nonetheless assessed to be strong .
The project's contribution to improving education management and plann ing in Jordan should also be
seen within the wider education sector and national context. UNESCO is currently supporting the MoE
in developing a five-year Education Sector Plan (ESP), which should be finalized by the end of 2017 .
The enhanced EMIS should strengthen the implementation of the ESP by provid ing the basis for the
effective monitoring of key indicators and the assessment of education sector performance - although
this will of course require further capacity building of key MoE stakeholders in the new project. The
effective use of education sector data is also a key priority identified as part of the w ider Education
Reform for a Knowledge Economy (ERfKE) initiative . Finally, given the large investments being made
by international donors in Jordan's education sector (particularly in the areas of school-building and
budgetary support to bolster the country's resilience to the impacts of the large influx of Syrian
refugees) , MoE is under considerable pressure to provide data to justify fund ing request s and
demonstrate achievements. 1 The OpenEMIS project - and particularly its aggregation and
dissemination of data on Syrian refugees in Jordanian schools - has made a strong contribut ion to
allowing MoE to meet these demands .
4 .1.2 . ASSESSMENT OF DEVELOPMENT OBJECTIVE INDICATOR
Indicator 1: Timely, relevant, integrated education data available on demand for decision
support
1 lnterv;ew with Constanza Fanna. D,rector of UNESCO Amman Office. 27 December 2017
Basis for assessment
• Secondary data
• Interviews with key project stakeholders and implementation and sub-contracted partners
Assessment
This indicator has been partially achieved : timely , relevant and integrated education data is
available through the enhanced EMIS , although its systematic exploitation for decision support and
evidence-based policy-making is yet to be fully realized .
The education data available through the Open EM IS platform is timely in that the system is
updated with data from the 2016-2017 school year (and data collection for the 2017-2018 school
year appears to be progressing well); relevant in that an extensive scoping phase and continuous
customization of the system ensured the type and format of data is coherent with the Jordanian
education system and meets MoE requirements; and integrated in that the OpenEMIS system is
linked to a series of other relevant public-sector databases.
That said, the core MoE EMIS team 's limited capacity for producing custom reports and indicators
for other MoE departments means that much of this data cannot in practice be said to be available
on demand for the MoE at large. Consequently, the extent to which it is used to systematically
support decision-making - beyond responding to ad hoc requests for data and the generation of a
limited number of simple reports - is limited .
Key activities Achievements of key activities
All N/A
4.2. IMMEDIATE OBJECTIVE: To ESTABLISH A GOVERNMENT-OWNED , COST-

EFFECTIVE EDUCATION DECISION SUPPORT SYSTEM
4.2 .1. ASSESSMENT OF IMMEDIATE OBJECTIVE

The project made a strong contribution to its immediate objective of establishing a government-
owned, cost-effective Education Decision Support System, without fully realizing it in practice.
The OpenEMIS platform has been adapted to the Jordanian education system and fully customized to
meet the needs of MoE , its roll-out across the country has been completed and the system is fully
operational. A variety of reporting and data visualization tools are available , and the system is integrated
with several other public-sector databases. The enhanced EMIS can be said to be government-owned
in the sense that commitment to using the system appears to be strong at all levels .
Yet while the core MoE team can manage some components of OpenEMIS independently, MoE does
not have the capacity to autonomously sustain the use of the system - meaning that it cannot be said
to be completely government-owned. In particular, the core MoE EMIS team is not yet self-sufficient in
fully exploiting the system's various reporting functionalities , so the extent to which the EMIS is being
exploited to its full potential for evidence-based policy-making in MoE is limited .
While there are some costs to maintaining OpenEMIS, the OpenEMIS platform is more cost-effective
than the previous EDUWAVE system, which cost MoE up to JD 500,000 per year to run and had much
reduced functionality. 2
4.2 .2. ASSESSMENT OF IMMEDIATE OBJECTIVE INDICATOR

Indicator 2: Evidence of appropriate use of education decision-support system for improved
planning and management
• Secondary data
Assessment
This indicator has been partially achieved : there are many examples of the Open EM IS platform
being used to respond to ad hoc requests for data and generating a limited number of basic
reports, as well as of workflows being used to facilitate some MoE administrative procedures;
however, there is little evidence of the EMIS being used systematically for evidence-based policy
formulation.
The Head of the MoE Strategic Planning Department reported that the most important ways in
which his department uses EMIS data is to determine whether a single-shift school should be
upgraded to a double-shift school (through an indicator derived from the number of students in
relation to the number of teachers and the sizes of classrooms), and to determine whether more
schools need to be built in a particular area {through observing the number of double-shift schools
in that area) .3
The CSF EMIS Programme Advisor and Developer provided a number of examples of how EMIS
data is regularly used to respond to ad hoc requests for data, including requests from the MoE
Infrastructure Department for data about school buildings and infrastructure when they receive
maintenance requests from schools, and regular requests from the MoE Donor Coordination Unit
for data on Syrian refugees in schools . He also described how the QRC EMIS team had recently
responded to a request from the Ministry of Youth for data on running water in school buildings in
order to inform planning for holding youth camps in public school facilities . In terms of reporting, the
CSF Developer noted that whenever the Minister for Education or senior MoE official visits a
school, EMIS data is used to generate an 'Educational Status Report' compiling various data on the
particular school. 4 The decision to use the Open EM IS system as the repository for official data on
Syrian refugees in Jordanian schools and the OpenEMIS system's role in assisting the Ministry of
Finance in its disbursal of 20JD to every schoolchild on behalf of the King are further examples of
the OpenEMIS system being used to respond to ad hoc data requests, albeit on a country-wide
scale.
At the Field Directorate level, staff interviewed reported using EMIS data to facilitate decision-
making around allocating scholarships and aid to students, as well as around when to build new
interv,ew with N1ihan Siam, Jordan EMIS Programme Ad Jisor/CSF 29 November 2017
·' Interview with Dr Abdullah Hassoneh Head of MoE Strategic Planning Department 28 November 2017
4 Interview with N,lhan S,am Jordan EMIS Programme Adv,sor/CSF 29 November 2017
classrooms and school buildings .5 At the school level , OpenEMIS does not appear to play a
prominent role in planning or management, although several principles reported that the system
helps them to communicate with parents by being able to easily look up every student's grades.
Some principles also reported using EMIS data to recommend students for scholarships. 6
Yet numerous key project stakeholders interviewed reported that the EMIS is not being used
systematically for evidence-based policy formulation , planning and management at the central
level. This is partly due to a lack of capacity among the core MoE EMIS team . While core MoE
EMIS team members have been given training in how to build custom reports , dashboards and
visualizations , capacity is still relatively weak in this regard and the MoE EMIS team is as yet still
not able to go beyond the limited number of pre-defined reports and dashboards and create custom
reports for each MoE department .7 The same is true for the development of custom indicators and
KP ls that would facilitate the monitoring of sector performance at a more systematic and strategic
level.
The weak link between the core technological aspects of the OpenEMIS system and the use of
educational data to drive evidence-based decision-making in MoE may be related to the decision to
transfer responsibility for the EMIS system from the Policy and Planning Department to the QRC .
QRC is focused on data collection (for which it had responsibility under the previous EDUWAVE
system) and the technical IT aspects of the Open EM IS system, but largely lacks the capac ity and
perspective for generating custom reports and indicators and linking EMIS data to planning and
decision-making processes .8 At the same time, due to a combination of staff turnover, lack of
capacity and lack of engagement with the Open EM IS project among certain key stakeholders, the
Policy and Planning Department is also not able to generate reports and drive the integration of
EMIS data in MoE planning and decision-making processes .9 A reported reluctance in QRC and
the Policy and Planning Department to work with each other only serves to weaken the link
between the core technological aspects of the system and the raw data on one hand , and reporting
and the strategic use of data for evidence-based decision-making on the othe r.
All N/A
~ Interview with Uwa Ban, Obeid Field Directorate 28 November 2017 1nterv1ewwith Madaba Field Directorate 29 November
2017
'' Interview with Principle Un· Tofayl Secondary School Amman 25 November 2017, interview with Principle Ai Hoson
Secondary School. lrb1d. 28 November 2017
Interview with UNESCO Amman Pro1ect Team 19 November 2017 Interview with Dr Omar AI-Jarrah Chair of OpenEMIS
Technical Steering Con1m11tee 29 November 2017
\ Interview with UNESCO Amman ProJect Team 27 November 2017 ROM Monitoring Report (June 2017): Support to Jordanian
EMIS System - Phase 2 (September 2017 - August 2020)· Pro1ect Document (May 2017,
9 Interview with UNESCO Amman Project Team. 27 November 2017

4.3. OUTPUTS/DELIVERABLES
4.3.1. OVERALL ASSESSMENT OF EXPECTED RESULTS
4.3.1.1. ER1: Comprehensive scoping of MoE needs for a more robust Education
Decision Support System
This Expected Result has been achieved. An initial needs assessment included consultations with a
wide range of stakeholders (MoE Planning , Research and Development Directorate, Queen Ran ia
Center , Educational Train ing Cente r, Examinations and Tests , Vocational Education, General
Education , selected field directorates and schools) , as well as a wide range of external stakeholders) ,
as well as a technical review comprising of a comprehensive IT infrastructure assessment from the
central (ministry) to the school level. The EMIS project team continued to accommodate MoE's evolving
needs for the EMIS throughout the lifetime of the project.
4.3.1.2. ER2: Open and user-friendly access to education data and indicators for
better informed decision-making at all levels
This Expected Result has been achieved in principle, but is not quite happening in practice . The
0penEMIS system has extremely rich data on a range of education sector components, along with the
tools for accessing , analyzing and sharing this data. Yet because these tools are not being utilized to
their full potential, the link between education data and better-informed decision-making at all levels is
relatively weak (see Indicator 7). Moreover, the core MoE EMIS team lacks the capacity to transform
data into useful indicators - including the development of KPls. 10
As such, education data from 0penEMIS is currently largely used to generate a limited number of
reports and to respond to ad hoc requests for information , rather than being used systematically to drive
better-informed decision-making at all levels.
4.3.1.3. ER3: New effective, easy to upgrade EMIS pilot-tested and rolled out
across all MoE institutions
This Expected Result has been achieved. The EMIS was successfully pilot tested in 64 schools , with
feedback from the pilot test used to refine and further customize the system (see Indicato r 9). Following
MoE approval the enhanced EMIS was rolled out throughout the entire country during the second
semester of the 2015-2016 school year - resulting in the eventual training of more than 110,000
Open EM IS users (see Indicator 10). The roll-out also included the training of key MoE staff in various
aspects of Open EM IS administration , management and usage .
The extensive consultation process during the scoping stage and protracted customization of the
system to be coherent with the Jordanian education system and meet the needs of various MoE
stakeholders ensured that the resulting EMIS is an effective tool for supporting evidence-based policy
formulation . The system is easy to upgrade both in the sense that custom fields , workflows , reporting
templates, visualizations and connections to other systems can easily be added , and in the sense that
the 0penEMIS platform is constantly being updated with improvements and new features.
1' ' lnterv,ew with Dr Omar AI-Jarrah Chair of OpenEMIS Technical Steering Committee. 29 November 2017. Interview with Dr
Mahmoud Smadi Statistics Consultant 29 November 2017: Support to Jordanian EMIS System - Phase 2 (September 2017 -
August 2020). ProJect Document (May 2017)
4.3.1.4. ER4: Fully-integrated Decision Support System with data from EMIS and
other relevant public-sector databases in place
This Expected Result has been achieved. At the close of the project MoE is in possession of a fully-
operational education decision support system (see also Indicator 5). This includes pre-existing
education datasets from the legacy EDUWAVE system as well as up-to-date educational data for the
2016 -2017 school year (see Indicator 6).
The OpenEMIS system has also been successfully integrated with other relevant public-sector
databases, including the Civil Service and Passports Department Database (CSPD) and the
Examinations Data Management Information System (EXAMIS) (See Indicator 11 ). The development
of an OpenEMIS Generic Application Programming Interface (API) allows OpenEMIS to connect with
other public-sector databases and systems in the future .
4.3.1.5. ER5: An EMIS system fully compliant to beneficiaries ' needs and
international standards in place
This Expected Result has been achieved. An extensive initial scoping , consultation and needs
assessment phase ensured that the OpenEMIS platform was fully customized to the Jordanian
education system and met the requirements of key MoE stakeholders . This was reinforced through the
iterative pilot testing phase and by the EMIS project team's accommodation of requests for additional
features and customization throughout the lifetime of the project. Moreover, the open and customizable
nature of the Open EM IS platform means that the system can easily be updated to comply with MoE's
future needs - from the creation of new KPls or dashboards to linking OpenEMIS with other public-
sector databases.
The project is replete with examples of the latest international standards being adhered to in the design
and implementation of the Open EM IS system - from the technical specifications of the servers that host
the system to the protocols for programming and software development. There are also examples of
consultancy services rendered under the 'Complementary Technical Advisory Services' project
component drawing on international best practice , such as the utilization of UNESCO Institute for
Education Planning and UNESCO Institute for Statistics methodology in the setting up of the National
Education Account in Jordan .
4.3 .2 . ASSESSMENT OF EXPECTED RES ULTS INDICATORS
Indicator 3: OpenEMIS operational and owned by the government
• Secondary data
• Field observation visits and interviews with benefic iaries at project implementation sites
Assessment
This indicator has been partially achieved : OpenEMIS is fully operational (see Indicator 10) and is
owned by the government in the sense that all the required hardware and software systems, data
and data collection processes (see Indicator 6), linkages with other systems (see Indicator 11) and
reporting functionalities (see Indicator 7) are in, place .
At the central level, MoE is assessed to possess the necessary hardware for supporting the
system, including 'optimum' servers (although the provision and quality of IT infrastructure at the
field directorate and school level is less consistent - a fact that was observed during field visits to
four schools as part of this evaluation) . 11 The system can also be said to be owned by the
government in the sense that commitment to using the system appears to be strong at all levels ,
from the Minister of Education to the field directorates and schools interviewed as part of this
evaluation. This observat ion is also confirmed by an EU mon itoring exercise carried out in June
2017 . 12
However, the government cannot be said to have ownership of OpenEMIS in the sense of MoE
being able to independently sustain the system following the closure of the project and withdrawal
of UNESCO support (see Indicator 4) .
All N/A
Indicator 4: OpenEMIS administration team with the capacity to sustain the use of the
system
• Secondary data
Assessment
This indicator has been partly achieved: the administration team can manage some components of
OpenEMIS independently , but MoE does not have the capacity to autonomously sustain the use of
the system .
QRC is reported as being able to competently manage OpenEMIS core, which is the component of
the system related to data and workflows pertaining to schools. 13 QRC is also driving and closely
monitoring the on-going data collection activities . 14 The QRC EMIS core team is also able to use
the system to produce basic reports and respond to ad hoc data requests (see Indicator 2 above) .
That said, the secondary data consulted as part of this evaluation as well as interviews with key
project stakeholders indicate that MoE lacks the capacity to autonomously sustain the use of the
system from both a technical and practical perspective .
From a technical perspective, an internal UNESCO capacity assessment of the core QRC EMIS
team entitled 'Support to Jordanian EMIS System - Phase 2 (September 2017 - August 2020) :
Project Document' conducted in May 2017 found that they had a generally weak command of tools
for system development, system design for linkages to other relevant data systems , system design
for data reporting , data utilization for key education management fields, and database
administration - as well as of the Linux programming language on which the whole system is based
(although it should be noted that selected members of the core QRC EMIS team subsequently
received additional training in several of these areas under the Accelerated Capacity Development
11 Support to Jordanian EMIS System Phase 2 (September 2017 August 2020) ProJect Document (May 20! 7)
l] ROM Monitoring Report (June 2017)
1-' Interview w:th Niihan Siam Jordan EMIS Programme AdvisorlCSF, 29 November 201 7
l-1 Interview w:tri UNESCO _Amman ProJect Team, 27 November 2017

Plan (ACPD). 15 The CSF developer confirmed that at the end of the project , despite the efforts of
the ACDP, the core QRC EMIS team still have weaknesses in underlying components of the
system such as programming and database management and hence are unable to autonomously
master the Open EM IS platform (see Indicator 21 ). 16 This was confirmed by the Director of the MoE
EMIS Division. 17 The Chair of the EMIS Technical Steering Committee also noted some remaining
technical capacity gaps in the core QRC EMIS team , including in building indicators, adding APls,
and environment management. 18
From a practical perspective , MoE is not yet self-sufficient in fully exploiting the system's various
reporting functionalities - including the generation of custom reports and the use of dashboards .
The UNESCO Amman Project Team reported that the QRC EMIS team has not yet been able to go
beyond the limited number of pre-defined report templates to generate custom reports for each
department, 19 while the Chair of the EMIS Technical Steering Committee noted that MoE is
currently only using one of the many dashboards that are built into the system. 20 Both of these
observations are echoed in the internal UNESCO capacity assessment of the core QRC EMIS
team conducted in May 2017 . The capacity of MoE to sustain the use of the system in MoE
decision-making processes is therefore limited.

All N/A
Indicator 5: Launch of an integrated education decision support system with data from
OpenEMIS and other relevant public-sector databases
• Secondary data
Assessment
This indicator has been achieved : at the close of the project MoE is in possession of a fully-
operational education decision support system. This includes pre-existing education datasets from
the legacy EDUWAVE system as well as up-to-date educational data for the 2016-2017 school
year (see Indicator 6). As is usual with any large primary data set, the 2016-2017 data is reported
to contain some gaps , errors and outliers .2 1 A statistics consultant has been retained by UNESCO
1' Support to Jordanian EMIS System •· Phase 2 (September 2017 - August 2020) ProJect Document (May 2017) Note that
some act1v1tiesimplemented under the ACDP drew on a d1fferent funding source
l(, Interview with N1lhan S1arn.. Jordan EMIS Programme Adv1sor/CSF 29 November 2017 P
1·· Interview with Marwar Turman Head of EMIS and E Learning Division. QRC, 5 December 2017
Interview w:tn Dr. Omar AI-Jarrah. Chair of OpenEM1S Technical Steer•ng Com m1ttee 29 November 20' 7
9 Interview w,th UNESCO An1man ProJect Teari. 19 Novern ber 2017
:,i, Interview with Dr Omar AI-Jarrah Chair of OpenEMIS Technical Steering Committee 29 November 2017
' 1 Interview with Dr Mahmoud Sr1ad1. Statistics Consultant 29 November 2017

to develop a capacity development plan for data utilization which will form a component of the
upcoming second phase of the project, of which data cleaning will be a key aspect.
The OpenEMIS system has also been successfully integrated with other relevant public-sector
databases, including the Civil Service and Passports Department Database (CSPD) and the
Examinations Data Management Information System (EXAMIS) (See Indicator 11).
All N/A
4.4. YEAR 1 ACT IVITIES
4.4.1. OVERALL ASSESSMENT OF YEAR 1 ACTIVITIES

Activities in Year 1 of the project were largely aligned with the first two project components: 'Component
1: Comprehensive scoping of MoE needs for a roust Education Decision Support System' and
'Component 2: Consolidation and Expansion of the Jordan Open EM IS indicators automation, reporting
and data visualization features.' These two project components correspond to Expected Results 1 and
2.
The indicators for Year 1 activities do not provide a very helpful basis for evaluating the contribution of
project activities towards the intended outcomes . That said, it can nonetheless be assessed that the
objectives of the first year of the project - namely the comprehensive scoping of MoE requirements in
relation to OpenEMIS, the migration of existing data to OpenEMIS, and the development of data
visualization and reporting tools and training of key MoE officers in their use - were generally achieved.
4.4.2 . ASSESSMENT OF YEAR 1 INDICATORS

Indicator 6: OpenEMIS loaded with existing education data
• Secondary data
Assessment
This indicator has been achieved: relevant historical datasets were migrated from EDUWAVE to
OpenEMIS, and the data was also cleaned. Historical Statistical Yearbooks were also migrated to
OpenEMIS. Reports on data migration and cleaning were submitted to the EMIS Steering
Committee and endorsed.
A precursor to Open EM IS being loaded with existing education data was the completion of a
comprehensive needs assessment (see the first key activity below). The project logframe does not
include an indicator related to this, but the activity is highly relevant to the achievement of Expected
Result 1: Comprehensive scoping of MoE needs for a robust Education Decision Support System .
Four relevant reports were submitted to the EMIS Steering Committee and endorsed (EMIS Needs
Assessment Report, Refining Key Performance Indicators Report, EMIS IT Infrastructure Report,
Staff Training Plan) .
Key activities I Achievements of key activities
' '
This activity corresponds largely to 'Project Component 1: Technical
reviews , needs assessments and inventories of human, technical ,
Policy and planning institutional capacities of procedural issues '.
activities including policy
The technical review component comprised a comprehensive IT
development , project
infrastructure assessment from the central (ministry) to the school
scoping, detailed needs
level, including a network connectivity assessment.
assessment and gap
analysis, data collection The needs assessment included consultations with a wide range of
analysis to assess the stakeholders (MoE Planning, Research and Development Directorate ,
needs and capabilities of Queen Rania Center, Educational Training Center, Examinations and
government institutions for Tests, Vocational Education, General Education, selected field
improved planning and directorates and schools , as well as a wide range of external
management of the stakeholders). The scope of the needs assessment included data
education system quality and tools, data quality issues, educations sector KPls and
data gaps, analysis of MoE business processes in relation to EMIS,
and reporting requirements .22
EDUWAVE data from the previous three school years was

successfully migrated to OpenEMIS, including school profiles, school
education programmes and shifts, academic structures (sections,
grades, subject classes), student directory , teachers and teaching
relations, staff positions, non-teaching staff (in school), school
infrastructure, student and staff attendance, and student marks.
The EMIS team also undertook a programme of extensively cleaning
Data migration of existing all the migrated historical datasets such that the data could be used
data to OpenEMIS system as a baseline for education sector KPls. The development and
to enable immediate use implementation of the data cleaning methodology also presented an
of the data opportunity to strengthen the capacity of the MoE staff involved in
statistical techniques for data cleaning and normalization .23
Discussions were also held with other partners to migrate relevant

thematic datasets collected under joint programmes to OpenEMIS -
including UNICEF and JEN (on WASH data on hyg iene and
sanitat ion in public schools), UNRWA (to link their EMIS data with
MoE OpenEMIS data), and UNHCR (on refugee data). 24
Indicator 7: OpenEMIS used with other data visualization and reporting tools to generate key
planning and management reports
• Secondary data
22 EMlS Interim Narrative Report Feb 2017 p 9
2 ·' EMIS Interim Narrative Report Feb 2007. p. 26
2-l EMIS lntenm Narrative Report Feb 2017 o.

Assessment
Note that this indicator corresponds closely to 'Project Component 2: Consolidation and expansion
of the Jordan OpenEMIS indicators automation, reporting and data visualization features.'
This indicator has been achieved in principle: all of the processes and enabling factors for
OpenEMIS to be used with other data visualization and reporting tools to generate key planning
and management reports are in place. MoE staff and beneficiaries are using EMIS data
visualization and reporting tools to answer requests for information and generate reports in certain
contexts. However , capacity gaps among MoE staff mean Open EM IS data visualization and
reporting tools are not being used to their full extent to generate key planning and management
reports.
The EMIS team developed an online data visualization and reporting

Data analysis for data
tool in both English and Arabic, which allows access to education
validation and quality
sector indicators and data reports (including tables, graphs and
assurance, business
maps). Calculation of indicators was also made more user-friendly.
intelligence cube and
query development, Business intelligence tools were also designed that cater for all the
interactive online technical needs of MoE for visualization and data analysis.
dashboard of key
A function called OpenEMIS dashboard was enabled for custom data
education data, publication
visualization, which allows MoE to build custom dashboards and
of educational statistical
share them with end-users.
report, education
programme monitoring of An online User Acceptance Test (UAT) environment was enabled to
key performance allow MoE to test and accept proposed changes to the Open EM IS
indicators, development of system, in line with international best practice.
education profiles to Three interactive dashboards were rolled out on the EMIS website
monitor and disseminate (Education System Overview; Indicator Mapper; Data Entry Monitor),
key data on the status of and a series of eight additional thematic dashboards were also
education in the country
developed. 25
The EMIS team began developing school profiles that are connected
to real-time OpenEMIS datasets, which will allow stakeholders (the
Use of Open EM IS system ministry, schools and families) to access a series of pertinent
to generate advocacy education sector indicators.
materials for dissemination
OpenEMIS data also contributed to an Education Sector Assessment
and visibility of the
(ESA) conducted in early 2017, from which a variety of shorter policy
education data
briefs will be developed. 26
The final status of this activity could not be determined .
Indicator 8: A team of key of MoE officers trained in use of education data for better
planning and management
• Secondary data
·' 5 EMIS lntenrn Narrative Report Feb 2017 p. 33-3'1
26 EM!S Interim Narra tive Report Feb 2017 . p

Assessment
This indicator has been achieved in principle : the project included extensive training and capacity-
building of key MoE officers in the use of education data for better planning and management.
That said , there still appears to be a capacity gap in the MoE - both in relation to the ownership
and management of the system, and in its use for informing and driving better planning and
decision-making.
Trainings and workshops held in Year 1 of the project included: (1)

OpenEMIS capacity training 12 MoE staff from the Directorate of Planning, Research and
building for education Development and Queen Rania Center on using data visualization
planners, administrators and reporting tools as well as data cleaning and indicator
and IT support in data normalization ; (2) training 17 MoE staff on updating and maintaining
analysis for use in the data visualization interface; (3) a set of advanced trainings on the
planning use of EMIS data for GIS school mapping , financial and budget
analysis, simulation models and education cost projections. 27
4.5. YEAR 2 & 3 ACTIVITIES
4.5.1. OVERALL ASSESSMENT OF YEARS 2 & 3 ACTIVITIES

Activities in Years 2 and 3 of the project were largely aligned with the third project component:
'Component 3: OpenEMIS enhancement, pilot-test, consolidation and deployment'. This project
component correspondent to Expected Result 3.
Both of the indicators for Years 2 and 3 were achieved: the pilot test was successfully completed and
OpenEMIS was subsequently scaled up to cover the entire country . Other key activities and
achievements during this phase included the full customization of the OpenEMIS platform to the
Jordanian education system and to meet the MoE's requirements, the comprehensive training of all
OpenEMIS users (from key MoE staff to more than 110,000 teachers), and the launch of data collection
for the 2016-2017 school year. Although not featured in the logframe indicators or activities for Years 2
and 3, the upgrading of MoE hardware infrastructure and subsequent verification of its ability to handle
the requirements for running Open EM IS was also an important component of this phase .
While all of these activities contributed towards the achievement of the indicators for Years 2 and 3, a
series of factors emerged during this phase that negatively impacted on the project timeline in the first
instance, and ultimately on the key objective of MoE being able to take full ownership of Open EM IS by
the end of the project (see 'Indicator 3: OpenEMIS operational and owned by the government' and
'Indicator 4: OpenEMIS administration team with the capacity to sustain the use of the system').
The first factor was the political decision to transfer responsibility for the EM IS from the Central Planning
Department to Queen Rania Center for Education Technology, which occurred just after the completion
of the pilot phase and before the full roll-out. This institutional uprooting of the Open EM IS project caused
time to be lost as responsibilities and decision-making processes within MoE became unclear.
Moreover, being strongly focused on information technology, QRC lacked the capacity and perspective
to be an effective counterpart on the reporting, •planning and decision-support aspects of the EMIS -
EM!S Interim Narrative Report Feb 2017 p. 23

leading to a capacity gap and loss of momentum in these key areas (although several members of the
Planning Department were eventually transferred to QRC) .
The second factor was the emergence of two areas of activity which drew substantial time and
resources away from the core project activities and objectives: providing EMIS data to the Ministry of
Finance so it could be used to monitor the disbursement of JD20 to every Jordanian student on behalf
of the King, and the configuration of the EMIS system as the repository for official data on Syrian
refugees in Jordanian schools . While both activities represent considerable achievements, and validate
the utility and functionality of OpenEMIS , fulfilling them drew time and resources away from the key
objective of building the capacity of the core MoE team to take ownership of Open EM IS.
The third factor is the multitude of requests for customization of the Open EM IS platform that extended
beyond the initial customization and pilot test phase (and indeed continue up to the present day). This
resulted in the development of a very rich system which became more difficult for key MoE staff to
master, while the process of accommodating these constant requests also drew time and resources
away from core capacity-building activities .
4.5.2. ASSESSMENT OF YEARS 2 & 3 INDICATORS

Indicator 9: OpenEMIS piloted in 50+ schools
• Secondary data
Assessment
This indicator was achieved : the EMIS was successfully pilot tested in 64 schools (before this a
closed pilot test was conducted in two schools). Feedback from the pilot test was used to further
refine and customize the system.
Extensive customization of the EMIS platform to fit the context of the Jordanian education system
as well as to meet the evolving priorities and requirements of the MoE took place throughout the
pilot phase and beyond.
A great deal of effort was invested in customizing the generic

UNESCO OpenEMIS platform to meet the needs of the MoE ahead of
the pilot test - including the customization of data entry forms and
adapting the system architecture to the Jordan ian educational system
and administrative structures.
In addition, two EMIS sub-modules were added on Teacher's
OpenEMIS customized
Professional Development Tracking and Quality Assurance for Early
and adapted to meet the
Childhood Education.
specific needs of the
government Numerous other additional data entry and management modules
were also incorporated after the initial customization phase, while the
survey data collection module was also expanded to be able
accommodate specific workflows in relation to schools, teachers and
staff, and students.
A substantial wo~stream was also opened up at this stage at the

request of the EMIS Steering Committee on the collection of data on
Syrian Refugees and the subsequent embedding of this data into the
core EMIS system .2 8
OpenEMIS piloted in a
representat ive sample of As Indicator 9.
schools
Indicator 10: OpenEMIS scaled up to cover all schools, students, teachers and staff
• Secondary data
Assessment
This indicator was achieved: the Minister of Education approved the roll-out of the enhanced EMIS
throughout the country during the second semester of the 2015-2016 school year.
The roll-out took place in three phases . In Phase 1, 40 regional-level trainers trained 608 field level
trainers. In Phase 2, these field level trainers trained more than 13,000 staff at the school level (two
teachers from each school) . In Phase 3, the two teachers from each school trained all other staff in
his/her school, resulting in the training of more than 110,000 OpenEMIS users.
Key MoE staff were also trained in numerous aspects of system administration, management and
usage . Open EM IS data validation and quality assurance processes were developed and
implemented, and OpenEMIS helpdesk was established .
Data collection for the 2016-2017 school year was launched in August 2016. Some difficulties were
encountered during data collection, due in large part to the decision to switch responsibility for the
EMIS (and hence data collection activities) from the Central Planning Department to Queen Rania
Center for Education Technology .
Following the extensive cascade training outlined above , the

enhanced EMIS was officially launched at an event at the Queen
Rania Center for Education Technology in August 2016 , which also
announced the launch of EMIS data collection operations for the
OpenEMIS scaled up to 2016-2017 school year.
cover all education
institutions in the country Two notable achievements from the roll-out of OpenEMIS were the
use of EMIS data by the Ministry of Finance to monitor the
disbursement of JD20 to each student from the King , and the use of
EMIS to collect and provide official data on Syrian refugees in
schools .
Data validation and quality assurance processes were developed ,

OpenEMIS data validation
operationalized and successfully tested during the pilot phase . These
and quality assurance
processes also underwent an independent audit by Amman Arab
processes operationalized
University ICT faculty .
' EMIS lntenrr, Narrative Report Feb 2017 p9. 41-43

Teams of education
planners, administrators,
Training provided to key MoE staff included : ( 1) training EMIS system
IT support personnel all
administrators and maintenance team; (2) training IT maintenance
provided appropriate
team; (3) training DEPR staff as trainers on the Open EM IS interfaces;
training to sustain the
(4) training of data entry staff and users on record-keeping, basic IT
administration of
skills, OpenEMIS data entry and user interfaces and basic statistical
OpenEMIS. Training of
literacy; and (5) school-based management training for supervisors,
trainers carried out to
principals, school board members and teachers on use of Open EM IS
enable cascade training
for decision-making.
from national level to
subnational levels
An OpenEMIS helpdesk
established within the MoE An online Support Service Desk was established to provide support to
for support to users within the central MoE team working on OpenEMIS.
the country
4 .6 . YEAR 3 & 4 ACTIVITIES

The core EMIS-focused activities in Years 3 and 4 were aligned with the fourth project component:
'Component 4: OpenEMIS expansion or connectivity to additional data sources'. This project
component corresponds to Expected Result 4. At the request of MoE and with the support of the EU,
Years 3 and 4 also saw the initiation of a set of 'Complementary Technical Advisory Services' which
sat outside of the core OpenEMIS workstream . This is reflected in the addition of a fifth project
component, 'Project Component 5: Specialized policy-level technical advisory services', which
corresponds to Expected Result 5.
Two evaluations conducted in the final year of the project indicated that MoE was further away than
anticipated from being able to take full ownership of the enhanced EMIS following the close of the
project and the withdrawal of UNESCO. As such, in the final six months of the project an additional set
of activities were launched under the auspices of an 'Accelerated Capacity Development Plan (ACDP)',
which aimed to accelerate knowledge transfer and intensively build the capacity of key MoE
stakeholders at the central and regional levels to take ownership of the enhanced EMIS following the
end of the project. 29
Both of the indicators for Years 3 and 4 for the remaining activities relating to the core Open EM IS roll-
out and implementation were achieved: the OpenEMIS platform was successfully integrated with a
series of other relevant public databases and key MoE staff received training in data analysis and the
use of the enhanced EMIS . In spite of this, the capacity of key MoE staff to manage and utilize the
enhanced EMIS to its full potential continued to lag behind in Years 3 and 4 due to the knock-on effect
of factors that emerged in Years 2 and 3 that detracted from the capacity-building and knowledge
transfer aspects of the project. The addition of the 'Complementary Technical Advisory Services' project
component, while highly useful to MoE and well-received in its own right, also drew time and resources
away from knowledge transfer and capacity building activities relating to the core OpenEMIS
workstream.
Indicators related to the 'Complementary Technical Advisory Services' were largely achieved: MoE
received the requested technical support and associated deliverables in the areas of teacher
professional standards and development, education sector finance, implementing a GIS platform to
support school mapping and planning, and impl~menting the MoE ICT strategy. One indicator relating
to the development of a vocational education work plan for implementing decisions stemming from the
_2q Note that son1e activities irnpiemented under the ACDP drew on a different funding source
IJ..; \
National Conference on Human Resource Development Strategy (HRD) could not be achieved due to
the postponement of the HRD Strategy; instead, UNESCO provided a consultant who prepared on-
demand technical notes for the Minister and provided strategic advice on TVET. Finally, it should be
noted that while a GIS platform for supporting school mapping and planning was established and is fully
operational, MoE appears to lack the capacity and resources to utilize the tool to its full potential to
support decision-making.
Indicator 11: OpenEMIS integrated with other public databases to establish an education
• Secondary data
Assessment
This indicator has been achieved : OpenEMIS is integrated with a series of other relevant public
databases.
Discussions are ongoing with the Ministry of Finance on the linking of Open EM IS with the
Government Finance Management Information System (GFMIS).
An Application Programming Interface (API) was developed to allow for the future integration of
OpenEMIS with other public databases and platforms.
OpenEMIS was successfully integrated with :

Data integration activities
to establish a complete • The Civil Service Database and Passports Department (CSPD),
and useful education allowing for the verification of every new student and staff record
decision support system, entered into EMIS against CSD data.
including OpenEMIS data , • The Examinations Data Management Information System
other public data and the (EXAM IS), allowing examination results from national tests and
development of mobile the national Tawjihi examinations to be loaded to OpenEMIS, as
apps to support data use well as facilitating additional workstreams around examination
administration .
Indicator 12: Education simulation modelling capacity building and training provided to
government officers
• Secondary data
Assessment
This indicator has been achieved : a Jordan Education Simulation Model (JESM) was delivered to
MoE.
Key activities Achievements C?fkey activities
Education data analysis Eight end-users of the JESM were trained and a restitution workshop
and management training was held in December 2016 involving key management staff from
to senior education staff in MoE.
the use of the education
Education and policy

planning activities to
support strategic planning
based on data
Indicator 13: Availability of a reviewed National Teacher Professional Standards
• Secondary data
Assessment
This indicator has been achieved : 'Teacher status and professional standards policies ' were
delivered by SOFRECO to MoE in March 2017 .
A SOFRECO expert undertook three missions between March and

November 2016 , consulting with a range of national government and
Technical Advisory non-government stakeholders and working closely with the MoE
Services to support Education Training Center (ETC) .
operalization of MoE
Teacher Policy Framework The expert gave two presentations to HE the Minister, and organized
a study tour to Georgia in August 2016 to enable MoE to learn from
teacher professional development reform processes in Georgia.
Indicator 14: Availability of generic frameworks for teacher professional development and
teacher induction and in-service programmes
• Secondary data
Assessment
This indicator has been achieved: 'Teacher continuous professional development policies ' were
delivered by SOFRECO to MoE in March 2017. A 'Teacher evaluation and appraisal system' and
'Career paths and ranking system for teachers' was delivered in April 2017.
Technical Advisory
Services to support As for key activities under Indicator 13.
opalization of MoE
Teacher Policy Framework
Indicator 15: Availability of an Education Sector Financing report
• Secondary data
Assessment
This indicator has been achieved : an 'Analysis of Government Expenditure' report was delivered by
an international expert to MoE in August 2016 .

The 'Analysis of Government Expenditure ' report outlined the key
Technical Adv isory trends in the current financ ing of Jordan's education sector . The
Services for the analysis found that 62 .2% of total education financing is directed
Rationalization of the toward the basic education, 15.3% toward secondary education,
Government of Jordan 11.1% toward higher education , and 6.3% toward general
Education Financing and administration costs.
Expenditure Framework
Indicator 16: Availability of National Satellite Account for Education
• Secondary data
Assessment
This indicator has been achieved: the foundations for a National Education Account were set up.
The setting up of the National Education Account involved

Technical Advisory consultation workshops with focal points from MoE, Ministry of
Services for the Finance, Directorate of Statistics , and the Ministry of Plann ing.
Rationalization of the
Government of Jordan The Jordan NEA followed international best pract ice from UNESCO
Education Financing and International Institute for Education Planning and the UNESCO
Expenditure Framework Institute for Statistics . A set of documents for maintaining the NEA
were drawn up and translated into Arabic .
Indicator 17: Availability of a cost simulation model
• Secondary data
Assessment
This indicator has been achieved : a Jordan Education Simulation Model (JESM ) was delivered to
MoE.

The JESM is a strategic planning tool allowing MoE to explore ,
Education simulation analyses and project the impact of a wide variety of socioeconomic
modelling to assist in parameters on the education system .
evaluating options, Eight end-users of the JESM were trained and a restitution workshop
planning and forecasting was held in December 2016 involving key management staff from
MoE . .
Indicator 18: Availability of a GIS platform to map education inequities, support decision-
making for rationalization of the school network and teacher allocation across schools
Assessment
This indicator has been partially achieved : a GIS platform - the Jordan Spatial Education Decision
Support System (JSEDSS)- is available and fully operational, although MoE appears to lack the
capacity to utilize the tool to its full potential to support decision-making .30
Technical Advisory
Services and Capacity
Development Tools in The JSEDSS supports a wide variety of workflows in relation to
GIS-based school school mapping and school planning, providing MoE with a tool to
mapping for equity and make evidence-based decisions on the rationalization of the school
rationalization of the network system .
education facilities
network across Jordan
Indicator 19: Availability of a workplan for implementation of MoE ICT strategy
• Secondary data
Assessment
This indicator has been achieved : an 'ICT Strategy Implementation Plan' was delivered to MoE .
Technical Advisory
Four National Experts provided technical advisory services to MoE on
Services on the
the operalization of the recently approved ICT Strategy in the
operalization of the MoE
teaching and learning process - with a particular focus on e-learning.
ICT strategy
Indicator 20: Availability of a MoE VE work plan for implementation of the decisions
stemming from the National Conference on Human Resource Development Strategy
• Secondary data
Assessment
This indicator was not achieved due to the postponement of the National Human Resources
Development Strategy, although a consultant still provided MoE with support and strategic advice
on TVET-related issues .
?O Interim Narrative Report p 83 See also interview \v!th Ms Safa AI-Beiruti

Technical Advisory
Services for the
incorporation of the Due to the postponement of the National Human Resources
National Conference on Development Strategy, a consultant hired by UNESCO instead
Human Resources prepared ad hoc technical notes for the Minister and provided
outcomes into MoE strategic advice on TVET.
Vocational Education
strategies and plans
Indicator 21: 80% of MoE EMIS officers at central and regional level satisfactorily trained on
their EMIS tasks
• Secondary data
Assessment
This indicator cannot be verified. However, interviews with numerous stakeholders, including with
the Director of the MoE EM IS and E-learning Division, indicate that at the close of the project MoE
EMIS officers at the central level require further training and capacity building support to be able to
satisfactorily carry out their EMIS tasks. 31

The Accelerated Capacity Development Plan aimed within the final
six months of the project to accelerate knowledge transfer and
strengthen MoE capacity at both the central and regional levels to
take full ownership of the enhanced EMIS and use it to its full
Accelerated Capacity potential following the phase-out of UNESCO.
Development Plan Specific areas of activity included: (1) Enabling the core EMIS team to
implemented perform its function and streamline its operation; (2) supporting the
expansion of EMIS use withn central ministry services ; (3) expanding
OpenEMIS features; (4) improving the quality, efficiency and
timeliness of EMIS field operations; and (5) supporting the EMIS IT
basement.
4 .7. OECD DAC CRITERIA
4.7.1 . RELEVANCE
The project was highly relevant to the beneficiaries' needs, originating from a direct request from
MoE in September 2011 to UNESCO for technical assistance in identifying gaps in the existing EMIS
system and improving it. The previous EDUWAVE system contained numerous technical weaknesses
and bugs, had very weak reporting functionality and was not able to generate indicators, and was
licensed to MoE by the vendor under commerciql terms whereby MoE had to pay up to JD 400,000 per
year in licensing and maintenance costs as well as additional fees whenever modifications or upgrades
' 1 Interview w;th Marwan Tu,rnal' Head of EMIS and E-Learn,ng D1vis1on,ORC 5 December 2017
were required. 32 Previous mIssIons by GOPA (2008), USAID (2011) and the World Bank (2011)
identified EDUWAVE as an Achilles' heel in MoE , and as partly responsible for MoE's extremely weak
capacity for evidence-based policy-formulation and decision -making . In contrast to EDUWAVE , the
UNESCO OpenEMIS system is license - and royalty-free , fully customizable, has strong reporting
functions , and is designed to be fully owned by countries adopting the system .
The project model was contextually appropriate and responsive to the needs of beneficiaries,
although the time and resources needed to train key MoE stakeholders and build capacity on
the management and use of the system was underestimated . An extensive consultation, scoping,
design and pilot-testing phase ensured that the OpenEMIS platform reflected the context and structure
of the Jordanian education system , while customization of the EMIS to meet the various data collection,
workflow and reporting requirements of MoE took place continuously throughout the lifetime of the
project. The basic project model - from design to implementation to capacity building - was appropriate
to the context. However, the project model underestimated the time and resources that would need to
be dedicated to the capacity-building component of the project. Delays in the implementation phase
meant that the capacity-building component was further squeezed . Attempts to accelerate capacity
building and knowledge transfer in the final six months of the project significantly boosted MoE's
capacity to manage and use various aspects of the system . However, at the close of the project MoE
does not have the capacity to autonomously manage the OpenEMIS platform and use it to its full
potential.
The project made a strong effort to adapt and respond to changes in the internal and external
context. The most significant internal change was the decision following the conclusion of the pilot-
testing phase by the then-Minister for Education to transfer responsibility for the Open EM IS project from
the Policy and Planning Department to QRC. The project team responded to this change by advocating
for the transfer of four Policy and Planning Department employees to the QRC to smooth this transition .
A second important change in the internal context was the mismatch between UNESCO and MoE's
initial conception of the EMIS as primarily a tool for storing and report ing on educational data and
statistics, versus the subsequent requirement from key MoE stakeholders that the OpenEMIS platform
also incorporate numerous Enterprise Resource Planning (ERP) features to facilitate various internal
MoE processes . The project team responded to this requirement by building a series of custom
workflows into OpenEMIS - including around student and teacher transfers and examinations
administration .
The most significant change in the external context was the strong focus that emerged on modifying
OpenEMIS to be able to collect, aggregate and disseminate data on Syrian refugees in Jordanian
schools . The project team responded to this change by dedicating time and resources to addressing
this issue in Project Years 2 and 3. The MoE's requirement for technical advice and support in a series
of areas outside of the core EM IS workstream - including GIS, education sector financing and teachers'
professional development - was accommodated through the incorporation of the additional
'Complementary Technical Advisory Services ' project component.
4 .7.2. EFFICIENCY
A series of internal and external factors negatively influenced the implementation of the core
EMIS elements of the project, resulting in delays. Yet given the high importance of addressing
these factors, such delays would seem to be reasonable. The project was originally forecast to last
33 months, from 1 February 2014 to 31 October 2016 - but was eventually extended by 13 months .
Due to delays with the implementation and customization of the EMIS , the main capacity building
-'2 Interview with N!lhan S1arn Jordan EMlS Programme Advisor-"CSF, 29 November 2017 Further details on the legacy
EDUWAVE system's weaknesses can be found in the Context and Justification section of the EU Ong,nal EMIS Project
Doc um ant (2013)
component did not start until around six months before the original project end date - leading to the
extension of the overall project timeline and the implementation of the 'Accelerated Capacity
Development Program' component.
The most sign ificant internal factor was the decision following the conclusion of the pilot-testing phase
by the then-Minister for Education to transfer responsibility for the OpenEMIS project from the Policy
and Planning Department to QRC . The disruption caused by this uprooting of the project was
compounded by the subsequent lack of clar ity on individual and departmental roles and responsibilities
in relation to the EMIS in MoE , as well as a lack of capacity in QRC in key skills required for managing
and using OpenEMIS to its full potential. Second, the mismatch between UNESCO and MoE 's initial
conception of Open EM IS and the subsequent requirements of key MoE stakeholders in relation to the
system (outlined above) also negatively affected the project's implementation . The additional time and
resources spent customizing OpenEMIS to accommodate a variety of different workflows had a
detrimental effect on the capacity building component, delayed project delivery overall , and resulted in
a platform that is more complex and difficult to maintain and manage than initially envisioned. The
internal factors referred to above - namely the coopting of the EMIS to provide data on Syrian refugees
and the addition of the 'Complementary Technical Advisory Services ' module - also negatively
influenced the project's implementation by drawing resources away from the core EMIS implementation
and capacity building .
The UNESCO core project team's resources appear to have been used efficiently, but there were
some inefficiencies in how CSF resources were deployed. The CSF developers being based in
Singapore meant that opportunities for knowledge-transfer to key QRC staff and on-the-job capacity
building were limited .33 Contact between CSF and QRC appears to have been largely through
UNESCO , resulting in inefficient communications processes and limited on-the-job training and
knowledge-transfer. 34 QRC staff praised the quality and efficiency of project activities delivered by
PALCO , a Jordanian company contracted by UNESCO on various aspects of the installation and
maintenance of the hardware associated with OpenEMIS .35
The amount of work delivered under the project is considerable, and as such cost efficiency is
assessed to be good. The recent EU ROM report is worth quoting in this regard : "The customization
of OpenEMIS to the Jordanian education system and the sheer amount of data that was processed
from 43 field directorates, approximately 8,000 schools, 100,000 schoolteachers and more than 1
million pupils is remarkable and required substantial resources from both the Implementing partner
UNESCO and the Beneficiary MoE ."36 A consultant contracted by the EU who is familiar with the
OpenEMIS project since its inception also remarked on the wide scope of what has been delivered. 37
4.7.3. EFFECTIVENESS
UNESCO's planning, management and coordination of the project was adequate overall,
although certain areas could have been improved. The initial design of the project appears to have
underestimated the sheer scale of the task of implementing Open EM IS in a country of Jordan's size
and the corresponding capacity building needs . It is also an open question whether the UNESCO project
---- -· ·- ·- -~.
J_; Interview w,th UNESCO Arnrnan ProJect Team 19 November 2017 Interview witll Marwan Turman. Head of EMIS and E-
L.ean,ng Division. ORC 5 De, errbe, 2017
l
1
l•1ter ✓1ew v.:th UNESCO Amma11ProJect Tearn 19 Nove<Yber 20 1 l. ROM Mon•tonng Report (June 2017)
1
' Interview with Marwan Turman Hea,j of El\:11Sand E-Le ctrning Division ORC 5 Decer,,ber 2017
1
- '' ROM Monitor:ng Report June 20171
;~ 1,,terv1eww·th Dr Joachim (Joe\ Friedrich Pfaffe EU consultant. 6 December 2017

team could have done more to resist or limit the MoE's many requests for additional features or
functionalities in the interests of ensuring that the EMIS that was handed over to MoE at the end of the
project was one it could independently manage and use . That said, once it became apparent that MoE
capacity to use OpenEMIS had progressed far more slowly than anticipated, measures were taken to
accelerate capacity building and knowledge transfer . Outputs from some of the missions and
consultancies delivered under the 'Complementary Technical Advisory Services ' do not appear to
always have met beneficiary or donor expectations, 38 although the extent to which this reflects
UNESCO 's management of the various consultants contracted to deliver the services is difficult to
assess. All MoE stakeholders reported that UNESCO coordination with MoE had been adequate,
although on the donor side a key stakeholder felt that the EU should have been more systematically
consulted and invited to meetings and events .39
The project M&E strategy suffered from some weaknesses. The project logframe is very weak. The
majority of the 21 indicators are expressed more as aspirations or aims rather than measurable and
verifiable indicators of success . The link between activities , indicators, expected results/outputs and
overall objectives is also unclear and often overlapping . The result is that the logframe is not an effective
tool for project management or for monitoring and assessing progress towards expected results and
outcomes. Assumptions and risks are also weakly formulated and appear not to have been properly
assessed . Due to these deficiencies , the logframe appears to have been largely discarded and replaced
by a detailed action plan consisting of over 60 activities as the key project management tool. As such,
project implementation seems to have become very activity-driven rather than objective-driven.
Monitoring was therefore mainly focused on the implementation of activities rather than on the
achievement of objectives .40 MoE also appears to have been a weak counterpart for the project's M&E
strategy and did not systematically produce expected progress reports on Open EM IS implementation. 41
Beneficiaries were overall satisfied with the services delivered, but recognize the need for
additional capacity building . All key MoE stakeholders interviewed felt that the implementation of
OpenEMIS had been a success, yet all emphasized that additional capacity building and knowledge
transfer is necessary to enable MoE to manage and use the system to its full potential. Key MoE
stakeholders also reported that they were satisfied with the services delivered under the
'Complementary Technical Advisory Services' module , although the previous EU ROM report , the
UNESCO EMIS project narrative report and a key EU stakeholder reported that quality had been
variable across these ad hoc consultancies . A key QRC stakeholder, while generally satisfied with the
way the project had been managed and delivered , highlighted two key weaknesses : the delays in
implementing and customizing the system to meet MoE requirements , and the fact that training and
knowledge transfer was carried out sporadically in an ad hoc manne r rather than implemented
systematically with predetermined curricula, written materials, and time allocated for direct supervision
and on-the-job training.42
An unintended negative consequence of the pressure to respond to external demands on the

EMIS was a dependency by the core MoE EMIS team on the UNESCO project team. MoE came
under heavy pressure from other government department and donors to provide data on Syrian
refugees in Jordanian schools . As QRC lacked the capacity to utilize the EMIS to provide the required
'' ROM Monitonng Report (June 201 lnterv,ew with J.3cob Arts. EU Programme Manager Educat,on and Youth Cooperation
Section 5 Decernber 20 1 7
1
'' Interview wih Jacob Arts. EU Programme i,1anager Education and Youth Cooperat,on Section 5 December 2017
This issue ,•,as aiso ,a1sed ,n the ROM Mon,tonng Repor t (June 2017)
./I Interview with UNESCO Amman Pro1ect Team 19 November 2017
-1.' Interview w,th Marwan Turman. Head of EMIS and E-Learrnng D1v1s1on.ORC 5 December 2017
/
data, the urgency of meeting these demands meant the UNESCO EMIS project team often had to step
in as the primary point of contact for EMIS data and reporting requests .43 An EU consultant who
periodically required EMIS data in order to monitor education sector performance also reported that , as
QRC were not able to respond to his requests for EMIS data, the UNESCO project team became his
primary interlocutors for EMIS data requests. 44
4.7.4. IMPACT
The project is well-positioned to generate meaningful change in the future. With the
implementation of the EMIS complete and the system fully operational , future support can focus strongly
on building MoE capacity to autonomously manage and utilize the system to its full potential. An
important impact of the project in the medium term will be the utilization of the EMIS to generate and
monitor KP ls during the implementation of the upcoming Education Sector Plan . The EMIS should also
support MoE in continuing a positive and constructive relationship with donors by allowing it to respond
effectively to requests for data and demonstrate evidence-based decision-making .
4.7.5 . SUSTAINABILITY
The core IT and data collection aspects of the EMIS are likely to be sustainable into the future,
although use of the EMIS for evidence-based policy-making needs to be further reinforced
before it is sustainable . The OpenEMIS platform is now successfully established and data collection
procedures across field directorates and schools are in place and likely to be sustainable . Further
support will be required to strengthen the link between the core IT and data collection activities and
reporting , such that the systematic incorporation of EMIS data into decision-making becomes
sustainable .
4.8. GENDER
A significant proportion of central MoE staff who have received training and capacity-building on
OpenEMIS are women . At the school level, most principals and teachers who have received training
on OpenEMIS and are largely responsible for OpenEMIS data collection are women .
4 ' Interview w:th UNESCO Amman ProIect Team 19 November 2017
" 4 Interview w,til Dr Joachim (Joe) Friedricl1 Pfaffe EU consultant, G December 2017
5. LESSONS LEARNED
1. The design of the project underestimated the magnitude of implementing OpenEMIS in a
country of Jordan's size and the corresponding capacity building needs of MoE. The full
customization of OpenEMIS to the Jordanian context and MoE requirements , roll-out across the
country and linking of the system to other public-sector databases took much longer than initially
anticipated. MoE's institutional capacity to absorb the OpenEMIS technology was also lower than
expected . Efforts towards the end of the project to intensively build MoE capacity and accelerate
knowledge transfer were not sufficient to ensure that MoE was able to independently manage and
utilize Open EM IS to its full potential by the close of the project.
2. A disconnect between UNESCO/MoE's initial conception of the EMIS and the subsequent
vision of other key MoE stakeholders for the system's purpose and functionality delayed
and complicated OpenEMIS implementation. The initial idea behind Open EM IS was that it would
be primarily a tool for aggregating, analyzing and reporting on educational data and statist ics to
provide a basis for and strengthen evidence-based policy- and decision-making. However , from the
pilot-testing phase onwards the UNESCO project team had to accommodate repeated requests
from MoE that OpenEMIS be customized to support various MoE internal processes and workflows
- from student and teacher transfers to examinations administration . The expansion of the scope
of Open EM IS to accommodate numerous Enterprise Resource Planning (ERP) features led to the
creation of a system that is more complex to manage and use than initially anticipated , and drew
time and resources away from building MoE capacity on the core data analysis and reporting
aspects of OpenEMIS.
3. A lack of clearly defined roles and responsibilities within MoE in relation to the EMIS
negativity impacted project implementation and sustainability . MoE's decision early in the
project to transfer responsibility for OpenEMIS from the Policy and Planning Department to QRC
caused delays to the project timeline and a lack of clarity around departments ' and individuals ' roles
and responsibilities in relation to OpenEMIS within MoE. The decision also served to weaken the
link between the core data storage and management aspects of OpenEMIS and the use of
educational data to drive evidence-based policy-making: QRC is focused on data collection and
management of the IT aspects of OpenEMIS, but lacks the capacity and perspective for generating
custom reports and indicators and linking EMIS data to planning and decision -making processes.
At the same time, due to a combination of staff turnover, lack of capacity and lack of engagement
with the OpenEMIS project among certain key stakeholders, the Policy and Planning Department
is also not able to generate reports and drive the integration of EMIS data in MoE planning and
decision-making processes .
4. The project adapted well to unanticipated external developments , but these had an
unavoidable negative impact on project implementation. The need to respond to out-of-scope
external requests in relation to the EMIS - most notably the requirement to modify OpenEMIS to
collect and disseminate data on Syrian refugees in Jordanian schools and to support the Ministry
of Finance in the one-off disbursal of government funds to every Jordanian student in a public
school - drew time and resources away from core Open EM IS implementation and capacity-building
activ ities. Yet meeting these requests served to increase OpenEMIS visibility and credibility among
key MoE and donor stakeholders . The same is true for the addition of the 'Complementary
Technical Advisory Services' project compQnent in the final year of the project - which provided
important support to MoE in a series of key strategic areas and reinforced UNESCO's credibility at
the policy level, but impacted negatively on the core EMIS workstream .
): :,
5. Top-level MoE engagement and support is crucial for the successful implementation and
sustainability of a large-scale technical assistance project such as OpenEMIS. The decision
by the former Minister for Education to transfer responsibility for the EMIS from the Policy and
Planning Department to QRC left the EMIS institutionally adrift and without clear ownership and
oversight of the project within MoE . This negatively impacted on the project timeline and left the
UNESCO project team without the most suitable interlocutors for building capacity on the data
utilization and reporting functions of OpenEMIS . The current Minister for Education's reportedly
stronger direct engagement with the EMIS provided a boost to the implementation of OpenEMIS
activities in the final year of the project (notably data collection for the 2017-2018 school year) , and
provides a positive foundation for the next phase of the project.
6. RECOMMENDATIONS
Any future iteration of the project should:
1. Continue building QRC's capacity to maintain and manage the core IT aspects of the
OpenEMIS platform, but identify a more suitable MoE counterpart for strengthening the link
between EMIS data and its utilization for evidence-based policy formulation. QRC will
continue to house the physical OpenEMIS hardware and technological infrastructure, and will
require additional capacity building to be able to independently maintain and manage the system .
QRC is also likely to have primary responsibility for driving EMIS data collection. However, QRC
lacks the capacity to use EMIS data to develop and monitor indicators and generate custom reports
for various MoE departments . As such, a more suitable MoE counterpart should be identified to
take responsibility for utilizing EMIS data for evidence-based policy formulation. Given the currently
weak capacity of the Policy and Planning Department in this regard, this is likely to entail identifying
suitable counterparts across other MoE departments or supporting the set-up and capacity building
of a new, dedicated MoE unit or sub-department for evidence-based policy formulation.
2. Incorporate broader institutional support for building a culture of evidence-based policy

making in MoE. Even with a bolstered QRC capacity to manage and maintain the system and the
identification of suitable MoE interlocutors for strengthening the utilization of EMIS data,
effectiveness and impact will still be limited without broader efforts to support the development of
an institutional culture of evidence-based policy making throughout MoE. Such activities might
include providing key MoE departments (outside of QRC and the Policy and Planning Department)
with training and capacity building on best practices in evidence-based policy-making, and
supporting cross-departmental sharing of data and collaboration on reporting and indicator
monitoring.
3. Conduct a comprehensive stakeholder consultation to ensure cross-MoE consensus on the

primary features and functionalities of the OpenEMIS platform, and clarify institutional roles
and responsibilities in relation to the EMIS. While a key advantage of OpenEMIS is that it can
be continually modified and customized, the project should urgently seek to establish cross-MoE
consensus on the primary features and functionalities of OpenEMIS. With the system considered
'settled', support can focus on capacity building and optimizing the use of EMIS data to its full
potential, rather than on continued modifications and customization . The project should likewise
seek to establish clearly-defined departmental and individual roles and responsibilities in relation to
the EMIS. The results of these processes could usefully be encapsulated in a written charter to
serve as a common point of reference .
4. Incorporate change management, communications and stakeholder management

workstreams into the core project management component. Sustainably embedding the EMIS
into a newly-established culture of evidence-based policy formulation in MoE will require a more
holistic project management approach that goes beyond the mere coordination of technical
assistance. A change management module will help to smooth the path for the broader utilization
and embedding of the EMIS across MoE, a communications plan will disseminate clear messaging
around the uses and benefits of OpenEMIS, and a stakeholder management plan will ensure buy-
in and support from key MoE stakeholders .
5. Ensure the project is coordinated and aligned with other donor support to MoE relating to
planning, M&E, research and evidence-based policy making . OpenEMIS capacity building and
support for evidence-based policy making should be careful coordinated with the development and
monitoring of indicators for the upcoming Education Sector Plan, as well as other relevant donor
'•,•, r
initiatives such as the DFID Educational Research Programme and the broader ERfKE framework .
Coordination with donors on approaches to supporting the integration of Syrian students in
Jordanian schools - and likely data needs - will also help to anticipate external demands on the
EMIS .
7. ANNEX 1: LIST OF SECONDARY DATA
The table below lists the documents and secondary data that was reviewed as part of the evaluation :
Document Source
EU project documents
Original EMIS Proje ct Document (2013) EU
Revised EMIS Project Document - Amendment 1 (2014) EU
Evaluations and assessments
Support to Jordanian EMIS System - Phase 2 (September 2017 -

UNESCO
August 2020): Project Document (May 2017)
ROM Monitoring Report (June 2017) EU
Narrative reports
EMIS Interim Narrative Report (February 2017) UNESCO
Project Brief Update (September 2017) UNESCO

8. ANNEX 2: LIST OF INTERVIEWS WITH KEY
PROJECT STAKEHOLDERS
The table below lists the key project stakeholders and implementing and sub-contracted partners with
whom interviews were conducted as part of the evaluation :
Name Position
Key stakeholder: Ministry of Education
Key project counterparts
MoE Secretary General for Administration and

HE Sarni Salaytah
Financial Affairs
Director of DCU (Development & Cooperation

Eng. Firyal Aqel
Unit)
Planning & Research Department
Dr. Saleh AI-Khalayleh Director of the Planning Department
Dr. Abdullah Hassoneh Director of the Strategic Planning Directorate
Head of the school mapping unit/Planning

Eng. Safa' Beiruti
department
Queen Rania Alabdullah Center for Technology in Education
Marwan Turman Head of EMIS & elearning Division/QRC
Key stakeholder: EU Delegation
Attache/Programme Manager Education and

Job Arts
Youth Cooperation Section
Dr. Joe Pfaffe EU Education Consultant
Key stakeholder: UNESCO Amman
Costanza Farina Director of UNESCO Amman Office
Yukiko Matsuyoshi Education Chief
Pierre Chapelet/Sa'eda Ramzi/Yuji Utsumi Project Manager & Project Team
Key stakeholder: EMIS Sub-Technical Committee & Individual Consultants
Chairman of JUST University (Jordanian

Dr. Omar AI-Jarrah University of Science & Technology; Chair of
OpenEMIS Technical Steering Committee.
Associate Professor / Department of

Dr. Mahmoud Smadi mathematics & Statistics/JUST University ;
Statistical Consultant
Implementation partner: Community Systems Foundation
Nilhan Siam Jordan EMIS Programme Advisor/CSF
Sub-contracted partner: PwC
Nawwar Abdul Majeed Project Manager/ PwC

9. ANNEX 3: LIST OF FIELD OBSERVATION VISITS
AND INTERVIEWS
The table below lists the locations for field observation visits and people with whom interviews were
conducted in Field Directorates and schools as part of the eva luation:
Name Position
Middle Region - Amman Governorate
Liwa AI-Jamma Field Directorate
Rodainah Salamat EMIS Officer in Field Directorate
Um Tofayl Secondary School
Aysha Shurbaji EMIS Officer in the school
Middle Region - Zarqa Governorate
Zarqa 2 Field Directorate
Zarqa 2 Field Directorate, Resources

Alaa Alzyood
Department
lskan Alhashmiya Mixed, Double-shift School
Maysoon Ali Omari Principle, Second Shift
Mervat Taib Alrosan Computer teacher, EMIS Officer
North Region - lrbid Governorate
Liwa Bani Obeid Field Directorate
Kholoud Shaawatah Head of Planning , Field Directorate
AI-Hoson Secondary School
Rania Shraideh EMIS Officer in the school
Middle Region - Madaba Governorate
Madaba Field Directorate
Mohammad Al Thawabieh Head of Planning, Field Directorate
Khadija Bent Khuwailed Primary School
Areej Khalil Alqaisi Principle
Zikra Fayed Quteish Computer Teacher , EMIS Officer

10. ANNEX 4: BENEFICIARY INTERVIEW
GUIDELINES
ENGLISH
Introduction to the evaluation and the context of the interview ,
1. Could you please tell me in your own words what you understand about EMIS? What is it? How
does it work? Why is it important?
2. Do you feel that EMIS has been well-explained to you and that you understand the system and its
uses?
3. Were you consulted on the design of the current EMIS? Was there an opportunity for you to have
input into what they system should look like and do?
4. How do you personally interact with EMIS? What are your responsibilities in relation to the system?
5. Did you receive training on using EMIS? Was this training helpful? Is there any way you think the
training could be improved?
6. If you have a problem with the system of there is something you don't understand , how do you
resolve it? Is there someone you can go to for help?
7. Do you feel that you have sufficient knowledge and understanding to use EMIS effectively? What
are the things you understand well about the system, and things you don 't understand so well?
Who would you ask if there is something you don't understand? Have you sought help on any
aspect of using EMIS before? What was it and who did you go to for help? Were you able to resolve
the issue?
8. How much time do you spend using or updating EMIS? Per day? Per week? Does this interfere
with your other work? Would you say that EMIS has any negative impacts on your job? If so, what
are they?
9. Do you find the system user-friendly? Do you experience any problems using the system? Is there
any way you think it could be improved?
10. How does your field directorate/school use EMIS? Can you give me some examples of how you
have used it for decision-making? Or how the system has been useful to you in doing your work?
11. In your job, what are the most important types of data that you need to make decisions? Do you
use EMIS to get this data? Are there other systems or means for obtaining this data?
12. What kind of data does EMIS provide on students? How do you update and use this data?
13. What kind of data does EMIS provide on teachers and staff? How do you update and use this data?
14. What kind of data does EMIS provide on the school and its facilities? How do you update and use
this data?
15. Do you feel that EMIS is useful? Do you think it improves and strengthens the education system in
Jordan? If you met a colleague from another country where they do not have an EMIS, would you
recommend it to them? Why/why not?
16. Are there any particular things that you feel make it difficult to successfully implement and run an
EMIS such as this in Jordan? What are they? How could the system be better suited to the local
context in Jordan?
ARABIC
I ~ ., . )
\. ....
't- ''t..
t
....
[,
C
~
....
[,
[,
''i.
i;-
....
~
b<';"
-~
J
L
~
i;-
(,,
·-~
G.,
'[ f ;~
;~
C
t f"' ~ r
(,,
.:-
'\
.c "
~-
~-
" ~
tf . ..l'''
t
t-· ~. (,,
7'... >!.- i;- .,.
T er
f.'I ~ -
~
t l· ~i
(: ·• E E r.
1 c ....i· .... ~t 1 f, -~.,. .r ·~
"l l '-__ '.··(. ~i;- I
.[. ~- :[[, .:,,
~ [
' ,c ~
...: f r=
.,;: t ~-
~ ''t.. ;; l { {
l ~- tr
' i.. ' .
~
l'- ••
~
....
_t,.
t.
:
;,t
E , .•·
t
. ~.
t ·. . ~
·<t.
'~
'"- 'Ee f
(,,
i;-
·_,
..
t
l 'r- l_ ~
: fc
-
f t ~t
·
r .,-:f.
t
{ r·( ·'t,. -
t 1 :l
·-~ _~-
1 ~£ r- l'L. ~ ....
i.,,
L - C... - -• ' .t
~ ''t.. \· ~ ~
.,. i.,, i.,, .,. - . - • "' L "'
.(•· . t .(·· e: r
' f l-
y; _ ·'t,.
~
L . \.
.~' ·'t,t. ...~, [~~ L F'
:_
_~F . );. f \;-· ,t·.. \." 1. ~~ ~~
c;, ..
i;- ,
~- ' ~
' t..
·t
L:
1(,, l { - l.t :E l
..
~
C _;
\ ·(;,( ~ ''t.- .... l. (,, f ' ;."' [
~
.
'r, ....·C
'L r-
....
I
.
t
I ·-~t
~ t'
t
l'o,
~-f
l- I:; tt \,f
. -
;."' f ~ f ·~ ~ l f t
t
.- ·
t t ,[
r1, 1E· l.: \•t, ~'i-: \-' .:,,
~ ~ r i
~ 'i. ~
~ ,r ,(_
ti? " 't
{J
ii' ~•
ff: ' ·f ' . ~- ~
~ ['-
- 1- en .§i k
~~... 'g
'f-£££
~~~~(SC
(,, ....t t t t );.:~
(,
~
t
fl'- 'ff':- :
~ ' f . [, tf
(SIS
,\:, l.
t'
-
t'~''i.
. ;."'
f fu [,
S
' . ;~ b
' - ~''- ~- ~t
~[·: E ~- ~ f. ~ < 1-.-~ [ ~ ~~ .~ f ~-.~ t- 1.
f. . t
r,,. ..
.c r;,"' 1
....1 'ii
. : '1::_ '1::_ '1::. (,,
- - - (,,
~
r .: e ..~. t, ' ~..(,,
''i,
b E. -~
~~-'!:
~-·- ' · ~ t C.
.... ~"
' '1, \- -~ '; ~
G{
. 'e-
,. [ ~- ....
I:
' i.. .:,, f
.. [,
~
-~ ~- i:: 1. l ·E,E t r ~..[: .~ ~ ~ r ~ 1 • t r ! t .[: f, 't 't; ::
C· ~ - [, [, ·<t, ·
[, -; - - t:,
't,_ ~g;~
~ ~
b
~
r-\'t. b
~
. e~~
~
.. ~
~[~~~ -~~~
~
.
b ••
,-I 0
"' -1 --< ~
-" > < ,-I 0
"' -1 --< ~
11. ANNEX 5: OECD DAC CRITERIA INDICATIVE
QUESTIONS
Criteria Indicative questions
Relevance
• When it was designed to what extent were the overall objectives of
the project and those of each component consistent with the needs
of the country and the beneficiaries' requirements?
• To what extent was the project model contextually appropriate and
responsive to the needs of the beneficiaries? How has this model
gained resonance with or generated support from local
stakeholders (government , private sector, communities)?
• As the context continuously changed was the Programme and each
component/activities still relevant? Were the changes made in
response sufficient?
• Was it appropriate to have a country project of six components and
multiple activities? Were the components and activities integrated
or not, and if not, should they have been?
• Which activities of the project should be flagged for further
exploration or analysis?
• To what extent has the project fostered coord ination with other
funders/initiatives and integration with other providers of similar
activities?
Efficiency
• Did any external/internal factors positively or negatively influence
the project's implementation? Were any problems or bottlenecks
encountered? If so, what were they?
• Were the outputs delivered in a timely manner? Where applicable,
were delays experienced by the project reasonable, given the local
context?
• What measured were taken to ensure that resources were
efficiently used?
• Was the project overall, and each component/activity, cost efficient
(funding, personnel, installations and resources?)
Effectiveness
• How well did UNESCO plan, manage, and coordinate the project
overall?
• To what extent were the objectives achieved? What major factors
influenced the achievement or non-achievement of the objectives?
How well were the objectives designed initially and what can be
improved for the design of future projects?
• Did the context in Jordan affect delivery of the project? Did
UNESCO plan well for potential changes?
• Were there examples of innovation and best practice?
• How effective and useful was the project's M&E strategy? How
adequate were data collection tools to obtain useful and
valid/reliable data? To what extent were data supporting project
related decision-making and allowing the program to make timely
adjustments?
• To what extent were beneficiaries (direct and indirect) satisfied with
the services delivered?
• What were the positive and negative, primary and secondary short-
term effects produced by the project, directly or indirectly, intended
or unintended?
' '
r
r
Impact
• To what extent is the project positioned to generate meaningful
change in the future, considering that the impacts of such
interventions are often longer term in nature?
• What are possible short- and medium-term positive/negative
impacts of the project?
• How can positive aspects be maximized or replicated/scaled up in
the future?
• To what extent are differences in impact explained by variations in
implementation?
Sustainability
• Will the benefits from the project continue into the future and what
is the probability of continued longer -term benefits?
• What aspects of the project are likely to be most sustainable into
the future?

OpenEMIS Jordan Evaluation 2017

Uploaded by

Copyright:

Available Formats

OpenEMIS Jordan Evaluation 2017

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

OpenEMIS Jordan Evaluation 2017

Uploaded by

Copyright:

Available Formats

,

1. EXECUTIVE SUMMARY ............................................................................................ 3

4. MAIN FINDINGS ....................................................................................................... 10

5. LESSONS LEARNED ............................................................................................... 34

4. Incorporate change management, commun ications and stakeholder management workstreams

• Interviews with key project stakeholders and implementation and sub-contracted

3.2. EVALUATION MATRIX

Table 1. Evaluation matrix

Development objective: To Indicator 1: Timely, • MoE (key • Qualitati ve • Qualitati ve

(field (UNESCO , EU,

Indicator 15: Availability

Indicator 16: Availability

Indicator 17: Availability

Indicator 19: Availability

Indicator 21: 80% of MoE

OECD DAC Criteria Please refer to Annex 5:

4.1.1. ASSESSMENT OF DEVELOPMENT OBJECTIVE

4 .1.2 . ASSESSMENT OF DEVELOPMENT OBJECTIVE INDICATOR

Key activities Achievements of key activities

4.2. IMMEDIATE OBJECTIVE: To ESTABLISH A GOVERNMENT-OWNED , COST-

4.2 .1. ASSESSMENT OF IMMEDIATE OBJECTIVE

4.2 .2. ASSESSMENT OF IMMEDIATE OBJECTIVE INDICATOR

Basis for assessment

Key activities Achievements of key activities

9 Interview with UNESCO Amman Project Team. 27 November 2017

4.3.1. OVERALL ASSESSMENT OF EXPECTED RESULTS

4.3 .2 . ASSESSMENT OF EXPECTED RES ULTS INDICATORS

Indicator 3: OpenEMIS operational and owned by the government

Basis for assessment

Key activities Achievements of key activities

Basis for assessment

l] ROM Monitoring Report (June 2017)

l-1 Interview w:tri UNESCO _Amman ProJect Team, 27 November 2017

Key activities Achievements of key activities

Basis for assessment

some act1v1tiesimplemented under the ACDP drew on a d1fferent funding source

9 Interview w,th UNESCO An1man ProJect Teari. 19 Novern ber 2017

' 1 Interview with Dr Mahmoud Sr1ad1. Statistics Consultant 29 November 2017

Key activities Achievements of key activities

4.4. YEAR 1 ACT IVITIES

4.4.1. OVERALL ASSESSMENT OF YEAR 1 ACTIVITIES

4.4.2 . ASSESSMENT OF YEAR 1 INDICATORS

Basis for assessment

Key activities I Achievements of key activities

EDUWAVE data from the previous three school years was

Discussions were also held with other partners to migrate relevant

Basis for assessment

22 EMlS Interim Narrative Report Feb 2017 p 9

2 ·' EMIS Interim Narrative Report Feb 2007. p. 26

2-l EMIS lntenm Narrative Report Feb 2017 o.

Key activities Achievements of key activities

The EMIS team developed an online data visualization and reporting

The final status of this activity could not be determined .

Basis for assessment

·' 5 EMIS lntenrn Narrative Report Feb 2017 p. 33-3'1

26 EM!S Interim Narra tive Report Feb 2017 . p

Key activities Achievements of key activities

Trainings and workshops held in Year 1 of the project included: (1)

4.5. YEAR 2 & 3 ACTIVITIES