EGR Benchmarking Workshop Phillipines FINAL PDF
EGR Benchmarking Workshop Phillipines FINAL PDF
EGR Benchmarking Workshop Phillipines FINAL PDF
Philippines
were the national averages for performance in
Filipino and English (since benchmarks in these
Results of the benchmarking two languages would be for performance across
workshop held on August 27, 2014 the whole country). In addition, data from the 2014
EGRA survey of reading performance in four
mother tongues were presented for use in the
INTRODUCTION workshop.2 The 2014 survey evaluated student
reading achievement in Ilokano (in Region I),
On August 27, 2014, USAID’s Education Data for
Hiligaynon (in Region VI), Cebuano (in Region VII)
Decision Making (EdData II) project and Basa
and Maguindanaoan (in ARMM). Data on student
Filipinas Project (Basa) jointly supported a
performance in each language were analyzed and
workshop to use the results from early grade
worked with separately so as to develop language-
reading assessments (EGRAs) conducted in the
specific benchmarks.
Philippines to develop benchmarks for reading
performance in the first years of elementary The approach to benchmarking followed a
education. The Philippine Department of Education methodology developed and used by EdData II in
(DepEd) assigned staff from the office of the several other countries. At the heart of this method
Undersecretary for Programs and Projects, the is the goal of having children learn to read well
Bureau of Elementary Education, and six regions to enough and early enough to be able to apply their
attend this workshop. The regions that participated reading skill in school and as lifelong learners.
included Region I, Region IV-A, the National Therefore, the emphasis is not just on the
Capital Region (NCR), Region VI, Region VII, and mechanics of reading, but on developing the ability
the Autonomous Region of Muslim Mindanao to read with understanding. To read with
(ARMM). A total of 49 participants spent the day comprehension, children need to recognize familiar
examining data from the EGRA surveys conducted words with automaticity and decode unfamiliar
in 2013 and 2014 and used those data to propose ones with minimal effort, group words quickly to
benchmarks for reading performance in grades 1 gain meaning from sections of text, and move
through 3 for Filipino and English and for four effortlessly enough through text to retain what they
regional mother tongues: Ilokano, Hiligaynon, have read. These skills are captured in the
Sinubuanong Binisaya (Cebuano), and assessments of students’ oral reading fluency
Maguindanaoan. (ORF). And the relationship between fluency and
comprehension is well established. In fact, based
Data were made available from the national EGRA
on the data from the 2014 survey in the Philippines,
survey conducted in February 2013.1 That study
oral reading fluency explains between 65 and 70
assessed the Filipino and English reading skills of a
percent of the variation in comprehension for all
sample of grade 3 students drawn from six
four languages. The approach for setting
geographic areas so as to show results at the
benchmarks therefore looks closely at the data for
subnational and national levels. The relevant data
each language to identify the specific levels of
for the purposes of the benchmarking workshop
reading fluency (particular to each language) that
1
The 2013 EGRA report can be found on the EdData II
2
website: The 2014 EGRA report can be found at:
https://www.eddataglobal.org/countries/index.cfm?fusea https://www.eddataglobal.org/countries/index.cfm?fusea
ction=pubDetail&ID=513 ction=pubDetail&ID=626
1
correspond to students being more fully able to benchmarks were developed. Following the
comprehend what they read. The types of data methods section are recommendations for reading
used and the process employed by the workshop performance benchmarks from each of the groups
participants are discussed further in the next (i.e., for each of the languages), including
section of this report. comments from the EdData II team regarding the
proposed benchmarks. Then readers will find the
One final introductory comment: At the end of the specific timeline for next steps as proposed by the
workshop, the participants were keenly interested workshop participants. The conclusion of the report
in discussing how the results of their work— discusses some issues worth further consideration.
proposed benchmarks for reading comprehension
and oral reading fluency—could be used in some THE BENCHARMARKING METHODOLOGY
official capacity by DepEd. Participants were AND PROCESS
particularly concerned about being able to engage
a broader cross-section of stakeholders in their As stated above, the workshop was held on August
respective regions in understanding the 27, 2014, and included 49 participants. Among
benchmarks and then building consensus around them were 5 to 7 participants from each of the
them so that the benchmarks could be incorporated regions, 5 people from the DepEd central office, 5
into regional strategies and plans for improving people from the Basa Filipinas project, and 2 from
learning outcomes. At the same time, they EdData II.3 EdData II and Basa collaborated on a
recognized the need to present their recommended presentation that reviewed the data from the 2013
benchmarks to the central leadership within the and 2014 surveys and that mapped out the process
Department of Education. Therefore, the group for developing benchmarks. Almost all the
proposed a process for carrying forward the work participants in the benchmarking workshop had
on benchmarks that included the following also attended the policy dialogue held on July 23,
sequence of next steps: 2014, during which the results of the 2014 EGRA
study of four mother tongues were presented and
Document the results of the workshop (i.e., discussed in detail. Therefore, only a quick
the writing of this report); overview/review of those findings was necessary.
Following that presentation and questions and
Share the report with the regional teams, answers related to both the available data and the
which would engage the leadership in each proposed methodology, the step-by-step process
region to validate the results produced in was laid out for each group to follow. The
the workshop; participants were organized into six working groups
Once validated by the regions, transmit the according to regional/language affiliation, so that:
documentation of the workshop results to
the Bureau of Elementary Education, which Participants from Region I worked on
will consolidate the regional benchmarking benchmarks for reading in Ilokano
recommendations for submission to the Participants from Region IV-A worked on
Undersecretary for Programs and Projects; benchmarks for reading in Filipino
The Office of the Undersecretary for Participants from Region VI worked on
Programs and Projects will make additional benchmarks for reading in Hiligaynon
inputs and/or comments as deemed
necessary prior to submitting the proposed Participants from Region VII worked on
benchmarks to Secretary of Education for benchmarks for reading in Sinubuanong
approval; Binisaya
Have the benchmarks for reading performance Participants from NCR worked on benchmarks
in Filipino, English, Ilokano, Hiligaynon, Cebu- for reading in English
ano, and Maguindanaoan officially finalized, Participants from ARMM worked on
approved, and issued as a DepEd order. benchmarks for reading in Maguindanaoan
In the context of wishing to respect and support the The process of setting the benchmarks was
above process, the present report was prepared. recognized to be in part analytical and in part
The methods and data used at the workshop are
presented next so that as the results are 3
communicated, others can see how the A full list of participants and their affiliations is included
in Annex A to this report.
2
common sense. Participants had to combine document. In addition, each group was given
information from the reading assessments with cumulative distribution tables for comprehension
their knowledge of what is possible in Philippine and oral reading fluency for grades 1 and 2 for
classrooms, bearing in mind in particular the mother tongues and for grade 3 for Filipino and
current difficulties administrators, teachers, and English. These tables showed the percentage of
students face in implementing the new mother- students reaching or exceeding different levels of
tongue based multilingual education (MTB-MLE) performance, so that when considering different
curriculum. The objectives of MTB-MLE regarding levels for a benchmark, each group could take into
acquisition of literacy in mother tongue, Filipino, account the percentage of students presently
and English were another important piece of meeting those levels (and thus weigh how
information that participants took into account. In ambitious the benchmark would be).
fact, those curricular objectives served as important
starting points for determining when students Each working group was charged with defining the
should be reading with comprehension in each following for reading comprehension (defined as
language. MTB-MLE stipulates the following the percentage of correct responses students
expectations. Students should: should attain) and for oral reading fluency (defined
as the number of words of grade-level text a child
Read and write in mother tongue by the end of should read correctly per minute):
grade 1 and continue literacy development
through grade 3. A long-range goal
Begin reading Filipino in the latter part of A benchmark (which might or might not be
grade 1, be reading and writing in Filipino in different from the long-range goal)
grade 2, and continue their literacy The percentage of students who would meet
development in grade 3. the benchmark by the end of school year (SY)
Begin reading English in the second semester 2015–2016
of grade 2 and be reading and writing in English The percentage of students who would score
in grade 3. zero by the end of SY 2015–2016
Each group was charged with developing Groups working on benchmarks for mother tongues
benchmarks for reading comprehension and oral began working on grade 2, then worked backward
reading fluency for the relevant grades for the from grade 2 to set benchmarks for grade 1, and
language on which they were working. Therefore, worked forward from grade 2 to set benchmarks for
workshop participants aimed to establish grade 3. For the mother tongues, the 2014 survey
benchmarks for reading fluency and provided data for grades 1 and 2. For grade 3
comprehension for mother tongues for grades 1 benchmarks, the working groups had to assume an
through 3, for Filipino for grades 2 and 3, and for additional year of “growth” in student performance.
English for grades 2 and 3. For Filipino and English, the groups began working
on grade 3, and then worked backward to set
In addition to the curriculum objectives defined by
benchmarks for grade 2. In this case, the 2013
MTB-MLE, data from the EGRA surveys were the
survey provided data on grade 3 performance, so
most important input to each group’s work. For
groups had to “subtract” from the grade 3
each language, participants were given a graphic
benchmarks to obtain what they assumed to be
summary of the relationship between oral reading
what one year less of reading skill development
fluency and comprehension. Box-and-whisker plots
would equate to.
were used to show the ranges of reading fluencies
associated with different levels of comprehension. THE BENCHMARKING RESULTS
The levels of comprehension were defined by the
number of questions students answered correctly The results of each group’s work are presented in
(from 0 through 5). In addition, tables summarizing the tables included in this section of the report. The
the quartile distributions were provided. The data in information is organized to show the four outputs
the tables corresponded to the 25th, 50th, and 75th defined by each group, for each relevant grade, for
percentiles of the oral reading fluency ranges for the two key indicators: reading comprehension and
each level of comprehension. The box-and-whisker oral reading fluency. The outputs include the long-
plots and accompanying table for Cebuano are range goal, the nearer-term benchmark (assuming
included as an example in Annex B to this a target date of the end of the 2015–2016 school
3
year), the percentage of students who would be Table 1. Proposed benchmarks for Ilokano
meeting that benchmark, and the percentage of Ilokano Grade 1 Grade 2 Grade 3
students who would score zero at the end of 2015– Comp. ORF Comp. ORF Comp. ORF
2016. The tables also include the levels of (% cor.) (cwpm) (% cor.) (cwpm) (% cor.) (cwpm)
performance as measured by either the 2013 or Average – 2014 data 17% 15 45% 30
2014 EGRA survey (in the shaded rows). These
Long-range goal 80% 40 80% 50 80% 60
existing data are presented to allow comparison of
Benchmark – end of 2015-16 60% 30 80% 40 80% 50
the projected benchmarks to current levels of
performance as one means to judge how ambitious
or conservative the groups were being in setting % Meeting Benchmark – 2014 data 16% 21% 29% 35%
their targets. % Meeting Benchmark – by end of 2015-16 75% 72% 86% 86% 97% 100%
4
ORF, the group essentially assumed a grade-to- For grade 2, the Region VI group set higher
grade improvement of 10 cwpm, which is at the expectations for both comprehension and ORF,
lower end of what EGRA studies in many countries and the proposed level of reading fluency did fall
have revealed. In fact, the 2014 data from Region I within the range of scores that corresponded with
showed a difference of 15 cwpm between the 60% comprehension in the 2014 data. But here
grade 1 and grade 2 average reading fluencies. also, the group’s proposed long-range goal may be
a better benchmark. Again, the proposed
The Region VI group chose not to set Hiligaynon percentage of students meeting the comprehension
benchmarks for grade 3 (Table 2), and the issue of benchmark is below the demonstrated existing
whether benchmarks for mother tongue are needed level of performance, and for fluency does not
and would be used beyond grade 2 is one that project very strong improvement.
DepEd should consider (discussed further in the
conclusion to this report). They also expressed that For zero scores, this group was less ambitious than
they were being cautious in setting their targets, as all the others in projecting targets for reducing the
the levels of performance revealed by the 2014 percentage of students at the bottom of the
EGRA were much lower than they would have distribution, especially in grade 1.
expected.
The group from Region VII, like their colleagues
Table 2. Proposed benchmarks for Hiligaynon from Region VI, chose to set lower comprehension
Hiligaynon Grade 1 Grade 2 Grade 3
benchmarks for Cebuano in both grades 1 and 2 –
40% and 60% respectively (Table 3). They did set
Comp. ORF Comp. ORF Comp. ORF
% Meeting Benchmark – by end of 2015-16 25% 50% 30% 50% Cebuano Grade 1 Grade 2 Grade 3
% Scoring Zero – 2014 data 64% 44% 31% 22% (% cor.) (cwpm) (% cor.) (cwpm) (% cor.) (cwpm)
% Scoring Zero – by end of 2015-16 40% 25% 15% 10% Average – 2014 data 26% 21 53% 41 na na
Note that cwpm stands for correct words per minute. Long-range goal 80% 45 80% 50 100% 55
goal may be more appropriate as a benchmark. % Scoring Zero – by end of 2015-16 25% 15% 15% 5% 10% 5%
And the level of oral reading fluency proposed is Note that cwpm stands for correct words per minute.
5
end of the 2015–2016 school year, this group programs specifically targeted to reducing
proposed some targets below the levels shown in nonreaders, conducting intensive trainings for
the data from the past school year. For this group, teachers focused on mother tongue reading, and
and probably for Region VI as well, it would appear providing coaching and mentoring for teachers and
that the low targets resulted from a misreading of instructional supervision all focused on MTB-MLE
the distribution tables for comprehension. implementation.
Therefore it is suggested that those values be
revisited, taking into account the correct 2014 Table 4. Proposed benchmarks for Maguin-
values shown in the tables included in this report. It dinaoan
is suggested that all groups consider the trade-off Maguindinaoan Grade 1 Grade 2 Grade 3
between a lower benchmark and a higher Comp. ORF Comp. ORF Comp. ORF
percentage projected to meet the benchmark, (% cor.) (cwpm) (% cor.) (cwpm) (% cor.) (cwpm)
versus a higher benchmark with a somewhat lower Average – 2014 data 8% 7 27% 21
(and only a 5 cwpm difference in the long-range % Scoring Zero – by end of 2015-16 40% 20% 25% 15%
goals), and a 13 cwpm increase from grade 2 to Note that cwpm stands for correct words per minute.
6
Table 5. Proposed benchmarks for Filipino However, for comprehension, the transition from L1
to L2 is more difficult, as children need to build up
Filipino Grade 1 Grade 2 Grade 3
sufficient working knowledge of Filipino as a
Comp. ORF Comp. ORF Comp. ORF
second language to be able to more fully
(% cor.) (cwpm) (% cor.) (cwpm) (% cor.) (cwpm)
comprehend what they read (and do so with only 2
Average – 2013 data 73% 68
years of instruction in that language).
Long-range goal 80% 75 100% 90
Benchmark – end of 2015-16 65% 60 75% 80 The group did propose eliminating zero scores in
grade 3, and indicated an expectation that 10% of
% Meeting Benchmark – 2013 data 70% 37% students might struggle to make the transition into
% Meeting Benchmark – by end of 2015-16 75% 65% 75% 80% Filipino in grade 2, and hence score at the bottom
of the distribution for both comprehension and
% Scoring Zero – 2013 data 6% 1%
fluency. The proposed benchmarks for grade 2—at
% Scoring Zero – by end of 2015-16 10% 10% 0% 0%
65% comprehension and 60 cwpm—seem
Note that cwpm stands for correct words per minute.
reasonable given all the issues mentioned above.
And a progression from 60 to 80 cwpm from grade
Even though the cohort tested in grade 3 in that 2 to grade 3 implies that performance will increase
year were students who had had three years of dramatically as students gain more practice in
instruction in Filipino, this group felt that current Filipino.
grade 1 students—who would have had only two
years of Filipino instruction when they reached The team from NCR that worked on setting
grade 3—could still demonstrate improved benchmarks and targets for English (Table 6)
performance. The groups reasoned that by the end recognized the challenge the system now faces in
of the 2015–2016 school year, schools, teachers, trying to build sufficient facility in English for
and students would have more experience with students to be able to read with comprehension by
MTB-MLE and the transition to Filipino, and grade 3. Given how low reading comprehension
teachers would have the tools and activities they was in English in 2013, when students would have
needed to provide effective instruction. The group been receiving English instruction starting in
also wanted to establish a strong standard that grade 1, the group was concerned about how
would reflect outstanding performance, and they
Table 6. Proposed benchmarks for English
were confident that teachers could have effective Grade 1 Grade 2 Grade 3
English
strategies, for example, for addressing the needs of
Comp. ORF Comp. ORF Comp. ORF
the few students at the bottom of the distribution
(% cor.) (cwpm) (% cor.) (cwpm) (% cor.) (cwpm)
(zero scores).
Average – 2013 data 32% 67
However, the benchmark proposed for reading Long-range goal 60% 60 80% 75
comprehension in grade 3—at 75%—is not Benchmark – end of 2015-16 40% 40 60% 70
reduces the amount of instruction in Filipino to % Meeting Benchmark – by end of 2015-16 30% 30% 40% 60%
7
comprehension, the NCR team also projected a Table 7. Next steps and timeline
significant reduction in zero scores in grade 3, Date Activity
targeting having less than half the proportion of
students scoring zero as did so in 2013. 1 Before Sept. 12, 2014 PhilED Data Project/ RTI International will prepare the
report and as suggested will share the benchmarking report
with the regional teams
In working backward to set standards for grade 2,
the group proposed a benchmark of only 40% 2 Before End Sept. 2014 The mother tongue regional teams will review the
benchmarking report with their respective regional
comprehension and only 40 cwpm for fluency. This directors for their information, additional input/support,
latter value actually falls well below the range of leading to finalization and endorsement of the
proposed/recommended benchmarks, which would then
fluency scores that were associated with 40% be officially submitted to the Bureau of Elementary
comprehension in 2013 (from 61 to 92 cwpm). Education at Central Office
Note that one could even ask, given how late 4 December 2014 Benchmarks for reading performance officially crafted,
English instruction is slated to begin under MTB- approved and issued by DepEd
8
First, one needs to consider which benchmarks are Another issue that arises when considering setting
needed for which grades. From the start, a benchmarks and targets for how many children will
parsimonious approach to benchmarking was meet them is how to manage the trade-off between
pursued. For example, since oral reading fluency these two standards. One can choose to set a high
represents a higher-order skill that encompasses benchmark; but then it is necessary to be realistic
the other important reading skills students are about the percentage of students who would be
developing, it was assumed that a benchmark in able to achieve that level of performance. For
fluency would indicate whether students were also example, the group from Region I set a benchmark
building skills in other areas, such as phonemic for Ilokano grade 2 oral reading fluency of 40
awareness, decoding, familiar word recognition, cwpm. This was higher than the 2014 average, so
and reading accuracy. Therefore it was not may be considered reasonable in terms of
necessary to have benchmarks for these other skill expectations for improvement. However, the group
areas.4 Furthermore, because comprehension and then proposed that 86% of grade 2 students be
fluency are strongly correlated, it is worth asking able to meet or exceed that benchmark in 2 years.
whether benchmarks are needed for both. At the That stands in contrast to only 35% of students
workshop, comprehension benchmarks were used having been able to read at 40 cwpm in the past
to identify the ranges of reading fluency that school year. If one wishes to maintain a relatively
corresponded to the desired level of understanding. high standard, then further discussions could focus
While one could argue that having a fluency on whether it is reasonable to assume that
benchmark therefore would also “indicate” a level teachers could more than double the percentage of
of comprehension, this author recommends their students able to meet that benchmark.5
strongly that comprehension continue to be
measured and monitored separately, to make sure The opposite approach would be to set the
that children are not just becoming mechanical benchmark lower, while being ambitious about how
readers. many students would be able to meet or exceed it.
This may be valid in some circumstances, but when
Continuing to take a parsimonious approach, it is considering whether to take this approach, it is
worth asking in which grades benchmarks should important to bear in mind that the benchmark
be set for the different languages. Here, what is should represent at least a minimally acceptable
important to consider is how best to monitor level of performance. In fact, some would argue
whether the MTB-MLE expectations for literacy that the benchmark should represent the desired
acquisition in all three languages are being met. level of high performance.
Clearly, benchmarks are needed to monitor the
development of reading in mother tongues in Lastly, an issue that came up during the workshop
grades 1 and 2. For Filipino, benchmarks are concerned the level of text that students were
needed to monitor whether literacy skills are being expected to be reading at benchmark levels. This is
transferred from mother tongue in grade 2. And it of concern for two reasons. First, when setting
could be argued that they are also needed for benchmarks, it is recommended that the level of
grade 3 to ensure that students are consolidating text for which the benchmark is set be well-defined.
their skills in a language that will continue to be Are grade 1 benchmarks based on reading grade 1
used as a medium of instruction in upper level material, grade 2 based on reading grade 2
elementary grades and beyond. For English, the level material, etc.? Second, DepEd therefore
issue is whether benchmarks (and hence needs to produce and reinforce standards for what
monitoring of progress) are needed at all in constitutes grade-appropriate text in each
grade 2. Benchmarks for English obviously are language. Participants evinced a fear that unless
needed for grade 3. there are clear standards for grade-level text,
schools could evaluate student performance using
reading material that is below what should be read
4
Of course, not having benchmarks for those skills does in a given grade (e.g., testing grade 3 students on
not mean that teachers have no interest in assessing grade 2 material).
whether their students are developing all the skills
needed to be good readers. The important distinction to
5
make is between benchmarks that could be used for Note, the Ilokano example is used here simply to
monitoring system progress—which is what this work illustrate the point about the trade-off between the
aimed at—versus those teachers would use to evaluate benchmark and the target percentage meeting it. It does
individual student or class progress in building reading not imply that the author is recommending a particular
skills. benchmark or target for Ilokano reading.
9
With appropriate review and additional input and
analysis, it is hoped that these initial benchmarks
can be honed and finalized as foreseen in the
process laid out in the previous section. How
DepEd can conduct such work for the other mother
tongues currently being used both as media of
instruction and as languages of literacy remains an
open question.
CONTACT INFORMATION
For additional information on the progress of
Philippines benchmarks for early grade reading:
Joseph DeStefano, RTI International
[email protected]
10
Annex A: List of Workshop Participants
DepEd Central Office Participants
1. Ms. Anne Choi Exec. Assistant (K to 3), Usec. for Programs and Projects
2. Dr. Marilette R. Almayda Director III, Bureau of Elementary Education (BEE)
3. Mr. Efren Dela Cruz Education Program Supervisor 2 (EPS 2), National Education
Testing and Research Center (NETRC)
4. Dr. Rosalina J. Villaneza Focal Person, MTB-MLE Program, BEE
5. Ms. Jocelyn S. Tuguinayo Senior Education Program Specialist, BEE
11
Region IV-A Calabarzon (EGRA Filipino)
34. Dr. Erick M. Jabijan Chief, CLMD, Region IV-A
35. Ms. Agnes Rolle EGRA Regional Coordinator
36. Dr. Lualhati O. Cadavedo City Schools Superintendent/OIC – Imus City
37. Ms. Andrea M. Abrencillo Education Program Supervisor
38. Ms. Lourdes M. Tan Principal III, Guinyangan District, Division of Quezon
39. Ms. Carina P. Jamilano Principal I, Plaridel District, Division of Quezon
Basa Pilipinas
40. Mr. Marcial Salvatierra Chief of Party, Basa Pilipinas, Education Development Centre,
Inc. (EDC)
41. Ms. Karen Cassidy International Project Associate, EDC
42. Ms. Nancy Clark-Chiarelli International Technical Advisor, EDC
43. Dr. Felicitas Pado Senior Consultant/Basa Pilipinas
44. Ilya Son Deputy Chief of Party
USAID Philippines
45. Mr. Mirshariff C. Tillah Program Management Specialist, Office of Education
RTI International
46. Mr. Joseph DeStefano Senior Technical Advisor
47. Mr. Rufino Jamilano In-Country Tasks Coordinator, PhilED Data Project
48. Ms. Rosa Rapanan Accountant, Science, Technology, Research and Innovation
for Development (STRIDE) Program
49. Mr. Julie Mar M. Mendoza Staff, STRIDE Program
12
Annex B: Example of Box-and-Whisker Plots and Accompanying Data
Tables (Cebuano)
Grade 1 Grade 2
50th 50th
# 25th percentile 75th 25th percentile 75th
Correct percentile (median) percentile Count percentile (median) percentile Count
Zero 0 0 6 189 0 3 13 78
1 21 26 29 57 20 28 38 36
2 27 32 36 59 25 35 45 70
3 35 39 45 40 36 40 51 67
4 43 51 57 25 47 54 64 84
5 57 61 67 22 55 65 75 80
13