EVALUATING A PROGRAM

PRESENTED BY

GROUP 6
HIJRAWATI S
FITRIANI HILAL

POST GRADUATE PROGRAM

ENGLISH EDUCATION DEPARTMENT
MUHAMMADIYAH UNIVERSITY OF MAKASSAR
2018

1
 PREFACE

Thank you to the Great Almighty Allah SWT who has given His bless to the
writers in good health for finishing the topic discussion entitled “Evaluating a
Program” for it has completed although it is far from perfect.

The writers also wish to express their deep and sincere gratitude for those who
have guided in completing this paper.

Hopefully, as students and also as teachers can work more professional by

using English. We hope some suggestions to improve this paper and it has
benefit for all of us.

Makassar, 22 January 2018

The writers

2
 Introduction

The study discussed in this paper to investigate two major approaches to

program evaluation: quantitative, experimental evaluation and qualitative,
naturalistic evaluation. The data came from the University of
Guadalajara/University of California, Los Angeles (UCLA) Reading English
for Science and Technology (REST) Project. The data will be used to
investigate and make recommendations concerning the methodology of
program evaluation. The focus and conclusions of this study will be the use of
quantitative and qualitative data and their associated methods of analysis in the
evaluation of language teaching programs in general.
Several methods are identified for each of the two general approaches.
The basic research design for the quantitative approach used in this study is the
non-equivalent control group (NECG) design. This is chosen because of the
nature of the REST setting (primarily the inability to randomly select and
assign to program and non-program conditions) and because it seems to be the
most feasible design for the majority of educational program settings. The
qualitative approach using a combination of data gathering and analysis
strategies that borrow features from three basic models: the illumination model,
the transactional model, and the decision-making model. In the analysis of
qualitative data (in this case, interview transcript, observation notes, journal
entries and correspondence), various type of displays, or matrices, are used to
reduce and interpret the data.

Method and procedures

Subjects and settings
The subjects are 116 students, four teacher/researchers, three assistants,
two program coordinators and several university professors and administrators
associated with the UDEG/UCLA REST project. The students are in their third
year at the Chemical Sciences Faculty of the UDEG, with an academic major in
Chemical Engineering. They ranged in age from 19 to 23 with an average age
of 21. There are 86 males and 30 females.
The most important need for English as a foreign language at the
university level in Mexico is the ability to read EST, to meet that need the
REST staff began to develop a curriculum and decided to focus on reading
skills and strategies as the core of the curriculum. They feel that a curriculum
which is focused on reading skills would have a better chance of success than
the traditional, four skills course. Perhaps part of a reluctance to such a focus
had been the unsatisfactory nature of its presentation in the classroom. In

3
Mexico, reading instruction had focused on word-for-word translation, which
made the reading activity a slow, tedious, and ultimately unproductive one.
Students are seldom taught how to obtain useful information from a reading,
nor have they been presented with relevant, authentic reading texts. In order to
counteract these problems with reading instruction, the design of the REST
curriculum began with a concern for motivating students with relevant,
authentic reading materials as well as a concern for applying current reading
theory to foreign language instruction.

The following represents an outline of the modules that is developed for the
first year of the REST curriculum:
I. Introduction
A. Rationale for reading skills course
B. Organization for the course
II. Grammar review
III. Previewing skills
A. Text survey – to form a quick, general idea of what the text is
about (avoiding word-by-word reading)
1. Using the title, subtitles, pictures, diagrams, and physical
layout of the text
2. Using cognates, repeated words, numbers, names, and dates
3. Prediction and hypothesis formation – using
prior/background knowledge
IV. Text sampling – skimming for main ideas
A. Skimming for the main idea of the text
B. Skimming for the main idea of each paragraph
V. Scanning for specific information
VI. Text structure
A. Organizational layout (overall text)
B. Paragraph organization
C. Rhetorical functions and structures (definition, classification,
process, cause and effect, etc.)
VII. Comprehensive reading
A. Interactive reading (author-reader)
B. Inferencing
C. Identifying the author’s point of view, tone, etc.
D. Critical reading

4
Data collection

In order to investigate both the traditional, experimental or quantitative

approach and the naturalistic of qualitative approach to program evaluation,
both types of data are collected. The quantitative data are collected using a
quasi-experimental (QE), non-equivalent control group (NECG) with pre-test
and post-test design (Cook and Campbell 1979). The experimental or treatment
group in this design consisted of the REST students (n=116) and the control
group consisted of a group of volunteer students at the same faculty (Chemical
Sciences) and in the same year (n=61). The qualitative data are collected using
a combination of models from the qualitative evaluation literature, limited to
those which do not require an outside evaluator who is unfamiliar with the
program.

The Quantitative Data

The major concern for quantitative data collection is the proper choice of a test
or other instrument for measuring or quantifying the underlying construct or
constructs that determine student achievement in the program. Many
researchers feel that standardized tests (generally referring to published, norm-
referenced (NRM) tests) are totally inappropriate for the evaluation of specific
educational programs due to their insensitivity to the instruction and learning
which has occurred. The need for tests that are ‘sensitive to instruction’ has
been the rallying cry of educators in favour of criterion-referenced
measurement (CRM) and domain-referenced measurement (DRM) for some
time (Hively et al.1973; Popham 1978), particularly in the context of language
for specific purposes programs (Bachman 1981).
There are convincing arguments for considering the use of NRM tests in
program evaluation, especially when used in conjunction with teacher-made
tests (Sax 1974). NRM tests, for the purposes of evaluation, allow for
references beyond the specific program or programs being evaluated or
compared. Discovering that the experimental program group performs better
than the ‘traditional’ program group will not necessarily mean that the
experimental program is effective. Both programs may be ineffective, with the
experimental one being statistically better than the traditional. Being able to
reference the test scores to the external norms would yield more information
regarding the effect of the program by allowing for comparisons with other,
proven programs. The groups and programs should be comparable, and the test
should be relevant, in general, to the curriculum or program objectives.
The REST Project evaluation carefully addresses these measurements issues,
they use both NRM and CRM tests to gather the quantitative data. The English

5
as a Second Language Placement Exam (ESLPE) Form A is selected as the
NRM test for the evaluation. A fill-in-the-blank cloze test with an every seventh
word deletion pattern is developed as an alternative NRM test. A multiple
choice cloze test with a rational deletion pattern is developed to serve as a CRM
test.
They chose ESLPE form as the NRM test for the quantitative data collection
primarily because it is the only reliable and valid instrument available to the
Project. While this test primarily measures general ESL proficiency, it is
developed within the context of English for academic purposes, to reflect the
UCLA ESL curriculum. Since the REST Project’s reading skills approach is
similar to the objectives of the UCLA curriculum, it is felt that meaningful
comparisons concerning student progress could be attempted.
The test is composed of five subparts – Listening (20 items), Reading
(18 items), Grammar (20 items), Vocabulary (20 items), and Writing Error
Detection (20 items). Due to problem with the listening audio-tape during
administration, only ten items (one passage) from the Listening subtest are used
in this analysis, resulting in 88 total items. In addition, a fill-in-the-blank cloze
test is developed to gather quantitative data for the program evaluation. While
cloze test may seem more closely related to reading ability than a battery of
subtests like the ESLPE, some would still classify the test as a ‘general
proficiency’ measure, lacking in authenticity and face validity (Spolsky 1985).
The fact that the fill-in-the-blank cloze tests also require language production
means that they cannot be considered as measures of reading independent from
other skills such as writing. However, as an alternative NRM test, it is felt that
an instrument like the cloze would be useful to the project. The fill-in-the-blank
cloze test is constructed using a pseudo-random deletion pattern of every
seventh word, resulting in 30 deletions/test items.

The Qualitative Data

The issue of measurement is fundamentally different in qualitative research and
evaluation and in the quantitative approach. Rather than developing test
instruments in order to quantify the variables or constructs being studied,
qualitative research tends to rely on people as the instruments of inquiry.
Problems of measurement are primarily related to the concept of reliability,
while one of the major strengths of qualitative research is its validity
(LeCompte and Goetz 1982). For the gathering of qualitative data, the REST
Project evaluation made use of journal entries, daily logs, observations, meeting
notes, questionnaire and interviews. These are organized into two basic files:

6
the Treatment File and the Administrative File, each of which consisted of
several sub-files.
a. The Treatment File: All of the Teacher/Researchers (T/Rs) and
Teacher/Researcher Assistants working on the REST Project are required to
keep a daily journal describing what happened in their classroom. A format
is developed over the first months of the Project which each T/R files out
and turns in each day. The module, the materials used, the types of activities
and exercises, this source of data describes the process of curriculum
implementation and characterization of the program.
The T/R Journal Sub-file, which also contains classroom observations
conducted by the REST coordinators, the observation sub-file. The format
for the observations is also the result of a trial-and-error development
process over the course of the first months of the project. It gives another
description of the implementation of the program at the classroom level,
from a non-participant observer’s point of view, as well as allowing for a
‘dialogue’ between the observer and teacher (the reactions from T/R to
observation comments’ section) concerning the class and lesson. The
observer normally takes extensive notes and then writes up the observation
using the format at a later time.
b. The Administrative File: The REST project coordinators also keep daily
journals. The UCLA coordinator keeps the journal in a daily log format
which is a sub-file of the Administrative File. The other major section of the
Administrative File is the Meetings Sub-file. This sub-file contains notes
from all project related meetings taken by the coordinators. The notes are
typed up from handwritten ‘field notes’ taken during the course of each
meeting.
Two related sources of data are interviews and questionnaires. For the
evaluation of the REST Project, these are kept as separate ‘sub-files’ technically
under the Administrative File. Interviews are conducted at two different times
during the year with the students and once at the end of the year with the T/Rs.
The students’ interviews are not recorded because the coordinators are worried
about intimidating students any more than they already seem to be. The student
interviews are unstructured in the sense that the coordinators ask very general
questions for the most part and try to get the students to tell whatever they have
to say about the program in their own words. During the second set of student
interviews the T/Rs select five to six students from each of their classes to
represent the range of EFL proficiency and REST class motivation and attitude
(they include good students and weak students, students with positive attitudes
and others with negative attitudes towards the REST course). The interviews

7
with the T/R are tape-recorded in addition to notes being taken. For the most
part, the T/R speak naturally and spontaneously about the Project, although they
have obviously given the issues involved some thought. For the purposes of the
qualitative analysis of the REST Project, only four questions from the
questionnaire are used: What is your personal understanding of the objectives
of the REST Project? What do you think the strengths of the REST Project are?
What do you think the weaknesses of the REST Project are? Do you have any
suggestions for the improvement of the Project?
The issues of validity and reliability are primarily addressed through the
use of multiple data sources: T/R journals, administrative daily logs, classroom
observations which included responses from the T/R being observed,
interviews, and meeting notes. The journal format is changed based on
suggestion from the T/R in order to encourage more and better information, and
the interviews formats are kept as open-ended as possible to encourage the
informants to express their perceptions of the program in their own words.

Data Analysis
Four different qualitative analyses are carried out: analysis of covariance
(ANCOVA), standardized change-score analysis, effect size analysis and the
Chi-square test. The qualitative data are reduced through a series of steps that
include the development of a thematic framework, coding of the data, and the
entering of selected data into displays (Miles and Huberman 1984). These
displays are in the form of matrices that present the qualitative data in a spatial
format that is compressed and systematic, making it easier to examine the data
for patterns and relationships.

The Quantitative Data

In the case of the analysis of covariance, the ESLPE total score and ESLPE
reading subtest score from the February administration are used as the post-
test/dependent variable and the September pre-test scores are used as the
covariate pre-existing differences between the REST group and control group.
The fill-in-cloze and the MC cloze (the May administration) are used as post-
tests/dependent variables and the ESLPE September pre-test as covariate. The
standardized change score analysis is an essentially correlational technique for
estimating the effect of programs being evaluated with a QE design (kenney
1975). If the treatment is having zero effect on the subjects in a group
(experimental or control), then the relationship between group membership and
performance on the tests should change significantly from pre-test to post-test.
The effect size analysis involving change or gain score analysis. This approach
calculates an effect size based on comparison of the means and standard

8
deviations for gain scores between the treatment and control groups. As with
the ANCOVA analysis, the pre-test and post-test used in the REST Project
evaluation are the September and February administrations of the ESLPE. The
fill-in-the-blank cloze and multiple-choice cloze could not be used in this
analysis because pre-test and post-test data do not exist for the control group.
The Chi-square test can be used to test for the effect of a program when the
requirements for parametric statistical tests cannot be met. This involves a two-
by-two Chi-square table in which the dependence between treatment
(experimental group, control group) and gain (positive change-score, negative
or zero change-score, on the dependent or outcome variable) is tested. In the
REST Project evaluation, the two variables being tested for dependence are
experimental group membership (REST versus control) and change in EFL
ability (positive change-score on the ESLPE from September to February, and
negative or zero change-score on the ESLPE from September to February.
Change-scores on both the ESLPE total score, the ESLPE Reading subtest, and
the ESLPE Reading and Vocabulary subtests are examined in this analysis.

The Qualitative Data

REST Project are analysed following a methodology presented in Miles and
Huberman (1984), it is the development of a thematic or conceptual framework
which is focused on the following:
1. Program goals
2. Program processes
3. Events
4. Setting
5. Participants
6. Outcomes
For the evaluation examples of the more specific themes/issues are explicit
grammar instruction versus reading skills, the use of English versus Spanish for
instruction and team teaching. As with every stage of qualitative data collection
and analysis, the coding of the data is an iterative process, which is to say that
it requires a continuous revision of both the codes and the thematic framework.
Some examples of the codes used in the REST Project evaluation are:
1 CLASSRM PROC (classroom process) – this code would sometimes be
associated with sub-codes, such as:
1a GRAM/READ (the use of explicit grammar instruction versus reading skills)
1b AUTH MAT’S (the use of authentic materials)
2 S ATT/MOT (student attitude and motivation) - this code is sometimes
combined with another.

9
 The effects matrix is used to characterize the various outcomes and
changes associated with the program. Focusing on different parts of one
particular outcome such as organizational change within a school or program:
sorting the outcomes or changes by degree of directness such as primary
changes intended by the program versus secondary or spinoff changes that are
not anticipated. the effect matrix for the project evaluation focused on ultimate,
end of year outcomes of the program implementation, categorizing these effects
into three types/columns in the matrix: objectives/goals (what is accomplished),
process/method (how is it accomplished), and relations/climate (how do the
program participants, T/R and students, interrelate, what is the social climate
like).
The final display used in project evaluation is the Site Dynamics Matrix.
This matrix attempt to address the question that follows from the Effect Matrix:
Why did these outcomes occur? It does this by displaying the links between
program outcomes of effects with the processes or aspects of the program that
lead to those effects. The focus decides upon for this matrix is the events or
situations in the data that could be depicted as dilemmas or problems. The
dilemma/problems that are chosen for the matrix are those that seem to have the
greatest impact on the project, based on the perceptions of the researcher and
their frequency of mention in the data.
In the interpretation of the effects matrix, one technique is a simple
counting of plus and minus signs in the matrix. As an example, even a quick
visual scan of the REST matrix reveals certain trends: there are many more
negative outcomes under process/methods than there are under objectives/goals.
Another approach to interpreting the matrix is to scan for more specific
different patterns. For example, in the REST matrix the UDEG Coordinator
notes many more positive outcomes than do the T/Rs with regard to
process/method. The effects matrix can also be interpreted by scanning for
similarity patterns. One example is the ability to use authentic English texts in
their Chemical Engineering courses, which is seen as an important outcome by
both the teachers and the students. This is all the more important an outcome
since it is a part of the original objectives for REST.
Another example of a similarity pattern is that both the T/Rs and the
students seem to agree that there is a certain lack of focus and organization in
the process/method of the REST course. Another similarity pattern is the
perception that there is a lack of motivation on the part of students. While the
reference to this lack of motivation and attendance does not appear as an
ultimate outcome for the T/Rs, it is heavily and continually referenced

10
throughout their journals. Furthermore, from the UCLA Teacher’s point of
view, it seems to result in a feeling of resentment on the part of the students.
The UDEG staff members, T/Rs assistants and coordinator, characterize the
outcome as a lack of understanding with cultural differences as the cause.

The Quantitative Data

There is a positive effect on the part of the program, although it has already
been pointed out that this may be saying nothing other than the program is
better than no program at all. However, the significance of the performance of
REST students in comparison to the control group, as indicated by the
ANCOVA and the standardized change-score analysis is useful information to
have. The effect size and Chi-square analyses support this interpretation of a
strong program effect. When considering ESL reading proficiency the ESLPE
reading subtest, the effect size and chi-square analyses contradict, to a certain
degree, the results of ANCOVA and standardized change-score analysis. These
analyses indicate that the improvement of the REST students in relation to the
control group is either non-significant (Chi-square) or moderate (effect size).
The REST students have not improved significantly in their ability to answer
detailed questions concerning a short academic English passage (which is what
the ESLPE subtest seems to measure). The rest curriculum did work with
academic reading texts, it did not deal with this type of reading. However, the
REST curriculum does not deal with general ESL proficiency (including
listening) either, and yet all forms of analysis indicate that the REST students
outgained the control group on their general English proficiency, even when
controlling for differences in pre-test ability. Another major problem with the
quantitative approach in this study is the control group. These students are
volunteers, who had no motivation for scoring well on the exam other than
pride. More than the issue of control group, it is needed for a better outcome
measure to evaluate the REST program. What is needed is measure of academic
reading skills for EST.

The Qualitative Data

Some interpretations of the data displays are expressed as tentative conclusions.
The reading strategies approach seems to have been negatively received for the
most part by the students. The teachers appear to have changed their emphasis
on these strategies to a greater focus on basic language skills as a result of the
low EFL proficiency level of most students and their reaction to the strategies
approach. There seems to be fairly negative relations between the students and
the teachers and problems of cooperation amongst the teachers. Other tentative
conclusions that can be drawn from the interpretation of the displays include

11
the observation that the REST curriculum does not appear to have resulted in
proficient readers of EST. The REST curriculum does appear to have
introduced its students to reading strategies and basic grammar and vocabulary
skills necessary for efficient reading of EST texts.

The independent evaluation of bilingual primary education

The discussion of this study is around the evaluation of BEP (bilingual

education project) based on the 1975-81 project evaluation run by SED
(Scottish Education Department) where it is the implications for the design of
an appropriate evaluation strategy. Next it is given of the development of the
design for the evaluation, involving interplay between academic, political and
practical considerations. This is followed by a description of experience in
implementing the evaluation strategy, and a limited account of external
perceptions of the evaluation. The research set about developing initial
proposals for the evaluation study which these proposals took account of a
number of principles as well as of a range of constraints to do with timing and
resources:
1. It is accepted that the concerns of the evaluation study should reflect those
of the original BEP as far as possible. That is, the evaluators would
concentrate on investigating to what extent the innovations proposed by the
BEP are actually operative in the schools, ten years on and to what extent
the hoped-for outcomes (in terms of pupils’ bilingual proficiency and
attitudes) are being achieved.
2. For practical reasons, the need to strike a balance between breadth and
depth of data collection is recognised. While limited data could be collected
in all schools, it is necessary to select smaller sample for extended
classroom observation and pupil assessment.
3. It is recognised that systematic, quantitative methods would have to be
adopted for the observation of bilingual teaching and learning.
4. It is also recognised that the timing of the evaluation project ruled out the
adoption of research designs which would permit the establishment of
unambiguous causal links between the innovation strategy and the school
curriculum.
5. The timing of the evaluation also makes it inevitable that the focus of the
work would lie in the schools.
6. It is however felt desirable to make one exception to the school focus of the
evaluation study; in the light of the BEP’s original aspirations to involve
parents and community members more fully in the bilingual curriculum, it

12
 is felt that a limited exploration of parental attitudes and involvement
should be carried out.
7. One suitably qualified, bilingual field worker could be recruited to the
evaluation team.

The full proposal envisaged a number of distinct phases:

1. A preliminary interview survey with the head teachers of all schools
involved in BEP. In a semi-structured interview format, it is proposed to ask
these teachers about the history of their school’s involvement with BEP,
their attitudes towards BEP policies, and the extent of implementation of
these policies in their school.
2. On the basis of preliminary survey, it is proposed to select a sample of
schools for further investigation. The main criterion involved in selecting
schools which are committed to the policies of BEP.
3. Select classrooms of each school for detailed study. A range of techniques
(teacher interviews, structured and unstructured observation of class teacher
and pupils) would be used to develop a full picture of aspects of classroom
life judged relevant to BEP policies.
4. In response to the outcome goals, it is proposed that pupils’ spoken and
written proficiency should be assessed, at two different age levels (midway
through primary schooling and towards its end). It is also proposed to
explore their perceptions of themselves in relation to native language and
culture, background, and their attitudes towards the BEP itself.
5. It is proposed that a limited sample of parents should be interviewed,
regarding their knowledge of the BEP and possible involvement with it as
well as their attitudes towards it.

Implementing the evaluation design

A bilingual field worker is required, with the personality, qualifications
and experience necessary to undertake the main responsibility for the collection
of data in schools. Such a person would ideally have experience both of
primary teaching and of research, as well as being fluent in both native and
English. While engaged in search for suitable staff, the researchers continue
with the development of more detailed plans for the evaluation. The evaluation
team begins its fieldwork, visiting original BEP schools and the preliminary
survey of head teacher opinion is carried out. Two members (Stirling-base
directors and fieldworker) of the evaluation team are involved in each of these
preliminary visits led a proportion of the semi-structured interviews.
Intensive preparation for the main phase of classroom-based research
begins with the basis of a systematic analysis of the head teacher interviews, ten

13
schools are identified as a potential main sample, two of the schools identified
have only one teacher, so that between them these schools would provide a total
of eighteen classrooms for the study. Meanwhile, they continue on the
development and trialling of the instruments to be used in the classroom based
work. It is decided that two separate systematic observation instruments are
needed, one teacher-focused, one pupil-focused.
The teacher focused instrument chosen is a slightly modified version of
the teacher record originally developed by Boydell (1974) and extensively used
in the large-scale project. The teacher record is a sign system in which the topic
and quality of teacher utterances (questions and statements) are documented. It
is considered that the instrument would yield evidence both on the extent to
which teacher-pupil interaction reflected a child-centred style of instruction and
on the range of purposes for which the teachers used the two languages, native
and English. It is felt necessary to devise an original instrument for pupil
observation, in order to answer a range of research questions regarding the
content of the curriculum, the activities in which pupils are engaged, and the
linguistics medium through which they worked.
A sampling procedure is therefore proposed, whereby eight members of
each class would be selected for individual observation during a fixed
proportion of the school day. Account is to be taken of pupils’ age, sex, and
bilingual proficiency, in selecting a balanced sample within each classroom, the
school day would also be systematically sampled, with a balance between
morning and afternoon observations. Two interview schedules (pre and post
observation) are developed for use with teachers, primarily to document aspects
of classroom life not observable within the space of a few days’ visit. The field
workers’ schedule of work in the main study schools also include periods of
unstructured observation, to document aspects of school life such as pupil
language use outside the classroom setting or environmental studies field trips.
They consider one substantial modification to the general plans for the
evaluation, given of the original BEP to the development of reading materials.
It is suggested that the team might undertake an investigation of children’s
reading skills, in addition to their speaking and writing. After consideration, it
is recognised that bilingual reading assessment materials are too great for the
resources of the evaluation team and thus they concentrate on developing
procedures for the assessment of speaking and writing. For this they heavily on
the language work of the assessment of performance unit. It is decided that all
primary pupils should be involved in the language assessment. For the
assessment of speaking, three main tasks are developed in parallel English and
native language (in this case Gaelic versions) where they talking about self,

14
talking about school work, and retelling a story. All the speaking tasks are to be
audio recorded for subsequent assessment using rating scales on a variety of
dimensions judged relevant to communicate proficiency. For the assessment of
writing, separate tasks are developed for use with primary 4 and primary 7
pupils. The P4 pupils are to be asked to write a simple narrative on the basis of
a series of four pictures, in either English or Gaelic. Two tasks are devised for
the P7 pupils; a story-writing task, and a non-fictional reporting task relating to
current school work.
The last element of the field work, the parent interviews, a semi
structured interview for administration to a small sample of parents of children
attending four main sample schools (two ‘high uptake’, two ‘low uptake’). The
field worker makes contact with the parent sample through the schools, while
revisiting them for the purpose of the language assessment work. All parents
are interviewed some in English, some in their native language (Gaelic), most
preferred not to be audio recorded.

The evaluation findings

The head teachers in all schools are well-informed about the aims of the BEP,
and generally supportive of them, though reported levels of implementation
varied considerably. In the ten main study schools, the overall curriculum
structure conformed well with BEP intentions. ‘Environmental studies’ with a
local orientation is well established, in the form of theme/project work, with a
wide range of practical activities. In language use, all teachers are bilingual,
English and Gaelic are both in widespread use as languages of instruction,
across the whole curriculum in most schools. Such pupils predominated Gaelic
(native) is the commonest language of whole-class instruction, fluent bilinguals
are addressed individually by their teacher in either language. The language
assessments provide rich information on the current state of children’s bilingual
competence. All the children possessed effective oral communication skills in
English, appropriate to their age level, whatever the first language of their
home.

15
Issues in evaluating input-based language teaching program
Description of the program
Theory behind the program
Krashen (1982,1985) hypothesizes that a language is acquired (picked
up in such a way that it can be used effortlessly and without conscious thought)
in only one way: a) by exposure to interesting, comprehensible input b) under
non-threatening conditions. These are Krashen’s first two hypotheses: the
‘input hypothesis’ and the ‘affective filter hypothesis’. A third hypothesis, the
production emerges hypothesis, c) speech emerges as the result of acquisition.
According to Krashen’s theory no production practice is needed for
subconscious language acquisition, nor is analysis or explanation of the target
language. In addition Krashen’s theory includes other hypotheses. For example,
his full theory includes a hypothesis that conscious learning (through analysis,
practice and explanation) is possible and is useful in certain ways, however,
conscious learning is not thought to be useful in developing acquired
competence and consciously learned material is not thought to become acquired
as the result of practice.

Version of the theory tested

In the experimental program, they decided to test as pure a version of Krashen
theory as possible. Therefore, they only provided interesting, comprehensible
input. They deliberately avoided activities that would tend to produce conscious
learning so that they might observe the effects of the subconscious acquisition
process. They also avoided requiring the students to speak for three reasons:
speaking might create stress, raise the affective filter, and block acquisition;
speaking (according to Krashen’s theory) is not required for speech to emerge,
and early speaking might lead to acquisition of deviant forms through exposure
to a large amount of imperfect German (Krashen 1985).

Why evaluate?
Why we are evaluating program? One is to find out whether our program is
feasible: could the teachers teach it and would the students put up with it?
Another as noted is to find out whether the program is productive. Whether the
program is appealing or would it be enjoyed? Primary reason for evaluating the
program is to find out more about language acquisition theory. Therefore, the
initial program design and any modifications made had to be consistent with the
pure version of Krashen’s theory. As a result, they accepted the possibility that

16
students would be resistant to some of the elements in the program, and they
decided not to modify the program in ways that would make conclusions about
the validity of the theory difficult to arrive at.

When to evaluate?
This evaluation is influenced by Krashen’s affective filter hypothesis, according
to this hypothesis, students acquire best under non-threatening conditions, and
large scale, formative evaluation might be likely to raise the filter (Krashen
1982). As a result, they evaluate the program in two phases: a small-scale,
somewhat informal formative evaluation of students’ attitudes toward the
program and a large-scale, formal, summative evaluation of the student’s
language abilities. The formative evaluation of attitudes would allow them to
keep track of attitude changes as they occurred, which would help them adjust
the input to keep it interesting and comprehensible. The summative evaluation
of students language ability to determine how well the students could perform
at the end of the program.

Comparative or independent group evaluation

Comparative evaluation
When think about how to evaluate the program, it comes to mind immediately
is a methods comparison study using experimental and control groups. It would
provide us a means of comparing results of the input-based program with the
results of traditional instruction. While comparative studies are interesting,
designing and interpreting them is difficult. Ideally, students should be assigned
to the two groups at random, attempts should be made to control for self-
selection. Also teacher variables are difficult to control. The results of the
Pennsylvania Study (Smith and Baranyi 1968) indicated that the teacher
variable was more important than the method used. The goals of programs
employing different language teaching methods are likely to be quite different,
which would make finding criterion measures appropriate to both programs
difficult (Beretta 1986).

Independent evaluation
The other alternative would be to evaluate the experimental program on its own
merits, which would involve stating the objectives of the program and
collecting data to evaluate how well these objectives had been met. Such a
study would avoid many of the problems described above, but it could not
provide the kinds of comparative data which many people want.

17
Solution
They decided to conduct both independent and comparative evaluation. To
evaluate the experimental treatment on its own merits, they specified objectives
for the experimental program, against which they could compare actual
outcomes. To compare the relative effectiveness of the experimental method
with something else, they created a control group of traditionally taught
students and gave the same set of proficiency tests to each group.

People affected by program outcomes

Whether a program is or is not considered successful, it succeeds if the students
learn something and are happy with the learning experience. In this case, it
measures or descriptions of student behaviour. Or they decided to evaluate the
program in terms of teachers’ attitudes. Are they pleased with their teaching
and with their perceptions of what their students learned? Or it might evaluate a
program’s success in terms of administrations’ attitudes. Or, as Rodgers (1986)
suggests, go outside of the educational setting altogether and evaluate the
program in terms of the attitudes of people within the community.

1. Cognitive domain
The cognitive domain consists of knowledge about the language. In input-
intensive acquisition based program, there is little to evaluate within this
domain. Such programs do not present grammar explicitly (cognitively). Nor do
they expect the subconscious acquisition process to lead to cognitive control.

2. Behavioural domain
To evaluate students’ behavioural performance, it is needed to start by
determining what kinds of outcomes would reasonably expect of students
completing the program. Krashen and Terrel discuss the role of grammar in
defining expected outcomes. While they indicate that beginning students may
simply string together appropriate lexical items in some logical order, they do
not expect that students will continue to use these simple stringing techniques.
They believe that the language acquisition that takes place in the classroom will
result in student acquiring grammatical competence. Thus, based upon Krashen
and Terrel’s guidelines, appropriate tests for the experimental group might
asses the students’ listening, speaking, and reading skills, as well as their
subconscious control of grammar.

18
3. Affective domain
In addition to evaluating behavioural outcomes, they also want to evaluate
attitudes of three populations (students, teachers and administrators) toward the
program.

STUDENTS
They divided questions to the students into three types. The first type consists
of questions about fairly general aspects of their attitude toward the program.
The process in preparing these questions include discussions with experts in
program evaluation, discussions with people involved in the program,
examination of other program evaluation studies, and examination of general
issues in program evaluation. The second type of question elicited students’
opinions about specific learning activities: Did they like the activities and find
them interesting? The researcher asked many questions of this type because
Krashen’s theory of language acquisition states that affect is one of the two
causal factors in language acquisition. The third type of question elicited
students’ general attitudes toward whatever aspect of the program concerned
them at the time. This kind of question is used to elicit feedback on issues
perhaps not taken into consideration when developing the list of specific
questions. Some topics are asked to elicit student’s opinion as follows:
1. Reasons for studying German at the beginning of the program and at
present; fun, curiosity, for use in future studies, for use in work, to get a
good grade.
2. Current satisfaction with their instruction in the areas of listening, speaking,
reading, writing and grammar
3. Confidence that they could cope with listening, speaking, reading and
writing German
4. How much fun they had in class
5. Confidence in the method
6. Opinions of each instructor’s control of language, ability to teach, and
personality.

TEACHERS
Teachers’ attitudes are also assessed on a variety of issues. In deciding what
questions to ask, they follow procedure similar to that used in deciding what
questions to ask of the students. Where possible to ask the teacher questions
which are similar to those asked the students so that they could determine the

19
extent to which students’ and teachers’ impressions and attitudes differed.
Some of the following topics are to obtain information on teachers’ attitude:
1. Satisfaction with the students’ competence in the four skills and grammar
2. Interest in the content of what they talked about in class
3. Satisfaction with the grading policy
4. Confidence in their ability to train others in method
5. Amount of fun they had teaching the course
6. Whether they would want to teach the same class again

Method effect
Bachman (1990) has proposed a system for the componential analysis of test
methods. The research on this study attempted to select methods that would not
be obviously disruptive. For example, in evaluating the student’s ability to read
German, they had students summarize a reading passage in English rather than
German. They knew perhaps the students’ ability to summarize would influence
the test results.

SPECIFIC METHODS
Two methods are used to evaluate the students’ German language proficiency:
traditional language tests and students’ self-ratings.
Listening: Lecture summary
Students are allowed to take notes during the lecture. They are instructed to
summarize the lecture as well as possible, in as much detail as possible.
Listening: Students’ self-rating
The students are asked to write brief descriptions of their ability to understand
spoken German. The descriptions are grouped into high, mid, and low
categories, counted the numbers of students in each category and provided
examples of students’ comments in each category.
Speaking: Oral interview
A native speaker interviews each student for five to eight minutes, the length
depending upon how proficient the students are. Questions are of the sort one
might ask someone when trying to become acquainted. For example: Where do
you live? Where do you work? Have you studied other languages before? In the
middle of the interview, the students are given some information to elicit by
asking questions in German. For example: Where does the interviewer come
from? What is the interviewer doing in America? Where does the interviewer
normally work?
Kramer scored these interviews using two criteria: sentence complexity and
well-formedness, and control of inflectional morphology. He then added the

20
two scores together and then converted these scores to ratings using a ‘special’
procedure different from the standard one. Kramer compared the students’
performance on the oral interview with their performance on the remaining tests
that had been rated using the standard procedures. He determined which
students tend to fail consistently into the high, mid, low categories on the other
tests and used this information, together with the original rating criteria, to
estimate high, mid, low break points for the interview scores.
Speaking: Students’ self-ratings
The students are asked to write brief descriptions of their ability to speak
German. Descriptions are high, mid and low categories, counted the numbers of
students in each category, and provide examples of students’ comments in each
category.
Reading: Translation
The students are given a humours passage about a child who threw a rock
through a window and then bragged about having answered the teachers’
question (about who threw the rock through the window) correctly. They then
write an English translation, which is rated using the standard procedure.
Reading: Summary
The students are given a written passage about German satirist written in
academic, unsimplified German, followed by a written summary of an
interview with the satirist. The students are given five minutes to write a
summary of the passage in English, which is rated using the standard
procedure.
Writing: Guided composition
The students are given a 79-word passage about the weather. They then write a
passage in German about the weather in America. These compositions are rated
using the standard procedure.
Writing: Dictation
A 65-word passage is dictated to the students. The dictation is scored by
marking one point off for each omitted word, extra word, and pair of permuted
words. Words are considered correct if the spelling is phonetically appropriate.
In addition, a half point is deducted for each error in inflectional morphology.
The compositions are then rated using the special procedure described for the
oral interview test.
Grammar: Rational cloze
The students are given a 118-word multiple-choice cloze test with rational
deletions (developed by Paul Kramer). The passage is taken from a German
reader. The subject matter is a humorous story about a student who went into a
restaurant and made a deal with the waiter that if the student could sing the

21
waiter a song that pleased the waiter, the student would get his dinner free. The
first sentence is left intact. Twenty-five words are rationally deleted from the
remainder of the passage to test grammar, discourse competency and
knowledge of the world. Deletions occurred at intervals ranging from four to
eleven words. Four answer choices are provided for each deletion. Kramer
scored the cloze test using the exact word method. He then assigned ratings
using the special procedure described for the oral interview test.
Vocabulary: Uncontextualized translations
The students are given a list of German words to translate into English. These
are scored and then rated using the special procedure described for the oral
interview test.
Vocabulary: Contextualized translations
The students are given a reading passage containing the same vocabulary words
used in the uncontextualized translation test. They translate these words into
English using the additional information they could obtain from the context.

For whom to evaluate?

As Rodgers (1986) pointed out, a variety of people have an interest in the
language teaching program evaluation reports. Students want to learn and have
an enjoyable experience in the process and might want to know what they could
expect of a specific program. Teachers want to know how effective instruction
proves to be and how enjoyable the teaching experience is. Administrators are
concerned with results and their effect on enrollments. Members of the
community may be concerned with the effect of the students’ language skills on
how well they function in the community. Researchers are concerned with the
results of applying language acquisition theory in the classroom. And program
evaluators may be interested in the report as a guide to future evaluation
studies.

Student attitudes toward the program

Students’ reactions to the method are very positive at the beginning of the
program, when they appreciate the lack of pressure to produce the language and
the absence of formal testing. As the program progressed, they begin to worry
about whether they would eventually be able to speak, and there is some
indication that they would like to have been tested in order to know that they
are indeed acquiring the language. Their comments indicated that a number of
them would have liked some specific encouragement to speak and make
mistakes in order to get used to this experience.

22
Student performance on behavioural measures
The experimental group considered by itself
Students’ abilities to produce German ranged from slight to quite surprising
(Kramer 1989). Students rated as low are functioning as one would expect of
acquires within the early stages of an expected interlanguage phase. For
students rated as high, evidence of acquisition is much more apparent, with
these students at times able to produce well-formed contextually appropriate
complete sentences.

Comparison between experimental and control groups

A MANOVA (Multiple Analysis of Variance) conducted by Kramer (1989)
indicated that the control group students (who received traditional instruction)
perform significantly better overall than the acquisition students. They also
perform significantly better on four of the seven tests (oral interview, reading
translation, writing summary, and vocabulary). The experimental students did
not perform significantly better on any of the tests (Kramer 1989).

Between-group comparison
Kramer’s finding that the control group performed significantly better than the
experimental group. What this study led to is an investigation of a number of
method-comparison studies, all of which dealt with the issue of the relative
efficiency of input-based and eclectic instruction (Kramer and Palmer 1990).
As a result of investigation, it is discovered an interesting pattern of interaction
among method of instruction, purity of instruction, and age of students. In
studies with impure experimental treatments, it is found no significant
differences. In studies with pure experimental treatments, they found significant
differences favouring the experimental (input-intensive) treatment for children,
but favouring traditional (eclectic) instruction for adults.

Problem-defining evaluation in a decade of eclecticism

Steven Ross

The spread of the communicative approach to foreign language pedagogy has

been accompanied by a gradual de-emphasis of comparative method studies
designed to contrast different pedagogical strategies. The focus of attention in
studies of the learning process has instead been placed on comparing the
discourse characteristics of teachers and learners in the classroom with that of
naturalistic second language acquisition contexts and with the processes

23
initiated by innovative methods. The concurrent trends of qualitative evaluation
and eclectic language teaching have led to a veritable glut of rival techniques,
methods and approaches from which the EFL practitioner may freely choose. A
laissez-faire approach to program administration has become a by-product of
the communicative movement in foreign language teaching in some countries
(e.g. Japan).

a. Evaluation in laissez-faire ELT

The proliferation of language teaching methods and approaches has stemmed
not primarily from empirical validation of novel ways of teaching language
(Richards 1984), but from two other sources of eclecticism. The humanistic
language teaching movement (Stevick 1976;1980) and implications from
second language acquisition research (Krashen and Terrell 1983) have been
most influential in fostering an attitude on the part of many language teachers
that evaluation is relevant to the extent it displays a similarity of the process of
the classroom to the ideals of the humanistic language teaching movement, or
to a current SLA theory (Gregg 1986). Given that one of the main aims of the
humanistic approaches is to improve the psychological environment in which
language is learned, product-oriented approaches have become relatively rare in
discussions of the efficacy of learner-centered methods. The newest wave of
classroom-centered research which describes the similarities of interlanguage
discourse to naturalistic encounters between native speakers and acquires
(Porter 1985; Duff 1986; Varonis and Gass 1985) has a whole conjectured that
if interlanguage discourse (the language used among learners in the classroom)
results in a negotiation of meaning resembling the conversational adjustments
made by native speakers to non-natives, a major criterion for adopting the task
based approach is satisfied.

b. Qualitative or quantitative evaluation

Both qualitative and quantitative types of evaluation can be utilized with a view
to defining a program, and as such are better described as program-formative
evaluation strategies which may provide alternative views of the same
classroom phenomena (Lett and Shaw 1986). Whichever evaluation approach is
chosen, there remains a subtle danger in attempting to directly compare the
effects of various materials or methods without first determining whether or not
they are being applied in an orthodox way. Long and Sato (1983) have
commented on the tendency for teachers to claim they are practicing a given
method while in fact they might be doing something quite different. The teacher
variable becomes essential in discussions of comparative method studies and

24
has led to some rather exaggerated attempts to eliminate the teacher from the
language lesson altogether (Freedman 1979). But as Beretta (1986) suggest,
there is a point at which an attempt to improve internal validity inevitably
results in a compromise in the external validity of the experiment.

c. The selection of materials

The materials used in this study are summarized as follows:
Audio-lingual materials represent late audio-lingualism. Dialogues and
comprehension questions are followed by massive pattern drills designed to
lead the students from mechanical to meaningful and ultimately communicative
ability in English.
Functional-notional materials are selected to represent a literal interpretation of
the Council of Europe recommendations for syllabus design. The materials
followed a limited set of language functions while at the same time included a
structural backbone.
Grammar-based materials follow the structure of the day principle whereby
each introductory segment of the lesson, a narrative story, is written to
demonstrate a particular grammatical point. Subsequent exercises allow
extended practice and review of previously covered grammar points in a
cyclical manner.
Self-access pair learning (SAPL) materials apply the principles of Lozanov’s
suggestopedia. Emphasis is put on relaxing the students by playing baroque
music while they engage themselves in pair work activities. The teacher
intervenes only when there is an apparent problem in an effort to reduce the
level of anxiety in the students.
Task-based materials provide a non-text-based alternative to the above
mentioned texts. The main characteristic of the task-based lesson is unplanned
discourse. An effort is made to create information gap activities (Doughty and
Pica 1986) and two-way problem-solving activities (Duff 1986) which require
students to create language appropriate for the communicative context.
Devising a coding scheme
Once the individual teachers agreed to participate in the study as both
classroom practitioners using their favourite materials and as observers of their
colleagues’ teaching, a coding scheme is developed which could take into
account the need for low inference categories of codings and at the same time
be limited to the kinds of activities which are most likely to transpire in the
various classes. The coding scheme which evolved encompassed information
indicating the main pedagogical strategies employed in the class and at the
same time provides information of interest in light of current second language

25
learning models. Four general categories of classroom activity are specified in
the observation scheme: student activities, sources of input to students, student
behaviour and the distribution of classroom time.
The focus of student activity section provides an initial basis for comparison of
the observed classes in terms of the type and frequency of aural and oral
language practice. The categories also denote the primary focus of the activity,
whether designed to allow the student to focus on the meaning of the message
the student formulates in a communicative activity with others or to focus on
more formal aspects of language usage.
The second category of codes represented both the source of input to the class
and the configuration of the class in each activity. Some materials required
students to respond to teacher-led drills and display/comprehension questions
for some activity types and to work in pairs or small groups in others. The
source of input is considered a potentially important characteristic of methods
in that it could reveal patterns of consistent types of practice embedded in
small-group configurations.
Here the category ‘student behaviour’ indicates the patterns of activities that the
majority of students engaged themselves in. The next category is the
distribution of classroom time. As a fundamental concept in communicative
language teaching is the exchange of messages, the division of the total time
spent in activities into two main classes – form and message focus is an
essential step in defining how different configurations of the classes, and the
activities introduced in the materials, actually could be considered
communicative. Each activity or teaching move is timed with the use of stop
watch so that the distribution of classroom time could be logged and
categorized in terms of time dedicated to pair or group work as opposed to time
taken by the teacher.
The observation schedule
After an orientation and several practice observation sessions, the five
teacher/observers begin the classroom observations on a rotational basis. Each
group using distinct materials is observed on four separate occasions during the
nine month academic year, each time by a different observer. Six classroom
hours out of a total of seventy hours of contact time are observed in this
manner.
Teacher talk as input
Each distinct activity is cross-referenced with a code signifying the sources of
input to the students. A large number of activities involved the students
listening to instructions or explanations, as well as listening in specific

26
exercises devised for that purpose in the text. Four principal categories of
listening comprehension activities are coded in the observations:
1. Listening to teacher speech or tape-recorded material for note taking, map
following, diagram completion
2. Listening to teacher or tape-recorded cues before completing a pair work
speaking task
3. Listening and responding to set patterns
4. Listening to the tape and questioning another student
The cumulative frequency of listening activities observed during the six hours
of observation in each class provides a basis for comparison of the five groups.

In this study, specific hypotheses based on the observation are evaluated

with the use of analysis of covariance. The results of the comparisons of
classroom focus and content imply that pair work leads to more practice time.
Contrary to recent claims (Ferguson 1978) there is no strong evidence to
suggest that pair work which is limited to the manipulation of canned language
in the form of dialogue and non-communicative routines will eventually result
in a general fluency. In contrast with pair work primarily focused on form, there
seems to be considerable potential for other forms of limited pair work which
stimulate the students’ use of unplanned discourse (Pica 1985; Ellis 1986).
The classroom observation data gathered demonstrate how a
longitudinal approach to the evaluation of intact classes can lead to a limited set
of testable hypotheses about product differences. In this study, specific
hypotheses based on the observations are evaluated with the use of analysis of
covariance. No evidence is found to support the position that focus on form
results in greater accuracy. Now was there any clear evidence found to support
a pair work centered method that assumes pair work practice will lead to more
fluent speech. The most direct match of process to product found is between the
incidence of listening input and the development of listening skills in the post-
tests.
The study demonstrates that program-defining evaluation can take place within
an institution by the participant observer/teachers, especially when a
collaborative effort to devise a local coding scheme such as that used in this
study is made. The orientation is towards the development of reliable low-
inferential codings which would lead to quantitative comparisons. The
observation schemata used here is designed to provide a limited account of the
classes. There is still an obvious need to supplement this type of approach with
a qualitative evaluation strategy, discourse analysis of teacher-to-student

27
interaction and student-to-student interaction as well as diary studies of teachers
and students alike.

28
 References

Bachman, L. 1981. Formative evaluation in Specific Purpose program

development.
Beretta, A. 1986. Program-fair language teaching evaluation. TESOL.
Boydell, D. 1974. Teacher-pupil contact in junior classrooms. British Journal
of Educational Psychology.
Cohen, J. and P. Cohen. 1983. Applied Multiple Regression/Correlation
Analysis in the behavioural sciences. Hillsdale, NJ: Lawrence Erlbaum
Associates.
Cook, T. D and D. T. Campbell. 1979. Quasi-Experimentation: Design and
analysis issues for field settings. Boston, MA: Houghton Mifflin
Company.
Davidson, F. 1985. The factor structure of the Fall 1984 ESLPE, Unpublished
paper, University of California, Los Angeles.
Galton, M., B. Simon and P. Croll. 1980. Inside the primary classroom.
London: Routledge and Kegan Paul.
Kenney, D. A. 1975. A quasi-experimental approach to assessing treatment
effects in the nonequivalent control group design.
Miles, M. B and A. M. Huberman. 1984. Qualitative Data Analysis: a source
book of new methods. Beverly Hills, CA: Sage.
Murray, J. and C. Morrison. 1984. Bilingual primary education in the Western
Isles Scotland. Stornoway: Acair.
Scottish Education Department. 1976. Educational Research 1976: A Register
of Current Educational Research Projects. Edinburgh.

Question.
My groupgot only one question from rospinah
Do you have the example of curriculum in reading project that is going to
evaluate.

29
The answer explained clearly in in the paper

30