Rubrics on Scoring English Tests for Four Language Skills

Pandiya A teacher at the Accounting Department, Polines Abstract: All language programs in instruction/education institution cover some components like input, process, and output. Input is materials to be processed, process is teaching and learning program and output is the result of process. One of teaching and learning programs is assessment or evaluation of the program. Assessment is carried out by testing/examining language learners. The test is done by objective test and or subjective test and referring to four language skills, i.e. listening, speaking, reading, and writing. Each of them can be done by holistic scoring system or analytic scoring system. The holistic scoring system is done by scoring language skills by single score, without referring to every element of language skills. The analytic scoring system, on the contrary, is scoring every language element of language skills, then totaling all scores to get final score. This is also done from the simplest step to the most complex one and from the micro-skills to the macro-skills. Instructors/teachers then may have one of those alternatives, and or combine of both. Keywords: assessment, objective test, subjective test, holistic scoring system, and analytic scoring system. INTRODUCTION The linguists describe language as means of communication. This is something fundamental about language. At least there are three theoretical views of language. The first is the view that language is a system of structurally related elements for the coding of meaning. The second is the functional view, i.e. language viewed as a vehicle for the expression of functional meaning. The third is interactional view, i.e. language viewed as a vehicle for the realization of interpersonal relations and for the performance of social transactions between individuals. Language is seen as a tool for the creation and maintenance of social relations (Richards and Rodgers, 1992: 17). This emphasizes the fundamental function of language universally, i.e. language as means of communication. Other linguists describe the functions of language further. Jeremy Harmer (2007: 76) described that a language function is a purpose one wishes to achieve when he/she says or writes something. By performing the function, one is performing an act of communication. If one says I apologize, he/she is performing the function of apologizing; the same thing when one says I promise, means he/she is performing the function of promising. Then for an invitation, there is a rather different expression, i.e. Dyou come to the cinema? or any expression like Dyou fancy coming around for a meal? Many functional exponents (patterns or phrases) are exactly the kind of lexical phrases. He stated further that the students who want to express themselves in speaking and writing, they need to know how to perform these functions, and even the appropriate given situations (whether formal or informal). James R Danis (1976: 67) stated that language is the key to understand a subject, so to understand the subject is to understand its language. And almost all of what we customarily call knowledge is language, which means that the key to understand a subject is to understand its language. In fact, that is Ragam Jurnal Pengembangan Humaniora Vol. 13 No. 1, April 2013

43

rather awkward way of saying it, since it implies that there is such as thing as a subject which contains language. It is more accurate to say that what we call a subject is its language. That is biology (for example), other than words? If all the words that biologists use were subtracted from the language, there would be no biology. This means, of course, that every teacher is a language teacher. The last is Gladys G Doty and Janet Ross (1973: 151) stated that language is an instrument to understand the background of people, activities, and values. The study of English as a foreign language nowadays becomes more complex and covers many areas of fields. This will comprise at least three points, i.e. teaching and learning of English as foreign language (EFL), English for Specific Purposes (ESP), and the last: English for Academic Purposes (EAP). Teaching and learning of EFL covers something like English for Adult learners, English for Young learners, English Teaching and Learning Methods, EFL Curriculum, EFL Teacher Education, and EFL Evaluation. ESP relates to something like Vocational English, English for Business and Trade, English for Society Development, English for Mass-media, Literary English, and Propaganda. And the last: English for Academic Purposes (EAP) relates to something like English in Articles/Papers, English for Presentation, Seminar, Thesis and Dissertation Writing. The study of English also relates to the aids for students in the quest for more knowledge from many international sources. There are two media for language communication: (1) receiving the message, i.e. reading and listening, (2) and sending the message, i.e. speaking and writing. The first is also called receptive skills, and the latter is productive skills. To understand how far the language learners master the materials given in an instruction program either the formal program or the informal one, the teachers or educators, and or instructors do some kinds of evaluation activity, i.e. by giving tests for the language learners. The tests themselves usually cover both oral test and written test, and four language skills, i.e. listening, speaking, reading, and writing. This paper aimed at describing any kinds of activity on how to give the tests to the language learners, especially on scoring test system. GOOD ENGLISH LANGUAGE TESTS When discussing the good test, it is an undeniable fact to refer to two sets of good standard test, i.e. reliability and validity. These two things are very important to determine whether the tests are good and adequate or not. The test is reliable if that test as a measurement instrument gives consistent results. In other words the reliability is the degree of consistency between two measures of the same thing (Mehrens and Lehmann in Saleh, 2008: 33). The next is the validity of test, i.e. concerning with the degree to which a test is capable of achieving certain aims. (Mehrens and Lehmann in Saleh, 2008: 33). The aims are usually two, i.e. to describe and predict test takers ability in doing certain tasks. In other words the validity of test is the accuracy of test to measure what must be measured. If the teacher wants to know the test takers ability in delivering speech, the appropriate test is speaking test or oral test, so the written test is not appropriate. For the English instruction program of institutes either formal or informal, the kind of test can be standardized test or standards-based test or both. The good standardized test itself is the product of a thorough process of empirical research and development. It dictates standard procedures for administration and scoring. And finally, it is typical of norm reference test, the goal of which is to place test takers on continuum across a range of scores and to differentiate test takers by their relative ranking (Brown, 2003: 67). The examples of standardized test are TOEFL, TOEIC, IELTS fulfilling the criteria of specifying a set of competencies (standards) for a given domain, and through a process of construct validation they program a set of tasks that have been designed to measure those competencies.(Brown, 2003: 67). For these tests, the instruction institute just joins and cooperates with other institutes who have authority in conducting those

44

Rubrics on Scoring English Tests for Four Language Skills (Pandiya)

kinds of tests, like Gajah Mada University, and Indonesia Australia Language Foundation (IALF) in Jakarta or Denpasar. The other alternative is using standards-based test, i.e. the test composed and prepared by considering any set of curriculum and or syllabus owned by the institutes themselves (Brown, 2003: 105). And the last is using both standardized test and standards-based test. SCORING SYSTEM OF SPEAKING TEST For this discussion the scoring system specifically tends to be for standards-based test. Speaking tests vary according to the language elements being assessed and the objectives of the test. Based on language elements, a language test in general can be classified into discrete-point test, integrative test, and pragmatic test (Oller in Mukminatien, 2000: 38). A discrete-point test measures one of the language components, such as pronunciation, intonation, grammar, vocabulary. An integrative test measures all the language components at a time. And a pragmatic test measures the learners ability in using target language for communicative purposes in a given context, i.e. as a functional speaking test. The functional speaking test can be done in an interactive communication or transactive one (Brown and Yule in Mukminatien, 2000: 39). The first is referring to the function of language to maintain social interaction like interview, and role play. The latter is referring to a type of communication that is focused on conveying the massage, and not on the interaction, like story telling, giving speech, reading an announcement, presenting a report, and many others. According to Underhill in Nur Mukminatien (2000: 39) there are two systems/approaches, i.e. analytic scoring systems and impressionistic system. The first is scoring the learners speaking ability by separating the components of speaking skill into sub skills, and the rater scores each component, and then sums the sub scores into final score. The latter is scoring/judging the learners speaking ability on the basis of the raters general impression on the learners performance without necessarily separating the speaking components. Thus, the rater directly comes to a single score without totaling the sub scores such that in the analytic system. Other experts like Lloyd Jones, White, Spandel, and Stiggins in Nur Mukminatien (2000: 40) called impressionistic system as holistic system/approach in writing assessment. For impressionistic system there are three categories of scoring scale as follows: 0 = inappropriate or seriously incorrect, 1 = relevant but entirely acceptable, 2 = appropriate and correct. For analytic system there are six elements/components to be assessed, i.e. fluency, grammatical accuracy, pronunciation of sentences, pronunciation of words and sounds, interactive communication, and vocabulary resources. The complete description is given as follows: (1) Description of Language Components
No. 1 2 3 4 5 Language Components Pronunciation Grammatical Accuracy Vocabulary Fluency Interactive Communication Description 1. Pronunciation of individual sounds and words 2. Pronunciation of sentences, the right intonation and stress Accurate use of structure, or how the learner gets his/her utterance correct The leaners ability in choosing appropriate words and how to solve the problems when he/she cannot find suitable words by explaining around the word 1. The ability to keep the conversation going 2. Read a text smoothly without hesitation, or inappropriate pause, or repeating words/lines The ability to get the meaning across the listener

Ragam Jurnal Pengembangan Humaniora Vol. 13 No. 1, April 2013

45

(2) Scale Criteria

Scale 0 Proficiency 10 39% Category Very Poor Pron GA Voc Flue IC Pron GA Voc Flue IC Pron GA Voc Flue IC Pron GA Voc Flue IC Pron GA Voc Flue IC : : : : : : : : : : : : : : : : : : : : : : Description of Criteria Many wrong pronunciation No mastery of sentence construction Little knowledge of English words Dominated by hesitation Massage unclear Frequent incorrect pronunciation Major problems in structure Frequent errors of word choice Frequent hesitation Disconnected idea Occasional errors in pronunciation Several errors in structure Occasional errors in word choice Occasional hesitation Ideas stand but loosely organized Some errors in pronunciation Minor problems in structure Minor errors in word choice Minor hesitation Clear and organized ideas No errors/Minor errors Demonstrates mastery of structure (few errors) : Effective/appropriate word choice : No hesitation : Well organized and clear ideas

40 50%

Poor

60 70%

Average

75 80%

Good

85 100%

Very Good

Other experts suggest slightly different scoring systems. HD Brown (2003: 166) suggests scoring guide of spoken English as follows:
No. 1 2 3 4 5 Rating Scale/ Category 60 50 40 30 20 Description of Criteria Communication almost always effective, task performed very competently, speech almost never marked by non-native characteristics Communication generally effective task performed competently, successful use of compensatory strategies, speech sometimes marked by non-native characteristics Communication somewhat effective task performed somewhat competently, some successful use of compensatory strategies, speech regularly marked by non-native characteristics Communication generally not effective, task generally performed poorly, ineffective use of compensatory strategies, speech very frequently marked by non-native characteristics No effective communication, no evidence of ability to perform task, no effective use of compensatory strategies, speech almost always marked by non-native characteristics

For language components, HD Brown (2003: 172-173) suggests six items, i.e. grammar, vocabulary, comprehension, fluency, pronunciation, and task. For other alternative is scoring from 0, 0+, 1, 1+, 2, 2+, 3, 3+, 4, 4+, and 5. And the leveling systems

46

Rubrics on Scoring English Tests for Four Language Skills (Pandiya)

can be superior, advanced, intermediate, and novice (Brown, 2003: 174-177). This, of course, refers to further complex speaking test or oral proficiency/oral production. SCORING SYSTEM OF WRITING TESTS Similar to scoring system of speaking tests, there are also two scoring systems of writing tests, i.e. analytic system and holistic system. The first is scoring the learners writing ability by separating the components of writing skill into sub skills, and the rater scores each component, and then sums the sub scores into final score. The latter is scoring/judging the learners writing ability on the basis of the raters general impression on the learners performance without necessarily separating the writing components. Thus, the rater directly comes to a single score without totaling the sub scores such that in the analytic system. HD Brown (2003: 219) describes any kind of genres of writing as follows: (1) Academic writing such as papers and general subject reports, essays, compositions, academically focused journals, short answer test responses, technical reports (e.g. lab reports), theses, and dissertation, (2) Job related writing such as messages (e.g. phone messages), letters/emails, memos (e.g. interoffice), reports (e.g. job evaluation, project reports), schedules, labels, signs, advertisements, and announcements, (3) Personal writing such as letters, emails, greeting cards, invitations, messages, notes, calendar entries, shopping lists, reminders, financial documents (e.g. checks, tax forms, loan application), forms, questionnaires, medical reports, immigration documents, diaries, personal journals, fiction (e.g. short stories, poetry). The three above aspects will also determine the scoring systems of writing ability. HD Brown (2003: 239) describes rating scale of writing ability by holistic system as follows:
Rating Scale/ Category 6 5 4 3 2 1 0 Description of Criteria Demonstrates clear competence in writing on both the rhetorical and syntactic levels, though it may have occasional errors Demonstrates competence in writing on both the rhetorical and syntactic levels, though it will probably have occasional errors Demonstrates minimal competence in writing on both the rhetorical and syntactic levels Demonstrates some developing competence in writing, but it remains flawed on either the rhetorical or syntactic level, or both Suggests incompetence in writing Demonstrates incompetence in writing Contains no response, merely copies the topic, is off the topic, is written in foreign language, or consists only of keystroke characters

For classroom evaluation learning is best server through analytic scoring in which as many as six major (or five) elements of writing are scored, thus enabling learners to home in on weaknesses and to capitalize on strengths (Brown, 2003: 243). The six major elements of writing, then cover organization, logical development of ideas, grammar, punctuation/spelling/mechanics, and style, and quality of expression, whereas the five major elements cover content, organization, vocabulary, syntax, and mechanics. Analytic scale for rating composition tasks suggested by Brown and Bailey in HD Brown (2003: 244-245) covers some points as follows:

Ragam Jurnal Pengembangan Humaniora Vol. 13 No. 1, April 2013

47

No. 1

Elements of Writing Organization, Introduction, Body, and Conclusion Logical of Development of ideas, Content Grammar

Category/Rating Scale 20 18, 17 15, 14 12, 11 6, 5 1 20 18, 17 15, 14 12, 11 6, 5 1 20 18, 17 15, 14 12, 11 6, 5 1 20 18, 17 15, 14 12, 11 6, 5 1 20 18, 17 15, 14 12, 11 6, 5 1

Punctuation, Spelling, and Mechanics Style and Quality of Expression

Description Excellent to Good, Good to Adequate, Adequate to Fair, Unacceptable, Not college level work Excellent to Good, Good to Adequate, Adequate to Fair, Unacceptable, Not college level work Excellent to Good, Good to Adequate, Adequate to Fair, Unacceptable, Not college level work Excellent to Good, Good to Adequate, Adequate to Fair, Unacceptable, Not college level work Excellent to Good, Good to Adequate, Adequate to Fair, Unacceptable, Not college level work

SCORING SYSTEM OF READING TESTS AND LISTENING TESTS Reading skill and listening skill are different from speaking skill and writing skill. The first is receptive skills, and the latter is productive skills, and or other alternative names, i.e. the first is receiving the message, while the latter is sending/giving the message. It seems that the first is also passive actions, and the latter is active actions, though both skills actually need serious and full energy effort. When referring to genres of reading, then people will cover to some points as follows: (1) Academic reading, such as general interest articles (in magazines, newspapers, etc.), technical reports (e.g. lab reports), professional journal articles, reference material (dictionaries, etc.), textbooks, theses, essays, papers, test directions, editorials, and opinion writing, (2) Job-related reading, such as messages (e.g. phone messages), letters/emails, memos (e.g. interoffice), reports (e.g. job evaluations, project reports), schedules, labels, signs, announcements, forms, applications, questionnaires, financial documents (bills, invoices, etc.), directories (telephone, office, etc.), manuals, and directions, (3) Personal reading, such as newspapers and magazines, letters, emails, greeting cards, invitations, messages, notes, lists, schedules (train, bus, plane, etc.), recipes, menus, maps, calendars, advertisements (commercials, want ads), novels, short stories, jokes, drama, poetry, financial documents (e.g. checks, tax forms, loan application), forms, questionnaires, medical reports, immigration documents, comics strips, cartoons (Brown, 2003: 186-187). For scoring scale of reading test, there are two alternatives, i.e. subjective test, and objective test. Scoring system of objective test is clear, just 1 for correct answer, and 0 for wrong answer. For scoring system of subjective test varies from the point of view of elements of reading, such as grammar, vocabulary, graphology (writing rules/styles), and contents. The variety of scoring scale can be based on the levels of cognitive domain, whether it is knowledge, comprehension, application, analysis, synthesis, or evaluation. This also refers to whether it is micro-skills of reading comprehension, or macro-skills of reading comprehension. For listening tests, scoring system is similar to reading tests, i.e. from the simplest level to the most complex one, or whether it is micro-skills of listening or macro-skills of

48

Rubrics on Scoring English Tests for Four Language Skills (Pandiya)

listening. This started from discriminating among the distinctive sounds of English, and then comes to developing and using a battery of listening strategies, such as detecting key words, guessing the meaning of words from context, appealing for help, and signaling comprehension or lack thereof. This also refers to scoring system by using objective tests or subjective tests. Even for elements of listening are similar to reading, i.e. grammar, vocabulary, contents, phonology aspects (intonation, stress, and or tone). The difference is on graphology (writing rules/styles) for reading, and phonology aspects for listening. The scoring scale of objective test is just 1 for correct answer, and 0 for wrong answer. For subjective tests, the variety lies on the kinds of elements of listening, and also the levels of cognitive domains, whether it is knowledge, comprehension, application, analysis, synthesis, and evaluation. CONCLUSIONS The best and fair scoring system of the test of four language skills lies on the teachers policy, institutions, and all authorized persons involved in education/instruction program. Thus they must collaborate and work together for creating qualified education/ instruction program, especially on evaluation/assessment activities. REFERENCES Bloom. 2004. Blooms Taxonomys Model Questions and Key Words. The UT Learning Center, The University of Texas at Austin. Brown, HD. 2003. Language Assessment: Principles and Classroom Practices. San Francisco State University, California. Danis, James R. 1976. Teaching Strategies for College Classroom. Westview Press. Harmer, Jeremy. 2007. How to Teach English, An Introduction to the Practice of English Language Teaching. England: Pearson Education Limited. Mukminatien, Nur. 2000. The Advantages of Using an Analytic Scoring Procedure in Speaking Assessment, TEFLIN Journal, Vol. XI, No. 1 Agustus 2000. Universitas Negeri Malang. Richards, Jack C and Theodore S Rodgers. 1992. Approaches and Methods in Language Teaching: A description and analysis. Cambridge: Cambridge University Press. Saleh, Mursid. 2008. Enam Tradisi Besar Penelitian Pendidikan Bahasa. Program Pascasarjana Universitas Negeri Semarang.

Ragam Jurnal Pengembangan Humaniora Vol. 13 No. 1, April 2013

49