COLLEGE OF EDUCATION AND LIBERAL ARTS Title : DEVELOPMENT OF ASSESSMENT TOOLS: KNOWLEDGE AND REASONING TYPES OF OBJECTIVE TESTS

ROXANNE M. HALILI ASSESSMENT OF STUDENT LEARNING (MSC 106) PITSBERG B. DE ROSAS

Presenter Subject Professor

: : :

I. INTRODUCTION Teachers or facilitators can use a wide variety of test formats for evaluating student understanding of key course topics. While the focus of this document are objective questions, it does not seek to promote one type of assessment over another. Traditional forms of assessment remain as valid today as they have ever been but, where appropriate, objective approaches have a role to play too. Neither does it seek to explain what you already know. More recently, there has been greater emphasis on objective testing due to its suitability for computerbased assessment; as a result, an increasing number of unit specifications involves an element of objective testing. This document discusses also some advantages and disadvantages of objective tests This guide will be of assistance to any teachers, facilitators or instructors involved in creating objective tests. Objective testing is a technical area with lots of jargon some of which is used inconsistently. This guide is the result of a wide-ranging literature review and seeks to harmonise our terminology with that used internationally. There are no rules for writing objective tests; theres only advice. II. PRESENTATION QUESTION TYPES Educators have been traditionally employed a variety of question types within Unit and Course assessment. These question types can be categorised under two headings: constructed response questions selected response questions.

Constructed Response Questions (SRQ) (also known as openended questions) are questions that require the candidate to create (construct) an answer. Examples of constructed response questions (CRQs) include short answer questions and essays. Example 1 Constructed Response Question Translate Good morning mother into Spanish. Write here: ________________ CRQs can be sub-divided into two sub-categories: Restricted Response Questions Are stricted response question (RRQ)is a question whose answer is limited to a few words. Examples of RRQs include complete-the-sentence, missing word and short answer questions Extended Response Questions An extended response question (ERQ) is one whose answer requires the candidate to write longer responses, normally consisting of two or more paragraphs. Examples of ERQs include reports, essays and dissertations. There is no hard-and-fast rule about where a restricted response question ends and an extended response question begins. Selected Response Question ( S R Q ) is a question whose answer is predetermined and involves the candidate choosing (selecting) the response from a list of options. Because the answer is pre-determined and there is only one correct answer, these types of questions are often referred to as objective questions. Examples of SRQs include true/false, multiple choice and matching questions. Example 2 - Selected Response Question The capital of the United States is New York. True/False Question papers typically consist of constructed response questions. Lower levels generally use restricted response questions and higher levels generally employ extended response questions (although sometimes a mixed approach is used). A limited number of subjects employ selected response questions. This guide focuses on SRQs, which are becoming increasingly popular for a variety of reasons. ADVANTAGES OF SRQs SRQs take less time to answer reducing the amount of time that candidates spend on assessment and increasing learning time. SRQs are quick to mark reducing the time teachers spend on assessment and increasing teaching time. SRQs are well suited to formative assessment since candidates responses can be analysed and used to provide detailed feedback. SRQs are good for assessing breadth of knowledge - they are ideal for assessing a broad range of topics in a short time.

SRQs are more reliable than CRQs - because they get around some of the marking problems associated with written answers.

The low writing load of SRQs means that the focus is on the candidates knowledge rather than the candidates writing or language skills which is a common problem with constructed response questions. Also, the speed of answering SRQs addresses another common criticism of assessment that it takes up too much time for both students and teachers. Research into the marking of CRQs and SRQs has shown significant differences in the reliability of the two approaches with objective tests proving to be significantly more reliable than written tests. This has been the major reason for the wide spread adoption of objective tests in the United States, where testing organisations operate in a more litigious environment. The compatibility of objective tests with computer-assisted assessment is a major driver for the renewed popularity of objective testing. SQA, along with other awarding bodies, is in the process of building banks of questions (item banks) which can be computerised and delivered to candidates over the Internet. DISADVANTAGES OF SRQs SRQs are not suitable for assessing certain abilities, such as communication skills or creativity. They are also not appropriate when candidates are required to construct an argument or provide an original response. SRQs may be less valid than CRQs and suffer from low professional credibility. SRQs that assess higher order skills are difficult (and time consuming) to produce. SRQs can be wordy and require high order reading skills.

The first and second disadvantages are linked. There is nothing inherent in the design of SRQs to make them less valid than CRQs but because they have often been used inappropriately (to measure skills that cannot be properly measured by this style of question) they have established a reputation for being invalid among some practitioners. TYPES OF SELECTED RESPONSE QUESTIONS There are several types of selected response questions (SRQs). Although they share some common characteristics, they each have unique features and applications. But they all share a fundamental characteristic they have one unambiguously correct answer. There are five types of SRQ. These are: 1. True / False Questions A true/false question (T/F) is a statement (not a question!) that is either true or false. The candidate must select one of two possible responses - true or false. Because candidates have a 50/50 chance of answering these questions correctly, this type of question is considered easy and is associated with low order knowledge. However, true/false questions can assess higher order skills;

and setting an appropriate pass mark can eliminate the effects of guessing. Example 3 True/false Question Electrons are larger than molecules. True or False?

2. Matching Questions This type of question requires candidates to match an object with one or more associated characteristics. Example 4 Matching Questions Match the list of storage technologies on the left with the list of memory characteristics on the right. Match each technology (1, 2, 3 or 4) with one characteristic (A, B, C or D) only. 1. Hard Disk 2. Flash Memory 3. RAM 4. ROM A. Non-volatile B. Volatile C. High capacity D. Low cost

The objects on the left are called stimulators and the matching statements on the right are called responses. No more than seven stimulators should be included in any one question. This type of question is often used to assess candidates knowledge of the characteristics of certain objects. It is particularly well suited to computer-based assessment since it can be implemented as drag-and-drop (dragging each response onto an associated stimulator). 3. Multiple Choice Question A multiple choice question (MCQ) consists of a question (or incomplete statement) followed by a list of possible responses from which candidates must select one. There are normally three to five options with four being the most common. Example 5 Multiple Choice Questions On the eve of the French Revolution, France was the wealthiest, most influential, and most populous nation in Europe. Its population was approximately a. 100 million b. 15 million c. 10 million d. 50 million 4. Fill in the Blanks Which asks examinees to supply a word, phrase, or number that answers a question or completes a sentence. Sometimes it is called a completion or fill-inthe-blank test. Example 6 Fill in the Blanks

_____ is the silicate mineral with the lowest melting temperature and the greatest resistance to weathering, and as a result, it makes up the great bulk of sand-sized particles. Fill in the Blank answers are evaluated based on an exact text match. Accordingly, it is important to keep the answers simple and limited to as few words as possible. Answers are not case sensitive, but are evaluated based on spelling. Although what a short-answer test item can assess is generally more limited to factual information, it does not require the development of plausible distracters. Moreover, short-answer items are much less susceptible to guessing than selected-response format items. 6. Enumeration An enumeration of a collection of items is a complete, ordered listing of all of the items in that collection. Example 7 Enumeration Enumerate the different parts of the computer. Some sets can be enumerated by means of a natural ordering (such as 1, 2, 3, 4, ... for the set of positive integers), but in other cases it may be necessary to impose a (perhaps arbitrary) ordering. The precise requirements for an enumeration (for example, whether the set must be finite, or whether the list is allowed to contain repetitions) depend on the branch of mathematics and the context in which one is working. TAXONOMIES OF LEARNING One of the key determinants in the selection of SRQs is the kind of knowledge or understanding that you are seeking to assess. For example, factual recall can be adequately assessed using true/false questions; deeper understanding may require more complex question types such as multiple response questions. As a starting point, we need a method of classifying knowledge and understanding. The most widely used classification system is Blooms Taxonomy. IDENTIFYING THE LEVEL OF A QUESTION Blooms Taxonomy can be used to categorise the cognitive demands of a question. For example, a question asking a candidate to describe something is normally associated with the knowledge domain; another question asking the candidate toexplain something is normally associated with analysis. In fact, the verb in the question can provide a clue to the questions intellectual demands. Table 1 - Verbs Associated with Bloom's Taxonomy Level Verbs Knowledge Comprehension Application Analysis define, describe, label, list, name, recall, show, who, when, where compare, discuss, distinguish, estimate, interpret, predict, summarise apply, calculate, demonstrate, illustrate, relate, show, solve analyse, arrange, categorise, compare, connect, explain, infer, order, separate

Synthesis Evaluation

arrange, combine,compose, create, design, hypothesize, integrate, invent, modify, plan

formulate,

assess, compare, decide, defend, discriminate, evaluate, judge, justify, measure, ran, recommend.

So, for example, a question that commences: Define is likely to assess basic knowledge; a question that begins Compare is likely to assess analytical or evaluative skills. CRITERIA FOR SETTING OBJECTIVE TEST ITEMS Table 2 - Criteria for Setting Objective Test Items TYPE OF OBJECTIVE CRITERIA FOR SETTING OBJECTIVE TEST ITEMS TEST Good directions Use relatively short statements Eliminate extraneous material Keep all true and false statement to approximately the same length, and be sure that there are approximately equal numbers of true and false items AVOID: using absolute terms (e.g., all, always, never, or only) using double-negative statements verbal clues and complex sentences broad general statements using terms denoting indefinite degree (e.g., large, regularly) placing items in a systematic order taking statements directly from the text MATCHING ITEMS Keep both the list of descriptions and the list of options fairly short and homogeneous Title the lists to ensure homogeneity Arrange the descriptions and options in some logical order Make sure that all the options are plausible distracters for each description to ensure homogeneity of lists The list of descriptions should contain the longer phrases or statements, while the options should consist of short phrases, words, or symbols Each description in the list should be numbered Include more options than descriptions In the directions, specify the basis for matching and whether options can be used more than once It is a good idea to introduce some sort of order chronological, numerical, or alphabetical to your list of options The stem of the item should clearly formulate a problem Include only the material needed to make the problem clear and specific Be sure that there is one and only one correct or

TRUE / FALSE ITEMS

MULTIPLECHOICE ITEMS

 FILL IN THE BLANKS / ENUMERATION

clearly best answer Be sure wrong answer choices (distracters) are plausible Eliminate unintentional grammatical clues Keep the length and form of all the answer choices equal Rotate the position of the correct answer from item to item randomly Sort options in ascending order Use negative questions or statements only if the knowledge being tested requires it Include three to five options to optimise testing for knowledge Items should require a single-word answer, or a brief definite statement Avoid statements that are so indefinite that they may be logically answered by several terms Be sure the question or statement pose a problem to the examinee Be sure the answer that the student is required to produce is factually correct Be sure the language used in the question is precise and accurate in relation to the subject matter area being tested Omit only key words Don't eliminate so many elements that the sense of the content is impaired Word the statement so that the blank is near the end of the sentence rather than near the beginning. This will prevent awkward sentences If the problem requires a numerical answer, indicate the units in which it is to be expressed

USES OF SELECTED RESPONSE QUESTIONS As previously mentioned, objective tests are used in a number of SQA summative assessments (such as Higher Physics and some HN units). This style of assessment is well suited to rapid, focussed assessment and is traditionally employed to assess factual recall and basic understanding. It is less commonly used to assess deeper knowledge and understanding, and there are few examples (within SQA or elsewhere) of objective tests being used to assess higher level skills. When used summatively, objective testing tends to be used for low-stakes assessment rather than high stakes assessment, which largely remains the preserve of constructed response questions. However, some subjects (such as Advanced Higher Biology) do employ objective testing and Higher Education has a long tradition in using objective testing for high-stakes summative purposes in some fields (such as Medicine). Objective testing is well suited to formative assessment since it is quick toad minister and assess (lack of time is often cited as the main reason for not using formative assessment). It is particularly suited to diagnostic assessment since it can be used to identify specific misunderstandings or weaknesses. Historically, objective testing has been widely used for psychometric testing (testing of intellect and attitudes) and, more recently, it has been widely

applied to job competence testing. It is also used in entry examinations for some professional bodies. Objective tests are widely used internationally including high stakes assessments such as the SAT in the United States, which is used for university entry. They are also widely used within vendor examinations (such as Microsofts global certification programme). Awarding bodies in every country are focusing on computer-assisted assessment, which has resulted in a renewed interest in objective testing. These organizations share the view that the increasing popularity of e-learning will drive demand for e-assessment which will be underpinned by item banks consisting of large numbers of selected response questions. MISCONCEPTIONS ABOUT OBJECTIVE TESTING Although this document does not seek to promote one type of assessment over another, it does aim to dispel commonly-held, but inaccurate, views about objective testing. Some of the most common misconceptions are rehearsed below. 1. Objective tests dumb-down education; objective tests are easy. Objective tests are as dumb or smart as you choose to make them. 2. Objective tests can only be used to assess basic knowledge. While this is largely true in practice, there is nothing inherent in the design of objective tests to make them unsuitable for assessing high level skills. 3. Objective tests encourage guessing. The problem of guessing can be resolved through one of a number of recognised techniques. 4. Writing an objective test is easy. While most teachers can create simple objective tests, the construction of high quality objective questions is highly skilled and requires significant knowledge and experience. 5. Objective testing is only fashionable because of e-assessment. Its true that objective tests are well suited to computer-assisted assessment but they are also valid and reliable forms of assessment in their own right. 6. Objective tests arent appropriate for my subject. While objective tests have traditionally been used in the physical and social sciences (such as Physics and Psychology), they can be used in any subject III. IMPLICATIONS Most teachers are comfortable with using SRQs to assess low order skills (such as factual recall, typified by Example 2). They are less comfortable with their use in assessing deeper knowledge and understanding. Most currently available examples of SRQs re-affirm this view by focussing on the assessment of surface knowledge;even examples of SRQs that are meant to assess deeper knowledge often only assess surface knowledge. Traditionally, the costs of carrying out assessment come at the end of the

process the setting of the question paper is relatively speedy, the time consuming part comes when the papers have to be marked. Objective tests reverse this model the time consuming bit is the production of the questions, with marking taking very little time. It is, therefore, something of a culture shock to move from traditional assessment to objective testing. Another criticism of SRQs is that they can atomise teaching and learning,encouraging teaching to the test and surface learning. This, combined with their efficiency in assessing large numbers of students in short periods of time, has resulted in them acquiring a reputation as weapons of mass instruction, with poor standing among many educationalists. Do whatever you think is right for your particular test. Assessment is an art not a science. There is no substitute for human judgement. IV. REFERENCES Objective Test Guide www.scribd.com Four Types of Objective Test Items www.scribd.com Objective test - Wikipedia, the free encyclopedia en.wikipedia.org

Prepared by: __________________ ROXANNE M. HALILI