Workshop on the Table of Specifications

Presented by Dr. Chan Chang Tik

Contents
The Table of Specifications and the

Bloom Taxonomy Construction of Examination Questions in line with the Learning Outcomes

Nov. 2006

Semester Break Training

Group Activity 1
Discussion
How do you know whether the examination questions you set are valid?

Nov. 2006

Semester Break Training

Valid?
Moderators comments:
I

think is a significant problem. The students cant answer the question properly on the basis of the information given . I consider the question is either unfair or unanswerable. Why it is here? I accept auditing is part of the learning outcomes but there seems to be too much auditing.
Nov. 2006 Semester Break Training 4

Test Validity
To be valid a test must measure what it

is intended to measure. Three points to consider in test validity:

Validity

is specific to some particular use Validity is a matter of degree Validity pertains to the results of the test

Nov. 2006

Semester Break Training

Types of Validity
Content validity

Criterion-related validity (concurrent and

predictive) Construct validity Which one to choose depends on the aims of your testing.

Nov. 2006

Semester Break Training

Content Validity
The test should cover a cross section of

the course. Emphasize important topics but give less attention to less important topics. The relative importance of topics in a test is directly proportional to the instructional emphasis given to the topics.
Nov. 2006 Semester Break Training 7

Table of Specifications Students Response Q

Step 1

Why?

Number of lectures/chapters/units
Number of questions in the final

exam One table of specifications for each question

Nov. 2006

Semester Break Training

Step 1 Cont.
17 lectures in a course and final exam

consists of 5 questions Q1 3 lectures Equally distributed? Q2 4 lectures Q3 3 lectures Grouping Q4 4 lectures Q5 3 lectures Discussion

Nov. 2006

Semester Break Training

Step 2
Learning objectives

Instruction Assessment List down the number of lectures List down the instructional objectives

Example (Mat250, Q1)

Nov. 2006

Semester Break Training

10

Step 3
Blooms taxonomy of cognitive skills:

Knowledge remembering facts, terms, definitions, and concepts. What?, list, name, define, describe Comprehension understanding the meaning of material. Explain, interpret, summarize, give examples, predict, translate
Semester Break Training 11

Nov. 2006

Blooms taxonomy

Application selecting a concept or skill and using it to solve a problem. Compute, solve, apply, modify, construct Analysis breaking material down into its parts and explaining the hierarchical relations. How does apply?, Why does work?, How does relate to ?

Nov. 2006

Semester Break Training

12

Blooms taxonomy (2)

Synthesis producing something original after having broken the material down into its component parts. How do the data support ? How would you design an experiment that investigates What predictions can you make based upon the data?

Nov. 2006

Semester Break Training

13

Blooms taxonomy (3)

Evaluation making a judgment based upon a pre-established set of criteria. What judgments can you make about ? Compare and contrast criteria for ?

Refer to the table of specifications

Nov. 2006

Semester Break Training

14

Table of Specifications SelectedResponse Q

Work as a team, for example each lecturer is

assigned 5 lectures to come up with 50 multiple choice questions. Store any extra questions in the question bank.

Nov. 2006

Semester Break Training

15

Why Table of Specifications

External Examiners Comments

Several topics covered in the lecture programme are not assessed by this exam. My analysis would indicate no assessment of: The Netherlands, France, Poland, Singapore, South Korea, Hong Kong (10 of the 28 weeks of teaching, 35.7%). The module descriptor, the lecture programme and the exam do not tie up very well.

Nov. 2006

Semester Break Training

16

Why Table of Specifications (2)

INTI-UC lecturers comment
I just want to say a million thanks for teaching us how to do the Table of Specifications in setting exam questions. It is only now that I am able to appreciate how useful that table is, now that I have managed to create my first set of final exam questions (deadline today lah!). I use the table and I am happy with the questions I set because I could justify them if I were asked. I think this part of the skill training (the Table of Specs part ) should be made compulsory for all new lecturers.
Nov. 2006 Semester Break Training 17

Table of Specifications Students Response Questions

Consider Question 1 of MAT250. There are 4 parts in this question taken from the following lectures: Learning Outcomes
Lecture 1 To define real numbers. To describe exponents and to produce the laws of exponents. To explain and simplify radicals. To define the various types of equations. To relate linear equation to its applications. To solve quadratic and radical equations. To discover some practical problems of equations. To explain some coordinates geometry topics. To define inequalities. To solve linear, quadratic and rational inequalities. To solve equations and inequalities involving absolute values. To produce partial fractions.

Lecture 2
Lecture 3

Lecture 4

Lecture 9

Nov. 2006

Semester Break Training

18

Bloom 3

Table of Specifications (2)

Knowledge
(a) To define and simplify exponents and radical expressions. (b) To solve radical equations

Comprehension

Application

Mark

5
(c) To solve rational and quadratic inequalities (d) To produce partial fractions

5
7 5 20

7 5

Total
1 2 17

Nov. 2006

Semester Break Training

19

Table of Specifications Selected Response Questions

Knowledge Comprehension Application No. of Questions 10 (20%) 15 (30%) 15 (30%) 5 (10%) 5 (10%) 50 ( 100%)

1 2 3 4 5

5 (50%) 2 (10%)

2 (20%) 6 (40%) 7 (50%)

3 (30%) 7 (50%) 8 (50%) 3 (70%) 3 (60%) 24 (48%)

1 (10%) 1 (20%) 9 (18%)

1 (20%) 1 (20%) 17 (34%)

Nov. 2006

Semester Break Training

20

Group Activity 2
Set three questions each at different level of

the Blooms Taxonomy using a table of specifications. Indicate clearly your learning outcomes. Present your questions to the participants.
Time : 45 minutes

Nov. 2006

Semester Break Training

21

Types of Assessment
Essay

Mathematical and

technical problem items

Nov. 2006

Semester Break Training

22

Essay
Use clear and concise questions
Measure effectively complex learning outcomes Establish scope of expected response

Handling of technical terms

Content versus mechanics of expression Use descriptive words of precise meaning as

directions No optional items

Nov. 2006 Semester Break Training 23

Clear and Concise Questions

Language usage and word choice are

particular important during question Example construction. The language dimension is critical because it controls the comprehension level of the item for the students and also it specifies the question parameters.

Nov. 2006

Semester Break Training

24

Clear and Concise Questions (2)

For example, Describe the origins of World War I
A better question is What were the principal

diplomatic events in Europe between 1890 and 1913 that contributed directly to the outbreak of World War I You have to narrowly specify, explicate, define, or otherwise clarify what it is that you want from the students.
Nov. 2006 Semester Break Training 25

Clear and Concise Questions (3)

The question set must have an answer that

expert could agree upon, thereby rendering it objective. For example, Describe the world and give two examples.

Example

Nov. 2006

Semester Break Training

26

Complex Learning Outcomes

Complex learning outcomes can be

measured effectively with essay questions. These are the abilities to:
Explain

cause-effect relationships Describe applications of principles Present relevant arguments Formulate tenable hypotheses Formulate valid conclusions
Nov. 2006 Semester Break Training 27

Complex Learning Outcomes (2)

These are abilities to:

State necessary assumptions Explain methods and procedures Produce, organise, and express ideas Integrate learning in different areas Evaluate the worth of ideas

Nov. 2006

Semester Break Training

28

Extended versus Restricted Response

From the instructors standpoint, an extensive

response to a few broadly based questions allows an in-depth sampling of a students knowledge, thinking processes, and problem-solving behaviour relative to a particular topic. The open-ended nature of the question posed by the instructor such as Discuss the bond and share markets in the Malaysian context is challenging to a student.
Nov. 2006 Semester Break Training 29

Extended versus Restricted Response (2)

Extensive response question is poorly

structured and its free-response essay item tends to yield a variety of responses from the students, in both content and organisation, and thus inhibit reliable grading.
Example

Nov. 2006

Semester Break Training

30

Extended versus Restricted Response (3)

A restricted-response question is preferable.

For example, Discuss the advantages and disadvantages of essay and short-answer question with respect to (a) reliability, (b) objectivity, (c) content validity, and (d) usability. This question presents a better defined task which lends itself to reliable scoring, yet allows students sufficient latitude to organise and express their thoughts creatively.
Nov. 2006 Semester Break Training 31

Content versus Mechanics of Expression

Factors such as expression, grammar, spelling,

and the like can be evaluated in conjunction with contents of the essay. Evaluation of such skills is legitimate provided they are included in the learning outcomes. You may point out such mistakes in the students coursework.

Nov. 2006

Semester Break Training

32

Directive Words
Compare

Defend
Trace Contrast Explain Differences

Nov. 2006

Semester Break Training

33

Non-directive Words
Review

Tell all that you know

Report your knowledge

Nov. 2006

Semester Break Training

34

Classic Case
Question (10 marks) Tell me all you know about Malaysia. Answer It is a beautiful country.

Nov. 2006

Semester Break Training

35

Example for Objectivity of Question

External Examiners comment:

The students cant answer the question properly on the basis of the information given as they dont know how to spread the finance charges and looking at the three main alternative methods is not part of the question. I consider the question is either unfair or unanswerable.

Nov. 2006

Semester Break Training

36

Example of Extensive Response

The relevance and the effects of accounting

practices, have been the subject of much debate. Discuss the impact on the accounting practices. External Examiners comments:

Give source. I dont think this is a clear question and I would be amazed if any student produced a response anything like the outline answer.

Nov. 2006

Semester Break Training

37

Example of Language Usage

A business or organisation can be positioned

as effective as products and services. Internal examiners comment:

The sentence does not make sense. I suggest Successful marketing of products and services often depends on effective positioning in the market place.

Nov. 2006

Semester Break Training

38

Group Activity 3
Look back at the essay or short-answer

questions you set in Group Activity 2. Do you think you need to make any adjustments? Present your questions to the participants.
Time : 20 minutes.

Nov. 2006

Semester Break Training

39

Mathematical and Technical Problem Items

Provide enough information and directions so

that students clearly understand the problem. Indicate the degree of precision expected in the response. Clearly specify the units for reporting the response. When using regular everyday words to present a problem, make sure the connection between the structured situation and the real world is genuine.
Nov. 2006 Semester Break Training 40

Enough Information
During the vacation Sue and Tai agreed to

paint their house. One side of the house is eight-meters long. They mark a point on the side that is four meters from each other. Each one starts at one end and paints toward the center mark, four meters from each other. Can we be sure that each painted the same area of that side of the house if each paints to the center mark?

Nov. 2006

Semester Break Training

41

Degree of Precision
In the following set of exercises for calculating

z-scores from the raw scores, use the mean and standard deviation values to the nearest hundredth of a raw score unit. The proportions that you report for the ratios should be given in decimal fractions to the nearest hundredth.

Nov. 2006

Semester Break Training

42

Units
Total elapsed time should be reported in

minutes and seconds. When reporting the perimeters, give your responses in meters.

Nov. 2006

Semester Break Training

43

Everyday Words
What words can you make from the letters in

TEA?
TEA, ATE, EAT, ETA, AET, TAE
Create a permutation for the three letters in

TEA.

Nov. 2006

Semester Break Training

44

Group Activity 4
Look at the mathematical and technical

questions set in Group Activity 2. Are the questions ambiguous? Do you want to re-set them? Present your questions to the participants.
Time: 20 minutes

Nov. 2006

Semester Break Training

45

Thank you
Nov. 2006 Semester Break Training 46