Teachers’ perceptions of student evaluations of teaching

This article was downloaded by: [University of Hong Kong Libraries] On: 25 June 2014, At: 18:12 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK Educational Research and Evaluation: An International Journal on Theory and Practice Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/nere20 Teachers’ perceptions of student evaluations of teaching a a a Cecilia K.Y. Chan , Lillian Y.Y. Luk & Min Zeng a Centre for the Enhancement of Teaching and Learning, The University of Hong Kong, Hong Kong Published online: 23 Jun 2014. To cite this article: Cecilia K.Y. Chan, Lillian Y.Y. Luk & Min Zeng (2014): Teachers’ perceptions of student evaluations of teaching, Educational Research and Evaluation: An International Journal on Theory and Practice, DOI: 10.1080/13803611.2014.932698 To link to this article: http://dx.doi.org/10.1080/13803611.2014.932698 PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions Educational Research and Evaluation, 2014 http://dx.doi.org/10.1080/13803611.2014.932698 Teachers’ perceptions of student evaluations of teaching Cecilia K.Y. Chan*, Lillian Y.Y. Luk and Min Zeng Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 Centre for the Enhancement of Teaching and Learning, The University of Hong Kong, Hong Kong (Received 21 January 2014; final version received 26 May 2014) Evaluation of teaching in higher education has drawn much attention due to the need for greater accountability and improvement in student learning. Our review of literature on Student Evaluation of Teaching (SET) surveys suggests that considerable controversy and criticism have surrounded its use, fairness, and validity. Yet, many universities in Hong Kong are depending primarily on student feedback in SET surveys to provide evidence for their improvement of teaching and learning, and for their decisions made on teachers’ pay-rise and tenure. If universities are to continue to use SET surveys as the main source of mechanism for measuring teaching effectiveness, teachers will have to understand and be aware of its purposes. But how much do we know about teachers’ perceptions of teaching and learning and, consequently, their opinions of using SET surveys as the primary standardized evaluation mechanism of teaching for promotion and tenure and for teaching effectiveness? Keywords: student evaluation; teaching effectiveness; higher education; SET survey; teacher perception of teaching and learning Introduction The evaluation of teaching effectiveness has become a major issue in higher education in recent years, due to the need for greater accountability and improvement in the quality of teaching and, more important, student learning (Leckey & Neill, 2001). However, the quality of teaching has not led to an improvement in academic quality but a degree of performance in peer reviews or dramaturgical compliance (Barrow, 1999) – an “implementation gap” between policies on quality and actual outcomes as described by Newton (2000). There are various mechanisms to evaluate teaching effectiveness, such as online questionnaires, class observation, peer review, surveys, and analysis of student drop-out rates (Slade & McConville, 2006). Student feedback using Student Evaluation of Teaching (SET) surveys is one of the several mechanisms commonly used to measure and improve teaching and learning. The survey usually employs the use of questionnaire items to evaluate teacher effectiveness and various areas of the course. Many universities in Hong Kong have been found to be overly reliant on SET surveys (Kember, Leung, & Kwan, 2002) to gather evaluation information for their course improvement. In particular, these universities tend to rely on SET survey results to provide benchmarking for teaching evaluation on *Corresponding author. Email: [email protected] © 2014 Taylor & Francis Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 2 C.K.Y. Chan et al. promotion and tenure decision-making in the absence of any standardized mechanism to evaluate good teaching (particularly where research is still viewed as the most important factor for promotion and tenure in research-intensive universities). Researchers such as Surgenor (2013) and Wachtel (1998) have suggested that faculty members have both positive and negative attitudes towards the purpose, validity, usefulness, and consequences of SET. Despite the inconsistencies in opinions towards the use of SET surveys, students’ rating of teaching effectiveness has long been playing a significant role in the evaluation of teaching in higher education institutions in Hong Kong. The present study examines science teachers’ perceptions of teaching evaluation using a SET survey mechanism at a research-intensive university in Hong Kong. In particular, the science discipline of this Hong Kong research-intensive university is of interest as this university is consistently ranked among the top-five institutions in Asia, and its science faculty has a large number of professors who have conducted world-renowned discipline research. For this university, the teaching and learning culture was not evident as teaching was not previously rewarded and there was no structural agreement on the evaluation criteria for teaching. In fact, only in recent years have a teaching and learning philosophy and university educational aims emerged. The SET instrument which the university has adopted incorporates different teaching and learning aspects including: (a) student learning information – the overall evaluation of the course including the use of English as the Medium of instruction; (b) the teacher – the overall evaluation of the respective teacher (s) who has/have delivered the main part of the course; and (c) the pedagogy – the evaluation of the delivery of the course including lecture/tutorial/laboratory in order to achieve the intended learning outcomes. Given the strong research culture of the university and its faculty, the aim of this research was to determine teachers’ perceptions of teaching and learning and, consequently, their opinions of using SET surveys as the primary and standardized mechanism for evaluating the effectiveness of teaching and learning, and for promotion and tenure. This study will also contribute to the knowledge about teachers’ interpretations of SET survey results in the Asian context and how they perceive the effectiveness and use of SET surveys. Such knowledge would be valuable for helping faculty members and policy makers evaluate, construct, and refine their current practices of SET surveys and other evaluation mechanisms for teaching and learning. Review of literature Student Evaluation of Teaching (SET) survey Reviews of the existing literature (e.g., Aleamoni, 1999; Marsh, 2007; Spooren, Brockx, & Mortelmans, 2013; Wachtel, 1998) have revealed inconsistent findings leading to an ongoing debate on the reliability and validity of SET surveys. Arguments against using SET surveys often raise issues related to (a) the definition of effective teaching; (b) other variables influencing student ratings, which may not be related to teaching ability; and (c) negative reactions from faculty members (Anderson, 2006; Scott, Stone, & Dinham, 2000). First, teaching effectiveness is a multifaceted construct (Marsh, 1984). Marsh (1984), Braskamp and Ory (1994), and Centra (1993), for example, identified six major components of effective teaching that are often included in SET surveys: course planning, communication skills, teacher–student interaction, course difficulty, grading, and student self-rated learning. However, there continues to be an absence of agreement on the definition of effective teaching (Casey, Gentile, & Bigger, 1997; Stehle, Spinath, & Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 Educational Research and Evaluation 3 Kadmon, 2012). Murphy, MacLaren, and Flynn (2009) have pointed out that it is difficult to measure quality of teaching because of its complex nature. Second, student ratings have been found to be influenced by other factors such as course characteristics, student characteristics, and teacher characteristics (Wachtel, 1998). For example, Marsh (1980, 2007) reported that students with a greater prior interest in the subject and students who were expecting higher grades tended to give higher teacher ratings. Apart from prior subject interest and expected grades, students’ perceptions of their teachers’ personality have been found to be related to their assessment of teacher effectiveness (Clayson, 2013; Tomasco, 1980). For instance, Shevlin, Banyard, Davies, and Griffiths (2000) identified “charisma” as a trait which influences students’ rating of their teacher. In a subsequent study conducted by Adamson, O’Kane, and Shevlin (2005), it was found that students’ perceptions of how entertaining a teacher was positively related to their ratings of teaching effectiveness. In fact, researchers such as Moore and Kuol (2005) doubted students’ ability to assess teacher performance due to their limited knowledge on teaching, and they expressed their concern towards student evaluation that is based on personal feelings and expectations, leading to students’ inability to make an objective judgement. Third, some teachers may try to get more favourable ratings by increasing students’ grades and lowering the standard of the course, causing grade inflation (Simpson & Siguaw, 2000). Thus, some teachers argue that SET surveys fail to serve their educational purpose and even violate the academic freedom and rights of teachers (Slade & McConville, 2006). Despite the criticisms directed towards using SET surveys, supporters of its use argue that effective use of feedback data (Hendry, Lyon, & Henderson-Smart, 2007; Kember et al., 2002) does provide useful information for teachers which helps improve student learning (Ballantyne, Borthwick, & Packer, 2000). A recent study conducted by Winchester and Winchester (2014) suggests that the use of weekly formative SET surveys helps teachers improve on their teaching through reflective practice, which in turn increases their summative SET score towards the end of the academic year. Aleamoni (1999) clarified 16 myths regarding SET surveys and proposed that students’ feedback does provide accurate information when data are collected using well-designed instruments. Lemos, Queiros, Teixeira, and Menezes (2011) demonstrated the reliability and validity of a students’ evaluation of teaching questionnaire, which was developed based on learning theory. Contrary to the findings in previous studies, Beran and Violato (2005) found that course and student characteristics have little influence on the rating of teachers and suggested that the ratings do measure teaching performance. It is also interesting to note that an analysis of more than 30,000 open-ended SET comments from students in a recent study conducted by Tucker (2014) revealed that the majority of the students provided meaningful feedback which was neither abusive nor offensive. However, inconsistencies across the different contexts of studies and research findings make it difficult for researchers and practitioners to agree on the reliability and validity of SET surveys. Although extensive research has been carried out on students’ feedback on the effectiveness of teaching, limited studies have examined teachers’ perceptions of SET surveys (Arthur, 2009). Teachers’ perceptions of the SET survey A review of the literature reveals that there is a lack of consensus in teachers’ attitudes towards the use of SET surveys (Beran & Rokosh, 2009; Nasser & Fresko, 2002). A national project conducted in New Zealand by Stein and colleagues (2012) and a study conducted by Surgenor (2013) at an Irish university have revealed that academics show both Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 4 C.K.Y. Chan et al. positive and negative attitudes towards the purpose, validity, usefulness, and consequences of the surveys. With regard to its purposes, teachers in these two studies mentioned that the surveys provided them with feedback that would help them to identify their own strengths and weaknesses in teaching. However, teachers in Surgenor’s (2013) study were concerned about the possible use of the SET survey as a means of surveillance, monitoring their performance for future reward or penalty. While more opportunities than challenges were identified with regard to the purpose of SET surveys, more barriers than opportunities have been identified with regard to the validity, usefulness, and consequences of SET (Surgenor, 2013). In terms of validity, some teachers in Surgenor’s (2013) study expressed concern towards sampling bias, which may result in questionable comments: extremely positive comments from dedicated students and extremely negative comments from discontented students. These teachers also noted that the survey may involve comments from both students who attend classes regularly and students who do not. Stein et al. (2012) found that some teachers were sceptical about the reliability of students’ feedback as they believe that students tend to be biased towards popular teachers. Despite doubts about the validity and reliability of SET surveys, a small proportion of participants in Surgenor’s (2013) study suggested that using the survey as a uniform evaluation tool across different faculties will raise the level of objectivity in the evaluation of teaching. In terms of the usefulness of the survey data, teachers were aware of the possible impact of extraneous factors on the SET survey ratings, and of the possible ways to increase their scores (e.g., increasing grades and lowering course standards). This finding coincides with that of Arthur’s (2009) study, in which teachers interviewed revealed those factors they believed to influence teaching evaluations including relationships between teacher and students, student expectations, course type, and students’ course experience. In terms of the consequences of SET surveys, teachers in Surgenor’s (2013) study acknowledged developmental opportunities that may emerge as a result of using SET surveys. For instance, they believed that using the survey will increase awareness of evaluation, which will provide them with an overview of their teaching and promote a reflective culture in teaching practice. However, they cast doubt on the degree of confidentiality and the use of data. Concerns about the confidentiality and the use of data were also reported in other studies (e.g., Beran & Rokosh, 2009; Nasser & Fresko, 2002). For example, there seems to be a general agreement that teachers should receive students’ feedback on their teaching (Nasser & Fresko, 2002) and that such data should not be made available to other staff members, students, or the public (Beran & Rokosh, 2009). On the other hand, there are conflicting views on the usage of the survey scores. Teachers interviewed by Balam and Shannon (2010) reported that the survey was used to improve their teaching; whereas in Nasser and Fresko’s (2002) study, only a few teachers mentioned that they made changes to their teaching in response to the SET data. Arthur (2009) found that whether or not teachers make any changes or improvements in response to the student feedback depends on a number of factors (e.g., the number of students reporting the same experience, teachers’ perceived importance of teaching or relevance of a certain topic, and the teaching/ research culture of the university). Arthur suggested that the decision to make any changes or improvements in teaching is dependent on teachers’ interpretations of the SET data; thus, this study aimed to understand teachers’ perceptions of SET surveys will provide information on how learning can be enhanced. Overall, teachers seem to have both positive and negative attitudes towards SET surveys. Nasser and Fresko (2002) point out that teachers who receive higher ratings Educational Research and Evaluation 5 Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 tend to display stronger beliefs and support for using the survey. According to Kwan (1999), science and engineering teachers tend to receive significantly lower ratings than language and business teachers do; therefore, teachers in different academic disciplines may differ in their attitudes towards SET surveys. Given the disciplinary differences, teachers’ perceived importance of teaching, and the teaching and research culture of the university, the present study aimed to investigate university science teachers’ perceptions of the use of a SET survey as the primary standardized evaluation mechanism of teaching and learning for promotion and tenure and for teaching and learning effectiveness. Methodology The objectives of this study are to: (1) investigate teachers’ perceptions to teaching and learning in the science discipline at a research-intensive university in Hong Kong; (2) analyse teachers’ current perceptions and opinions in the science discipline towards the use of a SET survey as the primary standardized evaluation mechanism of teaching and learning for promotion and tenure and for teaching and learning effectiveness at a research-intensive Hong Kong University. Quantitative and qualitative data were collected using questionnaires from teachers in the Science discipline at a research-intensive university in Hong Kong. The questionnaire included 17 close-ended items and one open-ended question. Responses to 13 close-ended items were elicited using a 5-point categorical scale (from strongly agree to strongly disagree), and 4 close-ended items were elicited using a “yes-and-no” scale. With the closed-ended items, teachers were asked about their perceived importance of teaching versus research, and their interest in teaching versus research. Related questions such as teachers’ awareness of the university’s teaching and learning support centre and their academic development activities were also included. Good teaching practices for student learning such as out-of-class contact with students, placing science teaching in a societal context, and enhancing course material in response to feedback were also incorporated into the questionnaire, to identify if the teachers realized good and effective teaching which reflected on teachers’ perceptions of teaching and learning. Only one general open-ended question was proposed, which was: “Comments (please feel free to tell us anything else you feel may be relevant to teaching and learning in the Faculty)”. The questionnaire was initially piloted with five science teachers. Certain items were revised based on the pilot to provide a more straightforward understanding and to ensure the questions were more related to the objectives according to the teachers’ feedback. Subject-matter experts were also consulted to help improve the content validity of the questionnaire items. From the qualitative comments collected, we identified a large number of comments directly related to the SET. The study received ethical approval from the university, and participant anonymity was assured. An information sheet was provided to the teachers for their inform consent and to avoid any further conflicts of interest within the faculty. An outside researcher (researcher from the teaching and learning centre) was invited to conduct the research study in order to ensure that teachers would not feel uncomfortable or threatened if the study was conducted by the faculty management. Participants from the faculty were invited to complete the survey using a purposive sampling method. To ensure sufficient questionnaire participation, both online and paper Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 6 C.K.Y. Chan et al. versions were distributed: An email was sent to the teachers to invite them to complete an online version of the questionnaire; a paper version of the questionnaire was also delivered to their letter boxes at their department. Personalized follow-up emails were sent to teachers to improve the response rate, and two research assistants were employed to help distribute the paper questionnaires directly to the teachers at their offices. Sixty-two out of 95 science teachers at the Hong Kong university participated in the study (65% response rate). Demographic information about the participants is provided in Table 1. The questions and the results of the questionnaire are summarized in Table 2 and are grouped together based on Objectives 1 and 2. For the teachers’ perceptions of teaching and learning (i.e., Objective 1), the quantitative data are analysed by calculating the percentage of responses for each category. The percentage of agreement (or disagreement) on the questionnaire items was calculated by adding up the responses Strongly agree (or Strongly disagree) and Agree (or Disagree). For the perception of the SET (i.e., Objective 2), in addition to the percentage of responses, the Spearman’s rank correlation coefficient was also calculated to investigate correlations between the different factors that influence teachers’ perceptions of SET. With the use of the cutting and sorting technique (Ryan & Bernard, 2003), key words and phrases in the qualitative data were identified and coded to reflect key concepts in the teachers’ comments. The codes were then sorted into categories (i.e., themes), which were named “perception of teaching versus research”, “perception of SET”, “alternative mechanism for teaching evaluation”, “curriculum reform for student learning”, and “challenges using English as a medium of instruction”. The themes from comments were compared with the quantitative data to determine any patterns (differences/similarities) in findings. Results Analysis of the quantitative data suggests that teachers had diverse attitudes towards teaching and research. Table 2 shows that 44% of the participants indicated that they enjoyed research more than teaching. The perceived time spent on research varies among participants, such that 37% indicated that they spent more time on research than teaching, 36% disagreed, and 27% held a neutral opinion. Despite participants’ diverse opinions in terms of their perceived time spent on research, most of them (56%) believed that research is a priority due to the performance review and tenure track system, and 44.3% believed that teaching is of less importance than research in their performance review. Although a generally positive attitude towards research was observed among the participants, most of them (67%) indicated that they had not engaged in any research activity related to teaching and learning. In fact, Table 1. Demographic information of participants. Background information Position Professor Associate Professor Assistant Professor Teaching Consultant Status Tenured Non-tenured % 19.7 39.3 26.2 14.8 52.5 47.5 Background information % Department Biological Sciences Chemistry Earth Sciences Mathematics Physics Statistics and Actuarial Science 19.7 9.8 19.7 18.0 14.8 18.0 Educational Research and Evaluation 7 Table 2. Teachers’ perception of teaching and research. Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 Frequency (%) Items on teachers’ perception on teaching & learning n I “enjoy” research more than I “enjoy” teaching. I spend more time on research than teaching. Due to the performance and tenure-track system, research is a priority for me. In my performance review, teaching is regarded less important than research. I am aware of the full scope of services provided by the teaching and learning centre such as support on course development, consultancy on learning and teaching, and educational research collaboration. I consider out-of-class contact with students as important. It is important to place science teaching in a societal context. I have engaged in a research activity related to teaching and learning. Have you ever applied for a Teaching Development Grant (TDG)? I have previously attended the teaching and learning centre seminars/workshops on learning and teaching. I update my course material each year to reflect advances in the subject. I update my course material each year in response to SET feedback. Agree Neutral Disagree 62 44 62 37 61 56 36 27 28 21 36 16 61 44 28 28 61 56 30 15 58 93 58 69 7 19 0 12 n Yes 61 33 No 67 62 29 61 64 71 36 57 95 5 57 83 18 71% of them indicated that they had never applied for teaching-related research or a development grant even though all of them would have applied for external or internal research grants in their research areas. This culture of teaching and learning is also captured in the qualitative data, where one of the teachers criticized the performance system: The system does not encourage good teaching. Teachers are only rewarded by research performance, i.e., IF (impact factor) journal papers and UGC grants. The current HR policy is killing high-quality teaching. The management style is focused on elementary numbers (SET) no matter how useless, rather than tackles the much harder job of actually finding out what goes on in the classroom. The quantitative data in Table 3 indicated the teachers’ perceptions towards the SET survey, as well as their opinions on the reasons for low SET scores. Of the participants (n = 62), 39% believed that the SET score reflected their teaching quality and ability, 40% disagreed, and 21% held a neutral attitude. In spite of the mixed attitudes toward SET, 83% of the participants used SET feedback for the refinement of their course materials. The major perceptions expressed by the participants (n = 62) for low SET scores were: the difficulty in subject matter (76%), students’ low interests in the subject matter (79%), and teachers’ low motivation (54%). Only 38% of the participants regarded students’ lack of interest in new learning approaches as the reason for low SET scores. Table 3 also presents teachers’ opinions on the reasons for low SET scores, based on whether they believed that the scores reflect their teaching quality and ability. Among those who believed that SET scores reflect their teaching quality and ability (n = 24), Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 8 C.K.Y. Chan et al. most agreed that the difficulty of the subject matter (75%) and student’s lack of interest in the subject matter (71%) were reasons for low SET scores. Most teachers (78%) agreed that teachers with low teaching motivation is a reason for low SET scores, and only 30% attributed this to students’ negative attitudes towards new learning approaches. Among those who had doubts on SET scores as a reflection of their teaching quality and ability (n = 25), 80% perceived that the difficulty of the subject matter would influence SET scores, 80% regarded students’ low interest in the subject matter as a reason for low SET scores, whereas only 44% agreed that students’ negative attitudes towards new learning approaches would result in low SET scores. Overall, the results indicate that regardless of whether teachers believed that SET scores reflect their teaching quality and ability, there is a general agreement that courses obtain low SET scores because of students’ lack of interest in the subject matter and the subject matter being too difficult. Compared to students’ interest and subject difficulty, fewer teachers agreed that a low SET score is a result of students’ negative attitudes towards new learning approaches. Although fewer teachers agreed that students’ acceptance of new learning approaches was a reason for a low SET score, qualitative data collected from the openended comments provide rich details on the relationship between SET score and students’ acceptance of new learning approaches, as perceived by the teachers. For example, one teacher believed that SET score is proportionally related to the way students perceive new learning approaches. Some courses with a heavy field-trip component will surely be welcome by the students and thus the SET score will be high. Others with more traditional or difficult components will be less welcome by the students and thus no matter how good the teacher was the score will just be around 70. When comparing to the former courses then the latter will be below average. Another teacher believed that “the students prefer to be spoon-fed and get easy concrete material, and many faculties focus this way to get high SET scores”. One teacher worried that new learning approaches may penalize their SET scores and careers, even though teachers agree that an authentic approach should be adapted for better student learning. I feel obliged to the students to adopt an industry-focused case-study approach for one of my courses, but it is becoming too “expensive” in harm to my career from SET to continue this. I have no choice but to switch to a grinning monkey style of teaching with all the material handed on a plate to the students. Teachers were aware that by lowering the standard of the course particularly on students’ requests, SET scores may rise, although they were also aware that this is a poor method of student learning. If the students keep on indicating that the course content was too difficult and the teacher has to lower the standard of the course we will eventually end up with substandard graduates who are not comparable to graduates of other international universities. While fewer teachers agreed on students’ acceptance of new learning approaches as a reason for low SET scores, there are more teachers who believed that low teaching motivation leads to low SET scores. This is especially true among those who believed that SET scores reflected their teaching quality and ability than those who disagreed. This finding suggests that teachers who believe in the reliability of SET in reflecting their teaching Attitude towards SET, opinions on the reasons for low SET scores, and use of SET feedback. SET n Disagree Neutral Agree All % 25 40 13 21 24 39 62 100 SM n 25 13 24 62 SI Disagree Neutral Agree (%) (%) (%) n 4 15 8 8 16 15 17 16 80 69 75 76 25 13 24 62 NA Disagree Neutral Agree (%) (%) (%) n 8 0 13 8 12 8 17 13 80 92 71 79 25 13 23 61 TM Disagree Neutral Agree (%) (%) (%) n 20 8 17 16 36 54 52 46 44 39 30 38 25 13 23 61 SETF Disagree Neutral Agree (%) (%) (%) n 36 23.1 13 25 28 31 9 21 36 46 78 54 22 12 23 57 Yes No (%) (%) 82 75 87 83 18 25 13 18 Notes: SET = “SET scores reflect my teaching quality and ability”; SM = “Subject matter is too difficult”; SI = “Students are not interested”; NA = “Students do not like new learning approaches”; TM = “Teachers are not motivated”; SETF = “I update my course material each year in response to SET feedback”. The items on SET, SM, SI, NA, and TM were assessed on a 5-point Likert scale with 1= Strongly disagree, 5 = Strongly agree, and 3 = Neutral. The percentage of agreement (or disagreement) on the questionnaire items was calculated by adding the responses Strongly agree (or Strongly disagree) and Agree (or Disagree). The participants were invited to choose “Yes” or “No” in response to SETF. Table 4. SET SM SI NA TM Correlations between opinion on SET and the reasons for low SET scores. SET SM SI NA TM 1.00 .001 −.13 −.16 .37** 1.00 .30* .17 .23 1.00 .37** −.11 1.00 −.09 1.00 Notes: SET = “SET scores reflect my teaching quality and ability”; SM = “Subject matter is too difficult”; SI = “Students are not interested”; NA = “Students do not like new learning approaches”; TM = “Teachers are not motivated”. **p < .01. *p < .05. Educational Research and Evaluation Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 Table 3. 9 Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 10 C.K.Y. Chan et al. are more likely to associate low SET scores with internal factors (e.g., teaching motivation) than those who do not. This finding is further supported by a significant correlation found between participants’ opinion on teachers’ motivation as a factor influencing SET scores, and SET scores as a measurement of teaching quality and ability (r = .37, p < .01, see Table 4), denoting that participants who believed that SET scores reflected their teaching quality and ability also believed teachers’ insufficient motivation was a reason for low SET scores. No statistically significant correlations were found between participants’ attitudes towards SET scores as a measurement of teaching quality and ability, and their opinions on the other possible reasons for low SET scores such as the perceived difficulties of the subject matters, the anticipation of students’ interests in the subject matters, and the anticipation of students’ acceptance of new learning approaches. Neither was a statistically significant correlation found between these factors and teacher’s motivation. However, there were statistically significant correlations identified among the three factors as shown in Table 4. A positive correlation was found between (a) perceived subject matter difficulty and perceived students’ subject matter interest (r = .30, p < .05) and (b) perceived students’ subject matter interest and perceived students’ acceptance of new learning approaches (r = .37, p < .01). Among the participants who believed that SET scores reflected their teaching quality and ability (n = 23), 87% updated their course material each year in response to SET feedback (see Table 3). For those who did not believe that SET scores reflected their teaching quality and ability (n = 22), 82% updated their course material each year in response to SET feedback. In other words, most teachers in this study took SET feedback seriously in informing their amendment/improvement of course material regardless of whether they believe that SET scores reflect their teaching quality and ability. The majority of teachers seemed to wish to improve their teaching quality, and some participants indicated that the university SET did not seem to be the best evaluation mechanism for identifying their teaching and learning issues: The Faculty of Science has refused, in writing, to implement mandatory peer assessment of teaching. The current SET in the Faculty of Science is not supported by any other assessment such as peer-observation of teaching. Discussion Previous research has shown a lack of consensus in teachers’ opinions on the use of SET. The results in this study indicate that within this science faculty, teachers generally perceived teaching and learning as less important than research. This is likely due to the heavy research weightings that are expected of the academics for their yearly performance reviews and for tenure and promotion. This perception of teaching and learning seems to persist even though more than half of the participants are already tenured and are either at professor or associate professor positions (at this university, only professor or associate professor positions may be considered for tenure). Participants also seemed to be aware of those features of an effective teacher, as almost all of them realized that out-of-class contact with students is an important factor, and they seemed to place science teaching in a societal context. Most of them have also reported that they update their courses regularly in response to feedback and curriculum changes. They also seem to be aware of the scope of support services provided by the university’s teaching and learning centre, and more than half of them have attended seminars and workshops on teaching and learning. In Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 Educational Research and Evaluation 11 relation to the perception of using SET surveys as the primary evaluation mechanism for promotion and tenure and for course effectiveness, diverse attitudes among the teachers were revealed. Science teachers who participated in our study generally believed that students’ lack of interest in the subject matter and students’ perceived difficulty of the subject matter were the two reasons for a low SET score, suggesting that teachers were aware that students’ subject interest and perceived subject difficulty could influence students’ rating of teaching effectiveness. Marsh (2007) supports the view that students with greater interest in a subject tend to give higher teacher ratings. Qualitative data from this study also reveal that teachers believe they can increase their SET survey score by adopting more traditional teaching approaches and by lowering the difficulty of their course content as requested by students. Teachers believe that students’ preferences of the teaching approaches and the standards of the curriculum are influencing their practice indicating this is because the university has been putting a lot of emphasis on the SET survey scores and this emphasis may cause harm to the teachers’ careers if students do not like the teachers’ approaches. A significant correlation was found between the three reasons for teachers’ opinions for low SET survey scores (i.e., subject matter difficulty, students’ interest in subject matter, and students’ positive attitudes towards new learning approaches). It was noted from the correlation that: (a) the more difficult the subject matter, the more likely students’ interest towards the subject decreases; (b) as students’ interest towards the subject decreases, so does their attitude towards the new learning approaches. However, these correlations are only moderate, suggesting the possibility of other extraneous factors, such as curriculum design, student grades, and course standards in influencing SET score (Surgenor, 2013). These other factors have not been investigated in the present study. The findings of this study reveal that teachers who believe in the reliability of SET in reflecting their teaching are more likely to associate low SET scores with teaching motivation than teachers who do not hold this belief. These findings indicate that factors that influence the SET scores are related to course, students, and teachers characteristics; and it is only teacher perception of their own teaching motivation that can be totally controlled by teachers – an internal factor – whereas the subject matter difficulty, students’ interest in subject matter, and students’ positive attitudes towards new learning approaches are external factors that are related to the course and student characteristics. These characteristics related to student motivation and engagement can only be marginally controlled by teachers. There has been a notable concern among the participants about using SET surveys as a dominant evaluation for tenure and promotion and as the only measurement of teaching in our study. Some teachers have expressed strong, negative opinions about using SET surveys as the primary mechanism for evaluating teaching. Some felt that the SET survey is “dumbling down” the university system and is a “threat” to improving teaching and learning quality, and this is consistent with earlier literature findings (e.g., Stein et al., 2012; Surgenor, 2013; Wong & Moni, 2014). In their opinions, SET feedback is possibly affected by the course and student characteristics such as difficulty of the subject, student interest in the subject, and student preferences in teaching and learning approaches. When SET surveys are used as the only measurement, teachers are likely to be forced to consider these factors and adjust their teaching unintentionally in order to obtain better student ratings. The teaching would then be solely driven by students’ preferences and interests, and might not necessarily align with the curriculum’s learning outcomes or instructional approaches advocated by the university. This shows that using SET surveys may have an opposite effect of demotivating teachers from trying new teaching approaches rather than encouraging for teaching improvement. Thus, our findings suggest that the inclusion of Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 12 C.K.Y. Chan et al. other direct evidence of quality teaching (see Australian University Teaching Criteria and Standards Framework, n.d.), such as peer evaluation of teaching (Lomas & Nicholls, 2005), in addition to SET surveys is a worthwhile consideration. Doing so may help triangulate SET surveys and provide teachers with information from various perspectives for the improvement of teaching. Most important, it may help balance the wash-back effect of SET on teaching. In fact, Stark-Wroblewski, Ahlering, and Brill (2007) proposed that other measures of student learning should be documented for promotion and tenure review in addition to SET surveys, in order to present a more comprehensive view on teaching effectiveness. Despite diverse opinions on the effectiveness of SET surveys as a measurement of teaching quality and ability and despite concerns on the use of survey results, there seems to be a dominant acceptance of the SET feedback as useful information for updating course materials among the participants. It implies that SET surveys could still benefit the teachers, especially when they are well designed. Previous researchers, after an extensive review of SET surveys, have suggested that reliable and valid student evaluation of teaching can be achieved if measurement and theoretical issues are carefully handled (Ginns, Marsh, Behnia, Cheng, & Scalas, 2009). First, a solid theoretical basis is required to start with. Second, as effective teaching is usually a hypothetical construct with a multidimensional structure, its measurement needs to be clearly defined, and it requires a robust factor structure. By attending to both of these issues, the survey can be reliable, valid, and less affected by variables such as expected student course grades, student interest in subjects, class size, and workload. It would then be able to provide university teachers with accurate data that help enhance teaching and learning. To further strengthen SET surveys as a tool to facilitate reflective practices (Winchester & Winchester, 2014), training can also be provided to faculty members to help ensure proper interpretation of SET data (Boysen, Kelly, Raesly, & Casner, 2013). Conclusions On the whole, the findings of this study remained consistent with that of previous research. While we have observed varied attitudes towards the effectiveness of SET surveys as a measurement of teaching among the participants, there was much concern about the aim and use of SET surveys, particularly for promotion and tenure decision-making. The findings suggest that it is important to ensure that students, teachers, and other stakeholders fully understand the aims and purposes of SET surveys and are aware that the survey is not simply a mechanism for benchmarking teaching, but that it serves and focuses on improving teaching and learning outcomes (as opposed to improving only the students’ perceptions of the teacher). SET surveys should not be the only data source for evaluating teaching and learning; rather, it should be part of an overall strategic plan that provides reliable triangulated evidence from different perspectives for the improvement of teaching and learning. The opinions on possible reasons for low SET scores among the participants are also similar to what has been documented in the previous literature. This study has also identified several correlation patterns among the variables. However, only limited numbers of variables have been included in the discussion. Awareness of the factors affecting the SET scores as well as the relationship among them can help inform judgement on the extent to which the scores could be used for any strategic decision-making. Thus, future studies may include more factors that will provide information required to facilitate the understanding of our observed correlation patterns between participants’ opinions on the Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 Educational Research and Evaluation 13 SET scores and their views on the effect of course, student, and teaching characteristics on student rating. In addition, it may also be worth surveying teachers from different disciplines for better controlled comparisons. Hence, it would be worth conducting a followup study by administering a modified questionnaire that is designed to examine academic staff members’ perceptions of the SET and to observe whether or not the current findings can be replicated. In order to measure and benchmark quality teaching, university management, university quality agencies, and academic teachers need to understand the definition of quality, and a structured, clear guideline needs to be discussed and developed in order to enhance student learning and not just to satisfy bureaucracy. Despite the limitations of our study, it has enriched the existing literature by providing insights into how Hong Kong science teachers – who work in a research-intensive context – perceive SET. This provides us with evidence to report alternative mechanisms, and for introducing ideas of how the process of SET can be improved in order to advance our understanding of SET in the context of Hong Kong. Notes on contributors Cecilia Ka Yuk Chan is an Assistant Professor at the Centre for the Enhancement of Teaching and Learning (CETL) at the University of Hong Kong (HKU). Prior to her appointment in HKU, she was an Assistant Professor and Programme Coordinator in the Faculty of Engineering at the Dublin Institute of Technology, Ireland. Cecilia has many practical experiences in relation to learning and teaching particularly in improving student retention rates. She has been involved in numerous researches and the development of engineering education. Her research fields are transferable skills in higher education, assessment and active learning, women in engineering, student retention, recruitment, and motivation. Cecilia holds a PhD in the field of Digital Signal Processing and Machine Vision in Electrical Engineering and also a Postgraduate Diploma and an MA in Higher Education. Lillian Yun Yung Luk is a senior research assistant at the Centre for Enhancement in Teaching and Learning, and she is also pursuing a PhD with the Faculty of Education at the University of Hong Kong. Her PhD research project is on students’ learning through internship experience. Her research interests include student motivation, experiential learning, assessment, and the development of transferable skills. Dr. Lily Min Zeng is a lecturer at the Centre for Academic Development at Victoria University of Wellington (VUW). Prior to her appointment as lecturer at VUW, she served as a lecturer at the Centre for the Enhancement of Teaching and Learning at the University of Hong Kong, where she oversaw the collection and analysis of research postgraduates’ research experience data at the university level, coordinated postgraduate supervision workshops for new supervisors of the research students, and coordinated certificate courses for tutors and lab demonstrators. She has over 13 years of working experience at tertiary level in Mainland China and Hong Kong. Her main research interests are conceptual change and teaching and learning, with a particular emphasis on these issues in a cross-cultural context. References Adamson, G., O’Kane, D., & Shevlin, M. (2005). Students’ ratings of teaching effectiveness: A laughing matter? Psychological Reports, 96, 225–226. Aleamoni, L. M. (1999). Student rating myths versus research facts from 1924 to 1998. Journal of Personnel Evaluation in Education, 13, 153–166. Anderson, G. (2006). Assuring quality/resisting quality assurance: Academics’ responses to “quality” in some Australian universities. Quality in Higher Education, 12, 161–173. Arthur, L. (2009). From performativity to professionalism: Lecturers’ responses to student feedback. Teaching in Higher Education, 14, 441–454. Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 14 C.K.Y. Chan et al. Australian University Teaching Criteria and Standards Framework. (n.d.). Overview of the Australian university teaching criteria and standards framework. Retrieved from http://uniteachingcriteria. edu.au/wp-content/uploads/2013/12/Framework_indicative-standards-arrange-by-criteria.pdf Balam, E. M., & Shannon, D. M. (2010). Student ratings of college teaching: A comparison of faculty and their students. Assessment & Evaluation in Higher Education, 35, 209–221. Ballantyne, R., Borthwick, J., & Packer, J. (2000). Beyond student evaluation of teaching: Identifying and addressing academic staff development needs. Assessment & Evaluation in Higher Education, 25, 221–236. Barrow, M. (1999). Quality-management systems and dramaturgical compliance. Quality in Higher Education, 5, 27–36. Beran, T., & Rokosh, J. L. (2009). Instructors’ perspectives on the utility of student ratings of instruction. Instructional Science, 37, 171–184. Beran, T., & Violato, C. (2005). Ratings of university teacher instruction: How much do student and course characteristics really matter? Assessment & Evaluation in Higher Education, 30, 593–601. Boysen, G. A., Kelly, T. J., Raesly, H. N., & Casner, R. W. (2013). The (mis)interpretation of teaching evaluations by college faculty and administrators. Assessment & Evaluation in Higher Education. Advance online publication. doi:10.1080/02602938.2013.860950 Braskamp, L. A., & Ory, J. C. (1994). Assessing faculty work: Enhancing individual and institutional performance. San Francisco, CA: Jossey-Bass. Casey, R. J., Gentile, P., & Bigger, S. W. (1997). Teaching appraisal in higher education: An Australian perspective. Higher Education, 34, 459–482. Centra, J. A. (1993). Reflective faculty evaluation: Enhancing teaching and determining faculty effectiveness. San Francisco, CA: Jossey-Bass. Clayson, D. E. (2013). Initial impressions and the student evaluation of teaching. Journal of Education for Business, 88, 26–35. Ginns, P., Marsh, H. W., Behnia, M., Cheng, J. H. S., & Scalas, L. F. (2009). Using postgraduate students’ evaluations of research experience to benchmark departments and faculties: Issues and challenges. British Journal of Educational Psychology, 79, 577–598. Hendry, G. D., Lyon, P. M., & Henderson-Smart, C. (2007). Teachers’ approaches to teaching and responses to student evaluation in a problem-based medical program. Assessment & Evaluation in Higher Education, 32, 143–157. Kember, D., Leung, D. Y. P., & Kwan, K. P. (2002). Does the use of student feedback questionnaires improve the overall quality of teaching? Assessment & Evaluation in Higher Education, 27, 411– 425. Kwan, K.-P. (1999). How fair are student ratings in assessing the teaching performance of university teachers?. Assessment & Evaluation in Higher Education, 24, 181–195. Leckey, J., & Neill, N. (2001). Quantifying quality: The importance of student feedback. Quality in Higher Education, 7, 19–32. Lemos, M. S., Queiros, C., Teixeira, P. M., & Menezes, I. (2011). Development and validation of a theoretically based, multidimensional questionnaire of students evaluation of university teaching. Assessment & Evaluation in Higher Education, 36, 843–864. Lomas, L., & Nicholls, G. (2005). Enhancing teaching quality through peer review of teaching. Quality in Higher Education, 11, 137–149. Marsh, H. W. (1980). The influence of student, course, and instructor characteristics in evaluations of university teaching. American Educational Research Journal, 17, 219–237. Marsh, H. W. (1984). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology, 76, 707–754. Marsh, H. W. (2007). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. In R. Perry & J. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319–383). Dordrecht, The Netherlands: Springer. Moore, S., & Kuol, N. (2005). Students evaluating teachers: Exploring the importance of faculty reaction to feedback on teaching. Teaching in Higher Education, 10, 57–73. Murphy, T., MacLaren, I., & Flynn, S. (2009). Toward a summative system for the assessment of teaching quality in higher education. International Journal of Teaching and Learning in Higher Education, 20, 226–236. Nasser, F., & Fresko, B. (2002). Faculty views of student evaluation of college teaching. Assessment & Evaluation in Higher Education, 27, 187–198. Downloaded by [University of Hong Kong Libraries] at 18:12 25 June 2014 Educational Research and Evaluation 15 Newton, J. (2000). Feeding the beast or improving quality? Academics’ perception of quality assurance and quality monitoring. Quality in Higher Education, 6, 153–162. Ryan, G. W., & Bernard, H. R. (2003). Techniques to identify themes. Field Methods, 15, 85–109. Scott, C., Stone, B., & Dinham, S. (2000, April). International patterns of teacher discontent. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA. Shevlin, M., Banyard, P., Davies, M., & Griffiths, M. (2000). The validity of student evaluation of teaching in higher education: Love me, love my lectures? Assessment & Evaluation in Higher Education, 25, 397–405. Simpson, P. M., & Siguaw, J. A. (2000). Student evaluations of teaching: An exploratory study of the faculty response. Journal of Marketing Education, 22, 199–213. Slade, P., & McConville, C. (2006). The validity of student evaluations of teaching. International Journal for Educational Integrity, 2(2), 43–59. Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83, 598–642. Stark-Wroblewski, K., Ahlering, R. F., & Brill, F. M. (2007). Toward a more comprehensive approach to evaluating teaching effectiveness: Supplementing student evaluations of teaching with pre-post learning measures. Assessment & Evaluation in Higher Education, 32, 403–415. Stehle, S., Spinath, B., & Kadmon, M. (2012). Measuring teaching effectiveness: Correspondence between students’ evaluations of teaching and different measures of student learning. Research in Higher Education, 53, 888–904. Stein, S. J., Spiller, D., Terry, S., Harris, T., Deaker, L., & Kennedy, J. (2012). Unlocking the impact of tertiary teachers‘ perceptions of student evaluation of teaching. Wellington, New Zealand: Ako Aotearoa National Centre for Tertiary Teaching Excellence. Surgenor, P. W. G. (2013). Obstacles and opportunities: Addressing the growing pains of summative student evaluation of teaching. Assessment & Evaluation in Higher Education, 38, 363–376. Tomasco, A. T. (1980). Student perceptions of instructional and personality characteristics of faculty: A canonical analysis. Teaching of Psychology, 7, 79–82. Tucker, B. (2014). Student evaluation surveys: Anonymous comments that offend or are unprofessional. Higher Education. Advance online publication. doi:10.1007/s10734-014-9716-2 Wachtel, H. K. (1998). Student evaluation of college teaching effectiveness: A brief review. Assessment & Evaluation in Higher Education, 23, 191–212. Winchester, T. M., & Winchester, M. K. (2014). A longitudinal investigation of the impact of faculty reflective practices on students’ evaluations of teaching. British Journal of Educational Technology, 45, 112–124. Wong, W. Y., & Moni, K. (2014). Teachers’ perceptions of and responses to student evaluation of teaching: Purposes and uses in clinical education. Assessment & Evaluation in Higher Education, 39, 397–411. View publication stats