Heuristic Evaluation

The role of academics has changed over the years from face-to-face teaching towards Open Distance Learning (ODL).  Teaching, assessment and moderation platforms have evolved to support virtual learning environments (VLE) platforms and moderating frameworks that support teaching and learning outcomes (Salmon 2003).  The manual moderation process is still widely used despite being tedious, time-consuming and cost-ineffective; it relies on paperwork and presents problems regarding feedback on the assessment of answer script of students (Van Staden, 2010). Online-moderation is an essential emerging technology in the era of online teaching but the application is still novel (Morgan, 2008), and the factors that determine the user experience in using eModeration have not been theorized in any depth.
The purpose of this paper is to investigate the user-experience aspect of eModeration by considering the fundamental realities and challenges faced by users with regard to eModeration information technologies.  The research is guided by the following question: How can the electronic moderation of examination scripts provide a better user experience in terms of functionality, usability, flow, interactivity, accessibility, effectiveness, efficiency, safety, learnability, memorability and user satisfaction for moderators and deans when this is compared to the paper-based moderation process?  In presenting this argument, different perspectives will be explored with a specific focus on the implementation of an eModeration system that is usable and ensures user experience.  Finally, the paper addresses the challenges faced by a Private Higher Education Institution (PHEI) named Midrand Graduate Institute (MGI), in successfully implementing online moderation and how this research can contribute to ensuring satisfactory user experiences from a managerial perspective.
Case studies on eAssessment and how Information Communication Technology (ICT) can support the formative assessment processes have been carried out, e.g. with reference to the submission of assignments online and regarding feedback between the lecturer as the eModerator and students (Bridge and Appleyard, 2008; Morgan, 2008; Vlachopoulos, 2008; Nicol, 2007; Salmon, 2003).  According to Bridge and Appleyard (2008) the online submission of assignments and the ability to provide feedback to the student enhanced the learning experience and assisted the lecturer with record keeping of assignments.  The focus of this particular study, however, is not automated marking but rather eModeration.  Furthermore the relationship between the eModerator and the dean of the faculty and not between the student and lecturer are investigated. 
The context is a Private Higher Education Institution in South Africa, named Midrand Graduate Institute (MGI).  MGI decided to investigate the possibility of moving towards an online-moderation system to replace the manual paper-based moderation system.  Initially the online moderation system was rolled out only in the Information Technology faculty.  MGI used its eLearn virtual learning environments (VLE) platform as a basis for eModeration and the principals of an eModerating framework as suggested by Salmon (2003).  The eModerate system is supposed to provide a user interface through which examination answer books can be submitted and graded electronically.  For the purpose of the researcher’s study the online moderation system was rolled out across five faculties at the institute to all moderators over a period of two examination sessions.
Creswell (2009) identified four different worldviews: post positivism, constructivism, advocacy participatory and pragmatism.  Social constructivism is often combined with interpretivism (Myers, 2009; Creswell, 2009) and is seen as an approach to qualitative research.  Social constructivists assume that individuals seek to understand the world in which they live and work (Creswell, 2009).  The philosophical worldview proposed in this study will be interpretivism.
The research strategy according to Oates (2006) is the overall approach to answering the research question.  Case study is one of the strategies that can be used (Myers, 2009; Olivier, 2009; Creswell, 2008;Oates, 2006). Interpretive case studies generally attempt to understand phenomena through the meanings that people assign to them (Myers, 2009).  Therefore the research strategy will involve a case study.
The data generation method is the means by which empirical (field) data or evidence are produced (Oates, 2006).  Data can be qualitative or quantitative (Myers, 2009; Olivier, 2009; Creswell, 2008 Oates, 2006).  By using more than one data generation method, the researcher will be able to look at the phenomenon of interest in different ways (Oates, 2009).  The reason for combining both quantitative and qualitative data is to bring about a better understanding of a research problem (Lazar et al., 2010)  by converging both quantitative numeric trends and qualitative detailed views of data and to advocate change (Creswell, 2008; Oates, 2006).  The user experience of the electronic moderation process is explored using interviews (qualitative instrument) with three deans from the three faculties at MGI.  For the purpose of this paper, however, only the interviews with deans are used as data sources.  In these situations, closed-ended quantitative data and open-ended qualitative data have been collected.  The results indicated that the proposed change in moderation has a positive impact on the turnaround time of moderators’ feedback and afforded moderators more flexibility according to the “I can moderate anywhere and at any time” principle.  The paper provides some guidelines towards the development of best practices to ensure a positive user experience of eModeration at an organizational level in an ODL environment.  The paper further contributes to the first quadrant of ODL research at UNISA called: research into ODL praxis with a focus on VLE, assessment and changing roles of academics and students.

The aim of this study was to identify and compare user experience issues and provide recommendations for the three airline websites. The websites were evaluated through heuristic evaluation and usability testing methods. Based on the heuristic evaluation findings, two relatively difficult task scenarios were generated for usability testing: A total of 168 student users participated in the study. The participants' performances for user testing were evaluated in terms of task completion time, success rate, and number of page views by a remote usability testing tool, Loop11. Additionally, a post-test questionnaire was administered online. Statistical analysis results indicated a significant ´website-design´ effect on task completion times. Overall, the task completion time significantly varied among the three airlines. Notably, ´gender´, ´education level´ and ´internet use frequency´ had no significant effect on task completion times for both tasks. Re-design recommendations are provided for a sample of cases.

99 analysts worked in-groups to apply the Heuristic Evaluation method to an office application’s drawing editor. The use of structured problem report formats eased merging of analysts’ predictions and subsequent matching to actual problems revealed by user testing with 16 participants. Actual problems were discovered via user tests, which were based on tasks designed to ensure that all predicted problems would be thoroughly addressed. Analysis of accurate and inaccurate predictions has supported the derivation of the DR-AR model for usability inspection method effectiveness. The model distinguishes between the discovery of candidate (possible) problems and their subsequent confirmation (or elimination) as probable problems. We confirm previous findings that heuristics do not support the discovery of possible usability problems. Our results also show that heuristics were only used appropriately to confirm possible problems that turned out to have low impact or frequency. Otherwise, heuristics are used inappropriately and, if acted on, would result in poor problem remediation. Our results also show that heuristics are very poor at eliminating improbable problems, and thus mostly incorrectly confirm predictions that could compromise the efficiency and effectiveness of usability activities. Overall, heuristics provide a poor analyst resource for the successful elimination/confirmation of im/probable problem predictions. Analysis of false predictions reveals that more effective analyst resources are knowledge of users, interaction, application domains, the application itself and design knowledge from HCI. Using the DR-AR model, we derive a strategy for UIM improvement.

Despite the emergence of many gameful design methodologies in the literature, there is a lack of methods to evaluate the resulting designs. Gameful design techniques aim to increase the user's motivation to interact with a software , but there are presently no accepted guidelines on how to find out if this goal was achieved during the design phase of a project. This paper presents the Gameful Design Heuristics, a novel set of guidelines that facilitate a heuristic evaluation of gameful software, with a focus on the software's potential to afford intrinsic and extrinsic motivation for the user. First, we reviewed several gameful design methods to identify the most frequently employed dimensions of motivational affordances. Then, we devised a set of 28 gamification heuris-tics that can be used to rapidly evaluate a gameful system. Finally, we conducted a summative empirical evaluation study with five user experience professionals , which demonstrated that our heuristics can help the evaluators find more motivational issues in interactive systems than they would without the heuristics. The suggested method fulfills the need for evaluation tools specific to gameful design, which could help evaluators assess the potential user experience of a gameful application in the early phases of a project.

Software development organizations consist of marketing, project management, development, design, and quality assurance team. It is important for the various teams within the organization to understand the benefits and limitation of incorporating various usability testing methods within the software development life cycle. Some of the reasons for poor usability include effort prioritization conflicts from project management, development and design team. The role of the usability engineer is to get involved as the heuristic evaluator and facilitate the development and design efforts are based on usability principles and at the same time adhering to the project time line. Two of the common usability inspection methods consist of user experience testing and expert review or more commonly known as Heuristic Evaluation (HE). This paper focuses on understanding the effectiveness of HE as a methodology for defect detection. The results show the effectiveness of the HE as a usability testing methodology in capturing defects and prioritizing development and design efforts. The results also reinforce the need for integrating traditional heuristics with modified heuristics customized to the domain or field of the project being tested such as E-Government.

Heuristic evaluation promises to be a low-cost usability evaluation method, but is fraught with problems of subjective interpretation, and a proliferation of competing and contradictory heuristic lists. This is particularly true in the field of games research where no rigorous comparative validation has yet been published. In order to validate the available heuristics, a user test of a commercial game is conducted with 6 participants in which 88 issues are identified, against which 146 heuristics are rated for relevance by 3 evaluators. Weak inter-rater reliability is calculated with Krippendorff's Alpha of 0.343, refuting validation of any of the available heuristics. This weak reliability is due to the high complexity of video games, resulting in evaluators interpreting different reasonable causes and solutions for the issues, and hence the wide variance in their ratings of the heuristics.

We describe the impact on analyst performance of an extended problem report format. Previous studies have shown that Heuristic Evaluation can only find a high proportion of actual problems (thoroughness) if multiple analysts are used. However, adding analysts can result in a high proportion of false positives (low validity). We report surprising interim results from a large study that is exploring the DARe model for evaluation method effectiveness. The DARe model relates the effectiveness of an evaluation method to evaluators' command of discovery and analysis resources. Previous work has shown that Heuristic Evaluation poorly supports problem discovery and analysis: heuristics tend to be inappropriately applied to problem predictions. We developed an extended problem report format to let us study analyst decision making during usability inspection. Our focus was on the quality of insights into analyst behaviour delivered by this extended report format. However, our first use of this format revealed unexpected improvements in validity (false positive reduction) and appropriate heuristic application. We argue that the format has unexpectedly led to more care and caution in problem discovery and elimination, and in heuristic application. Evaluation performance can thus be improved by indirectly 'fixing the analyst' via generic fixes to inspection methods. In addition, we provide the first direct evidence of how evaluators use separate discovery and analysis resources during usability inspection.