Effects On Out-Of-Class Activity

Computers & Education 125 (2018) 254–272
Contents lists available at ScienceDirect
Computers & Education

journal homepage: www.elsevier.com/locate/compedu
Implementing a theory-driven gamification model in higher

T
education flipped courses: Effects on out-of-class activity
completion and quality of artifacts
Biyun Huang∗, Khe Foon Hew
Division of Information and Technology Studies, Faculty of Education, The University of Hong Kong, Hong Kong
A R T IC LE I N F O ABS TRA CT:
Keywords: Flipped learning can provide more in-class time for students to practice and apply knowledge and
Gamification to receive feedback from peers and teachers. However, empirical studies have reported several
Flipped class problems that may occur with flipped classroom activities, including the failure of students to
Effects access out-of-class learning materials. Students who do not complete out-of-class work benefit
Engagement
little from the subsequent in-class discussion and problem-solving activities. This study offers a
Game elements
new contribution by exploring whether gamification could be a strategy to motivate students to
participate in more out-of-class activities without forfeiting quality of work. We applied crucial
aspects of five motivation theories to propose a goal-access-feedback-challenge-collaboration
(GAFCC) gamification design model. We then implemented and tested this theory-driven model
in two quasi-experimental studies involving postgraduate students. Collective results from the
two experiments revealed that (a) the GAFCC class completed significantly more pre- and post-
class activities than the control class and (b) the GAFCC class produced higher quality work than
the control class. Participants’ perceptions of gamification were also collected through inter-
views, and reported in this study. This evidence supports a call for further research into the use of
the GAFCC model in flipped classroom implementation.
1. Introduction
Flipped learning is a pedagogical approach in which students watch video lectures and complete pre-class activities (e.g., quizzes)
before class; during class, they engage in individual or class activities, and after class they finish additional activities (if any) as
homework (Nederveld & Berge, 2015). Watching video lectures before class allows students to learn content on their own time and at
their own pace. Students can repeatedly replay videos if they have difficulty understanding the content. Since in-class time is no
longer occupied by lectures, more time can be spent on problem-based learning and small-class learning activities with a teacher's
guidance.
However, the positive effects of flipped learning can only be realized if students are motivated to complete the out-of-class
activities. Although instructors may provide materials such as reading material or videos to students before or after class, not all
students will access the pre- or post-class resources (Gaughan, 2014; Hao, 2016a; Kim, Kim, Khera, & Getman, 2014). If students do
not access the flipped learning material provided by educators, they are unlikely to achieve better learning outcomes than in con-
ventional courses (Hao, 2016b; Lai & Hwang, 2016).
So how can we motivate learners to access and complete out-of-class activities? Some researchers have suggested providing
∗
Corresponding author.
E-mail addresses: [email protected] (B. Huang), [email protected] (K.F. Hew).
https://doi.org/10.1016/j.compedu.2018.06.018
Received 9 August 2017; Received in revised form 9 May 2018; Accepted 19 June 2018
Available online 20 June 2018
0360-1315/ © 2018 Elsevier Ltd. All rights reserved.
B. Huang, K.F. Hew Computers & Education 125 (2018) 254–272
incentives for students to prepare for class, such as giving marks for quizzes (e.g., Enfield, 2013; Tune, Sturek, & Basile, 2013) or
allocating low-stakes marks for submitting questions (e.g., Albert & Beatty, 2014; Gilboy, Heinerichs, & Pazzaglia, 2015; Kim et al.,
2014). Although grades may reinforce participation, students may feel pressured or anxious about completing flipped learning
activities outside of class (Marcum & Perry, 2015). Very few studies have specifically examined whether gamification can be in-
tegrated in a flipped learning course to motivate students to complete out-of-class activities.
In this study, we explore whether gamification can encourage participation in flipped activities in the higher education sector. We
applied crucial aspects of five motivation theories to propose a goal-access-feedback-challenge-collaboration (GAFCC) gamification
design model. This theory-driven model was then tested in two quasi-experimental studies involving postgraduate students.
Collective results from the two experiments revealed that (a) students in the GAFCC class completed significantly more pre- and post-
class activities than the control class and (b) students in the GAFCC class produced higher quality work than the control class.
The rest of this paper is organized as follows. First, we briefly introduce the concept of gamification. Second, we discuss the gaps
in previous gamification research, and propose a gamification design model based on the five motivation theories. We then report the
results of the two quasi-experimental studies. Finally, we discuss how our empirical findings contribute to theoretical development
and improve our understanding of how gamification can be used in flipped classroom contexts.
2. Literature review
Gamification is the utilization of game-like design elements in a non-game context (Deterding, Dixon, Khaled, & Nacke, 2011) to
motivate people and solve problems (Zichermann & Cunningham, 2011). This definition helps differentiate “gamification” from
“serious games” and “full-fledged games” by emphasizing “elements” and “non-game contexts.” Game design elements are the
building blocks of games and characteristics that play a crucial role in gameplay (Deterding et al., 2011). Although classifications of
game design elements differ, there is general agreement among scholars that the most basic and concrete elements include points,
badges, leaderboards, etc., which can be used to trigger particular behaviors among users and respond to their psychological needs
(e.g., Werbach & Hunter, 2012).
2.1. Gaps in previous gamification studies
Several gaps were identified from our review of previous empirical gamification studies. These include the following:
(a) Insufficient descriptions of the context and process. Researchers have criticized many empirical gamification studies for providing
insufficient detail about the process (Hamari, Koivistor, & Sarsa, 2014) and context (Falkner & Falkner, 2014) of how gamifi-
cation was actually deployed in education settings.
(b) Inadequate exploration of theoretical foundations. Many previous studies lack a theoretical explanation to describe the con-
nection between gamification and motivational effects (Sailer, Hense, Mayr, & Mandl, 2017; Seaborn & Fels, 2015). Seaborn and
Fels (2015) reviewed gamification research in different disciplines, such as education, health and wellness, and crowdsourcing,
and reported that a majority of studies were not grounded in theory. Nacke and Deterding (2017) found that recent studies had
mainly focused on self-determination theory and goal-setting theory, and argued that it is necessary to explore other theories.
(c) Many previous studies either blend popularized versions of self-determination theory with other models into untested “home-
grown” motivation models (Deterding, 2015, p. 311), or merely propose some form of untested gamification design framework
without any theoretical foundation. For example, although Simões, Redondo, and Vilas (2013) developed a social gamification
framework for K6 students, no empirical data were reported in their study. Hence, it is not clear how effective this framework is
in guiding gamification design. Several researchers (e.g., Rodrigues, Costa, & Oliveira, 2016; Werbach & Hunter, 2012) have
proposed their own gamification design frameworks, but these focus more on business promotion or e-banking rather than
educational purposes. Other design models (e.g., Hunicke, LeBlanc, & Zubek, 2004; Rodrigues et al., 2016) appear more suitable
for IT technicians, rather than for scaffolding teachers' use of gamification strategies. Klevers, Sailer, and Günthner (2016)
developed a GameLog Model, which focuses on gamifying the crowdsourcing process (e.g., order picking) by defining superior-
and behavioral-level goals from the employer's and staff's perspectives. This model seems more suitable for business processes
than for helping instructors to gamify their teaching and learning practices.
(d) Insufficient evidence for the effectiveness of gamification due to the methodological limitations of study designs and analysis
strategies (Hamari, 2017; Sailer et al., 2017; Seaborn & Fels, 2015). Many previous studies merely used self-reported survey data,
a class intervention without a pre-test, or a two-class comparison study that did not compare students from the same course
(Hamari, Koivisto, & Sarsa, 2014; Çakıroğlu, Başıbüyük, Güler, Atabay, & Memiş, 2017). Other studies were correlational in
nature, providing indirect evidence that gamification may enhance “time-on-task” (e.g., Landers & Landers, 2014), and that
“time-on-task” is correlated with better academic performance. Correlational studies cannot, however, establish causal effects.
In response to these research gaps, this study further explores the theoretical foundation of gamification design. It proposes a
theory-driven gamification design model targeted at the flipped classroom higher education setting, and empirically examines ef-
fectiveness of this model in two quasi-experimental studies involving postgraduate students. The contexts and processes of the studies
are elaborated in detail. The following research questions guided the current study.
1) Is a gamification design based on the GAFCC design model effective in motivating learners to complete more out-of-class learning
255
activities?
2) Is a gamification design based on the GAFCC design model effective in motivating learners to produce higher quality artifacts in
out-of-class learning activities?
3) What are the students' perceptions of gamification?
The following hypotheses are proposed.

H1. Students in the gamified flipped learning condition complete more pre-class learning activities than in the control condition.
H2. Students in the gamified flipped learning condition complete more post-class learning activities than in the control condition.
H3. Students in the gamified flipped learning condition produce higher quality pre-class artifacts than in the control condition.
H4. Students in the gamified flipped learning condition produce higher quality post-class artifacts than in the control condition.
3. Towards a theory-driven gamification design model
The most commonly referenced theory in gamification studies is self-determination theory (Deci & Ryan, 2000). Based on this
theory, Nicholson (2012) recommended a user-centered meaningful gamification design framework focusing on cultivating learners’
intrinsic motivation. In recent years, goal-setting theory has also become a guiding theory (e.g., Landers, Bauer, & Callan, 2017).
Richter, Raban, and Rafaeli (2015) reviewed nine intrinsic and extrinsic motivation theories, and suggested an intrinsic-social-
extrinsic model of motivation in games, classifying it into three categories: needs-, social-, and rewards-based. There seem to be many
overlapping elements across different motivation theories, and it may be confusing for researchers and educators to refer to so many
theories before designing a gamification implementation. Their work thus inspired us to reflect on the most relevant theories un-
derlying gamification, and extract the essential elements to construct our gamification design model.
To this end, we first review five key theories that were most commonly used to explain people's motivation needs: flow theory,
goal-setting theory, social comparison theory, self-determination theory, and behavior reinforcement theory. In what follows, we
introduce the main motivation needs emphasized by these theories, and how each theory can guide the design of a motivating
experience.
3.1. Flow theory
Individuals enjoy playing games due to the resulting state of “flow” (Csikszentmihalyi, 1978; Malone, 1981). Flow is a state of full
involvement, complete absorption, and intrinsic enjoyment when performing an activity (Csikszentmihalyi, 1978; Nakamura &
Csikszentmihalyi, 2009, pp. 195–206). The flow state requires the following conditions: 1) clear and proximal goals; 2) immediate
feedback on performance and progress; 3) suitable level of challenges; and 4) perceived usefulness of the challenges in building up
existing skills (Nakamura & Csikszentmihalyi, 2009, pp. 195–206; Shernoff, Csikszentmihalyi, Shneider, & Shernoff, 2003).
3.2. Goal-setting theory
A goal is an objective or purpose that one consciously strives to attain (Locke & Latham, 2002). It influences students' motivation
and academic achievement (Locke & Latham, 2002; Schunk & Swartz, 1993). Goals affect task performance through four major
mechanisms: directing attention to goal-relevant activities, mobilizing the degree of effort, increasing persistence in pursuing the
goal, and promoting the development of goal-relevant plans or strategies (Locke & Latham, 1984, 2002; Woolfolk, 1998). Schunk
(1991) suggested that to build a motivating environment, a teacher should 1) set up achievable long- and short-term goals for
students, 2) provide feedback on their performance, and 3) assist them in evaluating their own progress. Empirical studies have
shown that game design elements such as badges and leaderboards can direct learners’ attention to targeted learning activities. For
example, Anderson, Huttenlocher, Kleinberg, and Leskovec (2014) reported that badges motivated students in a MOOC course to do
more voting and reading than the control class, but students did not complete more tasks when no badges were assigned. Landers
et al. (2017) reported that when leaderboards were used to set up goals and provide feedback, learners were motivated to produce
more opinions than the do-your-best and easy goal conditions.
3.3. Social comparison theory
Festinger (1954) states that people have an innate drive to evaluate their own opinions and abilities. For purposes of self-
evaluation, people compare their opinions and abilities with those of others if objective and non-social means are not available
(Festinger, 1954). As individuals have an innate drive for self-evaluation, providing learners with a means to compare themselves
with peers and evaluate their own performance provides a chance for them to improve. The publicly visible badges or ranks in a
leaderboard earned by different users can function as social markers (Hamari, 2017), thus enabling comparisons between people.
256
3.4. Self-determination theory
In self-determination theory, according to Deci and Ryan (1985), intrinsic motivation involves doing something out of inherent
interest or joy. When people are intrinsically motivated, they are more likely to exhibit a high level of engagement and persistence
and actualize high-quality learning and creativity (Deci & Ryan, 2000). Autonomy, competence, and relatedness are the three essential
elements that can facilitate intrinsic motivation (Deci & Ryan, 1985, 2000). Autonomy is the need for a sense of free will, choice, and
psychological freedom when participating in an activity (Deci & Ryan, 2000; de Charms, 2013). Competence is the need to feel
effective when interacting with the environment (Deci & Ryan, 2000; Nicholson, 2012). Relatedness refers to people's need for
feelings of security, attachment, and belongingness (Deci & Ryan, 2000). In a social context, people may be extrinsically motivated to
do things that do not initially interest them, when such actions are prompted or modeled by people to whom they feel strongly
connected (Deci & Ryan, 2000). Sailer et al. (2017) found that game elements such as badges, leaderboards, performance graphs were
associated with higher levels of psychological satisfaction of competence than in the control condition, and game elements like
avatars, storylines, and guilds were associated with higher levels of social relatedness than in the control condition.
3.5. Behavior reinforcement theory
In the 1950s, the behaviorist B.F. Skinner (1953) proposed that rewarding people's new behavior positively reinforced it and thus
helped people to develop the corresponding habit. Skinner recommended that, at the beginning stage, a continuous reinforcement
schedule be adopted to reward an individual's new behavior and thus reinforce every correct response (Skinner, 1953, 1989;
Woolfolk, 1998). In the subsequent stage, when a subject has mastered a new behavior, an intermittent reinforcement schedule can
be used (Skinner, 1953, 1989; Woolfolk, 1998). This approach should help sustain learners' behavior and increase learners' curiosity.
In a gamified setting, badges can serve as a form of virtual achievement by a user – they provide positive reinforcement for the
desired targeted behavior (Kumar & Herger, 2013).
Table 1 provides an overview of motivation theories and the core elements that contribute to a motivating experience. From
Table 1, we may see that the motivating elements, while overlapping, can be summarized as 1) goal, 2) access, 3) feedback, 4)
challenge, and 5) collaboration.
3.6. A theory-driven gamification design model
Based on our synthesis of the five theories of motivation, we developed a GAFCC gamification design model (Fig. 1). This model
posits five crucial motivating elements in gamification design: goal, access, feedback, challenge, and collaboration. These five mo-
tivating elements can be reified in items such as badges, and leaderboards.
3.6.1. Reifying goal

A goal can be either long or short term. In a gamified setting, game design elements can be used to set up proximal and reasonable
goals. When learners are clear about long- and short-term goals, their motivation may be enhanced. To enable goal-setting, it is
helpful to reward students (e.g., with badges or points), direct their attention to goal-relevant activities (Anderson et al., 2014),
promote their goal-related plans, and encourage them to persist (Mekler, Brühlmann, Tuch, & Opwis, 2017). For example, an in-
structor may use participation-based badges such as early bird to motivate learners to complete the pre-class activities before a
specified deadline.
3.6.2. Reifying access

Access means offering learners the autonomy to choose suitable challenges for themselves. Game design elements such as leveling
up guide learners from easier tasks to more difficult ones and help to build competency (e.g., Iosup & Epema, 2014; Li, Grossman, &
Fitzmaurice, 2012, October). To enable access, it is helpful to provide a variety of optional challenges or tasks so that learners can
Table 1
Motivation theories and motivating elements.
Motivation theories & motivating elements
Flow Theory Goal Setting Self-Determination Social Comparison Behavior

(Csikszentmihalyi, 1978) (Locke & Latham, 2002; (Deci & Ryan, 2000) (Festinger, 1954) Reinforcement
Schunk & Swartz, 1993) (Skinner, 1953, 1989)
1) clear goals (goal) 1) set up long-term and 1) allow learners to choose between 1) self-evaluation 1) reinforcement
short-term goals (goal) several courses of action (access) (feedback) (feedback, goal)
2) immediate feedback on 2) provide feedback on 2) offer opportunities for learners to 2) offer opportunities for
performance and progress their performance compete with their own selves or with learners to compete with
(feedback) (feedback, competence) peers (access, competence, feedback) peers (access, challenge)
3) suitable level of challenges 3) offer opportunities for learners to
(challenge, competence) work together to achieve a shared
goal, or to interact with each other
(collaboration)
257
Fig. 1. Motivation needs, motivating elements, and enablers (game design elements).
choose a difficulty level that matches their own perceived skills (Csikszentmihalyi, 1978).
3.6.3. Reifying feedback

Feedback means providing instant or summative feedback to learners so that they know their own progress and achievements and
the progress and achievements of their peers. Feedback enables learners to engage in self-evaluation and self-correction. A leader-
board could for instance be used to provide feedback on their achievements and their status relative to the whole class (e.g., Hew,
Huang, Chu, & Chiu, 2016). Participation-based badges can also be used to provide feedback on their efforts (Hew et al., 2016), and
skill-based badges could provide feedback on the quality of their artifacts or contributions (e.g., Hew et al., 2016).
3.6.4. Reifying challenge

Challenge means providing opportunities for learners to compete with themselves or their peers. This would meet individuals'
need to excel themselves or surpass others, and raise participants’ curiosity. For example, Barata, Gama, Jorge, and Gonçalves (2013)
integrated challenges in a multimedia content production course, and found that students were motivated to produce more postings;
consequently, those students who took more challenges scored higher final grades.
3.6.5. Reifying collaboration

Collaboration means providing chances for learners to work together to achieve a shared goal or interact with each other (e.g.,
commenting and replying). Creating chances for learners to interact with each other could make them feel related and connected and
help them to learn more from peers (e.g., Hew et al., 2016; Sailer et al., 2017). Positive reinforcements can be used to encourage
individuals to participate in collaborative activities.
3.7. Flowchart for designing a gamified course
In the preceding part, we introduce the theoretical components of the GAFCC model. The following introduces the procedure for
incorporating this theory into practice. To ensure the alignment of motivation theories, gamification strategies, and instructional
objectives, we recommend following a five-stage gamification design procedure. The five design stages are examine, decide, match,
launch, and evaluate (see Fig. 2 for a flowchart of the design procedure).
• First, examine the specific instructional objectives, learner context, and technology affordances of a particular online platform
such as learning management system.
• Second, decide what motivating elements (i.e., goal, access, feedback, challenges, collaboration) to strengthen or introduce.
258
Fig. 2. Flowchart of five-stage gamification design procedure.
• Third, match motivating elements with game design elements and learning activities, and decide which gamification strategies to
adopt.
• Fourth, implement the design in actual classes.
• Fifth, evaluate the design. After launching the design, reflect on the implementation result and investigate whether this design
needs improvement.
In the following sections, we describe two quasi-experimental studies that utilized the GAFCC model and flowchart.
4. Research design and results
The gamification design model was implemented in two tertiary-level postgraduate flipped courses: a three-week basic statistics
course and an eight-week introductory library sciences course. The implementation contexts and results are discussed in what fol-
lows.
4.1. Study 1
4.1.1. Context
Study 1 was a quasi-experiment that involved 21 participants in the treatment class (Year 1, Master of Information Technology in
Education students), and 19 participants in the control class (Year 1, Master of Library and Information Management students). The
participants self-enrolled in a Basic Statistics and SPSS course based on their own timetabling schedule. This course is one of the
259
mandatory courses in both the Master of Information Technology in Education (MITE), and the Master of Library and Information
Management (MLIM) program. Participants in the MITE program generally have some background knowledge in education-related
subjects (e.g., primary school education), while those in the MLIM program generally have some background knowledge in in-
formation management subjects.
Both classes of the Basic Statistics and SPSS course were taught by the same instructor but in two different days. The treatment
class used gamified course activities, while the control class used the same activities but without any gamification. According to the
course instructor, none of the participants had previous experience in a flipped or gamified course prior to the Basic Statistics and
SPSS course.
The Basic Statistics and SPSS course is a relatively challenging course, and as the in-class learning session only lasted for three
hours the instructor decided to set up flipped learning activities for students before and after the in-class session to extend their
capacity for learning. In the instructor's previous experience, only a small number of students completed the pre- and post-class
learning activities. Therefore, we conducted a comparison to examine whether gamification design following the GAFCC model could
be effective in stimulating students to participate in more flipped learning activities and produce higher quality artifacts.
To evaluate whether the participants in the control and treatment classes exhibited any difference in prior knowledge, a pre-test
on the statistical and SPSS content was administered since the participants were not randomly assigned to either classes. Altogether,
16 participants from the control class and 19 from the treatment class completed the pre-test. A normality test has been conducted to
examine the normality of data distribution, and the result showed that the data were normally distributed. Therefore, an independent
t-test was administrated to compare the prior knowledge of the two groups. The independent t-test result showed that there was no
significant difference in pre-test scores between the control class (M = 5, SD = 1.26) and treatment class (M = 4.69, SD = 1.86), t
(33) = 35, p = 0.57. This result indicated there was no significant difference in the students’ prior knowledge.
4.1.2. Designing the basic statistics and SPSS module

In designing this module, we followed the GAFCC design model and the five-stage gamification design procedure (see Fig. 2) of
examine, decide, match, launch, and evaluate.
4.1.2.1. Stage I. Examine course context. This stage involved examining the instructional objectives, learner context, and technology
affordances.
Instructional objectives. In this course, the instructional objectives were classified into basic level and higher level (i.e., extra-
curricular level). See Table 2 for a brief description of the basic- and higher-level objectives.
Learner context. The learners were postgraduate level students, and most had little prior knowledge of statistics. They might attend
in-class lectures, but lacked the motivation to complete the flipped learning activities.
Technology affordances. The course was launched on Moodle (a learning management system), which supported gamification
features such as badges, points, and leaderboards.
Table 2
Instructional objectives, learning activities, and gamification strategies.
Instructional objectives Learning activities Gamification strategies
Basic Level: Pre-class: Pre-class:

● Distinguish basic terms (e.g., mean, ● Video: SPSS 101. ● Reward students with Early Bird Badges if they
median, independent sample t-test, ● Pre-course task: Enter data in SPSS. complete the pre-course task (Enter data in SPSS)
paired sample t-test). ● Think Activity before class. A leaderboard was used to display the
● Enter data in SPSS. In-class: names of the students who obtained the badge
● Run data in given background. ●Learn terms related to paired sample t-test and Enabled motivating elements: Goal, feedback.
● Analyze and report t-test result. independent sample t-test. Post-class:
Higher Level: ● Learn how to operate paired sample t-test and ● Reward students with points if they do optional post-
● Distinguish parametric tests and non- independent sample t-test. class activities. The activities have five difficulty
parametric tests. ● Learn how to report paired sample t-test and levels. Students gain 1 point for Level 1, 2 points for
● Recognize Anova, Mann-Whitney U independent sample t-test result. Level 2, 3 points for Level 3, 4 points for Level 4, and
test and Wilcoxon signed rank test. ● Conduct paired sample t-test and independent t-test 5 points for Level 5.
● Collect own data, run data and report in classs, and report result. ● Reward students two points if they can find a mistake
data. Post-class: in their peer's SPSS data report and help correct it
● Analyze and evaluate peers' data ● Individual challenges (six optional levels) (i.e. level 6).
reports. L1: Enter data and analyze data; Enabled motivating elements: Goal, access, challenge,
L2: Report data using APA style; collaboration, feedback.
L3: Search for three other t-test studies, report the
findings;
L4: Collect data, analyze it, and report it in APA
style;
L5: Extracurricular learning video (on parametric/
non-parametric tests, including Anova, Mann-
Whitney U test and Wilcoxon signed rank test);
L6: Find mistakes in peer's report and correct them.
● Extra reading.
260
Fig. 3. Moodle page, badge rule, and point scheme.
4.1.2.2. Stage II. Decide elements. Based on the learner context, instructional objectives, and technology affordances, designers could
decide which motivating elements (e.g., goal, challenges) to strengthen or introduce. Given the course context, we decided to include
all five motivating elements in this module, i.e., goal, access, feedback, challenges, and collaboration.
4.1.2.3. Stage III. Match motivating elements with game mechanics. For the pre-class activities, to address goal and feedback, an early
bird badge was used to direct learners' attention to this desired goal (i.e., complete Enter data in SPSS activity) and reward their
participation if the learners completed the activity before the specified deadline. For the post-class activities, to address access and
challenges, challenges of various difficulty levels were provided in the individual challenges activity, so that learners could choose a
suitable level for themselves. The challenges were carefully aligned with the course instructional objectives. A point scheme for these
individual challenges guided learners in setting up the desired goal for themselves. To address feedback and collaboration, extra
points were used to encourage peers to participate in the level 6 challenge (i.e., find peers’ mistakes and correct them) so as to
promote peer feedback and collaboration.
4.1.2.4. Stage IV. Implement (launch) design. After matching the motivating elements, game design elements, and learning activities,
we launched this design in the class. See Fig. 3 for screenshots of the Moodle page, badge rule, and point scheme.
4.1.2.5. Stage V. Evaluate design. After launching this design, we reflected on the implementation results and investigated whether
this design needed improvement.
4.1.3. Data collection and analysis tools

The treatment class (N = 21) had access to the gamified flipped learning course, whereas the control class (N = 19) had access to
the non-gamified flipped learning course. The interfaces of both web pages were similar; there were only two differences. First, when
the treatment class participants completed the pre-task on time, they would see an “early bird badge” on the top right of the webpage.
Participants in the control class would not see any badges. Second, participants in the treatment class could see predefined points for
each challenge level, while participants in the control class could only see challenges of different levels without any predefined
points. Both the gamified and non-gamified classes were informed that they had autonomy to complete or not complete the post-class
individual challenges.
To examine whether more students would complete pre- and post-class tasks under the gamified condition than under the non-
gamified condition, the rates of activity completion were compared. To examine whether students under the gamified condition
produced higher quality pre- and post-class artifacts than those in the control class, students’ pre- and post-class task artifacts (e.g.,
261
Fig. 4. Pre- & post-class activity completion rate.

Note: “G” refers to “gamified activity,” “NG” refers to “non-gamified activity.” Pre-class activities were “Video: SPSS 101,” “Enter data in SPSS,” and
“Think activity.” Post-class activities were “Individual challenges: Level 1–6” and “Extra reading material.”
submissions of individual challenges) were analyzed and compared.
4.1.4. Results
4.1.4.1. Effects on pre-class activity completion. Students’ activity completion in the pre-class stage was compared between the
treatment and control class. According to the descriptive data, the activity completion of the treatment class (N = 21) was higher
than that of the control class (N = 19) in all activities (see Fig. 4). The video-watching activity was completed by 81% of the
participants from the gamified class and only 58% from the control class. The pre-class task (Enter data in SPSS) was completed by
67% of the treatment class participants, but only 26% of the control class (N = 19). Lastly, 14% of the gamified class posted answers
for the Think Activity, while in the non-gamified class no one (0%) posted answers.
Chi-square tests of independence were conducted to examine whether there were statistically significant differences in activity
completion. Statistically significant differences were observed in all the gamified activities. For example, in the gamified Enter data in
SPSS, a significant difference was found (x2 (1) = 6.51, p = 0.01) between the treatment and control class. That is, the treatment
class (67%) was more likely to complete the pre-class task (Enter data in SPSS) than the control class (26%). For the non-gamified
activities, the percentages of completion for the treatment class were higher but not statistically significant. For example, a chi-square
test of independence revealed that there was no statistically significant difference between the two classes (x2(1) = 2.53, p = 0.11) in
the non-gamified video-watching activity. Similarly, the chi-square test of independence for the non-gamified Think Activity in-
dicated that there was no statistically significant difference between the two classes (x2(1) = 2.93, p = 0.09). Overall, the treatment
class completed significantly more gamified pre-class activities than the control class. However, there was no significant difference
between the treatment and control classes in terms of non-gamified pre-class activity completion.
4.1.4.2. Effects on post-class activity completion. In the post-class stage, the activities consisted of gamified activities, individual
challenge task levels 1–6, and a non-gamified activity: extra reading material. The task completion rates for the individual challenge
task levels 1–5 were 95% for the gamified class and 0% for the control class. A chi-square test of independence was administered to
compare the completion rate for the individual challenge task levels 1–5 between the gamified and non-gamified classes, and a
significant difference was found (x2(1) = 36.19, p = 0.00). The task completion rate for the individual challenge task level 6 was
29% for the gamified class and 0% for the control class. A chi-square test of independence indicated a significant difference between
the two classes (x2(1) = 6.39, p = 0.01). The completion rate for the non-gamified extra reading material differed only slightly
between the treatment class (62%) and control class (58%). A chi-square test of independence revealed no statistically significant
difference between the two classes (x2(1) = 0.07, p = 0.80). The result for the post-class activity was similar to that for pre-class
activity in that statistically significant results could only be observed for gamified activities. The treatment class was thus more likely
to complete more gamified activities than the control class.
262
4.1.4.3. Effects on the quality of pre-class activity artifacts. To examine the quality of students' pre-class artifacts, students’ submissions
for the pre-class task (Enter data in SPSS) were collected and analyzed. In this task, students needed to enter data into SPSS and
correctly define the variables; the full score for this task was 10. In the data analysis stage, scores for students who did not submit
answers were recorded as zero. A normality test was conducted to examine the normality of data distribution, and it showed that the
data were not normally distributed. Therefore, non-parametric Mann-Whitney U test was conducted to compare the results of two
groups. The non-parametric Mann-Whitney U test showed a significant difference between the scores of the treatment class
(Mdn = 10) and control class (Mdn = 0); U = 120, p = 0.03. Therefore, in the gamified condition, students produced higher quality
pre-class artifacts than the control class on average.
4.1.4.4. Effects on the quality of post-class activity artifacts. To examine the quality of students' post-class artifacts, students'
submissions for the individual challenge tasks (level 1–6) were collected and analyzed. In the treatment group, 20 out of 21 students
(95%) attempted at least one challenge task. Among them, 15 students (71%) completed the tasks from level 1 to level 5. Three
students (14%) completed all the challenge tasks (Level 1–6). The full score for the individual challenge levels 1–5 was 15. The
treatment class's (N = 21) average score for the levels 1–5 challenge was 10.23. In the control class (N = 19), no student (0%) was
motivated to accomplish the individual challenge tasks. Therefore, no one in the control class received any score for this activity. This
result indicated that under the gamified condition, learners were motivated to make more efforts, and produced higher quality post-
class activity artifacts than the control class.
In general, the quantitative data revealed that the gamified course following the proposed theory-driven model motivated learners
in the treatment class to complete more pre- and post-course activities, take the more difficult challenges, and produce higher quality
post-class artifacts than the control class. However, Study 1 was limited by the short duration of the intervention, which lasted only
three weeks.
4.1.4.5. Students’ perception of gamification. Students in the gamified condition were invited to participate in an interview, among
them twelve students consented to be interviewed individually. The interviewees shared their views on the impact of gamification,
and provided suggestions on improving the gamification design.
Nearly all the interviewees (N = 11, 92%) expressed their preference to learn in a gamified environment. The participants ex-
plained that in a gamified environment they would receive more feedback, be clearer about their target, have more social interactions
with peers, and have fun. For example, interviewee (S1) said, “I think every course should at least have some form of participation-based
badges …, just give learners some hints that they have achieved these …” A majority of the interviewees reported that gamification
motivated them to set higher goals and complete more tasks. For example, interviewee (S2) stated, “it [gamification] helps increase
motivation. We were motivated to take initiatives to complete the learning tasks.” Another interviewee (S3) remarked, “The participation
badge (e.g., early-bird) signifies a task I have to complete or a target that I need to achieve. It really shows me what the important part is …”
Still another interviewee (S7) commented, “If that task did not have any badge attached to it, we might not work on it. But if it has badge,
we know that if we do it, we have the chance to win a recognition.” The use of a leaderboard for the pre-class activity provided a means for
the participants to track and compare their performances with peers’ performances. This mechanism stimulated social comparison
among learners, as highlighted by interviewee (S8), “Now that a leaderboard is implemented, we can know the progress and performance
of our classmates … when seeing the leaderboard, we will feel the peer pressure. When seeing others doing something, I myself will also do it.”
Participants also remarked that giving “access for optional difficulty levels”, gave them a sense of autonomy. For example, “it gives
me the freedom to choose, and this increases my motivation … It's encouraging us to do it, not forcing us to do it.”(S9) The use of gami-
fication also motivated participants take the more difficult challenges: “if there is no point system, I would not attempt the difficult ones.
But with points, I would choose the most challenging one.” (S6) Participants also mentioned that gamification brought them fun. For
example, “gamification is fun. You will not feel bored to complete those tasks, as its not the course instructor that assigns you tasks but you,
yourself, choose to do those tasks.” (S10).
Despite the overall positive comments about the impact of gamification, two suggestions were made by the interviewees to further
improve the use of gamification in future courses. First, one interviewee (S1) reported that he wanted to link gamification to as-
sessment scores (e.g., to link the number of badges a learner earned to a proportion of the learner's total course assessment scores).
Second, several interviewees suggested more game elements to be used in future gamified courses. One interviewee (S5) re-
commended the addition of a progress bar to the course page. This bar will show all the required tasks and the proportion of
completed tasks. Another interviewee (S11) suggested adding the up-vote and down-vote elements to the course forum, so that
students could show their appreciation or disagreement towards an answer or question posted in the forum.
4.2. Study 2
4.2.1. Context
Study 2 was a quasi-experiment carried out in two eight-week long master level course. This study involved 25 participants in the
treatment class (Year 1, Master of Library and Information Management students), and 15 participants in the control class (Year 1,
Master of Library and Information Management students). These participants self-enrolled in a Library and Information Science
Foundation course based on their own timetabling schedule. Both classes were also taught by the same instructor but in different
days. According to the course instructor, none of the participants had previous experience in a flipped or gamified course prior to this
course. The purpose of selecting a different course in Study 2 was to see whether the gamification design model applied to the
previous study (Study 1) would generate similar results in a different context. In this course, students were taught many theories and
263
Table 3
Instructional objectives, learning activities, and gamification strategies.
Instructional Objectives Learning Activities Gamification Strategies
Basic Level: Pre-class: Pre-class:

● Identify the properties of information, and ● Think activities. ● Reward students with early bird badges if they
distinguish information and knowledge. ● Pre-class reading materials or complete a pre-class Think Activity before a specified
● Compare trends and issues pertaining to the videos. deadline.
information society. In-class: Enabled motivating elements: Goal, feedback.
● Analyze and enumerate the characteristics, user ● Learning concepts and theories of Post-class:
needs, and differences of information organizations. information management. ● Reward students with super-efficient badges if they
Higher Level: ● Learning cases of information complete the quizzes within a specified deadline.
● Analyze and evaluate information management management. ● Reward students with communicator badges if they
practices in organizations and society ● Class discussion and hands-on initiate or reply 3 postings.
● Apply information models and create information practices. ● Reward students with truth-seeker badges if they raise
organization evaluation reports. Post-class: or reply 2 questions on the discussion forum.
● Quizzes. Enabled motivating elements: Goal, access, feedback,
● Forum discussions. challenge, and collaboration.
concepts in the information management discipline.

The treatment class took the gamified course, while the control class took the same course without any gamification. Both the
treatment and control classes used similar learning materials (i.e., the learning content was the same). Course topics were introduced
in the same order and assignments were scheduled at almost the same time. The treatment class accessed the gamified Moodle
webpage, and the control class accessed the conventional Moodle webpage. On the gamified Moodle course main page, students
would see 1) the rules for getting rewards (e.g., different types of badges) and 2) the normal learning materials. On the student profile
page, each student would see the badges he or she had won. On the non-gamified Moodle course main page, students would see the
normal learning materials. A pre-test was conducted to see whether there were any differences in learners' prior knowledge. A
normality test indicated that the data were not normally distributed, so a non-parametric Mann-Whitney U test was administered to
compare the differences of the two groups. The Mann-Whitney U test results indicated no significant difference in students' pre-test
scores between the treatment class (Mdn = 3.0) and control class (Mdn = 3.5), U = 117, p = 0.50. This result showed no significant
difference in students’ prior knowledge.
4.2.2. Designing the information management course

We followed the GAFCC design model and a five-stage gamification design procedure. The five design stages were as follows.
4.2.2.1. Stage I. Examine course context. Instructional objectives. The instructional objectives for this course were: 1) identify the
properties of information and distinguish information and knowledge; 2) interpret and compare trends and issues pertaining to the
information society; 3) analyze and enumerate the characteristics, user needs, and differences of information organizations; 4)
analyze and evaluate information management practices in organizations and society; and 5) apply information models and create
information organization evaluation reports (see Table 3).
Learner context. The learners in this core course were master's students, who were required to complete this course in the first
semester. As mentioned earlier, students might have had motivation problems in completing the pre-class think activities and the
post-class quiz activities. Sometimes students rushed to complete all of the out-of-course activities in one or two days at the end of
term.
Technology affordances. The learning management platform for this course was Moodle, which supported gamification features
like badges, points, and leaderboards.
4.2.2.2. Stage II. Decide elements. Based on the previous information, we decided to gamify this course without increasing the
pressure on students, as master's students are typically adult learners and have busy schedules. The purpose of using gamification was
simply to make this course more interesting, remind them to participate in pre- and post-class activities, and provide feedback when
they engaged in positive behavior. Meanwhile, we gave the students autonomy in participating in activities, rather than forcing them
to complete all of them. Therefore, the motivating elements included in this module were goal, access, feedback, (mild) challenges,
and collaboration.
4.2.2.3. Stage III. Match motivating elements with game mechanics. For the pre-class Think Activities, to address goals and feedback,
early bird badges were assigned to signify the successful completion of the desired goals (i.e. to complete the weekly Think Activities)
before a specified deadline each week. The weekly pre-class Think activities required students to first watch a video or read materials
(e.g., instructor's PPT slides), and then post their responses to open-ended questions related to the video or readings. An example of a
pre-class Think Activity was: “In your opinion, what is the most important issue to consider before recording information?”.
Completion of the pre-class Think Activities would help students to be better prepared for the subsequent in-class discussion with the
instructors and peers.
For the post-class activities, super-efficient badges were assigned to learners if they completed the quizzes within a specified time.
264
Fig. 5. Screenshots of Moodle page, badges, and badge rules.
These quizzes were optional; learners had the autonomy to complete them or not depending on their own learning motivation. The
quizzes helped foster the experience of access to additional learning opportunities, but learners had the choice to decide whether or
not to complete them. To address challenges and collaboration, communicator badges were rewarded to students when they chose
to initiate or reply to three postings in each module, and truth-seeker badges were used to reward students who asked or replied to
two questions in the discussion forum.
4.2.2.4. Stage IV. Implement (launch) design. After matching the course context, motivating elements, and game design elements, we
launched this design in the actual classes. See Table 3 for the instructional objectives, learning activities, and gamification strategies.
See Fig. 5 for screenshots of the Moodle page, badges, and badge rules.
4.2.2.5. Stage V. Evaluate the design. After launching this design, we reflected on the implementation results and investigated whether
this design needed improvement.
4.2.3. Results
4.2.3.1. Effects on pre-class activity completion. In this course, there were seven pre-class Think Activities. One pre-class Think Activity
invited students to introduce their background and expectations for the course. As this was a general question rather than one seeking
to determine students' reasoning and thinking ability in the area of information management, students’ output in this activity was not
included in data analysis. Another Think Activity was not opened to students until the in-class session had started and so was
excluded from analysis. The outputs for the other five activities were included in the analysis.
265
Fig. 6. Pre-class Think Activity completion.
Descriptive data indicated that the pre-class Think Activity completion rate for the gamified class was higher than that for the
non-gamified class (see Fig. 6). An interesting trend was that the activity completion rate in the control class declined as time went
on, while in the treatment class more students completed the activities as time went on. Chi-square tests of independence were
conducted to examine whether there were statistically significant differences in pre-class activity completion. In Think Activity 1, a
significant interaction was found (x2 (1) = 6.33, p = 0.01) between the treatment class (71%) and control class (29%). Statistically
significant results could be found in all other chi-square tests. See Table 4 for the results of chi-square tests for the pre-class activities.
The gamified class completed significantly more pre-class activities than the non-gamified class.
4.2.3.2. Effects on post-class activity completion. In the post-class stage, students’ completion of the quizzes was compared between the
gamified and non-gamified class. The percentages of quiz completion for the treatment class were higher than for the control class for
all of the examined weeks (see Fig. 7). Chi-square tests of independence were conducted to examine whether there were statistically
significant differences in post-class activity completion. In Quiz 1, a significant interaction was found (x2 (1) = 10.67, p < 0.001)
between the treatment class and control class. Statistically significant chi-square test results could also be found in Quizzes 3, 4, and
5, but not Quiz 2. See Table 5 for the results of chi-square tests of independence for the post-class activities. The Chi-square results
revealed that the treatment class was more likely to complete the post-class activities than the control class.
4.2.3.3. Effects on the quality of pre-class activity. To examine the quality of students' pre-class artifacts, students' submissions for the
think activities were collected and analyzed. In the think activities, students were asked to respond to questions relevant to the week's
topic. For example, the questions for one week were as follows: “a. In your opinion, what's the most important issue to consider before
recording information? b. In your opinion, why is it important to have a standard format for student assignments?” The quality of
students' submissions was marked according to the following criteria: 1) clear and meaningful content; 2) sufficient information
provided in support; 3) connecting concepts; 4) linkage between theory and examples; and 5) original ideas. The full score for each
activity was 10. Descriptive data indicated that the mean scores of the treatment class were higher than those of the control class in
all of the think activities. A normality test indicated that the data were not normally distributed, so a non-parametric Mann-Whitney
U test was administered to compare the differences of the two groups. For Think Activity 1, a non-parametric Mann-Whitney U test
indicated a significant difference between the scores of the treatment class (Mdn = 8) and control class (Mdn = 6.5), U = 71,
Table 4
Chi-square tests of independence for pre-class activities.
Pre-class activities d.f. N X2 p
Think Activity 1 1 40 6.33 = 0.01

Think Activity 2 1 40 6.01 = 0.01
Think Activity 3 1 40 16.00 < 0.001
Think Activity 4 1 40 10.58 < 0.001
Think Activity 5 1 40 16.05 < 0.001
266
Fig. 7. Post-class quiz completion.
Table 5
Chi-square tests for post-class activities.
Post-class activities d.f. N X2 p
Quiz 1 1 40 10.67 < 0.001

Quiz 2 1 40 1.14 > .05
Quiz 3 1 40 4.42 = 0.04
Quiz 4 1 40 9.34 < 0.001
Quiz 5 1 40 8.71 < 0.001
p = 0.001. For Think Activity 2, a non-parametric Mann-Whitney U test showed a significant difference between the scores of the
treatment class (Mdn = 8.4) and control class (Mdn = 6.2), U = 71, p = 0.001. For Think Activity 3, a non-parametric Mann-
Whitney U test showed a significant difference between the scores of the treatment class (Mdn = 8.1) and control class (Mdn = 0),
U = 69, p < 0.001. For Think Activity 4, a non-parametric Mann-Whitney U test showed a significant difference between the scores
of the treatment class (Mdn = 8.2) and control class (Mdn = 0), U = 43.5, p = 0.005. For Think Activity 5, a non-parametric Mann-
Whitney U test showed a significant difference between the scores of the treatment class (Mdn = 7.8) and control class (Mdn = 0),
U = 90.05, p < 0.001. Overall, the Mann-Whitney U-tests showed that the overall quality of the treatment class's activities was
higher than that of the control class's activities (See Table 6).
4.2.3.4. Effects on the quality of post-class activity. To examine the quality of students' post-class artifacts, students' scores for the
quizzes were collected and analyzed. Students were allowed to try the quizzes multiple times. In the analysis stage, each student's
highest score for each activity was collected and analyzed. For Quiz 1, a non-parametric Mann-Whitney U test indicated a significant
Table 6
Scores of pre-class think activities.
Class N Median Minimum Maximum Significance
Pre_T1 Treatment 25 8.00 0.00 10.00 U = 71.00, P = 0.001

Control 15 6.50 0.00 8.50
Control 15 6.20 0.00 8.20
Pre_T3 Treatment 25 8.10 0.00 9.50 U = 69.00, P < 0.001
Control 15 0.00 0.00 7.60
Control 15 0.00 0.00 8.60
Pre_T5 Treatment 25 7.80 0.00 9.50 U = 90.05, P < 0.001
Control 15 0.00 0.00 8.30
Note: Significance was computed using Mann-Whitney U-tests.
267
Table 7
Scores of post-class quizzes.
Class N Median Minimum Maximum Significance
Quiz 1 Treatment 25 9.60 0.00 10.00 U = 109.00, P = 0.023

Control 15 0.00 0.00 10.00
Control 15 0.00 0.00 10.00
Control 15 0.00 0.00 10.00
Control 15 0.00 0.00 10.00
Control 15 0.00 0.00 10.00
Note: Significance was computed using Mann-Whitney U-tests.
difference between the scores of the treatment class (Mdn = 9.60) and control class (Mdn = 0), U = 109, p = 0.023. For Quiz 2, a
non-parametric Mann-Whitney U test showed a significant difference between the scores of the treatment class (Mdn = 10.00) and
control class (Mdn = 0), U = 109, p = 0.035. For Quiz 3, a non-parametric Mann-Whitney U test showed a significant difference
between the scores of the treatment class (Mdn = 10.00) and control class (Mdn = 0), U = 116.5, p = 0.007. For Quiz 4, a non-
parametric Mann-Whitney U test showed a significant difference between the scores of the treatment class (Mdn = 9.00) and control
class (Mdn = 0), U = 98.5, p = 0.006. For Quiz 5, a non-parametric Mann-Whitney U test showed a significant difference between
the scores of the treatment class (Mdn = 10.00) and control class (Mdn = 0), U = 94, p = 0.003. Overall, the Mann-Whitney U-tests
showed that the overall quality of the treatment class's activities was higher than that of the control class's activities (see Table 7).
4.2.3.5. Students perception of gamification. Students in the gamified group were invited to participate in an interview. In this case, 11
participants volunteered to be interviewed. The interview participants expressed their views about the impact of gamification, and
provided suggestions for improvement.
Similar to study one, a majority of the students expressed that gamification motivated them to set higher goals and complete more
tasks. For example, interviewee (S1) said, “with gamification, I would set up goals each week as to the type of badges that I wanted to earn
and the total number of badges I wanted. As such, I would complete my work early and it also motivated me to check in on Moodle to see how
my peers were doing as well.” Interviewee (S2) described, “I think that early bird badge really brought a positive impact, because in order to
get the badge, everyone read the power point and answer the question before coming to class.” Interviewee (S6) reported that, “Receiving a
badge gives me a kind of accomplishment. And I would tell myself oh I have completed another task.” Students' effort to complete the pre-
class Think Activities influenced their subsequent performance in-class as described by (S2), “when the professor started to talk about
the different subjects that he introduced in the subsequent face-to-face class, they [students] came up many good questions during the class. We
wouldn't have been able to ask the questions, if we had not read the power point before class. So I think it's a very good idea of using
gamification to motivate students to prepare beforehand for the face-to-face class sessions.”
Gamification also stimulated students to take more challenges. Interviewee (S3) stated, “If given 3 questions, and I am able to answer
only two, and one question is beyond my ability to answer, I will spend time to search more literature and resources to find answer to that one
question. But if without the badge system, I may just answer the two questions that I know. I would not attempt to answer the one question that
I find difficult.”
Gamification also seemed to bring the class together. The use of badges (e.g., communicator badge) motivated students to initiate
or reply to three other forum postings. This helped increase the number of comments or questions-related postings that function as a
starting point to ground the rest of the online discussion (Schellens, Keer, & Valcke, 2005). Interviewee (S2) explained it this way, “I
think overall, the entire gamification process really brought the class together to do the online discussion …, and at the end we began to talk to
each other.” Another interviewee (S7) stated, “I was worried about my English but I still tried to post more on the forum. The communicator
badge gave me a sense of recognition.”
Despite the overall positive comments about the impact of gamification, several suggestions for improvement were proposed by
the interviewees. Although learners in the gamified condition overall completed significantly more out-of-class activities and pro-
duced higher quality work than the control condition, one interviewee (S8) lamented that a few participants seemed to “play the
game” in the online forum to get the badges, especially at the end of the course. For example, some participants simply made postings
to earn the badges but their contributions were not particularly insightful. One interviewee (S9) therefore suggested that some form
of quality control mechanism should be implemented (e.g., asking the instructor to monitor the postings). Another possible option is
to ask students to peer-evaluate one another's contributions in the online discussions (e.g., using an up-vote or down-vote button to
indicate users' opinions about a posting).
Another interviewee (S7) suggested that the instructor add the game element of leveling-up. Leveling-up indicates the progress and
status that a user has made. They typically convey the mastery of a particular task. The changes in levels occur when the participant
reaches a set point threshold, indicated by the accumulation of a certain number of points or the collection of a new badge. Reaching
the set threshold serves to “unlock” access to next higher level content or activity. One interviewee (S11) also suggested the instructor
to link gamification to some tangible material rewards such as number of badges earned for some participation marks that will count
toward the course total score.
268
5. Discussion
Finding 1. Gamification design based on the GAFCC model motivated learners to complete significantly more out-of-class
activities than in the non-gamified condition.
H1 and H2, which stated that students in the gamified course would be encouraged to complete more course activities, were
supported. In the short-term three-week basic statistics and SPSS case, the treatment class completed significantly more pre- and post-
class activities than the control class. For the gamified pre-class task (Enter data in SPSS), the percentage of activity completion for
treatment class was 67%, while for the control class it was 26%. For the gamified post-class activity individual challenge task levels
1–6, the percentage of attempts for treatment class was 95%, while for the control class it was 0%. It is particularly noteworthy that
although the post-class individual challenge tasks were difficult tasks requiring time and effort to complete, the completion rate for
these challenges in the treatment class was high. This finding suggests that, under the gamified condition, learners were motivated to
take on more challenging tasks than the control class.
This pattern could also be observed in the longer-term eight-week introductory information management course. In the pre-class
think activities, learners were provided with some challenging questions. The percentage of completion for this task grew week by
week in the treatment class, from 71% (i.e., Think Activity 1) to 95% (i.e., Think Activity 5). However, the percentage of completion
declined dramatically in the control class, from 29% (i.e., Think Activity 1) to 5% (i.e., Think Activity 5). This phenomenon cor-
roborates the observations from Case 1, and provides evidence that learners were encouraged to take on more difficult tasks than the
control class. Gamification design based on the GAFCC model motivated learners to complete significantly more out-of-class activities
than in the non-gamified condition.
Finding 2. Gamification design based on the GAFCC model motivated learners to produce higher quality out-of-class
activity artifacts than in the non-gamified condition.
H3 and H4, which stated that students in the gamified course would produce higher quality pre-class and post-class artifacts, were
also supported. In Study 1, the quality of pre-class and post-class activity artifacts indicated that on average the treatment class
produced higher quality artifacts than the control class. For example, the mean score for level 1–5 activity in the treatment class was
10.23, while in the control class no one gained any score. In Study 2, the average quality for pre- and post-class activities outputs in
the treatment class was also higher than in the control class. For example, the median scores for Quiz 1 was 9.60 for the treatment
class and 0 for the control class. Gamification design based on the GAFCC model motivated learners to produce higher quality out-of-
class activity artifacts than in the non-gamified condition.
Overall, the present study showed that gamification has a positive effect on motivating participants to complete more out-of-class
activities, and produce higher quality out-of-class artifacts than the non-gamified courses. Several reasons have contributed to this
positive effect.
First, game mechanics such as badges (e.g., early bird badges) served as specific goal signifiers and reminded learners that
completing tasks before the deadline was a desirable goal. This is in line with the notion that goal setting directs people's attention to
goal-relevant activities and increases their persistence in completing the desired task (Locke & Latham, 1984, 2002). Previous re-
search has suggested that when users are given a clear and specific goal such as individuals who contribute in Wikipedia will be
awarded a “Barnstar” badge, their contribution increased by 60% compared to users who were not given a specific goal (Restivo &
van de Rijt, 2012).
Second, after completing the tasks on time, the badges also provide positive feedback to learners, thus helping reinforce their
behavior. They confirm the learners’ achievements, and visibly show their accomplishment of goals (Antin & Churchill, 2011).
Skinner (1953, 1989) argued that such feedback serves as a positive reinforcement that increases the probability of behavior re-
occurrences. Our results also suggested that such positive feedback contribute to better work quality.
Third, the gamified courses provided learners access to different levels of challenge. Different levels were offered but learners
could choose to decide the level of effort they preferred to make. For example, in Study 1, we broke down the post-class task into six
tasks, ranging from level 1 (entering data and analyzing data) to level 6 (finding mistakes in peer's report and correcting them). This
gave learners an opportunity to exercise their sense of autonomy (i.e., experiencing decision freedom in choosing between several
courses of action) (Sailer et al., 2017). Self-determination theory posits that satisfaction of the need for autonomy helps increase
intrinsic motivation in performing a task (Deci & Ryan, 2000). This is also consistent with the need achievement theory which posits
that most people desire to seek different challenges (McClelland, 1961), which leads to feelings of competence. A heightened per-
ception of competency encourages a higher level of engagement (Deci & Ryan, 2000).
Finally, the careful alignment of instructional objectives, learning activities, and gamification strategies (Tables 2 and 3) pro-
moted participation in the gamified activities. According to Jenkins (2016), the most common pitfall of gamification is adding game
elements to a course without linking them to some kind of learning objectives; this wastes the learners’ time. Gamification design
without a careful alignment of learning objectives usually results in shallow gamification and may not yield positive results. Spe-
cifying clear objectives should be the first most important requirement for any successful gamification projects (Morschheuser,
Werder, Hamari, & Abe, 2017). As De-Marcos, Domínguez, Saenz-de-Navarrete, and Pagés (2014) elaborated, “the bottom line is, in
our opinion, that a careful instructional design driven by clear objectives is essential for a meaningful integration of gamification in e-
learning approaches” (p. 91). In our two studies, the alignment among instructional objectives, learning activities, and gamification
strategies helped strengthen the perceived usefulness of completing the out-of-class learning activities.
The two studies show that gamification following the GAFCC model has a positive impact on engaging learners, and this was
because a design following GAFCC model make students feel that it is helpful for their learning. Based on the empirical findings, we
suggest the following principles for applying gamification in flipped learning settings.
269
● Make game rules clear at the beginning of the flipped course. Although in video games “uncertainty” (Malone, 1981) is an
important element in motivating players, in learning settings making the rules clear for obtaining rewards (e.g., badges or points)
helps students be clearer about the behaviors desired, and thus helps them set short- and long-term goals. Without clear rules, they
may become confused or frustrated, or not put effort into the most necessary parts.
● Align instructional objectives, learning activities, and gamification strategies. Perceived usefulness (Nakamura &
Csikszentmihalyi, 2009) is an important element driving learners to put effort into pursuing badges. Gamification strategies
isolated from learning objectives may not be effective in motivating greater participation and higher quality outputs, especially
for mature learners. As students stated in the interview that they enjoyed the gamification not because the single badges or points,
but because the overall gamification design can help them learn.
● Use gamification to provide recognition and encourage collaboration. Students stated in interviews that when they did the out-of-
class activities, they usually experienced little interaction with the teacher and peers. Many students did not habitually view
others' posts after posting on the forum. When gamification was applied, they were happy to receive the feedback and recognition
provided by the badges. When necessary, adding peer-review badges or points could encourage them to interact with each other
and read each other's postings.
● Differentiate difficulty levels and provide leveling-up tasks. Splitting complicated or difficult tasks into parts and using rewards to
encourage learners to complete them one level at a time could be more effective than simply presenting a whole complicated task
to learners.
● Make sure students have equal resources or access for completing gamified tasks. As gamification may introduce competition
between learners, not having enough resources to participate in the activity may frustrate learners. For example, a student
mentioned that she could not gain access the SPSS software to complete the pre-class activity, and thus felt unmotivated at the
beginning.
6. Conclusion and future research
Examining how gamification can be used to facilitate flipped learning is still at its initial stage (Dicheva, Dichev, Agre, &
Angelova, 2015; Seaborn & Fels, 2015). This study is one of the first to synthesize a broad spectrum of motivation theories underlying
gamification and to propose a theory-driven gamification design model. It is also one of the first to empirically validate this theory-
driven gamification design model in a higher education setting, using rigorous quasi-experimental designs in both short- and long-
term flipped courses.
From the theoretical perspective, the comparison and categorization of the five relevant theories can help researchers and practi-
tioners gain an overview of the underpinning motivation mechanism that connects learner motivation, potential behaviors and
gamification strategies. Applied studies have rarely provided a rationale for how motivation theory could guide gamification design.
Among the small number of theory-guided empirical studies, most have referred to merely one or two theories, i.e., self-determi-
nation theory or goal-setting theory. Although self-determination theory and goal-setting theory may help interpret psychological or
behavioral phenomena, other relevant theories can broaden our understanding of learners' motivation and how to enhance learners’
motivation and experience (Nacke & Deterding, 2017).
From the pedagogical perspective, this study contributes to understanding the implementation context and procedures in detail.
Incorporating gamification into flipped courses needs considerable technological pedagogical knowledge, psychological knowledge,
and content knowledge. It may be overwhelming for educators and pre-gamification practitioners to decide when and how to gamify
learning activities. Detailing the context would help them understand which factors may influence the implementation results.
Theories or frameworks, especially those in educational field, need to tell researchers and practitioners how the ideas could be
applied to the real world (Mishra & Koehler, 2006). In this study, the five-stage gamification design procedure illustrates how the
gamification design model can be applied in real classrooms in a step-wise way, and can help practitioners streamline their design
procedure. The empirical data provided evidence that the GAFCC model is effective at stimulating higher completion rates and higher
quality artifacts in flipped activities.
As this study is an initial exploration on building a gamification design model based on five motivation theories and testing its
effectiveness, it has some limitations. First, the sample sizes of the implementations were small. Due to the characteristics of post-
graduate education, class size is usually small. Moreover, the students in the two courses came from convenient samples. We were
unable to randomly select and assign students into the treatment and control class. In the future, researchers may consider validating
the design model using randomly selected samples, in larger common core courses, and compare its effectiveness with the results
presented here. Second, this model was tested only in the basic statistics and information management disciplines, and it is not clear
how effective it would be in guiding the design of courses in other disciplines. Researchers and practitioners should examine the
application of this model to other disciplines. Third, in this study the course instructor did not participate in the online activities to
interact with students on Moodle, it is uncertain if teachers' involvement with the online activities would better enhance students'
engagement. It would be meaningful to explore the level of teacher involvement in gamified activities, and its impact on students’
motivation and engagement. Fourth, this model mainly targeted learners in a higher education setting. Future studies may explore
how this model could be refined to meet the needs of K-12 students.
Finally, in the present study, we utilized the quasi-experimental research methodology to help draw causality conclusions. We
have also used qualitative research method such as participant interviews to provide richer data to help provide possible explanations
for why an intervention might have an effect. However, there are other research methodologies that could be employed in future
studies to examine and evaluate the GAFCC model in flipped classroom implementation. One such possible research design is design-
270
based research (Anderson & Shattuck, 2012). Design-based research allows a researcher to iteratively adjust and improve a gamified
course over a longer period of time while focusing on and advancing its theoretical underpinnings at the same time. This could
potentially yield more generalizable practical design principles for using gamification in a flipped classroom setting.
References
Albert, M., & Beatty, B. J. (2014). Flipping the classroom applications to curriculum redesign for an introduction to management course: Impact on grades. The Journal
of Education for Business, 89(8), 419–424. http://dx.doi.org/10.1080/08832323.2014.929559.
Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2014). Engaging with massive online courses. Proceedings of the 23rd international conference on World
wide web (pp. 687–698). ACM.
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16–25.
Antin, J., & Churchill, E. F. (2011). Badges in social media: A social psychological perspective. Paper presented at the CHI 2011, Vancouver.
Barata, G., Gama, S., Jorge, J., & Gonçalves, D. (2013). Improving participation and learning with gamification. Paper presented at the proceedings of the Gamification’13,
2013. ACM.
Çakıroğlu, Ü., Başıbüyük, B., Güler, M., Atabay, M., & Memiş, B. Y. (2017). Gamifying an ICT course: Influences on engagement and academic performance. Computers
in Human Behavior, 69, 98–107. https://doi.org/10.1016/j.chb.2016.12.018.
de Charms, R. (2013). Personal causation: The internal affective determinants of behavior. Routledge.
Csikszentmihalyi, M. (1978). Intrinsic rewards and emergent motivation. In M. R. Lepper, & D. Greene (Eds.). The hidden costs of reward: New perspectives on the
psychology of human motivation (pp. 205–216). Wiley.
De-Marcos, L., Domínguez, A., Saenz-de-Navarrete, J., & Pagés, C. (2014). An empirical study comparing gamification and social networking on e-learning. Computers
& Education, 75, 82–91.
Deci, E. L., & Ryan, R. M. (1985). The general causality orientations scale: Self-determination in personality. Journal of Research in Personality, 19(2), 109–134.
Deci, E. L., & Ryan, R. M. (2000). The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227–268.
Deterding, S. (2015). The lens of intrinsic skill atoms: A method for gameful design. Human-computer Interaction, 30, 294–335.
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: Defining gamification. Proceedings of the 15th international
academic MindTrek Conference: Envisioning future media environments (pp. 9–15). ACM.
Dicheva, D., Dichev, C., Agre, G., & Angelova, G. (2015). Gamification in education: A systematic mapping study. Educational Technology & Society, 18(3).
Enfield, J. (2013). Looking at the impact of the flipped classroom model of instruction on undergraduate multimedia students at CSUN. TechTrends, 57(6), 14–27.
Falkner, N. J., & Falkner, K. E. (2014). Whither, badges? or wither, badges! A metastudy of badges in computer science education to clarify effects, significance and
influence. Proceedings of the 14th Koli calling international conference on computing education research (pp. 127–135). ACM.
Festinger, L. (1954). A theory of social comparison processes. Human Relations, 7(2), 117–140.
Gaughan, J. E. (2014). The flipped classroom in world history. The History Teacher, 47(2), 221–244.
Gilboy, M., Heinerichs, S., & Pazzaglia, G. (2015). Enhancing the student engagement using flipped class. Journal of Nutrition Education and Behavior, 47(1), 109–114.
Hamari, J. (2017). Do badges increase user activity? A field experiment on the effects of gamification. Computers in Human Behavior. https://doi.org/10.1016/j.chb.
2015.03.036.
Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? A literature review of empirical studies on gamification. Proceedings of the 47th Hawaii inter-
national conference on system science (HICSS) (pp. 3025–3034). IEEE.
Hao, Y. (2016a). Exploring undergraduates' perspectives and flipped learning readiness in their flipped classrooms. Computers in Human Behavior, 59, 82–92. https://
doi.org/10.1016/j.chb.2016.01.032.
Hao, Y. (2016b). Middle school students' flipped learning readiness in foreign language classrooms: Exploring its relationship with personal characteristics and
individual circumstances. Computers in Human Behavior, 59, 295–303.
Hew, K. F., Huang, B., Chu, K. W. S., & Chiu, D. K. (2016). Engaging Asian students through game mechanics: Findings from two experiment studies. Computers &
Education, 92, 221–236. https://doi.org/10.1016/j.compedu.2015.10.010.
Hunicke, R., LeBlanc, M., & Zubek, R. (2004). MDA: A formal approach to game design and game research. Proceedings of the AAAI Workshop on Challenges in Game
AI, 4(1).
Iosup, A., & Epema, D. (2014). An experience report on using gamification in technical higher education. Proceedings of the 45th ACM technical symposium on computer
science education (pp. 27–32). ACM.
Jenkins, J. (2016). Top 4 gamification problems to avoid. Retrieved from: https://elearningindustry.com/top-4-gamification-problems-avoid.
Kim, M. K., Kim, S. M., Khera, O., & Getman, J. (2014). The experience of three flipped classrooms in an urban university: An exploration of design principles. The
Internet and Higher Education, 22, 37–50. http://dx.doi.org/10.1016/j.iheduc.2014.04.003.
Klevers, M., Sailer, M., & Günthner, W. A. (2016). Implementation model for the gamification of business processes: A study from the field of material handling.
Simulation and gaming in the network society (pp. 173–184). Springer Singapore. http://dx.doi.org/10.1007/978-981-10-0575-6_14.
Kumar, J. M., & Herger, M. (2013). Gamification at work: Designing engaging business software. Aarhus:, the Netherlands: The Interaction Design Foundation.
Lai, C. L., & Hwang, G. J. (2016). A self-regulated flipped classroom approach to improving students' learning performance in a mathematics course. Computers &
Education, 100, 126–140.
Landers, R. N., Bauer, K. N., & Callan, R. C. (2017). Gamification of task performance with leaderboards: A goal setting experiment. Computers in Human Behavior, 71,
508–515.
Landers, R. N., & Landers, A. K. (2014). An empirical test of the theory of gamified learning: The effect of leaderboards on time-on-task and academic performance.
Simulation & Gaming, 45(6), 769–785.
Li, W., Grossman, T., & Fitzmaurice, G. (2012, October). GamiCAD: A gamified tutorial system for first time AutoCAD users. Proceedings of the 25th annual ACM
symposium on user interface software and technology (pp. 103–112). ACM.
Locke, E. A., & Latham, G. P. (1984). Goal setting: A motivational technique that works!. Prentice Hall.
Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705.
http://dx.doi.org/10.1037//0003-066X.57.9.705.
Malone, T. W. (1981). Toward a theory of intrinsically motivating instruction. Cognitive Science, 5(4), 333–369.
Marcum, T. M., & Perry, S. J. (2015). Flips and flops: A new approach to a traditonal law course. Journal of Legal Studies Education, 32(2), 255–286.
McClelland, D. C. (1961). The achieving society. Princeton, NJ: Van Nostrand.
Mekler, E. D., Brühlmann, F., Tuch, A. N., & Opwis, K. (2017). Towards understanding the effects of individual gamification elements on intrinsic motivation and
performance. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2015.08.048.
Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017.
Morschheuser, B., Werder, K., Hamari, J., & Abe, J. (2017). How to gamify? A method for designing gamification.
Nacke, L. E., & Deterding, S. (2017). The maturing of gamification research. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2016.11.062.
Nakamura, J., & Csikszentmihalyi, M. (2009). Flow theory and research. Handbook of positive psychology.
Nederveld, A., & Berge, Z. L. (2015). Flipped learning in the workplace. Journal of Workplace Learning, 27(2), 162–172. https://doi.org/10.1108/JWL-06-2014-0044.
Nicholson, S. (2012). A user-centered theoretical framework for meaningful gamification. Paper presented at the proceedings of games + learning + society, Vol. 8, 1.
Restivo, M., & van de Rijt, A. (2012). Experimental study of informal rewards in peer production. PLoS One, 7(3), Retrieved from: http://journals.plos.org/plosone/
article?id=10.1371/journal.pone.0034358.
271
Richter, G., Raban, D. R., & Rafaeli, S. (2015). Studying gamification: The effect of rewards and incentives on motivation. Gamification in education and business (pp. 21–
46). Springer International Publishing.
Rodrigues, L. F., Costa, C. J., & Oliveira, A. (2016). Gamification: A framework for designing software in e-banking. Computers in Human Behavior, 62, 620–634.
https://doi.org/10.1016/j.chb.2016.04.035.
Sailer, M., Hense, J. U., Mayr, S. K., & Mandl, H. (2017). How gamification motivates: An experimental study of the effects of specific game design elements on
psychological need satisfaction. Computers in Human Behavior, 69, 371–380. https://doi.org/10.1016/j.chb.2016.12.033.
Schellens, T., Keer, H. V., & Valcke, M. (2005). The impact of role assignment on knowledge construction in asynchronous discussion groups. Small Group Research,
36(6), 704–745.
Schunk, D. H. (1991). Self-efficacy and academic motivation. Educational Psychologist, 26(3–4), 207–231.
Schunk, D. H., & Swartz, C. W. (1993). Goals and progress feedback: Effects on self-efficacy and writing achievement. Contemporary Educational Psychology, 18(3),
337–354.
Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International Journal of Human-computer Studies, 74, 14–31. https://doi.org/10.1016/j.
ijhcs.2014.09.006.
Shernoff, D. J., Csikszentmihalyi, M., Shneider, B., & Shernoff, E. S. (2003). Student engagement in high school classrooms from the perspective of flow theory. School
Psychology Quarterly, 18(2), 158.
Simões, J., Redondo, R. D., & Vilas, A. F. (2013). A social gamification framework for a K-6 learning platform. Computers in Human Behavior, 29(2), 345–353.
Skinner, B. F. (1953). Science and human behavior. Simon and Schuster.
Skinner, B. F. (1989). The origins of cognitive thought. American Psychologist, 44(1), 13.
Tune, J. D., Sturek, M., & Basile, D. P. (2013). Flipped classroom model improves graduate student performance in cardiovascular, respiratory, and renal physiology.
Advances in Physiology Education, 37(4), 316–320. http://dx.doi.org/10.1152/advan.00091.2013.
Werbach, K., & Hunter, D. (2012). For the win: How game thinking can revolutionize your business. Wharton Digital Press.
Woolfolk, A. (1998). Educational psychology (7th ed.). Allyn & Bacon.
Zichermann, G., & Cunningham, C. (2011). Gamification by design: Implementing game mechanics in web and mobile apps. O'Reilly Media, Inc.
272

Effects On Out-Of-Class Activity

Uploaded by

Copyright:

Available Formats

Effects On Out-Of-Class Activity

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Effects On Out-Of-Class Activity

Uploaded by

Copyright:

Available Formats

Computers & Education 125 (2018) 254–272

Contents lists available at ScienceDirect

Computers & Education

Implementing a theory-driven gamiﬁcation model in higher

A R T IC LE I N F O ABS TRA CT:

2.1. Gaps in previous gamiﬁcation studies

The following hypotheses are proposed.

3. Towards a theory-driven gamiﬁcation design model

3.1. Flow theory

3.2. Goal-setting theory

3.3. Social comparison theory

3.4. Self-determination theory

3.5. Behavior reinforcement theory

3.6. A theory-driven gamiﬁcation design model

3.6.1. Reifying goal

3.6.2. Reifying access

Flow Theory Goal Setting Self-Determination Social Comparison Behavior

3.6.3. Reifying feedback

3.6.4. Reifying challenge

3.6.5. Reifying collaboration

3.7. Flowchart for designing a gamiﬁed course

Fig. 2. Flowchart of ﬁve-stage gamiﬁcation design procedure.

4. Research design and results

4.1.2. Designing the basic statistics and SPSS module

Basic Level: Pre-class: Pre-class:

Fig. 3. Moodle page, badge rule, and point scheme.

4.1.3. Data collection and analysis tools

Fig. 4. Pre- & post-class activity completion rate.

submissions of individual challenges) were analyzed and compared.

Basic Level: Pre-class: Pre-class:

concepts in the information management discipline.

4.2.2. Designing the information management course

Fig. 5. Screenshots of Moodle page, badges, and badge rules.

Fig. 6. Pre-class Think Activity completion.

Think Activity 1 1 40 6.33 = 0.01

Fig. 7. Post-class quiz completion.

Quiz 1 1 40 10.67 < 0.001

Pre_T1 Treatment 25 8.00 0.00 10.00 U = 71.00, P = 0.001

Note: Signiﬁcance was computed using Mann-Whitney U-tests.

Quiz 1 Treatment 25 9.60 0.00 10.00 U = 109.00, P = 0.023

Note: Signiﬁcance was computed using Mann-Whitney U-tests.

6. Conclusion and future research

You might also like