NurAnisIdayu CHAPTER 3 REVISED
NurAnisIdayu CHAPTER 3 REVISED
NurAnisIdayu CHAPTER 3 REVISED
THE EVALUATION OF
USER EXPERIENCE (UX)
ABOUT AN INTERACTIVE ANIMATED COMIC OF
PAEDOPHILE AWARENESS FOR CHILDREN
(I-ComPedo)
MSc
October 2021
TABLE OF CONTENT
CHAPTER THREE: RESEARCH METHODOLOGY
3.1 Introduction 1
3.2 Research Approach 1
3.3 Research Design 1
3.4 Research Framework 3
3.4.1 Phase 1: Planning and Preparation 4
i
LIST OF TABLES
ii
LIST OF FIGURES
iii
CHAPTER THREE
RESEARCH METHODOLOGY
3.1 Introduction
In Chapter 3, the entire research technique was covered in greater depth. The methodology
was a system of methods used to conduct research that consists of a set of strategies and
processes to be followed to achieve research objectives and ensure that research was completed
on time. Specific procedures for completing this study would describe in the following
categories.
1
The research was based on a quantitative method, which would be evaluating user
experience about interactive animated comics of paedophile awareness for children (i-
ComPedo). The table below shows the research design of this study.
Table 3.1 Research Design Summary
RESEARCH RESEARCH METHODS/ TOOLS/ DELIVERABLES PHASE
QUESTIONS OBJECTIVES TECHNIQUES
RQ1 RO1 To identify the appropriate topic 1. Describe research Planning and
and title for the research. background, problem Preparation
What is the To analyze user statements, research
user experience 1. Discussion with the objectives, research
experience (UX) with the i- supervisor about the research scope, research
with the i- ComPedo topics and problems. significance.
ComPedo application. 2. Information collection and 2. A literature review has
application? Literature review through been made.
journals, thesis, articles, 3. Define the sample of
books regarding the user respondents.
experience (UX), animated
comics for children.
3. Research on the sample of
respondents.
RQ2 RO2 To make the data analysis The output from the data Data Analysis
that has been analyzed.
How can we To recommend 1. Analyze the data by using an
improve the or suggest an Attrakdiff online platform
user interface improved user a) Tool
for a better interface for - Attrakdiff
user better user - Microsoft Excel File
experience experience b) Techniques
2
(UX) with the (UX) with the i- - Distribute questionnaire to Suggestions or
i-ComPedo ComPedo teachers. recommendations to
application application. 2. Data transformation has been improve the user interface
used for data analysis. of the i-ComPedo app.
a) Tool
- Data analysis tool
b) Techniques RO2: User interface
- Using Portfolio suggestion and
presentation, average value, improvements
and description word-pairs.
- Smiley-o-meter scale
3
Figure 3.1 Research Framework adapted from Subbarao, A., & Mahrin, M. N. (2020) with Research
Framework of Evaluation Model
a) Sample of Respondents
Ravindra, K. (2020) states that a focus group was a selected group of people who
discusses a given topic or issue in-depth, made by the manager, an outside expert. It’s a
great way to bring together people from similar backgrounds or experiences to discuss a
particular topic of interest. In this research, the sample of respondents was from primary
school children from 9 to 12 years old, Sekolah Kebangsaan Kuala Besut, Terengganu.
4
3.4.2 Phase 2: Data Collection
This section would focus on data collection for this study. In this research, data
collection would use a "self-administered web survey design and implementation", in which
the authors suggest that short, concise, and understandable terms should be included in the
study questions (Jong, J., et al., 2016; Gonzales- Bañales, et al., 2007; Rada, V., et al., 2019).
According to Parveen, H., et al (2017) and Sajjad Kabir, S., et al (2016), data collection is one
of the most important stages of conducting research. This method of data collection is
extensively used in quantitative research. Simple random sampling techniques were used in
this study for each topic or unit of the population selected and have an equal chance. The
researcher needs to avoid bias towards respondents in conducting research sampling.
Even though the face-to-face interview is the most common form of communication, over
the past few years it has been shown that most interviews were done by focus groups, by phone,
email, internets such as messenger apps, and electronic form (Dworkin, 2012). Telephone
interviews were useful when collecting information from remote participants on a costly and
time-consuming basis compared to face-to-face interviews and open-ended questions that give
5
participants more productivity without having more freedom to share their ideas (Kallio et al.,
2016; Dworkin, 2012). Another advanced online interview method such as email and
messenger can also be done in focus group planning; videoconferencing, chat rooms, and
discussion groups. In this research, due to the limited time and movement during the Movement
Control Order (MCO) that took place during the data analysis phase, the face to face session
was conducted where the meeting allowed an unanswered question to be discussed with
relevant and limited participant which was only 40 student and 5 primary school teachers. The
researcher retrieved participant feedback used to support previously obtained findings in the
review and review of the literature.
The questionnaire for the primary school teacher’s representative to address the following:
The findings would calculate into three groups on the Attrakdiff online platform: Portfolio
Presentation, The Average Value, and Description Word-pairs. Researchers would be able to
establish whether data was feasible and consistent as a source of information based on these
findings (User Experience rating). Besides, the Attrakdiff online platform calculation, the
Smiley-o-meter scale, and the main results would further be discussed below.
a) Portfolio Presentation
The portfolio presentation would represent the confidence level when the participant
using the application. The vertical axis of the portfolio presentation view displays the hedonic
quality while the horizontal axis shows the pragmatic quality. The results were displayed on a
quadrant. The Confidence rectangle was the small square inside the larger one. A tiny
confidence rectangle indicates that the results were more dependable and less coincidental. The
more diverse the evaluation ratings, the larger the confidence rectangle. The confidence
rectangle indicates whether or not users were satisfied with the product. The portfolio
presentation diagram was shown below as an example.
6
Figure 3.2 Diagram portfolio presentation
7
c) Description Word-pairs
The final output returned by the Attrakdiff online platform was description word-pairs.
The mean values of the word pairings were shown here. The extreme values were of great
relevance. These indicate which features were particularly critical or well-resolved. These
findings would demonstrate whether or not the application may raise the participant's
awareness of paedophiles. The description word-pairs diagram was shown below as an
example.
d) Smiley-o-meter Scales
The results of Smiley-o-meter were “scored from 1 to 5 (1=Awful (Extremely unlikely);
2=Not very good (Very unlikely); 3=Good (likely); 4=Really good (Very likely); 5= Brilliant
(Extremely likely)”. The Smiley-o-meter feedback would divide into two (2) parts:
i. “Positive feedback” (good, really good, and brilliant)
ii. “Negative feedback” (awful, not very good).
8
Besides, the feedback and suggestions from the sticky notes were also collected and
sorted into two categories:
i. The “positive experiences” such as the comment that define satisfaction such as "I
like", "Happy", "Thank you", and others.
ii. The “negative experiences” such as the comment that describes dissatisfaction for
example “Boring”, “Upset”, “Annoying”, and others.
e) Main results
After processing at the Attrakdiff online platform stage, the major results can be
displayed in their corresponding spreadsheet. The portfolio presentation, average value, and
description word pairs would all be displayed in a bar chart. After the user experience stage
was evaluated, the smiley-o-meter scale for each participant was displayed in a table. As a
result, each item's mean and standard deviation was determined. Below were the details of the
calculation results.
9
criteria for research, and was also easy to identify the area as it is close to the researcher’s
home.
Besides, the AttrakDiff would use Smiley-o-meter scales which consist of (5) point Likert
scales and pictorial representation with (10) attributes. The example of (10) attributes of
AttrakDiff was shown in Figure 3.3. Thus, below displays details about the categorization of
AttrakDiff and Smiley-o-meter scale.
10
Table 3.2 Main Groups in the current version of AttrakDiff
11
Source: Example of i-ComPedo questionnaire using Attrakdiff, adapted from Giardi, et al. (2019)
3.8 Summary
This chapter describes the methodology that would use in this research design. The
main goal of this chapter was to explore research methodology and methods to bring impact
on the research design. Each section of the research framework for this study was briefly
12
described in this chapter and uses a quantitative research method in the form of online user
experience (UX) on the product of the i-ComPedo application. This chapter also highlights the
research instruments and areas that would use in this study. Discussions in this way were
discussed throughout the chapter.
13
REFERENCES
Abbas, J., Aman, J., Nurunnabi, M., & Bano, S. (2019). The impact of social media on
learning behavior for Sustainable EDUCATION: Evidence of students from Selected
universities in Pakistan. Sustainability, 11(6), 1683. doi:10.3390/su11061683
Ahmad, S., Mohd Nor, N., & Maesin, A. (2020). An interactive animated comic of
paedophile awareness for children. International Journal of Recent Technology and
Engineering, 9(1), 332-337. doi:10.35940/ijrte. a1501.059120
Alhussayen, Asma & AL-Rashed, Wea'am & Mansor, Evi. (2015). Evaluating the User
Experience of Playful Interactive Learning Interfaces with Children. Procedia Manufacturing.
3. 10.1016/j.promfg.2015.07.378.
Almarabeh, H., & Amer, E. F. (2015). The effectiveness of multimedia learning tools in
education. Retrieved April 14, 2021, from
https://www.researchgate.net/publication/290429349_The_Effectiveness_of_Multimedia_Le
arning_Tools_in_Education
Aristizábal, L. F., Cano, S., Collazos, C. A., Moreira, F., Alghazzawi, D. M., & Fardoun, H.
(2017). Tools and methods applied in interactive systems to evaluate the user experience with
deaf/hard of hearing children. Proceedings of the 5th International Conference on
Technological Ecosystems for Enhancing Multiculturality. doi:10.1145/3144826.3145365
14
Begum, M. (2019). One in 10 Malaysian children are sexually abused, usually by those they
trust. The Star Online. https://www.thestar.com.my/lifestyle/family/2019/11/29/betrayed-by-
the-people-they-trust-the-most
Brane, A. (2016). User Experience Design for Children. Retrieved 2021, from
https://umu.diva-portal.org/smash/get/diva2:973772/FULLTEXT01.pdf
Bunyan, J. (2018, July 24). 50,000 sex crimes, domestic violence cases since 2013. Malaysia
| Malay Mail. https://www.malaymail.com/news/malaysia/2018/07/23/50000-sex-crimes-
domestic-violence-cases-since-2013/1655182
Castro, Á., Ibáñez, J., Maté, B., Esteban, J., & Barrada, J. R. (2019). Childhood Sexual
Abuse, Sexual Behavior, and Revictimization in Adolescence and Youth: A Mini-Review.
Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.02018
Cheah, I. G.-S., & Yuen, C. W. (2016). A Review of Research on Child Abuse in Malaysia.
http://www.e-mjm.org/2016/v71s1/research-on-child-abuse.pdf.
Childhelp National Child Abuse Hotline. (2016). What is Child Abuse? The Issue of Child
Abuse. Retrieved from https://www.childhelp.org/child-abuse/
Chuang, H., & Cheng, M. (2018). Learning processes for digital storytelling scientific
imagination. EURASIA Journal of Mathematics, Science and Technology Education, 15(2).
doi:10.29333/ejmste/100636
Collins, C. S., & Stockton, C. M. (2018). The Central Role of Theory in Qualitative
Research. International Journal of Qualitative Methods, 17(1), 160940691879747.
https://doi.org/10.1177/1609406918797475
Díaz-Oreiro, I., López, G., Quesada, L., & Guerrero, A. (2019). Standardized Questionnaires
for User Experience Evaluation: A Systematic Literature Review. Proceedings, 31(1), 14.
https://doi.org/10.3390/proceedings2019031014
15
Dworkin, S. L. (2012). Sample Size Policy for Qualitative Studies Using In-Depth
Interviews. Archives of Sexual Behavior, 41(6), 1319–1320. doi:10.1007/s10508-012-0016-6
Fatimah, A. S., Santiana, S., & Saputra, Y. (2019). DIGITAL comic: An innovation of Using
toondoo as media technology for teaching English short story. English Review: Journal of
English Education, 7(2), 101. doi:10.25134/erjee. v7i2.1526
Feng, L., & Wei, W. (2019). An Empirical Study on User Experience Evaluation and
Identification of Critical UX Issues. Sustainability, 11(8), 2432.
https://doi.org/10.3390/su11082432
Feng, L., & Wei, W. (2019). An Empirical Study on User Experience Evaluation and
Identification of Critical UX Issues. Sustainability, 11(8), 2432.
https://doi.org/10.3390/su11082432
Foundation, I. D. (2021). What is user experience (UX) design? Retrieved April 15, 2021,
from https://www.interaction-design.org/literature/topics/ux-design
Gerwinn, H., Weiß, S., Tenbergen, G., Amelung, T., Födisch, C., Pohl, A., … Kruger, T. H.
C. (2018). Clinical characteristics associated with paedophilia and child sex offending –
Differentiating sexual preference from offense status. European Psychiatry, 51, 74–85.
https://doi.org/10.1016/j.eurpsy.2018.02.002
Giardi, A. (2019). Evaluate children’s User eXperience with AttrakDiff method: USiena
experience. Form@re - Open Journal Per La Formazione in Rete, 19(3), 155-174.
https://doi.org/10.13128/form-7464
Gonzalez-Bañales, D. L., & Rodenes, M. (2007). (PDF) Web Survey Design and
Implementation: Best Practices for Empirical Research.
https://www.researchgate.net/publication/252577588_WEB_SURVEY_DESIGN_AND_IMP
LEMENTATION_BEST_PRACTICES_FOR_EMPIRICAL_RESEARCH.
16
Greathouse, S. M., Saunders, J. M., Matthews, M., Keller, K. M., & Miller, L. L. (2015). A
review of the literature on sexual assault perpetrator characteristics and behaviors. RAND
Corporation.
Greenhow, C., & Askari, E. (2015). Learning and teaching with social network sites: A
decade of research in k-12 related education. Education and Information Technologies, 22(2),
623-645. doi:10.1007/s10639-015-9446-9
Hall, L., Hume, C., & Tazzyman, S. (2016). Five Degrees of Happiness. Proceedings of the
The 15th International Conference on Interaction Design and Children.
https://doi.org/10.1145/2930674.2930719
Handayani, T., & Kurniawati, N. (2017). Fostering Community Awareness in Preventing and
Handling Pedophilia. Proceedings of the International Conference on Ethics in Governance
(ICONEG 2016). doi:10.2991/iconeg-16.2017.25
Hashim, M. E. A. B., Idris, M. Z. B., & Said, C. S. B. (2021). The Potential of Integrating
User Experience (UX) and Aesthetic Experience (AX) in Augmented Reality Comic (AR
Comic). International Journal of E-Education, e-Business, e-Management and e-Learning,
11(1), 34–41. https://doi.org/10.17706/ijeeee.2021.11.1.34-41
Hellweger, S., & Wang, X. (2015). What is User Experience Really: towards a UX
Conceptual? What is User Experience Really: towards a UX Conceptual Framework.
https://www.researchgate.net/publication/272828093_What_is_User_Experience_Really_tow
ards_a_UX_Conceptual_Framework.
17
Heron, R. L., Schwiekert, L., & Karsten, J. (2021). Meeting a person with paedophilia:
Attitudes towards paedophilia among psychology students: A pilot study. Current
Psychology. https://doi.org/10.1007/s12144-021-01384-5
Hogan, M., & Strasburger, V. C. (2018). Social media and new technology: A primer.
Clinical Pediatrics, 57(10), 1204-1215. doi:10.1177/0009922818769424
Hornbæk, K., & Hertzum, M. (2017). Technology acceptance and user experience. ACM
Transactions on Computer-Human Interaction, 24(5), 1-30. doi:10.1145/3127358
Huan, R. (2017). Evaluating engagement in digital narratives from facial data. Proceedings of
the 19th ACM International Conference on Multimodal Interaction.
https://doi.org/10.1145/3136755.3137036
Kallio, H., Pietilä, A. M., Johnson, M., & Kangasniemi, M. (2016). Systematic
methodological review: Developing a framework for a qualitative semi‐structured interview
guide. Journal of advanced nursing, 72(12), 2954-2965
Kirchoff, J. (2017). Using digital comics to develop digital literacy fostering ... Retrieved
April 14, 2021, from https://files.eric.ed.gov/fulltext/EJ1162692.pdf
Lehnert, F. K. (2019). User experience challenges for designing and evaluating Computer-
Based assessments for children. Proceedings of the 18th ACM International Conference on
Interaction Design and Children. doi:10.1145/3311927.3328724
18
Mäkelä, S., Bednarik, R., & Tukiainen, M. (2013). Evaluating user experience of autistic
children through video observation. CHI '13 Extended Abstracts on Human Factors in
Computing Systems on - CHI EA '13. doi:10.1145/2468356.2468438
Mispa, K., Mansor, E. I., & Kamaruddin, A. (2019). Evaluating children's user Experience
(ux) towards mobile application. Proceedings of the 5th International ACM In-Cooperation
HCI and UX Conference on - CHIuXiD'19. doi:10.1145/3328243.3328250
Moen, O. M. (2015). The ethics of pedophilia. Etikk i Praksis - Nordic Journal of Applied
Ethics, 9(1), 111–124. https://doi.org/10.5324/eip.v9i1.1718
Othman, A., & Wan Yahaya, W. A. (2015). MULTIMEIDA design principles in developing
multimedia learning application (mmla) to increase children awareness of child sexual abuse.
Jurnal Teknologi, 75(3). doi:10.11113/jt. v75.5036
Quayle, E. (2020). Prevention, disruption and deterrence of online child sexual exploitation
and abuse. ERA Forum, 21(3), 429-447. doi:10.1007/s12027-020-00625-7
19
Recine, D. (2013). Comics aren't just for fun anymore: The practical use of comics by
TESOL professionals. Retrieved April 14, 2021, from
https://minds.wisconsin.edu/handle/1793/65479
Rutta, C. B., Schiavo, G., Zancanaro, M., & Rubegni, E. (2020). Collaborative comic-based
digital storytelling with primary school children. Proceedings of the Interaction Design and
Children Conference, 426–437. https://doi.org/10.1145/3392063.3394433
Santoso, H., Schrepp, M., AG,, S., Kartono Isal, R., Utomo, A., & Priyogi, B. (2016).
Measuring User Experience of the Student-Centered e-Learning Environment. The Journal of
Educators Online, 13(1). doi:10.9743/jeo.2016.1
Schrepp, M., Hinderks, A., & Thomaschewski Jörg. (2017). Design and Evaluation of a Short
Version of the User Experience Questionnaire (UEQ-S). International Journal of Interactive
Multimedia and Artificial Intelligence, 4(6), 103. https://doi.org/10.9781/ijimai.2017.09.001
Schrepp, M., Hinderks, A., & Thomaschewski, J. (2014). Applying the user Experience
Questionnaire (UEQ) in different EVALUATION SCENARIOS. Design, User Experience,
and Usability. Theories, Methods, and Tools for Designing the User Experience, 383-392.
doi:10.1007/978-3-319-07668-3_37
Schrepp, M., Hinderks, A., & Thomaschewski, J. (2014). Applying the User Experience
Questionnaire (UEQ) in Different Evaluation Scenarios. Design, User Experience, and
Usability. Theories, Methods, and Tools for Designing the User Experience, 383–392.
https://doi.org/10.1007/978-3-319-07668-3_37
Sim, G., & Horton, M. (2012). Investigating children's opinions of games. Proceedings of the
11th International Conference on Interaction Design and Children - IDC '12.
https://doi.org/10.1145/2307096.2307105
20
Educational Comics (GRAFMIC) based on Markerless Augmented Reality. 2019
International Conference on Electrical, Electronics and Information Engineering (ICEEIE),
220–225. https://doi.org/10.1109/iceeie47180.2019.8981439
Subbarao, A., & Mahrin, M. N. (2020). Research Framework of Evaluation Model to Assess
the Effectiveness of Coordination Processes in Global Software Development Projects.
Journal of Physics: Conference Series, 1529, 052064. https://doi.org/10.1088/1742-
6596/1529/5/052064
Takacs, Z. K., Swart, E. K., & Bus, A. G. (2015). Benefits and Pitfalls of Multimedia and
Interactive Features in Technology-Enhanced Storybooks : A Meta-Analysis, 85(4).
https://doi.org/10.3102/0034654314566989
Wahab, N. A., Yahaya, W. A., & Muniandy, B. (2015). The use of multimedia in increasing
perceived knowledge and awareness of cyber-bullying among adolescents: A pilot study.
Procedia - Social and Behavioral Sciences, 176, 745-749. doi: 10.1016/j.sbspro.2015.01.535
Zaman, B., Vanden Abeele, V., & De Grooff, D. (2013). Measuring product liking in
preschool children: An evaluation of the Smileyometer and This or That methods.
International Journal of Child-Computer Interaction, 1(2), 61–70.
https://doi.org/10.1016/j.ijcci.2012.12.001
Zarour, M. (2017). User Experience Aspects and Dimensions: Systematic Literature Review.
International Journal of Knowledge Engineering, 52–59.
https://doi.org/10.18178/ijke.2017.3.2.087
21
APPENDICES
22