Conference

Congress UPV
9th International Conference on Higher Education Advances (HEAd’23)
The contents of this publication have been evaluated by the Scientific Committee according to the
procedure described in the preface. More information at http://www.headconf.org/
Scientific Editors
Josep Domenech
David Menéndez Álvarez-Hevia
Alicia Martínez-Varea
Rosa M. Llácer-Iglesias
Domenico Brunetto
Cover design by Gaia Leandri
Publisher
2023, Editorial Universitat Politècnica de València
Cno. de Vera, s/n.
46022 Valencia (SPAIN)
www.lalibreria.upv.es / Ref.: 6379_01_01_01
ISSN: 2603-5871
ISBN: 978-84-1396-085-2
Print on-demand
DOI: http://dx.doi.org/10.4995/HEAD23.2023.16871

This book is licensed under a Creative Commons Atribution-NonCommercial-ShareAlique-4.0
International license
Editorial Universitat Politècnica de València http://ocs.editorial.upv.es/index.php/HEAd/HEAd23
Universitat Politècnica de València, València, 2023
Preface
Josep Domenech1, David Menéndez Álvarez-Hevia2, Alicia Martínez-Varea3, Rosa M.
Llácer-Iglesias1, Domenico Brunetto4
1
Universitat Politècnica de València, Spain, 2Universidad de Oviedo, Spain, 3La Fe
University and Polytechnic Hospital, Spain, 4Politecnico di Milano, Italy.
Abstract
This is the preface for the ninth edition of the International Conference on
Higher Education Advances (HEAd). HEAd'23 brought together researchers,
educators, students, and professionals from around the world to exchange
ideas and research findings on student preparation and higher education
systems. The conference facilitated thought-provoking discussions, innovative
ideas, and collaborative efforts to advance teaching, learning, and the overall
higher education landscape. The scientific program, curated by a committee
of 188 members from 44 countries, received 326 full paper submissions. Of
these, 98 papers were accepted for full-paper presentation, while 40 short
papers and 35 posters were also included. The contributions covered diverse
topics such as pedagogical approaches, technology integration, student
engagement, assessment methods, curriculum design, and educational
policies. Additionally, three keynote speeches and three pre-conference
workshops enriched the conference experience.
Keywords: Higher education; innovative materials; educational technology;
evaluation and assessment; globalization in education.
This work is licensed under a Creative Commons License CC BY-NC-SA 4.0

Editorial Universitat Politècnica de València
I
Preface
1. Introduction to HEAd’23
This volume contains the selected papers of the Ninth International Conference on Higher
Education Advances (HEAd’23), which was held in Valencia, Spain, from 19 to 22 June
2023. HEAd'23 provided a dynamic platform for researchers, educators, students, and
professionals from around the world to come together and exchange ideas, experiences, and
research results regarding the preparation of students and the organization of higher education
systems. The conference served as a catalyst for thought-provoking discussions, innovative
ideas, and collaborative endeavors, all aimed at advancing teaching, learning, and the overall
higher education landscape.
The selection of the papers for the scientific program was conducted by a team of 188
committee members representing 44 countries on all five continents. Following the call for
papers, the conference received 326 full paper submissions with authors from 48 different
countries. All the submitted papers were reviewed by at least two program committee
members under a double-blind review process. Finally, 98 papers were accepted as full
papers for oral presentation during regular sessions, representing an overall full paper
acceptance rate of 30%, reflecting the conference's commitment to showcasing the highest-
quality work. Additionally, 40 submissions were accepted for short paper presentations and
35 for poster presentations, all of them receiving high review scores, and were published by
UPV Press in this volume. The organization committee congratulates all the authors for
having their papers accepted in the proceedings of such a competitive conference.
The contents of the program represent a diverse and comprehensive range of topics and
research interests in the field of higher education. The selected papers cover various aspects
of teaching and learning, including pedagogical approaches, technology integration, student
engagement, assessment methods, curriculum design, and educational policies. This rich and
varied collection of research contributes to the advancement of knowledge in higher
education and offers valuable insights and innovative practices that can inform and enhance
teaching and learning experiences worldwide.
The conference program featured three keynotes that overview important and current topics.
The first keynote, “Pedagogical Practices in Large Classes in Higher Education: Challenges
and Opportunities,” delivered by Anna Logan and Ann Marie Farrell (Dublin City University,
Ireland), examined the unique challenges faced by educators in large class settings and
explored innovative pedagogical approaches to enhance student engagement and learning
outcomes. The second keynote, “What Higher Education can learn from Games – more than
fun!” by Martin R. Wolf (FH Aachen University of Applied Science, Germany), highlighted
the transformative potential of incorporating game-based learning principles into higher
education, emphasizing the benefits of immersive, interactive, and experiential learning
environments. Lastly, the keynote titled “The Transformation of the University: Contributing
II
Josep Domenech et al.
to Integral Human Development,” presented by Josep Gallifa Roca (Ramon Llull University,
Spain), explored the evolving role of universities in fostering holistic development and
societal transformation, emphasizing the importance of integrating academic pursuits with
social impact and personal growth. These keynotes provided valuable insights and sparked
thought-provoking discussions, contributing to the overall richness and impact of HEAd’23.
HEAd’23 also hosted three pre-conference workshops to offer participants the opportunity
to explore specific topics relevant to the conference. The workshops included “The future
higher education classroom: Introducing new types of learning, executive function processes,
and strategies to foster students’ motivation and academic success,” organized by Genny
Villa (Université de Montréal, Canada), “Reinventing University: the Digital Challenge in
Higher Education,” led by Stefania Capogna (Link Campus University, Italy) and Erika
Zuperkiene (Klaipeda University, Lithuania), and “Learning experience design in the age of
metaverse,” organized by Eman AbuKhousa (Higher Colleges of Technology, UAE).
The conference was supported and hosted by the Faculty of Business Administration and
Management of the Universitat Politècnica de València, which has been recently ranked as
the best technical university in Spain by the Academic Ranking of World Universities
(ARWU) 2022.
The organizing committee would like to thank all of those who made this year’s HEAd a
great success. Specifically, thanks are indebted to the invited speakers, authors, scientific
committee members, reviewers, session chairs, presenters, sponsors, supporters, and all the
attendees. Our final words of gratitude must go to the Faculty of Business Administration
and Management of the Universitat Politècnica de València for supporting, once again, the
HEAd conference, making it possible to become a great event.
2. Organizing Committee
General chair
Josep Domènech, Universitat Politècnica de València, Spain
Scientific committee chair

David Menéndez Álvarez-Hevia, Universidad de Oviedo, Spain
Publicity chairs
Gareth Bramley, University of Sheffield. United Kingdom
Daniela Zehetmeier, Munich University of Applied Sciences, Germany
III
Preface
Publication chair
Alicia Martínez-Varea, La Fe University and Polytechnic Hospital, Spain
Workshops chair
Domenico Brunetto, Politecnico di Milano, Italy
Workshops organization
Eman AbuKhousa, Higher Colleges of Technology, UAE
Stefania Capogna, Link Campus University, Italy
Genny Villa, Université de Montréal, Canada
Erika Zuperkiene, Klaipeda University, Lithuania
Local arrangements chair

Rosa M. Llácer-Iglesias, Universitat Politècnica de València, Spain
Local organization
Eduardo Cebrián Cerdán
Sergi Domenech de Soria
Francesc Domenech Villalba
Pablo de Pedraza García
Ignacio Reyes Vázquez
Ana Rubio García
3. Sponsors and Supporters
Silver Sponsor
SOWISO
Supporters
Universitat Politècnica de València
Facultad de Administración y Dirección de Empresas
Departamento de Economía y Ciencias Sociales
4. Scientific Program Committee

M’hammed Abdous, Old Dominion University, USA
Felix Acebes Andreu, University of Oviedo, Spain
Mifrah Ahmad, Deakin University, Australia
IV
M. Ángeles Alcaide, University of Valencia, Spain

Gabriella Aleandri, Roma Tre University, Italy
Filomena Almeida, ISCTE IUL, Portugal
Daniel Alonso-Martinez, Universidad de León , Spain
Asier Aranzabal Maiztegi, University of the Basque Country (UPV/EHU), Spain
Azucena Arias-Correa, Universidade de Vigo, Spain
Jose Luis Arquero, University of Seville, Spain
Raquel Ayala Carabajo, Universidad Católica de Santiago de Guayaquil, Ecuador
Marina Milić Babić, University of Zagreb, Croatia
Alice Barana, University of Turin, Italy
Virginia Barba-Sanchez, University of Castilla-La Mancha, Spain
Elvira Barrios Espinosa, Universidad de Málaga, Spain
Alexander Bartel, Neu-Ulm University of Applied Sciences, Germany
Paula Bartel, Kempten University of Applied Sciences, Germany
Evgeniia Beliauskene, Tomsk Polytechnic University, Russia
José V. Benlloch-Dualde, Universitat Politècnica de València, Spain
Maria Eliza Bernardes, Universidade de São Paulo, Brazil
Marnie Binder, California State University, Sacramento, USA
Ignacio Bosch Roig, Universitat Politècnica de València, Spain
Gregory D. Breetzke, University of Pretoria, South Africa
Domenico Brunetto, Politecnico di Milano, Italy
Barbara Bruno, University of Hawaii, USA
Geoff Bunn, Manchester Metropolitan University, UK
Eliseo Bustamante, Universitat Politècnica de València, Spain
Laura Cabeza-García, University of León, Spain
Marisol Calabor, Universitat de València, Spain
Sabrina B. Caldwell, The Australian National University, Australia
Conrado Javier Calvo Saiz, Universitat de València, Spain
Lourdes Canós-Darós, Universitat Politècnica de València, Spain
Noelia Carmona Vicente, University of Valencia, Spain
Adolfo Carrillo Cabello, University of Minnesota, USA
Teresa Carvalho, University of Aveiro and CIPES, Portugal
Dimitris Chassapis, National and Kapodistrian University of Athens, Greece
Alberto Ciolfi, ANVUR, Italy
Jamie Cotler, Siena College, USA
John Cowan, Edinburgh Napier University, Scotland
Daniela-Maria Cretu, Lucian Blaga University od Sibiu, Romania
Jose Luis Cruz Muñoz, University of Valencia, Spain
Raúl de Arriba, Universidad de Valencia, Spain
V
Preface
Diego Víctor de Mingo-López, Universitat Jaume I, Spain

Wietse de Vries, Benemérita Universidad Autónoma de Puebla, Mexico
Ignacio J. Díaz-Maroto, University of Santiago de Compostela, Spain
Marilyn Dono-Koulouris, St. John’s University, USA
Pablo Durán Santomil, Universidad de Santiago de Compostela, Spain
Jan Elen, KU Leuven, Belgium
Jana Erina, Riga Technical University, Latvia
Mara Escrivá, Universitat de València, Spain
Shir Etgar, Columbia University, USA
Joaquim Filipe Ferraz Esteves Araujo, University of Minho, Portugal
Bruno Miguel Ferreira Goncaves, Instituto Politécnico de Bragança, Portugal
Sandro Nuno Ferreira Serpa, University of the Azores, Portugal
Margarida Figueiredo, University of Évora, Portugal
Sylwia Izabela Filipczuk-Rosińska, Polish Air Force University, Poland
Björn Fisseler, FernUniversität in Hagen, Germany
Maria Assunção Flores, University of Minho, Portugal
Nuno Flores, University of Porto, Portugal
Francesco Floris, Univeristy of Turin, Portugal
Elena Fraj-Andrés, University of Zaragoza, Spain
Mark Frydenberg, Bentley University, USA
Thomas Fuhrmann, OTH Regensburg, Germany
Josep Gallifa, Ramon Llull University, Spain
Gonzalo García Ros, Universidad Politécnica de Cartagena, Spain
Consuelo Garcia Tamarit, Universidad Internacional de la Rioja (UNIR), Spain
Suzanne Gatt, University of Malta, Malta
Daniela Gil-Salom, Universitat Politècnica de València, Spain
José Luis Godos-Díez, Universidad de León, Spain
Nuria González-Álvarez, Universidad de León, Spain
Mª de Fátima Goulão, Universidade Aberta, Portugal
Sue Gutierrez Berciano, University of Oviedo, Spain
Katrin Herget, University of Aveiro, Portugal
Ainhoa Herrarte, Universidad Autónoma de Madrid, Spain
Monica Herrero Vázquez, University of Oviedo, Spain
Booi Hon Kam, RMIT University, Australia
Fethi A. Inan, Texas Tech University, USA
Juan Carlos Jiménez Muñoz, University of Valencia, Spain
Beatriz Jiménez Parra, Universidad de León, Spain
James Johnston, University of the West of Scotland, UK
Jorge Joo-Nagata, Metropolitan University of Science of Education, Chile
VI
Libor Juhaňák, Masaryk university, Czech Republic

Aleksandra Klašnja-Milićević, University of Novi Sad, Serbia
Bernd Kleimann, German Centre for Higher Education Research and Science Studies and
University of Kassel, Germany, Germany
Sevda Kucuk, Atatürk University, Turkey
Aleksandra Kulpa-Puczyńska, Cardinal Stefan Wyszyński University in Warsaw, Poland
Natalia Lajara Camilleri, Universitat Politècnica de València, Spain
Virginia Larraz, Universitat d’Andorra, Andorra
André Leblanc, Dalarna University, Sweden
Clotilde Lechuga, University of Malaga, Spain
Maria Limniou, University of Liverpool, UK
Rosa M. Llácer-Iglesias, Universitat Politècnica de València, Spain
María Lorenzo-Rial, Universidad de Vigo, Spain
Matthias Ludwig, Goethe-University, Germany
Nicolaas Luwes, Central University of Technology, South Africa
Elsa María Macías López, Universidad de Las Palmas de Gran Canaria, Spain
Brenda Mallinson, Rhodes University, South Africa
Sathiamoorthy Manoharan, University of Auckland, New Zealand
Mónica Martínez Gómez, Universidad Politécnica de Valencia, Spain
Alicia Martínez-Varea, La Fe University and Polytechnic Hospital, Spain
Konstantina Martzoukou, Robert Gordon University, Scotland
Inmaculada Méndez Mateo, University of Murcia, Spain
Marek Milosz, Lublin University of Technology, Poland
María del Mar Miralles Quirós, Universidad de Extremadura, Spain
Ulisses Miranda Azeiteiro, University of Aveiro, Portugal
Darlinda Moreira, Universidade Aberta, Portugal
Michelle Morgan, University of East London, UK
Nektarios Moumoutzis, Technical University of Crete, Greece
Estefanía Mourelle, Universidade da Coruña, Spain
Ana Isabel Muñoz Alcón, Catholic University Saint Teresa of Avila, Spain
Fabio Nascimbeni, European Training Foundation, Italy
Raquel Niclòs Corts, University of Valencia, Spain
Rosella Nicolini, Universitat Autònoma de Barcelona, Spain
Michael Niemetz, OTH Regensburg, Germany
Luis Nobre Pereira, University of Algarve, Portugal
Omid Noroozi, Wageningen University and Research, Netherlands
María Isabel Núñez-Peña, University of Barcelona, Spain
Martin O’Brien, University of Wollongong, Australia
Abeer Ali Okaz, Pharos University in Alexandria, Egypt
VII
Preface
Cesar Ortega-Sanchez, Curtin University, Australia

Viacheslav Osadchyi, Borys Grinchenko Kyiv University, Ukraine
Julieth E. Ospina-Delgado, Pontificia Universidad Javeriana Seccional Cali, Colombia
Miriam Ossevoort, University of Groningen, Netherlands
Gonzalo Pajares, University Complutense of Madrid, Spain
Candelas Paniagua Correas, University of Malaga, Spain
Cristina Pardo-Ballester, Iowa State University, USA
Elena Paunova-Hubenova, Inst. Information and Communication Technologies, Bulgaria
Dieter Pawelczak, UniBw München, Germany
Luís Pedro, University of Aveiro, Portugal
Maria Rosario Perello-Marín, Universitat Politecnica de Valencia, Spain
Uxio Perez-Rodriguez, University of Vigo, Spain
Anja Pfennig, HTW Berlin, Germany
Robert A Phillips, University of Manchester, UK
Julián Pinazo-Dallenbach, Universidad Europea de Valencia, Spain
Pablo Pinazo-Dallenbach, Universitat Politècnica de València, Spain
Luísa Helena Pinto, University of Porto, School of Economics, Portugal
Soner Polat, Kocaeli University, Turkey
Luis Porcuna-Enguix, Universitat Politècnica de València, Spain
Rubén Porcuna-Enguix, Universitat de València, Spain
María Isabel Pozzo, National Technological University, Argentina
Dimitri Prandner, Johannes Kepler University, Austria
Natalija Prokofjeva, Riga Technical University, Latvia
Fazlul K. Rabbanee, Curtin University, Australia
Sergio Rabellino, University of Torino, Italy
Martin Ramirez-Urquidy, Universidad Autónoma de Baja California, Mexico
Timothy Read, Universidad Nacional de Educación a Distancia, Spain
Emanuela Reale, CNR IRCRES, Italy
Mercedes Ruiz Lozano, Universidad Loyola Andalucía, Spain
Demetrios G. Sampson, University of Piraeus, Greece
Esther Sanabria Codesal, Universitat Politècnica de València, Spain
Juan Francisco Sánchez Pérez, Universidad Politécnica de Cartagena, Spain
Carmen M Sánchez-Arévalo, Universitat Politècnica de València, Spain
Yvonne Sedelmaier, SRH Wilhelm Löhe Hochschule, Germany
Jordi Segalàs, Universitat Politècnica de Catalunya, Spain
Elies Seguí-Mas, Universitat Politècnica de València, Spain
Begoña Serrano Arnáez, Universidad de Granada, Spain
Abdulhadi Shoufan, United Arab Emirates University, United Arab Emirates
Álvaro Suárez Sarmiento, Universidad de Las Palmas de Gran Canaria, Spain
VIII
Fátima Suleman, ISCTE – Instituto Universitário de Lisboa, Portugal

Valentina C. Tassone, Wageningen University, Netherlands
Jesús Tejada, University of Valencia, Spain
Francisca Tejedo-Romero, University of Castilla-La Mancha, Spain
Andreia Teles Vieira, Nova University Lisbon, Portugal
Sabu M. Thampi, Kerala University of Digital Sciences, Innovation and Technology, India
Susan Thomas, RMIT University and Okanagan College, Canada
Iman Tohidian, Allameh Tabataba’i University (ATU), Iran
Łukasz Tomczyk, Jagiellonian University, Poland
Guillermina Tormo-Carbó, Universitat Politècnica de València, Spain
Jani Ursin, University of Jyväskylä, Finland
Enric Valor, University of Valencia, Spain
Mercedes Varela-Losada, Universidade de Vigo, Spain
Jesús Vázquez Abad, Université de Montréal, Canada
Maria Verdeja, University of Oviedo, Spain
Henrique Vicente, University of Évora, Portugal
Lidia Vidal-Meliá, Universitat Jaume I, Spain
Cristina Vilaplana Prieto, University of Murcia, Spain
María Cinta Vincent Vela, Universitat Politècnica de València, Spain
Martin Wolf, FH Aachen, Germany
Jiun-Yu Wu, National Yang Ming Chiao Tung University, Taiwan
Nowreen Yasmin, Noakhali Science and Technology University, Bangladesh
Joanne Yip, The Hong Kong Polytechnic University, Hong Kong
Jorge Agustín Zapatero Ayuso, Universidad Complutense de Madrid, Spain
Katerina Zdravkova, Faculty of Computer Science and Engineering, North Macedonia
Daniela Zehetmeier, Lufthansa Aviation Training, Germany
LeChen Zhang, Stockholm University, Sweden
Yin Zhang, Kent State University, USA
Roza Zhussupova, Gumilyov Eurasian National University, Kazakhstan
Ana Zorio-Grima, Universitat de Valencia, Spain
External reviewers
Vitor Manuel Barrigão Gonçalves
Elena Borsetto
Alessio Di Paolo
Ana Estima
Jon Fanning
Juan Carlos Farah
Jose Figueiredo
IX
Preface
Michael Fowle
Tuba Gezer
Nomfundo Gladys Khoza
Willie Golden
Célio Gonçalo Marques
Patricia Gutierrez Moreno
Amilia Hasbullah
Gabor Kiss
Giray Kolcu
Andreas Körner
Moises Librado Gonzalez
Diana Lopez
Rebeca Macedo Mendes
Réka Matolay
Rosalina Pisco Costa
Laura Torres Caro
Augusto Zaide Macalalag
X
Index
Examining university student satisfaction and barriers to taking online remote exams ......... 1
Maria Aristeidou, Simon Cross, Klaus-Dieter Rossade, Carlton Wood, Patrizia Paci
Measuring Student Development Using Points ...................................................................... 9
Munir Mandviwalla, David Schuff, Laurel Miller, Manoj Chacko
University sustainability assessment and reporting: Preliminary findings from the
Italian context ....................................................................................................................... 17
Marta Fundoni, Gianfranco Pischedda
Creating the “Examination Kiosk”: Blended Assessment to Foster Self-Directed
Learning in Higher Education .............................................................................................. 27
Anne Jantos, Claudia Albrecht, Sebastian Pannasch, Vincent Zipper
Pro-f-quiz: increasing the PROductivity of Feedback through activating QUIZzes ............ 35
Kris Aerts, Wouter Groeneveld
Adverse effects of personalized automated feedback........................................................... 45
Jan Riezebos, N. Renting, R. van Ooijen, A.J. van der Vaart
Authentic Assessment in Higher Education: A Collaborative Approach ............................. 51
Angela Siobhan Wright
Mind the Gap: Factors Which Inhibit Supporting Student Teachers to Engage in
Action Research While on School Placement ...................................................................... 59
Maria Moore, Dermot O'Donovan, Pauline Logue
Ability of Spanish preservice teachers to differentiate between creative and
reproductive activities in the classroom ............................................................................... 67
Eva Izquierdo Sanchis, Joan Josep Solaz Portolés, Antonio Martín Ezpeleta,
Yolanda Echegoyen Sanz
XI
Special Needs Pre-Service Teachers Digital Competencies: An Exploratory Study at

the University of Foggia ....................................................................................................... 75
Piergiorgio Guarini, Guendalina Peconio
Advancing Internationalization Agenda Amidst the War in Ukraine: Kindness and
Trauma-Informed Teaching Project in Teacher Education .................................................. 83
Svitlana Anatoliivna Kuzmina, Sue Fan Foo, Olha Vasylivna Matviienko, Tamara
Volodymyrivna Glazunova
How University Teachers Can Support the Linking of Theory and Practice in Lesson
Debriefings with Student Teachers ...................................................................................... 91
Christine Ladehoff, Jan Pfetsch, Diemut Ophardt
An Empirical Survey of Employment and MTI Competence in China ................................ 99
Xian Wang, Wenxiang Si, Linxin Ge, Qiuhan Wang
Intercultural interaction modulating implicit attitudes towards disability and cultural
competence in higher education ......................................................................................... 107
Ludmila Tokarskaya Valerievna, Zijun Li
Developing Supply Chain Competencies Through Experiential Learning and Games ...... 113
Sandra Maycotte-Felkel, Heriberto Garcia-Reyes, Elena Isabel Victoria Quijano-
Dominguez
Students’ experience of Double Degree programmes: motivations and perceptions of
skill acquisition .................................................................................................................. 121
Elena Borsetto, Chiara Saccon
A Suggested Program for Developing First Year University Students’ Agency................ 129
Nevine Mahmoud Fayek El Souefi
Information skills instruction in Higher Education students using the 4C/ID model ......... 137
Consuelo Garcia, Esther Argelagos, Jesús Privado
A Systematic Review: Foundations for Online Course Design in Higher Education ........ 145
Jacelyn Smallwood Ramos
E-learning and economics: Knowledge dissemination through social networks................ 153
Roberto Ivan Fuentes Contreras, Moises Librado Gonzalez, Yadira Zulith Flores
Anaya
The digital platform for the Unite! Alliance: the metacampus ........................................... 161
Jesus Alcober, Farnaz Haji Mohammadali
XII
Index
Nurturing the human connection: Increasing student engagement and personal

connection in an asynchronous language course ................................................................ 171
Scott Despain, Jennifer A Despain
Critical analysis of the potential of social robotics in higher education for the
management of illness and bereavement ............................................................................ 179
Laia Riera, Francisco José Perales, Francisca Negre
Enhacing Personalization and Experiential Learning in Higher Education through
the VR Mobile Application, I’m IN - HKUST................................................................... 185
Kasina Wong
Student Response Systems: Enabler of active learning in a large class ............................. 195
Willie Golden
A Mixed Reality Laboratory for Developing Competencies in Control Engineering ........ 203
Manuel Navarro-Gutiérrez, Carlos Renato Vázquez, Alejandro Guajardo-Cuellar,
Noe Marcelo Yungaicela-Naula
How Students Manage Peer Feedback Through a Collaborative Activity in a CS1
Course ................................................................................................................................ 211
Géraldine Brieven, Laurent Leduc, Benoit Donnet
Facilitating international transdisciplinary collaboration in a virtual academic
exchange project................................................................................................................. 221
Martin Leibinger, Alexandra Regan Toland
Introducing a collaborative learning strategy in a hybrid and traditional laboratory
for undergraduate computer science students..................................................................... 229
Sam Ivan O'Neill, Aidan Mooney
School, University and Community Collaboration for Inclusivity: the Turin
Experience with the University Courses “Expert in the Processes of Inclusive
Education” and “Expert in Educational and Didactic Processes” ...................................... 237
Alessandro Monchietto, Paolo Bianchini, Cecilia Marchisio
Experiential learning using short-term global virtual team projects .................................. 245
Stephanie Swartz
Adopting Learning Circle approaches to equip academic staff for Community
Engaged Research and Learning practices ......................................................................... 255
Réka Matolay, Linde Moriau, Emma McKenna, Andrea Toarniczky, Judit Gáspár,
Márta Frigyik, Brecht Van der Schueren, Sinead McCann, Caroline McGowan,
Catherine Bates
XIII
Interdisciplinary Week in Game Design: A Learning Experience ..................................... 263

Inês Barbedo, Bárbara Barroso
“Strategy just isn’t like that”. A case study on the use of a coopetitive game to
experience the strategy roller coaster. ................................................................................ 273
Jon Fanning
Combining computer-based training, virtual, or augmented reality with peer teaching
in medical and biotechnological education ........................................................................ 279
Christian Hanshans, Moritz Mourice Rick Faust
They want to fly! – International students attitudes concerning the climate crisis and
their air travel behavior. ..................................................................................................... 287
Dimitri Prandner
A Project-Based Learning Approach For Engaging Undergraduate Students In UN
SDGs Using GIS ................................................................................................................ 295
Moustafa Ahmed Baraka
Take the Challenge: Compute the CO2e emissions of your Programming Course ............ 303
Maria Serna, Joaquim Gabarro, Maria Josep Blesa, Amalia Duch
Investigating Learners Perceptions of Completion and Certification in MOOCs .............. 311
Meghan Perdue
An Augmentation Framework for Efficiently Extracting Open Educational
Resources from Slideshows ............................................................................................... 321
Balthasar Teuscher, Zhouyi Xiong, Martin Werner
ChatGPT in Higher Education: The Good, The Bad, and The University ......................... 331
Marius Schönberger
ChatGPT in the Classroom: Friend or Foe? ....................................................................... 339
Josep Domenech
"Building" knowledge by creating manipulatives with the 3D printer: A course for
mathematics student teachers ............................................................................................. 349
Tim Läufer, Matthias Ludwig
The complexity of grading student work and the reconstruction of the meaning of
criterion-referenced assessment ......................................................................................... 357
Liao Liang
Reggio Childhood Studies PhD as a learning community ................................................. 367
Marcella Colacino, Roberta Mineo
XIV
Index
Financial literacy decision tree game: A system development exposé ............................... 377
Melany Lotter, Chioma Okoro
Fairness matters in higher education: Student classroom justice perceptions and
behavioral responses .......................................................................................................... 385
Rebecca M. Chory
University graduates enrolled in Higher VET in Spain: An upskilling or reskilling
choice? ............................................................................................................................... 393
Iván Diego Rodríguez, Juan P. Gamboa
Quality Management in Italian Universities: A Case Study in the University of
Cassino and Southern Lazio ............................................................................................... 401
Ilenia Colamatteo, Ilenia Bravo, Lucio Cappelli, Patrizia Papetti
Robust estimation method for the economic subsidies to educational institutions in
Chile ................................................................................................................................... 411
Valentina Ugalde, Sebastián Dávila-Gálvez, Óscar Vásquez
Evolution of academic dishonesty in computer science courses ........................................ 421
Katerina Zdravkova
Place-based Teaching amidst a Global Pandemic? ............................................................ 429
Barbara Cabezal Bruno, Daniela Bottjer-Wilson, Jennifer Engels
Surviving and Thriving in COVID..................................................................................... 437
Mary Jo Parker
A Post-Covid comparison of students’ usage of an online learning platform .................... 447
Robert A. Phillips
Digital multitasking during academic lectures: Did the Covid-19 lockdown change
the students’ behavior? ....................................................................................................... 455
Rebeca Macedo Mendes, Ana Isabel Veloso
The role of financial aid in college performance: The importance of class attendance,
aid amount and type of aid ................................................................................................. 465
Isabel de Sivatte, Patricia Gabaldón
Changing Higher Education Governance in Latin America: The Cases of Chile and
Ecuador .............................................................................................................................. 475
Mario Alarcón, José Joaquín Brunner
Resource orchestration and the higher education programme director .............................. 483
Hans Frederik
XV
Non-formal Faculty Development. Conceptual considerations and implementation in

practice ............................................................................................................................... 491
Birgit Hawelka, Regine Bachmaier
Teaching olfaction at the time of the 'sensual turn'. The case of Pierre Bénard ................. 499
Sandra Cadiou
Internal quality assurance systems in Namibian higher education: Stakeholder
perceptions and guidelines for enhancing the system ........................................................ 507
Marien Alet Graham, Toini Tuyeimo Ndapewoshali Angolo, Celeste Combrinck
Lecturer language: EMI students’ experiences on first- and second-cycle degrees ........... 517
Jane Helen Johnson, Mariangela Picciuolo
Learning Analytics Dashboard to Support Instructors: A Literature Review .................... 525
Dhatri Padakanti, Marcia Moraes
Which career should I choose?. Aplication of a pre-university vocational guidance
platform based on the Ikigai methodology ......................................................................... 533
Marta Retamosa, Ángel Millán, Jorge García-Unanue
Quest-based Gamification In A Software Development Lab Course: A Case Study ......... 541
Nuno H. Flores, Rui Pinto
Retail Design Education. Designing New and Reframed Learning Tools for
Experience-based Learning ................................................................................................ 549
Mariagiovanna Di Iorio, Alessandra Spagnoli
Analysis of the feasibility of investment projects in real assets with PBL: A very real
experience .......................................................................................................................... 559
Isabel Abinzano, Harold Bonilla, Pilar Corredor, Cristina Del Río, Elena Ferrer,
Ana González, José Manuel Mansilla, Beatriz Martínez, Luis Muga
A Framework for Developing Mathematical Tasks for Automatic Formative
Assessment in Higher Education........................................................................................ 567
Andreas Körner, Clara Horvath, Lana Medo
Insights from a transgender student in the EFL classroom: from individual
perspective to institutional change ..................................................................................... 577
Sergio Peñalver-Férez
English for Engineering: Intercultural formal letter writing............................................... 587
Begoña Bellés-Fortuño, Adrián Pla-Ángel
XVI
Index
The explicit teaching of vocabulary in French L2: theoretical models, teaching

practices and experimental itineraries ................................................................................ 595
Silvia Domenica Zollo
Mathematics in Economic and Business Science: how to reach the top without a
pathway .............................................................................................................................. 605
Yeray Rodríguez, Ana Munarriz, Maria Isabel Goicoechea, Maria Jesus Campion
The role of education in fostering entrepreneurial intentions among business
students............................................................................................................................... 615
Laila Cekule, Andrejs Cekuls, Margarita Dunska
The learning motives of business students and postponement ........................................... 623
Merle Küttim, Tarvo Niine, Jelena Hartšenko, Anna Kornijenko, Juhan Loorits
Online Repository for Facilitating Teaching and Learning of Undergraduate
Statistical Modeling Tools ................................................................................................. 631
Qing Wang, Xizhen Cai
The MILAGE LEARN+ app on Higher Education ............................................................ 639
Mauro Figueiredo, Custódia Fonseca, Paula Ventura, Marielba Zacarias, José
Inácio Rodrigues
Leveraging AI to Instruct Architecture Students on Circular Design Techniques and
Life Cycle Assessment ....................................................................................................... 647
Toktam B.Tabrizi, Ozgur Gocer, Arash Sadrieh, Anastasia Globa
When a test-taking strategy is better? An approach from the paradigm of scheduling
under explorable uncertainty. ............................................................................................. 657
Cristóbal Alfredo Mauricio, Sebastian Davila-Gálvez, Óscar Carlos Vasquez
Microcredentials: an opportunity towards the digital transformation ................................ 665
Fernando J. P. Caetano, Diogo G. Casanova, Darlinda M. P. Moreira
Fostering Cooperation Between Lecturers and TLC Staff to Improve Digital
Teaching. Experiences with a Pool of Discipline-specific Experts .................................... 675
Petra Panenka, Ann-Christine Hadam, Hans-Martin Pohl
(Digitally) transforming education in a large university .................................................... 683
Iris Peeters, Sylvia Grommen, Hans Tubbax
Enhancing Online Teaching: Addressing the Challenges Faced by Early-Career
Academics at Vietnam National University, Hanoi ........................................................... 691
Le Thi Thuong, Nghiem Xuan Huy, Tran Thi Hoai, Nguyen Thai Ba, Tuan Thu
Trang
XVII
Steps towards Enabling Health Professionals through Future Skills .................................. 701
Yvonne Sedelmaier, Dieter Landes
“Map ourselves through Digital Storytelling”: Pedagogical tool in the development
of self-knowledge skills on Advanced practice psychiatric-mental health nursing ............ 709
Carlos Laranjeira, João Gomes, Paula Carvalho, Ana Querido
Competency-based education advances in Higher Education in Health ............................ 717
Pau Martinez-bueso, Olga Velasco-Roldán, Iosune Salinas-Bueno, Inmaculada
Riquelme-Agulló, Elisa Bosch-Donate
Myth-busters at work: Development of engineering identity and employability
through student research..................................................................................................... 723
Jeroen Lievens
Development and piloting of a micro-credential programme in research ethics and
integrity leadership – an example from Estonia ................................................................. 733
Anu Tammeleht, Kertu Rajando, Margit Sutrop
Higher education student work placement and employability ........................................... 741
Badroonesha Aumjaud, Brinda Ramasawmy, Brigitte Marie Francoise Driver,
Deena Ramful-Baboolall
When Intercultural Education is Problematic: The Case of Russian as a Foreign
Language ............................................................................................................................ 749
Linda Torresin
Digital Pedagogy for the Present: An Artificial Intelligence Methodology for
Curriculum Development ................................................................................................... 757
Michelle Anne Rufrano, Jean-Ezra Yeung
New Model to Evaluate Values, Beliefs and Assumptions in the Recognition of
Prior Learning .................................................................................................................... 765
Phil O'Leary
Academic rankings as a source of metrics and benchmark tools for continuous
improvement at Técnico Lisboa ......................................................................................... 773
Carlos Carvalho Carvalho, João Fernandes
Undergraduates as researchers in Humanities and Social Sciences courses:
Articulating assessment by means of micro and macro cooperative and integrated
tasks.................................................................................................................................... 783
Mireia Trenchs-Parera, Andreana Pastena
XVIII
Index
Threshold Tests as a way to encourage long-term, self-regulated learners in

Engineering ........................................................................................................................ 793
Danica Solina, Chris Wong, Kate Crawford, Elaine Huber
The PASSt Project: Predictive Analytics and Simulation of Studies aimed at Quality
Management and Curriculum Planning .............................................................................. 801
Gabriel Wurzer, Shabnam Tauböck, Markus Reismann, Christian Marschnigg,
Sukrit Sharma, Karl Ledermüller, Julia Spörk, Maria Krakovsky
An innovative, technology-enhanced instructional approach to address the diverse
competencies of STEM students in math classes ............................................................... 809
Andreas Körner, Corinna Modiz
Development of scientific skills in Higher Education with a flipped classroom-
contest approach ................................................................................................................. 817
Miguel Garcia-Bosque, Carlos Sánchez-Azqueta, Concepción Aldea, Esther
Cascarosa, Santiago Celma
The essentials of science communication in a course engaging both for students and
professionals....................................................................................................................... 825
Maurizio Dabbicco, Franco Liuzzi, Sandra Lucente, Massimo Trotta
Teachers of Natural Sciences: some challenges and perspectives in Brazil ....................... 835
Cristina Leite
Designing co-curriculum experiential learning practice in permaculture for studying
science classics ................................................................................................................... 843
Ming Li
Relationship between time management and class attendance in university students:
clustering techniques for detection of profiles ................................................................... 851
Santiago Porras Alfonso, Julio César Puche Regaliza, Athénaïs Marianne Sauvée,
Paula Antón Maraña, Silvia Casado Yusta, Joaquín Antonio Pacheco Bonrostro
The engagement in university students: Preliminary psychometric analyses of the
Spanish version of the Engagement vs. Disaffection with Learning Scale. ....................... 861
Alba Aza, Georgina Guilera, Chuen Ann Chai, Juana Gómez-Benito, Estefania
Guerrero, Maite Barrios
Increasing student engagement with COIL Padlet ............................................................. 871
Barbara Therese Miller, Susan Göldi, Meng Hsien (Jenny) Lin
Positive effects of mindfulness practices on academic performance and well-being ......... 879
Katia G. Karadjova-Kozhuharova, Ruth L. Baker
XIX
Peer coaching in a leadership development program: The role of developmental

relationships ....................................................................................................................... 887
Laura Cortellazzo, Sara Bonesso, Fabrizio Gerli, Chaima Lazreg
The current HE classroom: Promoting new types of learning, executive function
processes and strategies to foster students’ motivation and academic success. ................. 895
Genny Villa
The value of the teaching quality innovation projects in the universities. The case of
Quid Sapienza .................................................................................................................... 909
Barbara Mazza, Elena Valentini, Cristina Sofia
The search for values as a didactic tool - An interdisciplinary perspective........................ 919
Andrea Fenice, Renzo Mocini
Design and manufacturing of a LIGHTBOARD - Combining the peer-to-peer idea
with project based teaching ................................................................................................ 929
Anja Pfennig
The impact of technology adoption in teaching and learning within ODeL ...................... 937
Thandiwe Bongani Radebe, Ronny Tebeta, Rendani Wilson Maladzhi
Moodle-based e-learning courses for introduction to critical thinking in a
multidisciplinary perspective ............................................................................................. 947
Jean-François Bodart, Anne-Frédérique Paul-Antoine, Aline Stevenoot, Corinne
Vézirian
Tailoring professional development to postgraduate students: It’s not the chicken,
it’s the egg .......................................................................................................................... 957
Marianne Elisabeth Savory
Academic dishonesty, essay mills, and Artificial Intelligence: rethinking assessment
strategies............................................................................................................................. 965
Simon Sweeney
Thinking outside the box - Virtual, intercultural labs in engineering education ................ 973
Julia Salzinger, Ika Kurniawati, Lars Abrahamczyk, Rüdiger Höffer
Students experience of blended teaching formats in pre-calculus courses ......................... 981
Domenico Brunetto, Giulia Bernardi, Caterina Bassi
Should we be afraid of open book exams? Our experience................................................ 989
Antoni Alegre-Martínez, María Isabel Martínez-Martínez, José Luis Alfonso-
Sanchez
XX
Index
Experiencing Third Spaces in between University and Society: Transdisciplinary

Learning Experiences in a Shopping Center ...................................................................... 997
Thorsten Philipp, Katarina Marej
Adaptation of the Interprofessional Collaborative Competency Attainment Scale for
Usage Across Professions ................................................................................................ 1005
Simon Schmitt, Monja Pohley, Aldin Strikovic, Eveline Wittmann
Story/No Story: a narrative design exercise for shared ways of seeing............................ 1013
John Stevens
Collaborative knowledge construction during computational lab activities in
Financial Mathematics ..................................................................................................... 1021
Adamaria Perrotta, Alice Barana, Marina Marchisio, Matteo Sacchet
Learning agroecology through the serious game SEGAE in an online lesson:
unveiling its impact on knowledge articulation................................................................ 1029
Mireille De Graeuwe, Lavena Van Cranenbroeck, Rémy Parent, Clément Serdobbel,
Yves Brostaux, Kevin Maréchal
Managing European interuniversity collaboration: A bottom-up approach to identify
digital education challenges from below .......................................................................... 1039
German Varas, Alvaro Pina Stranger, Gaelle Mobuchon
Deep Learning: a Study on Marine Renewable Energy and Sustainability Education
in an Irish Context ............................................................................................................ 1047
Pauline Anne Logue, Roisin Nash, Michael Walsh, Michael Faney, Dearcán Ó
Donnghaile
Plant and plan, care and grow. A hands-on exercice using the (inner) sustainable
development goals to teach research methodology to final year sociology students ....... 1055
Community intervention model: social entrepreneurship education as a strategy for a
sustainable development .................................................................................................. 1063
Yadira Zulith Flores Anaya, Martín Arturo Ramírez Urquidy, Roberto Iván Fuentes
Contreras
Exploring second language viewers’ use of cognitive strategies in learning Chinese
through multimedia learning resources with captions and social annotations.................. 1071
Shuzhou Lin, Ting Wang, Wei Wei, Jiahui Yang, Sara Cañizal Sardón
Diversity in (word) meaning: Reducing the risk of bias in foreign language
vocabulary teaching using prototype theory..................................................................... 1081
Anna Malena Pichler
XXI
Enhancing Language Self-Efficacy of EFL University Students through Experiential

Learning: A Study of the Learning League Project.......................................................... 1089
Fiona Sze Han Ho, Nick Wong, Angie Wing Chi Li, Lo Lau
Data-Driven Project-Based Learning in Specialized Translation Classes – The Case
of Comparable Corpora .................................................................................................... 1097
Katrin Herget, Teresa Alegre
Cultural heritage and its dissemination through linguistic and technical learning in
Higher Education ............................................................................................................. 1105
Daniela Gil-Salom, Eliseo Marzal-Calatayud, Damián López-Rodríguez
Institutional influencers and support for tutoring in a South African higher education
institution ......................................................................................................................... 1113
Chioma Sylvia Okoro, Nelson Bakali Phiri
Globalisation vs diversity in national languages in HE context: Case of Estonia ............ 1123
Peep Nemvalts, Triin Roosalu, Eve-Liis Roosmaa, Helena Lemendik
Bottom-up Curriculum Innovation through Grants for Lecturers .................................... 1131
Miriam Ossevoort, Robert Inklaar, Jan Riezebos
Designing a Metaverse for an Immersive Learning Experience ...................................... 1139
Mark Frydenberg, Shivam Ohri
Technology-enhanced learning: Cloud Computing to implement cooperation among
schools .............................................................................................................................. 1147
Ilaria Veronesi, Paolo Musmarra, Francesco Saverio Tortoriello
Inverting the classroom using on-demand lightboard micro lecture films (learning
glass) ................................................................................................................................ 1155
Anja Pfennig
Let’s Experience Learning in the Metaverse .................................................................... 1163
Eman AbuKhousa
Innovation in Education by Design Thinking .................................................................. 1171
Iris Schmidberger, Sven Wippermann
Combining integrated curriculum and project-based learning: A short film case
study from media and communication students ............................................................... 1179
Hibai Lopez-Gonzalez, Maria-Jose Masanet, Maddalena Fedele, Sandra Rius,
Jaume E. Vilaseca, Lydia Sánchez, Sergio Villanueva Baselga, Adrien Faure-
Carvallo, Anna Marquès, Jorge Franganillo, María Ángeles García Asensio
XXII
Index
Teamwork and student engagement during practical sessions in laboratories ................. 1187
Chalak Omar, Sarah Plumb
The Impact of Practical Training on Student Understanding of Plagiarism ..................... 1197
Mairéad Hogan
Evolution of a Continuous Assessment and Feedback Concept in a Computer
Science 101 Course .......................................................................................................... 1207
Dieter Pawelczak, Antje Neve
Black Student Achievement Plan Evaluation and Assessment ........................................ 1215
TrVel Lyons, Mabel Hernandez, Sy Doan, Shafiqa Ahmadi, Darnell Cole
Reflective Practice ePortfolios: A digital teaching tool to enhance third year BA
Culinary and Gastronomic Science students’ professional learning experiences ............. 1225
Clare Gilsenan, Marie English
Job Demands and Resources in the Work of University Teachers in Central and
Eastern Europe ................................................................................................................. 1233
Klára Kovács, Beáta Dobay, Szabolcs Halasi, Tamás Pinczés
From training practice in the professional world to university: characterization of
Vocational Training students to retain them in university studies ................................... 1241
Ana Munarriz, Yeray Rodriguez, Maria Isabel Goicoechea, Maria Jesus Campion
An Examination of Significant Factors Influencing College Student Employment
Cognition .......................................................................................................................... 1251
Yinggui Qiu, Qiaoguan Ye, Ningjie Wu, Sandy Chen, Hongjie Zhang, Xuan Zhang
Holey Moley Guacamole! Understanding Foreign Currency Exposure ........................... 1261
Valeria Martinez
Creating a Bridge to Post-Traditional Male Student Success at a Community College ... 1271
Theresa Marie Dereme
EPIC in Action, Measuring Entrepreneurial Competencies in Higher Education ............ 1279
Connie O'Regan, Neil Ferguson, Michelle Millar, Jenny Mullery, Natalie Walsh,
Tony Hall
The MEM project: 5 years of experiences, challenges, and outcomes of an
international double master-level degree ......................................................................... 1287
Roberto Montanari, Abdel-Malek Layek, Gino Ferretti, Alessandro Bernazzoli,
Eleonora Bottani, Andrea Volpi, Federico Solari, Letizia Tebaldi, Giorgia Casella,
Natalya Lysova, Michele Bocelli, Claudio Suppini, Lucia Orlandini, Monia Bertoli,
Andrea Pintus
XXIII
Multiple-Value Governance in (German) Higher Education: a New Paradigm? ............. 1297

Bernd Kleimann
Development of the “Complex Project” course in the Transport engineering
education .......................................................................................................................... 1305
Gábor Horváth
Digital identity and body identity: the mutation of the university environment in
pandemic times................................................................................................................. 1313
Lucia Pallonetto, Rosanna Perrone, Carmen Palumbo
Reflective practice to bridge the theory-gap practice in Human-Computer interaction
classes............................................................................................................................... 1321
Moretlo Tlale-Mkhize, Janet Liebenberg
The use of TAM in evaluating the effectiveness of network simulation tools in
Internet Technologies subject........................................................................................... 1331
Dina Moloja
Symbiosis between Learning Analytics and digital transformation ................................. 1341
Hans Tubbax, Iris Peeters
Six colours for inclusion: results of an explorative activity in a simplexity approach ..... 1349
Alessio Di Paolo, Iolanda Zollo, Michele Domenico Todino, Maurizio Sibilio
Cybersecurity education in European higher education institutions ................................ 1357
Mirva Salminen, Niko Candelin, Kaisa Cullen, Sari Latvanen, Marianne Lindroth,
Teemu Matilainen
The impact of digital platforms enhancement on global virtual teams’ engagement
across selected business schools ...................................................................................... 1365
Emil Velinov, Juergen Bleicher, Vincent Montenero
Evaluating the Impact of Strategies on Students’ Perceptions of Digital
Transformation – A Case Study of a Swedish Higher Education Institution ................... 1375
Henning Brink, Ellen Mårtensson, Carina Ström Hylén, Sven Packmohr
My First Six Months: Industrial Design Students Perceptions of their First Semester
Experience of Learner-Centred Design Studios ............................................................... 1383
William Dim, Helen McLean, Soumitri Varadarajan, Caroline Francis
Student perceptions of a remotely operated motor-driven generator in engineering
education .......................................................................................................................... 1391
Nicolaas Johannes Luwes, Leanri van Heerden, Walter Commerell
XXIV
Index
Ideas on Digitally Supported Individualization of Teaching and Learning for

Evolving Competency Requirements ............................................................................... 1399
Thomas Fuhrmann, Michael Niemetz
Using Technology Innovation and Blended Delivery for Student-centred Learning in
Large Undergraduate Classes ........................................................................................... 1407
Dominik Holzer, Alberto Pugnale
Postgraduate degree program in Social Business as a new knowledge tool in
entrepreneurship and social impact .................................................................................. 1417
Karina Isabel Salinas Solis, Natanael Ramírez Angulo, German Osorio Novela
Embeddedness of students with special educational needs in higher education .............. 1425
Anett Hrabéczy, Gabriella Pusztai
Black Digital Humanities in Interdisciplinary Undergraduate Teaching on Diversity,
Gender, and Sexuality ...................................................................................................... 1433
Magdalena J Zaborowska, Juan J Rodríguez Barrera
Audiovisual and Accessible Translation from a Transdisciplinary Insight:
Curriculum Design and Professional Practice for Intercultural Communication ............. 1441
Mercedes Enríquez-Aranda
Setting Learners up for Success: A Universal Design for Learning Approach to
Industry Placement Assessment ....................................................................................... 1451
Kate Dunne, James Corbett
The UNIVAC project: Implementing the user journey approach in accessibility
research at university........................................................................................................ 1459
Blanca Arias-Badia, Irene Hermosa-Ramírez, Mar Ogea, Mireia Oliver, Daniel
Segura, Ana Tamayo, Sergi Torner
About change: How institutionally aligning online pedagogy, design and technology
impacts higher education teachers .................................................................................... 1467
Paula Charbonneau-Gowdy, Caro Galdames
Psychometric Properties of the Spanish-Language Version of the Agentic
Engagement Scale (AES): A Preliminary Study .............................................................. 1477
Estefania Guerrero, Maite Barrios, Chuen Ann Chai, Juana Gómez-Benito, Alba
Aza, Georgina Guilera
Education 4.0: MiReBooks - Development and Usage of Mixed Reality for
Teaching Mining in Higher Education ............................................................................. 1485
Mariaelena Murphy, Manuel Labrador Ortega, Peter Moser, Susanne Feiel
XXV
DOI: http://dx.doi.org/10.4995/HEAd23.2023.16117
Examining university student satisfaction and barriers to taking

online remote exams
Maria Aristeidou1, Simon Cross1, Klaus-Dieter Rossade2, Carlton Wood3, Patrizia Paci3
1
Institute of Educational Technology, The Open University, UK, 2Faculty of Wellbeing,
Education and Lanugage Studies, The Open University, UK, 3Faculty of Science,
Technology, Engineering and Mathematics, The Open University, UK.
Abstract
Recent years have seen a surge in the popularity of online exams at
universities, due to the greater convenience and flexibility they offer both
students and institutions. Driven by the dearth of empirical data on distance
learning students' satisfaction levels and the difficulties they face when taking
online exams, a survey with 562 students at The Open University (UK) was
conducted to gain insights into their experiences with this type of exam.
Satisfaction was reported with the environment and exams, while work
commitments and technical difficulties presented the greatest barriers.
Gender, race and disability were also associated with different levels of
satisfaction and barriers. This study adds to the increasing number of studies
into online exams, demonstrating how this type of exam can still have a
substantial effect on students experienced in online learning systems and
technologies.
Keywords: Online exams; assessment; universities; Covid-19; online learning.

Editorial Universitat Politècnica de València 1
Examining university student satisfaction and barriers to taking online remote exams
1. Introduction
Online exams have become increasingly popular in universities in recent years, as they offer
students and institutions greater flexibility and convenience. Despite the growing popularity,
there is still limited research that addresses student satisfaction with online exams, and in
particular students in distance-learning universities. This paper examines the evidence on
student satisfaction and barriers to online exams at universities and explores how students in
distance learning universities engage with this new exams model.
Recent studies suggest that students are generally satisfied with online exams, as they allow
for more flexibility and convenience. For example, Bashir et al. (2019) found that students
are more satisfied with online exams as they are allowed more time to prepare, and the exams
were more accessible. However, the increased use of online exams in universities has raised
questions about students' potential challenges when taking such tests. Technological issues
are a major barrier for many students taking online exams. These include slow internet
connection, lack of access to the necessary technology, and lack of technical knowledge
(Barrot et al., 2021). In addition to technical challenges, students face psychological
difficulties, such as stress and anxiety, when taking online exams. For instance, Almossa
(2021) noted that students felt overwhelmed and anxious during online exams due to the lack
of face-to-face interaction with the instructor.
Recent studies have shown that online exams are more beneficial to certain student groups
than others. Dikmen (2022) examined the effect of online exams on female medical students
and found that they were more likely to experience anxiety and lower self-efficacy when
taking online exams than their male counterparts. This was because they felt they had less
control over the exams and less access to resources. Further, Tai et al. (2022) note that with
online exams, disabled students can take the exam in a location of their choosing, adjust the
font size and background colour, and have more time to read, write and review answers.
However, the literature also highlights some drawbacks of online exams for disabled
students, such as feelings of isolation (Tai et al., 2022). This can be particularly problematic
for those with disabilities that require more support. Finally, Tran and Reilly (2019) suggest
that online exams can create challenges for Black, Asian or Minority Ethnicity (BAME)
students due to their lower access to technology, less access to tutoring or other academic
supports, and lower levels of comfort and confidence with the technology.
This study explores students’ satisfaction and barriers to taking online exams at The Open
University (OU), an institution with a long tradition of distance learning and mature student
enrolment in the UK. The OU delivers its courses via virtual learning environments, online
tutorials and tutor groups. Pre-pandemic, 24% of the OU courses ended with a face-to-face
exam, which has now been replaced with mainly remote open book style exams (e.g.,
multiple choice questionnaires, essays, equations and numerical workings). These exam
2
interactions range from timed exams (2-4.5 hours) to 7-day submission windows. Motivated
by the current lack of an empirical basis for insights into distance learning students’
satisfaction and barriers to taking online exams, we explored the following research questions
(RQs):
1. To what extent are distance learning students’ satisfied with the different elements
of online exams?
2. What are distance learning students’ barriers to taking online exams?
3. Are there any significant differences among student groups (gender, race, disability)
in their satisfaction and barriers to taking online exams?
2. Methods
2.1. Development of survey instruments

The instruments used in this study consist of the ‘online exam satisfaction’ and ‘barriers to
taking online exams’, hereby mentioned as ‘satisfaction’ and ‘barrier’ instruments,
respectively. The ‘satisfaction’ instrument was developed as part of a bigger assessment
project by the Student Experience of Feedback, Assessment and Revision (SEFAR) team at
the OU (Cross, Whitelock & Mittelmeier, 2016) and was validated through an institutional
survey administered in 2015 (n = 281) and thirteen interviews. The instrument includes items
developed from constructs tested previously by others (e.g., Vattøy, Gamlem & Rogne, 2021)
and, in part, composed of newly created items designed to probe additional themes such as
anxiety (Falchikov & Boud, 2007) and exam preparedness (Payne & Brown, 2011).
The ‘barrier’ instrument was developed as part of a Covid-19 evaluation project at the OU
(Aristeidou & Cross, 2021). It consists of items identified in the literature as reasons for
students’ disrupted studies during the pandemic. Some examples of these items include a lack
of technical equipment and great demand for devices and the internet in the household (Barrot
et al., 2021), childcare and other caring responsibilities (Chirikov et al., 2020). Both
instruments were in the form of 5-item Likert scales, ranging from 1 (strongly disagree) to 5
(strongly agree), with an extra ‘not applicable’ (N/A) option.
2.2. Data collection

Data collected include undergraduate students self-reports on the online exams ‘satisfaction’
and ‘barriers’ survey instruments. Demographics and other student characteristics were
retrieved from the university database, such as gender (female or male), declared disability
(yes/no), and BAME (yes/no). The dataset was anonymised on the 30th of May 2022, prior
to initiating the process of data analysis. Ethical approval was obtained from the authors’
university ethics committee. The participants provided us with a written consent.
3
Table 1. Demographics of survey respondents
Background All With online exams experience

Characteristics (n = 562) (n = 190)
Gender
Female 336 90
Male 226 100
Declared disability
No 485 161
Yes 77 29
BAME
No 485 160
Yes 60 21
Unknown 17 9
2.3. Data analysis

Visualisations of the Likert scales were used to describe students’s satisfaction with different
aspects of online exams (RQ1) and the barriers to taking online exams (RQ2). Percentages
of participants were presented for each frequency selection, and items were presented in order
of negative impact on the frequency of agreeing to a particular statement. Data presented in
RQ1 include only the students who had an experience of taking online exams (n = 190), while
data presented in RQ2 includes all survey participants (n = 562).
Then, to determine how gender, disability, race, previous qualification and qualification
intention relate to each ‘satisfaction’ or ‘barrier’ statement (RQ3), chi-square tests were
performed. For the tests, dichotomous variables were used for each statement, in which
option 1 included all the ‘agree’ and ‘strongly agree’ responses to the statement and 0 all the
‘disagree’ and ‘strongly disagree’. N/A and ‘neither agree nor disagree’ were excluded from
the chi-square test analysis. An alpha level of .05 was used for all the analyses.
3. Findings
3.1. Satisfaction with online exams

The survey respondents who had an experience with online exams (n = 190) were mainly
found to agree that they were able to find a quiet space to take the exam (86%), they were
satisfied with the quality of that space (85%), the type of exam allowed them to demonstrate
their learning (78%), thought that the exam questions were clear (77%), felt a sense of
achievement upon completion of the exam (76%), were satisfied with the mark received
(72%), felt prepared just before starting the exam (61%), felt anxious during the exam (58%),
thought that the exam was harder than they expected (48%), and enjoyed the exam (31%)
(Figure 1).
4
Figure 1. Student satisfaction with different areas of online exams, presented in ascending order of agreement
with each statement
Findings from examining student experience during and with the online exams and among
students in different groups indicated that BAME students were less likely to feel prepared
immediately before starting the exam (χ2 = 4.53, df = 1, p = 0.03) than those of a white
background. There were also borderline differences between students who declared disability
and those who did not (χ2 = 3.74, df =1 , p = 0.053). Gender-wise, female students were less
likely to report that the exam questions allowed them to demonstrate what they had learnt (χ2
= 4.32, df = 1, p = 0.04). Moreover, female students (χ2 = 6.48, df = 1, p = 0.01) and students
with declared disabilities (χ2 = 5.53, df =1 , p = 0.02) were less likely to be satisfied with the
mark they received. Further, students with declared disabilities (χ2 = 8.72, df =1, p = 0.003)
were less likely to be satisfied with the quality of the home space they used.
There were also no significant differences among different student groups as to whether (a)
the exam questions were clear, (b) the exam to be harder than they expected, (c) they felt
anxious when doing the exam, (d) they felt a sense of achievement when completing the
exam and (e) they were able to find a quiet space to take the exam.
3.2. Barriers to taking online exams

The survey respondents, both those with and without an experience with online exams (n =
562), selected employment commitments as the most significant barrier to taking online
exams at home – chosen by about one in three students (30%) (Figure 2). This barrier was
followed by a lack of confidence in dealing with technical difficulties, expressed by 29% of
the survey respondents, when dealing with potential technical difficulties during the online
assessment. The third most chosen challenge, selected by 28% of the respondents, was the
reliability or quality of the internet connection. Other less selected barriers involved a lack
of quiet working space (20%), childcare responsibilities (17%), mental health issues (13%),
5
confidence in setting up and using technologies (13%), access to an internet connection

(10%), and other caring responsibilities (10%).
Figure 2. Student barriers to taking online exams, presented in ascending order of agreement with each statement
Findings from examining barriers to taking online exams among students in different groups
indicated that female students are more likely to report a lack of confidence in setting up and
using technology (χ2 = 10.00, df = 1, p < 0.01) and dealing with potential technical difficulties
during assessment (χ2 = 11.07, df = 1, p < 0.01). In addition, female students (χ2 = 4.51, df
= 1, p = 0.03) were more likely to choose mental health issues as a barrier to taking online
exams at home. Students with declared disabilities (χ2 = 9.01, df = 1, p < 0.01) are more
likely to choose other caring responsibilities as a barrier to taking online exams at
home. Finally, students with declared disabilities (χ2 = 35.32, df = 1, p < 0.01) were more
likely to choose physical disability as a barrier to taking online exams at home.
There were no differences among student groups in the following statements: employment
commitments or constraints, reliability or quality of internet connections, lack of quiet
working space, childcare responsibilities, access to an internet connection, and household
competition for access to devices. Furthermore, there were no differences between students
of different races and faculties.
4. Discussion
Distance learning university students at the OU demonstrated, like traditional university
students (e.g., Bashir et al., 2019), high levels of satisfaction with the environment in which
they took the exam and the exam procedure, including the type of exams and the clarity of
the questions. However, similarly to other studies (e.g., Almossa, 2021), they felt anxious
during the exam and thought it was harder than expected. Barriers to taking exams involved
various areas, with the most affected being employment commitments –expected in a
university with students of a higher average age. Likewise other studies (e.g., Barrot et al.,
6
2021), technical difficulties also ranked high, with barriers related to dealing with unexpected
software and internet issues. Other important barriers related to finding a suitable
environment for the exam and arranging childcare – which can relate again to the fact the
students have different responsibilities compared to the average university student.
Our findings align well with previous studies wherein online exams are less beneficial for
specific student groups than others. In line with Dikmen (2022), female students were found
to experience more issues than their male counterparts, including more lack of confidence in
technology, and mental health issues. They also thought the type of exam impacted their
performance. We have also verified that students with declared disabilities face some
drawbacks when taking online exams, including lack of access to assistive technologies or
mental health issues (e.g., Tai et al., 2022). However, our results go beyond previous reports,
showing how their environment and caring responsibilities may also be barriers. A similar
conclusion to Tran and Reilly (2019)’s was reached on BAME’s students’ confidence with
online exams, with findings in the current study concerning their feelings of unpreparedness.
Our work adds to a growing corpus of research on online exams, showing how this type of
exam can still impact students familiar with online learning formats and technologies.
Overall, the literature and our findings suggest that online exams can offer several advantages
over traditional exams. However, it is also important for universities to recognise the
potential drawbacks, and carefully consider the needs of students when planning online
exams, ensuring they are adapted to meet the different student groups’ requirements.
In addition to the greater convenience and flexibility, online exams have also provided a
pragmatic and workable solution to challenges as a result from crises such as the recent global
pandemic. In this context, online exams can, as indeed they did during 2020 and 2021,
provide means to secure academic business continuity and student success. The findings from
this and similar studies can give direction to universities about which processes and policies
they need to put in place. The barriers to student satisfaction identified here seem, from an
institutional perspective, perfectly manageable and do not require unaffordable or not yet
existing high tech solutions: acknowledging employment commitments can be addressed
through assessment load and scheduling, anxieties about technology can be lowered through
training and responsive support services. There is, in fact, little in these findings that could
not be implemented at speed: it would therefore all the sooner address the barriers, secure
confidence in the robustness of assessment, and offer a better service overall to students.
References
Aristeidou, M., & Cross, S. (2021). Disrupted distance learning: the impact of Covid-19 on
study habits of distance learning university students. Open Learning: The Journal of
Open, Distance and e-Learning, 36(3), 263-282.
7
Almossa, S. Y. (2021). University students’ perspectives toward learning and assessment

during COVID-19. Education and Information Technologies, 26(6), 7163-7181.
Barrot, J. S., Llenares, I. I., & Del Rosario, L. S. (2021). Students' online learning challenges
during the pandemic and how they cope with them: The case of the Philippines. Education
and information technologies, 26(6), 7321–7338.
Bashir, A., Bashir, S., Rana, K., Lambert, P., & Vernallis, A. (2021). Post-COVID-19
adaptations; the shifts towards online learning, hybrid course delivery and the
implications for biosciences courses in the higher education setting. Frontiers in
Education, 6, 310. Frontiers.
Chirikov, I., Soria, K. M., Horgos, B., & Org, E. (2020). Undergraduate and graduate
students’ mental health during the COVID-19 pandemic. (SERU Consortium Report).
Center for Studies in Higher Education, UC Berkeley.
https://escholarship.org/uc/item/80k5d5hw
Cross, S., Whitelock, D., & Mittelmeier, J. (2016). Does the quality and quantity of exam
revision impact on student satisfaction and performance in the exam itself?: Perspectives
from undergraduate distance learners. EDULEARN16 Proceedings, 5052–5061.
Dikmen, M. (2022). Test anxiety in online exams: scale development and validity. Current
Psychology, 1-13.
Falchikov, N., and D. Boud (2007). Assessment and Emotion, the Impact of Being Assessed.
In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education, Learning
for the Longer Term (pp. 144–156). Routledge.
Payne, E. and Brown, G. (2011). Communication and practice with examination criteria.
Does this influence performance in examinations? Assessment and Evaluation in Higher
Education, 36(6), 619-626.
Tai, J., Mahoney, P., Ajjawi, R., Bearman, M., Dargusch, J., Dracup, M., & Harris, L. (2022).
How are examinations inclusive for students with disabilities in higher education? A
sociomaterial analysis. Assessment & Evaluation in Higher Education, 1–13.
Tran, D., & Reilly, D. (2019). Extending conversations about what is an inclusive curriculum.
Educational Developments, 20(4), 23-25.
Vattøy. K.-D., Gamlem, S.M. & Rogne, W.M. (2021). Examining students’ feedback
engagement and assessment experiences: a mixed study. Studies in Higher Education,
46(11), 2325-2337.
8
Measuring student development using points

Department of Management Information Systems, Fox School of Business, Temple,
University, Philadelphia, USA.
Abstract
While higher education is effective at measuring the acquisition of knowledge,
it is less successful in quantifying other types of learning such as learning to
know, live together, and to be. This is a problem because it makes it difficult
for institutions to implement and sustain student development programs. In this
paper, we describe how to measure student development using points. Using
data from 623 students over 7.5 years, we show how our points system was
used to improve student development over time by focusing on the quantity,
variety, quality, and distribution of development activities. Based on our
findings, we recommend avenues for additional research.
Keywords: Higher education; student development; measurement.

1. Introduction
Higher education is typically associated with physical, cognitive, and personality
development of young adults (Kail and Cavanaugh, 2018). Yet, a major United Nations study
concluded that “formal education systems tend to emphasize the acquisition of knowledge to
the detriment of other types of learning; but it is vital now to conceive education in a more
encompassing fashion” (Delors et al. 1996, p. 37). However, higher education still primarily
focuses on one aspect of human development – academics – as measured by GPA.
Delors et al. (1996) place human development at the core of higher education, the principle
of which has been widely adopted by universities (Kilpatrick 2019). We interpret this to mean
going beyond pedagogy (e.g., teaching tips, GPA), scale (e.g., MOOCs), and reach (e.g.,
distance learning) to focus on value generation activities that develop students. Student
development has always been a major goal of universities (Kilpatrick 2019), and availability
of development activities is typically included in institutional assessment (Zilvinskis et al.
2017). Yet, to date there are no systematic and scalable measures of student development.
Therefore, our research question is: How can we measure student development? This
question is important because for student development to become a core activity and move
beyond a nice-to-have but difficult-to-attain goal, it must be measurable.
2. Student development
According to Delors et al. (1996), learning can be broadly conceptualized as: learning to do
- the acquisition of skills and competencies, learning to know - the ability to think and
integrate new information, learning to live together - understanding others, managing
conflicts, and learning to be - developing one’s personality and judgment. We adopt this
broad view of student development. Student development is also related to forming an
identity. Chickering and Reisser (1993) conceptualize establishing identity by proposing
seven vectors that include competence, emotions, autonomy and interdependence,
interpersonal relationships, identity, purpose, and integrity. Student development and identity
relate to employability, which Yorke and Knight (2006, p. 8) define as “A set of achievements
– skills, understandings, and personal attributes – that make individuals more likely to gain
employment and be successful in their chosen occupations, which benefits themselves, the
workforce, the community, and the economy.”
Another important stream of research focuses on student involvement (also termed
engagement), which is “the amount of physical and psychological energy that the student
devotes to the academic experience” (Astin 1984, p. 518). According to Astin (1984), student
learning and personal development are a function of the quality and quantity of student
involvement. Today, involvement is typically seen as an indicator of institutional excellence
as well as the effectiveness of education policy and practice (Axelson and Flick 2011).
10
Involvement includes traditional academic activities such as studying - learning to do - as

well as experiences such as discovery, engagement, and feedback of ideas, cultures, places,
and others that relate to Delors et al.’s other forms of learning. Zilvinskis et al. (2017) show
that these different forms of student engagement increase the Delors et al. expanded view of
learning. While the above research is important for understanding the meaning and
importance of student development, it provides little guidance on measuring, implementing,
and weighing different forms of development.
Integrating the above literature, we define student development as the achievement of
Delors et al. (1996) outcomes of learning to do, know, live together, and be which establish
identity and increase employability. To establish a boundary and to acknowledge its
formative and emergent nature, we conceptualize development as a longitudinal process that
involves internal and external activities of varying depth and value, which in turn
differentially influence aspects of the above outcomes.
3. Measuring student development

Engagement, involvement, and development activities are currently measured with survey
instruments upon graduation (Astin 1984; Kuh 2001), while others have explored proxy
measures such as learning management system (LMS) log files. These are cross-sectional
perceptual measures. We instead follow a process view and operationalize student
development as a series of activities accomplished at different times measured by points. The
point value of each activity is associated with its development value.
Points are common in gamified systems (Liu et al. 2017). They are tangible and provide
feedback, which in turn influences motivation by fostering competence, relatedness, and
autonomy (Ryan and Deci 2000). Point-based rewards can also produce recurrent behavior
(Liu et al. 2017). Points communicate recognition and a sense of accomplishment for activity
and task completion, and the quantification enables social comparison and competition.
Points are also more objective than the perceptual surveys used in the involvement literature
(Astin 1984; Kuh 2001), and related survey measures of acceptance, usage, and satisfaction.
Overall, points work well as an individual measure that can quantify the number and weight
of development activities. For example, if a student completes an internship, they might be
awarded 300 points, while attending a talk by a speaker might yield only 25 points -
reflecting the differential developmental value of each respective acttivy. The total points
that a student has earned per term (points per term) and over the course of their degree
program (total points per student) provide a summary of an individual student’s development.
The average of total points per student provides an aggregate measure of performance, at the
academic department (our focus) or insititutional university level.
11
The above measures are important because they focus on the quantity of student development
activities, but they are not sufficient. Point totals do not address variety - the number of
different activities (e.g., completing a project and an internship vs. attending two lectures),
quality – the intrinsic development value of different activities (e.g., leadership vs. attendance
activities), and distribution (e.g., all the activities in one term vs. spread out across multiple
terms). For example, a student earning 500 points may seem very good, but perhaps they
earned those points by completing only a few, and potentially similar, high-value activities
in their last term before graduation. Addressing variety, quality, and distribution is important
to ensure that development is holistic in addressing different aspects of learning (e.g.,
learning to live together vs. learning to know). Further, our goal is to measure development
as a process, in which students develop over time by engaging in a variety of high-quality
activities.
4. A field experiment in a living lab

We implemented points as a student development measure through a novel web-based self-
service technology platform. The platform manages and records the points earned by each
student for activities such as experiences (e.g., internships), career awareness (e.g., career
fairs), leadership (e.g., officer position in club), enrichment (e.g., study abroad,
competitions), communication (e.g., conferences, social activities), team work (e.g.,
community service, team projects), and workplace readiness (e.g., mentoring, site visits).
The authors’ home department served as a living lab to implement and study the use of points
as a measure for development. Living labs are a real world test and experimentation
environment which enable co-creation of innovation among stakeholders and creators. Living
labs offer incremental and visible improvements that reduce fear of failure and co-opt sources
of resistance into co-designers (Hyysalo and Hakkarainen 2014; Mandviwalla et al. 2008) by
exposing stakeholders to successive prototypes (Mandviwalla 2015). The stakeholders
included students, friends and family, faculty, staff, college and university administration,
and employers. The stakeholders participated in the development of each iteration of the
platform through feedback and use.
The project evolved considerably over a decade of incremental improvements involving more
than 7500+ students. To the stakeholders, we positioned student development as a program
of learning that complements but is separate from academics in which students were expected
to gain 1000 points prior to graduation. Over time, we integrated variety, quality, and
distribution into the system as follows. Variety – we placed restrictions on how many times
an activity can be repeated for credit, promoted activities offered across the university and
the local community, and adjusted point values so that a student can only meet point
expectations by participating in more than a few activities. We also allowed students to
12
propose new activities. Quality - We adjusted the point value of activities to reflect their
development potential (e.g., internship receives more points than attending a lecture).
Distribution - We restricted how many times a student can get credit for an activity (e.g.,
receive points for attending a lecture only once a term).
5. Results
We collected data from 623 students over a 7.5-year period of using the platform. For each
student, we recorded the points they earned for the eight consecutive terms leading up to their
graduation as well as the total number of points they earned upon graduation. Table 1 shows
the percentage of students attaining point levels in the years following the platform’s
implementation, in which each year includes the group scheduled to graduate that year (only).
As the program was designed for students to achieve 1,000 points before graduation, we
chose point cutoffs to designate low, average, above average, advanced, and very advanced
achievers. The results show that over time a greater percentage of students achieve higher
point levels with 100% achieving the 1000-point expectation by the fifth year. Close to 100%
of graduating seniors achieved at least 1,000 points by year 4, compared to a little over 9%
in year 1. Over the course of the five years, the percentage above 1,400 points grew from
1.3% to 12.6%.
Table 1. Percentages of students and points levels
Point levels/Year 1 2 3 4 5
Poor (0-300) 76.0% 6.4% 0.0% 2.0% 0.0%
Low (300-999) 14.7% 7.3% 1.1% 0.0% 0.0%
Average (1000-1200) 6.7% 49.1% 59.8% 65.3% 80.0%
Advanced (1201-1400) 1.3% 10.0% 7.6% 8.2% 7.4%
Very Advanced (1400+) 1.3% 27.3% 31.5% 24.5% 12.6%
The results suggest that (a) students will, over time, adopt and embrace new measures of
development, and (b) that once you start measuring an activity, perceptions change, and
development becomes more frequent. Further, given the structures we put in place, students
engaged in a variety of high quality development activities. We measured variety by
reviewing the average number of activities. In addition, we adjusted the point values so that
to meet expectations students had to participate in high value point activities (quality).
However, it was unclear if the activities were distributed over time or bunched together.
Figure 1 shows activity across eight terms (T1 – T8) which is about 2.5 years. The data was
standardized to enable comparison so that for a student graduating in any year, T1 is when
13
they started participating (eight terms in the past). The later years (3, 4, and 5) show more
activity with less variance compared to the earlier years. The figure suggests that compared
to year 1, year 5 graduates are more developed because they completed more activities each
term at a consistent level, reflected by the lower variance.
4.5
4.0
3.5
3.0
Activity
2.5
2.0
1.5
1.0
0.5
0.0
T1 T2 T3 T4 T5 T6 T7 T8
Term
Year 1 Year 2 Year 3 Year 4 Year 5
Figure 1: Development activity
While illuminating, the above analysis requires qualitative interpretation of graphs. We drew
inspiration from Shannon’s diversity index (Shannon 1948), which has been used to assess
species diversity in biology and ecology (Spellerberg and Fedor 2003). We developed a novel
application of the Shannon Diversity Index to measure Development Distribution (DD). DD
is expressed as: − ∑𝑁 𝑖=1 𝑝𝑖 log 2 𝑝𝑖 , where N is the number of terms and 𝑝𝑖 is the proportion of
activities completed in a particular term. DD is useful because summing the number of
activities only measures what is termed richness, as opposed to diversity, which is the
function of the relative frequency of different species (Keylock 2005, p. 203). In our case,
DD calculates the activity distribution across terms, i.e., the development process. So that
higher DD implies activities are more evenly spread out across more terms. For example,
consider six activities completed over five terms. Completing two activities every alternative
term across 5 terms (2,0,2,0,2) generates a DD score of 1.585. In contrast completing two
activities in the first term and one in each term thereafter (2,1,1,1,1) generates a DD score of
2.251. Another scenario where activities are concentrated in terms 2 through 4 (0,1,4,1,0)
generates an DD score of only 1.252. DD is not affected, however, by the total number of
activities if the distribution remains the same. Consider a scenario where the distribution of
14
activities is (3,0,3,0,3). Even though there are now nine activities instead of six, the DD score
remains the same (1.585). This means that DD is a more useful measure when combined with
total points earned. In sum, DD rewards better distribution, meaning more time to absorb,
reflect on, apply, and experience the benefits from each activity as well as apply what was
learned to the next activity (e.g., an officer position one term, followed by taking the lead
role in a competition in the following term).
Table 2. DD scores over five year period
Year DD
1 1.57
2 1.84
3 1.97
4 2.09
5 2.18
Table 2 summarizes DD scores of 623 graduates participating in 6,474 activities totaling

617,558 points across five years, in which the minimum possible score is 0 and 4.31 is the
theoretical maximum. The gradual increase in DD scores suggests that students were
completing activities that were more evenly distributed across terms. This result matches
what is observable in figure 1, providing an intuitive validation of the efficacy of the measure.
DD needs to be further improved since it treats all activities as equal, even though it is likely
that at different times, for different students, the importance of learning to do, know, live,
and be will vary. In addition, we don’t know which factors motivate and influence the
trajectory of student development. Finally, we also need additional research on how to
benchmark points and DD scores at the individual and institutional level.
6. Conclusion
In this study, we show how we developed a measure of student development in a living lab
using a technology platform. For student development to move from the nice-to-have to the
essential activity conceptualized by Delors et al. (1996), it must be measurable. The results
show that points can serve as a measure of student development, including the quantity,
variety, quality, and distribution of development activites. Overall, as far as we know, we are
the first to measure student development across time using direct rather than perceptual or
proxy measures.
References
Astin, A. W. (1984). Student involvement: A developmental theory for higher
education. Journal of college student personnel, 25(4), 297-308.
15
Axelson, R. D., & Flick, A. (2010). Defining student engagement. Change: The magazine of
higher learning, 43(1), 38-43.
Chickering, A. W., & Reisser, L. (1993). Education and Identity. The Jossey-Bass Higher
and Adult Education Series. Jossey-Bass Inc., Publishers, 350 Sansome St., San
Francisco, CA 94104.
Delors, J., Amagi, I., Carneiro, R., Chung, F., Geremek, B., Gorham, W., ... & Suhr, M. W.
(1996). Learning. The treasure Within. Paris.
Hyysalo, S., & Hakkarainen, L. (2014). What difference does a living lab make? Comparing
two health technology innovation projects. CoDesign, 10(3-4), 191-208.
Kail, R. V., & Cavanaugh, J. C. (2018). Human development: A life-span view. Cengage
Learning.
Keylock, C. J. (2005). Simpson diversity and the Shannon–Wiener index as special cases of
a generalized entropy. Oikos, 109(1), 203-207.
Kilpatrick, S. W. (2019). Visualizing the Future: Work-Integrated Learning and the
Psychosocial and Professional Development of Undergraduates.
Kuh, G. D. (2001). The National Survey of Student Engagement: Conceptual framework and
overview of psychometric properties.
Liu, D., Santhanam, R., & Webster, J. (2017). Toward Meaningful Engagement: A
Framework for Design and Research of Gamified Information Systems. MIS Quarterly,
41(4).
Mandviwalla, M. (2015). Generating and Justifying Design Theory. Journal of the
Association for Information Systems, 16(5), 314.
Mandviwalla, M., Patnayakuni, R., & Schuff, D. (2008). Improving the peer review process
with information technology. Decision Support Systems, 46(1), 29-40.
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic
motivation, social development, and well-being. American psychologist, 55(1), 68.
Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical
journal, 27(3), 379-423.
Yorke, M., & Knight, P. T. (2006). Embedding employability into the curriculum (Vol. 3).
York: Higher Education Academy.
Zilvinskis, I. F., & Fedor, P. J. (2003). A tribute to Claude Shannon (1916–2001) and a plea
for more rigorous use of species richness, species diversity and the ‘Shannon–
Wiener’Index. Global ecology and biogeography, 12(3), 177-179.
Zilvinskis, J., Masseria, A. A., & Pike, G. R. (2017). Student engagement and student
learning: Examining the convergent and discriminant validity of the revised national
survey of student engagement. Research in Higher Education, 58(8), 880-903.
16
University sustainability assessment and reporting: preliminary

findings from the Italian context
Marta Fundoni1, Gianfranco Pischedda2
1
Department of Economics and Business, University of Sassari, Italy, 2Department of
Economics and Business, University of Sassari, Italy.
Abstract
With universities playing a key role in creating and promoting sustainable
development, they are inevitably called upon to report to stakeholders their
active commitment to facing social and environmental issues using
comprehensive and clear reporting tools. GRI Standards provide guidelines to
help write these reports, yet these can lead universities to prepare
heterogeneous documents that are difficult to understand and compare in
terms of both information and results. Considering Italian state universities,
this paper examines the materiality matrix, a tool used to disclose social,
environmental and economic efforts toward the stakeholders. Preliminary
results seem to show that somehow universities have misunderstood the GRI
2016 guidelines concerning the materiality matrix. The paper concludes by
outlining the limitations of the study and offering suggestions for future
research.
Keywords: Sustainability; reporting; Italian higher education system; GRI
standards; materiality matrix.

University sustainability assessment and reporting: Preliminary findings from the Italian context
1. Introduction
In the last decades, sustainable development has attracted the attention of organisations, both
private and public, including educational institutions at all levels. In this context, the
relevance of higher education institutions (HEIs) has been widely recognised as meaningful
in that they act as drivers of sustainable change (Barth and Rieckmann 2012; Disterheft et al.
2013; Lozano et al. 2013). In particular, universities not only have the duty to teach and
transmit knowledge, but they must also act as responsible institutions by advancing research
to stimulate sustainable development and change (Barth and Michelsen 2013). Some
initiatives to foster universities’ sustainable activities have taken hold, such as the new global
university rankings aimed at assessing universities for their contribution to sustainable
development goals (e.g., The Times Higher Education Impact Rankings). Moreover, some
governments have developed sustainability indicators to determine their funding lines to
motivate universities to act sustainably. HEIs are thus called upon to improve the dialogue
with government and society by reporting and disclosing their sustainable activities and
efforts that foster sustainable development. However, the lack of standardised standards has
led to the production of different documents across HEIs, making it difficult to clearly
disclose or compare information (Fiorani and Di Gerio 2022). Starting from these
assumptions, this study attempts to analyse sustainable reports in the Italian higher education
system (IHES) by investigating if and how state universities are adopting the so-called
materiality matrix and how the standards are being applied. A materiality matrix is a tool that
considers key aspects and indicators that reflect an organisation’s social, environmental, and
economic impacts, or those that influence stakeholder decisions, thereby identifying material
issues that deserve to be included in the report (Hsu, Lee, and Chao 2013).
The paper is organised as follows: section 2 presents a brief literature review on sustainability
reporting in HEIs; section 3 explains the methodological approach; section 4 describes the
findings; and the conclusions are provided in section 5.
2. Literature review
Due to the necessity to act sustainably, HEIs play an essential role in providing and promoting
helpful pathways to sustainable action. HEIs, indeed, have a twofold responsibility:
delivering trustful results that can help both citizens and firms on the path to sustainability,
and being reliable institutions and points of reference when it comes to representing virtuous
actions in this sense and stimulating organisational change (Ceulemans, Lozano, and Alonso-
Almeida 2015). Especially for the latter, universities should externalise their institutional role
by communicating a coherent and reliable image that represents all the virtues expected of
them. In this sense, reporting activities, intended as means and measures to collect, process,
and present information, help organisations reduce the complexity of information by making
18
data accessible and understandable in a simplified way for specific target groups and
stakeholders. From the sustainability viewpoint, reporting is an accounting practice that aims
to communicate all the efforts that an organisation makes in terms of the environment,
society, and economy, tracking the impacts and actions related to these aspects, and engaging
stakeholders during the investigation process. From a legal perspective, countries are
developing numerous directives and laws to provide clear and trustful guidance. It is widely
accepted that financial and non-financial information have the same importance and must be
subject to regulations, resulting in the double-materiality concept. For example, the European
Union recently moved from the NFRD (Non-Financial Reporting Directive) (Directive
2014/95/EU) to the CSRD (Corporate Sustainability Reporting Directive) (Directive (EU)
2022/2464). However, the path to providing clear instructions is long and diversified, and
each institution can use one of many available standards (e.g., from the Global Reporting
Initiative (GRI) or the International Accounting Standards Board (IASB)) to map its
sustainability actions. GRI Standards are the most widely adopted worldwide, but despite
their importance, they do not provide specific guidelines for HEIs and more generally for
other types of organizations. This might explain why sustainability reporting is still at an
early stage in this field (Herzner and Stucken 2020). For this reason, academics questioned
whether GRI Standards are adequate when writing sustainability reports for HEIs and
proposed alternatives and modifications (Amiano Bonatxea, Gutiérrez-Goiria, Vazquez-De
Francisco, & Sianes, 2021; Lozano, 2006), such as the use of a materiality matrix. A
materiality matrix is a tool that illustrates which material topics are salient for an organisation
based also on the perspectives and opinions of its stakeholders. GRI Standards state that an
issue is material when it represents “the organization’s most significant impacts on the
economy, environment, and people, including impacts on their human rights” (Global
Reporting Initiative, 2021a), whereby material topics are “topics that represent the
organization’s most significant impacts on the economy, environment, and people, including
impacts on their human rights” (Global Reporting Initiative, 2021b). While an example of a
materiality matrix is lacking in the most recent set of standards, one can be found in the GRI
Standards published in 2016. This states that a materiality matrix “shows the two dimensions
for assessing whether a topic is material; and that a topic can be material based on only one
of these dimensions. The use of this exact matrix is not required; however, to apply the
Materiality principle, it is required to identify material topics based on these two
dimensions.” The topics appear to be relevant according to the two dimensions: if they are
reasonably important in reflecting the organisation’s economic, environmental, and social
impacts, or in influencing the decisions of stakeholders. However, only the 2016 GRI
Standards mention the two dimensions in the report (Global Reporting Initiative, 2016). In
the 2021 GRI Standards, the example of the materiality matrix was removed and replaced by
more general indications about the process of determining material topics (Global Reporting
Initiative, 2021a). This backing down about the materiality matrix between 2016 and 2021
19
could have been caused by the problematic definition of materiality (herein termed
materiality haziness) and by the fact that proposing a materiality matrix became more
difficult following the 2016 guidelines. However, even though the materiality matrix is no
longer recommended by GRI, it continues to be employed both by companies (De Cristofaro
and Raucci 2022) and universities. The GRI 101: Foundation 2016 presents a graphical
materiality matrix example, wherein the two previously cited dimensions are respectively
associated with the x-axis and y-axis. This inconsistency was already noted by Taubken and
Feld (2018), who found that is not so uncommon for the materiality matrix to be subject to
misunderstandings and mistakes due to both the materiality haziness and the fact that the
matrix is part of a method that is imprecise overall. Problems and misunderstandings, also
related to the materiality haziness, moreover seem common in HEIS, to the point that recent
literature underlined that HEIs’ sustainable reports have a low quality and quantity
(Ceulemans, Stough, and Lambrechts 2018) and are mainly limited to the environmental side
without properly engaging all stakeholders (Disterheft, Caeiro, Azeiteiro, & Filho, 2015).
These issues related to materiality haziness introduce several discrepancies when mapping
and identifying key topics for the organisation. The problem of defining when a certain topic
is material for a given institution is an object of debate, both for the standards’ providers and
legislators, to the extent that the definitions of materiality differ slightly in meaning and
content, causing several misunderstandings in creating a materiality matrix. The lack of
clearness regarding materiality assessment thus has a subjective effect (Bellantuono,
Pontrandolfo, and Scozzi 2016; Calabrese et al. 2019). Moreover, assessing the materiality
to create a matrix is even more complex. Indeed, materiality must not only determine the
relevance of the social, economic and environmental impacts of the institutions but also
quantify the influence of these topics on stakeholders in terms of choices and evaluations
when making decisions concerning the institution itself.
3. Methodology
Starting from this scenario, and according to the GRI Standards and the GRI’s materiality
concept, this work aims to investigate if and how Italian state universities incorporate the
materiality matrix in their sustainability reports. Although, as seen in the previous section,
there is no specific reference to HEIs, this study considers the GRI Standards as the most
frequently used within the sustainability reporting of each business area, regardless of the
organisation. To address this research topic, and per GRI Standards, this work tries to
understand how Italian universities build materiality matrices and if the results thereof are
reliable. The methodology used in this paper is content analysis, which is a qualitative
research technique used to interpret and evaluate textual material such as sustainability
reports (Aggarwal and Singh 2018). The content analysis resulted in a conceptual analysis
conducted through Nvivo 12 Plus, aiming to shed light on the concept of materiality along
20
with the materiality matrix. A list of all Italian public universities was considered to conduct
this analysis (N = 67), whereby 19 universities were found to produce sustainability reports,
of which only four (University of Brescia, University of Tor Vergata, University of Tuscia,
and University of Torino) disclosed a materiality matrix. For each university, only the most
recent published report has been considered, regardless of the academic years under
consideration. Subsequently, analyses and comparisons of the materiality matrices in the
sustainability reports were conducted to detect their peculiarities and characteristics, as well
as any inconsistencies with the GRI Standards.
4. Results and Discussion

The analysis showed that of the 19 identified documents, 17 cite the word “materiality”,
although only 4 provide a materiality matrix. All 19 reports state that they have been drawn
up based on GRI Standards 2016 (see Table 1).
Table 1. Presence of the materiality concept, materiality matrices and GRI Standards.
Yes No Total
Materiality 17 2 19
Materiality Matrix 4 15 19
GRI Standards 2016 19 0 19
The results show a common implementation of the GRI Standards by the entire sample. Yet
while the term “materiality” is used in 17 of the 19 reports, only 4 provide a matrix. This
could be due to the materiality haziness perceived by the staff in charge of the materiality
assessment. Due to the different existing definitions provided by the standards and the
legislator, the interpretation of what is material can vary greatly. In any case, even though
the definition of materiality is still unclear for academics and regulators, the adoption of the
GRI Standards should assume that the organisations that choose to use them automatically
share the materiality definition proposed by the GRI. Assuming that it is necessary to follow
GRI 2016 instructions when constructing a materiality matrix, and considering that all four
universities with a materiality matrix followed GRI Standards 2016, it goes without saying
that they embrace the definition of materiality proposed by those standards. Consequently,
they should build a matrix based on these. Focusing on the analysis of the four identified
materiality matrices, it emerged that there is no alignment between GRI 2016 and the results
presented in the graphics produced by these Italian universities. Table 2 shows the
discrepancies between the examples of materiality provided by the GRI Standards of 2016
and what is present in the materiality matrices of the Italian universities, demonstrating that
what they map slightly differs semantically from what was stated as the basis in the GRI 101
Foundation.
21
Table 2. Discrepancies between the GRI example and the materiality matrices of four Italian
universities.
Example of a University of University of University of University of

materiality matrix Brescia Tor Vergata Tuscia Torino
by the GRI 2017-2019 2021 2021 2019/2020 -
Standards (2016) 2020/2021
Significance of Relevance for Relevance for Relevance for Relevance for
economic, the University the university/ the university the University
x- environmental, of Brescia internal of Torino
axis & social impacts stakeholders
Influence on Relevance for Relevance for Relevance for Relevance for

y- stakeholder stakeholders external stakeholders the University
axis assessments & stakeholders of Torino’s
decisions stakeholders
Based on the indications provided by the GRI Standards, the identified materiality matrices
did not precisely follow the suggestions provided by the standards. This assumes that the
universities did not properly conduct the materiality analysis according to the two
dimensions, doing it in terms of tracking the importance of the topics for the institution and
its stakeholders. Moreover, the topic prioritizations of the four matrices are not used as
guidelines for the sustainability reports. What they missed probably links to the materiality
haziness, which can cause an effective misinterpretation of what is important to map, leading
to incorrect questions being asked, and consequently influencing the meaning of what is
being investigated (Taubken and Feld 2018). This is also the result of a non-mandatory way
of conducting the materiality analysis. With the 2016 and 2021 standards, the GRI merely
outlined an overall pathway to follow, without forcing the organisations to adhere to a defined
methodology, even though the provided path was quite specific and method-oriented. As a
result, the materiality analysis is not a materiality analysis conducted based on the
recommendations provided by the GRI Standards. These matrices result in tracking the
relevance of specific topics from both the university’s and the stakeholders’ points of view.
In other words, the analysed matrices do not seem to provide reliable results according to the
recommendations of the GRI Standards, and misunderstandings seem to emerge. The
prioritisation of topics is based on an estimation of the relevance from both the internal and
external perspectives, i.e. from the points of view of the university and its stakeholders.
However, the materiality matrix should prioritise issues based on their influence on
stakeholder decisions and on the organisation as well as the impacts that the organisation has
in social, environmental, and economic terms – the latter issues are currently neglected.
22
5. Conclusion
This paper is a preliminary investigation into the coherent use of the materiality matrix in
sustainability reports disclosed by HEIs in the context of the IHES. Preliminarily, the results
show that those universities that used the matrix deliberately chose to do so based on their
free interpretation, thereby producing matrices that, while referring to GRI 2016, bear no
relevance from the point of view of the prioritisation of issues. the matrices seem to be based
on improper investigations, compared to what was prescribed by the GRI 2016 indications.
Moreover, the results prove that the matrix conceived in the 2016 GRI Standards created
problems for users and was not easy to put into practice. This is consistent with the retraction
by the GRI, which does not further mention the materiality matrix in the new 2021 standards.
The difficulties in realising and applying the materiality matrix are also consistent with the
notion of materiality haziness. Nevertheless, the matrix, in line with the indications of the
materiality analysis, could still be relevant if it is developed using a tailored methodology.
Finally, although the research is still in the preliminary stage, it still presents some
limitations, which mainly relate to the sample comprising only Italian state universities that
provide a materiality matrix. For this reason, future studies should be extended to the
European context or include Italian private universities. This would provide more relevant
and comparable information that could help researchers, university managers, and
policymakers find the best practices for building the materiality matrix.
Acknowledgments
This study contributes to the developmental project of the Department of Business and
Economics of the University of Sassari (Dipartimenti di Eccellenza 2018–2022) financed by
the Italian Minister of Education.
References
Aggarwal, P., & Singh, A. K. (2018). CSR and sustainability reporting practices in India: An
in-depth content analysis of top-listed companies. Social Responsibility Journal.
Amiano Bonatxea, I., Gutiérrez-Goiria, J., Vazquez-De Francisco, M. J., & Sianes, A. (2021).
Is the global reporting initiative suitable to account for university social responsibility?
Evidence from European institutions. International Journal of Sustainability in Higher
Education, 23(4), 831–847. https://doi.org/10.1108/IJSHE-04-2021-0129
Barth, M., & Michelsen, G. (2013). Learning for change: An educational contribution to
sustainability science. Sustainability Science, 8(1), 103–119.
https://doi.org/10.1007/s11625-012-0181-5
Barth, M., & Rieckmann, M. (2012). Academic staff development as a catalyst for curriculum
change towards education for sustainable development: An output perspective. Journal
of Cleaner Production, 26, 28–36. https://doi.org/10.1016/j.jclepro.2011.12.011
23
Bellantuono, N., Pontrandolfo, P., & Scozzi, B. (2016). Capturing the Stakeholders’ View in
Sustainability Reporting: A Novel Approach. Sustainability, 8(4), 379.
https://doi.org/10.3390/su8040379
Calabrese, A., Costa, R., Levialdi Ghiron, N., & Menichini, T. (2019). Materiality analysis
in sustainability reporting: A tool for directing corporate sustainability towards emerging
economic, environmental and social opportunities. Technological and Economic
Development of Economy, 25(5), 1016–1038. https://doi.org/10.3846/tede.2019.10550
Ceulemans, K., Lozano, R., & Alonso-Almeida, M. D. M. (2015). Sustainability Reporting
in Higher Education: Interconnecting the Reporting Process and Organisational Change
Management for Sustainability. Sustainability, 7(7), 8881–8903.
https://doi.org/10.3390/su7078881
Ceulemans, K., Stough, T., & Lambrechts, W. (2018). Pioneering in sustainability reporting
in higher education: Experiences of a Belgian business faculty. Handbook of
Sustainability Science and Research, 211–231.
De Cristofaro, T., & Raucci, D. (2022). Rise and Fall of the Materiality Matrix: Lessons from
a Missed Takeoff. Administrative Sciences, 12(4), 186.
https://doi.org/10.3390/admsci12040186
Directive (EU) 2022/2464 of the European Parliament and of the Council of 14/12/2022
amending Regulation (EU) No 537/2014, Dir. 2004/109/EC, Dir. 2006/43/EC and Dir.
2013/34/EU, as regards corporate sustainability reporting, 322 OJ L § (2022).
Directive 2014/95/EU of the European Parliament and of the Council of 22/10/2014
amending Dir. 2013/34/EU as regards disclosure of non-financial and diversity
information by certain large undertakings and groups, 330 OJ L § (2014).
Disterheft, A., Caeiro, S., Azeiteiro, U. M., & Filho, W. L. (2015). Sustainable universities –
a study of critical success factors for participatory approaches. Journal of Cleaner
Production, 106, 11–21. https://doi.org/10.1016/j.jclepro.2014.01.030
Disterheft, A., Caeiro, S., Azeiteiro, U. M., & Leal Filho, W. (2013). Sustainability science
and education for sustainable development in universities: A way for transition. In
Sustainability assessment tools in higher education institutions (pp. 3–27). Springer.
Fiorani, G., & Di Gerio, C. (2022). Reporting University Performance through the
Sustainable Development Goals of the 2030 Agenda: Lessons Learned from Italian Case
Study. Sustainability, 14(15), 9006. https://doi.org/10.3390/su14159006
Global Reporting Initiative. (2016). GRI 101 Foundation. Amsterdam.
Global Reporting Initiative. (2021a). GRI 3 Material Topics. Amsterdam.
Global Reporting Initiative. (2021b). GRI Standards Glossary. Amsterdam.
Herzner, A., & Stucken, K. (2020). Reporting on sustainable development with student
inclusion as a teaching method. The International Journal of Management Education,
18(1), 100329. https://doi.org/10.1016/j.ijme.2019.100329
Hsu, C.-W., Lee, W.-H., & Chao, W.-C. (2013). Materiality analysis model in sustainability
reporting: A case study at Lite-On Technology Corporation. Journal of Cleaner
Production, 57, 142–151. https://doi.org/10.1016/j.jclepro.2013.05.040
24
Lozano, R. (2006). A tool for a Graphical Assessment of Sustainability in Universities

(GASU). Journal of Cleaner Production, 14(9), 963–972.
https://doi.org/10.1016/j.jclepro.2005.11.041
Lozano, R., Lukman, R., Lozano, F. J., Huisingh, D., & Lambrechts, W. (2013). Declarations
for sustainability in higher education: Becoming better leaders, through addressing the
university system. Journal of Cleaner Production, 48, 10–19.
https://doi.org/10.1016/j.jclepro.2011.10.006
Taubken, N., & Feld, T. Y. (2018). Impact measurement and the concept of materiality—
New requirements and approaches for materiality assessments. Nachhaltigkeits
Management Forum | Sustainability Management Forum, 26(1), 87–100. Springer.
25
Creating the “examination kiosk”: blended assessment to foster

self-directed learning in higher education
Claudia Albrecht1, Anne Jantos1, Sebastian Pannasch2, Vincent Zipper2
1
Center for Interdisciplinary Learning and Teaching Technische, Universität Dresden,
Germany, 2Faculty of Psychology Technische, Universität Dresden, Germany.
Abstract
The field of higher education is continually evolving; with it, examination
strategy must also adapt. Especially self-directed learning will be a focus. We
are developing an innovative solution to address the changing needs of
university students and their exams. We found current challenges and new
requirements for examination strategies based on extensive research analysis
of relevant literature in the field and student-based feedback. The solution is a
didactical methods in the form of an examination kiosk designed specifically
for portfolio examinations. The purpose of the examination kiosk is to promote
self-directed learning. It will allow for greater flexibility and individuality in
the assessment process. We present the examination kiosk in detail and provide
insights into its creation and potential applications.
Keywords: Portfolio; higher education; self-directed learning; assessment;
examination; examination kiosk.

Creating the “examination kiosk”: blended assessment to foster self-directed learning
1. Introduction
Given the constantly changing conditions of the labor market and the development of
technology (e.g. Chatgpt), university graduates need a variety of hard and soft skills.
Therefore, it is important to adapt the daily routine in university teaching and the associated
examination culture to test in a competence-oriented manner (Dede, 2010; Ayaz and Gök,
2020). We pay particular attention to self-direction, which refers to students’ ability to set
and pursue goals, make decisions, and manage their learning and development, independent
of external guidance or direction (Morris, 2019), which graduates will need in future
professional careers. Accordingly, the design of a holistic exam strategy is of high relevance.
In the recently established research project „Portfolio Profis“ at the faculty of psychology at
Technische Universität Dresden in Germany (TUD), we will design, implement, and evaluate
new portfolio strategies for higher education in an iterative process. The search for new
solutions became necessary to meet the requirements of the new Psychotherapy Act for the
study program in psychology. The conversion of this highly demanded degree program with
120 students each year required the adaptation of the study and examination regulations.
Fifteen out of the 16 modules in the study program now have the portfolio as the form of
examination. Each of the modules is completed by a graded portfolio based on 30 to 60 hours
of work. Portfolio work was chosen as the form of exam to give the teaching staff a high
degree of flexibility in designing the exams. It is also intended to reduce the huge amount of
feedback and grading required at the end of the semester by assessing performances. It is
intended to support students in continuously designing their learning process. However, the
basic understanding of portfolio exams at the faculty of psychology differs greatly from that
of portfolio work. To improve the quality of teaching, it is necessary to develop and test
concepts that combine portfolio work with the requirements of portfolio examinations.
However, so far there are no proven concepts for graded portfolio examinations for large
groups of students, especially if they aim at a reflective mapping of self-directed learning
processes. The current paper, therefore, addresses the following question:
How to develop the concept of portfolio examinations to support self-directed learning
processes based on fundamental aspects of portfolio work?
To answer this question, the following sections compare portfolio exams and portfolio work
concerning various aspects. Subsequently, a new concept for the implementation of portfolio
exams is designed from the comparison and discussed with students of the bachelor's program
in psychology at TUD.
28
Claudia Albrecht, Anne Jantos, Sebastian Pannasch, Vincent Zipper
2. Comparison of Portfolio Work and Portfolio Examinations

An extensive literature search enabled us to compare portfolio work with portfolio
examinations regarding their structure, use, and potential. We focused the search on
publications from German-speaking countries to achieve the highest possible fit for synthesis
with the portfolio examinations. Additionally, we filtered based on availability online and
date of publication after the year 2000. We found many different types and implementations
of portfolios, making a single, universally applicable definition difficult (this is described in
Knauf, Behrend, & Knutzen (2020) and Keplinger (2014), among others). To answer our
research question and to further develop portfolio examinations towards a more reflective
design supporting self-directed learning, it does not seem necessary to differentiate the
concept of the portfolio work to different criteria but to elaborate on the basic elements.
Therefore, the definition by Hornung-Prähauser et al. (2007) serves as the basis for the
present description:
A portfolio is "a (digital) collection of "skillfully made works" (=lat. artifacts) of a person
who thereby wants to document and illustrate the product (learning outcomes) and the
process (learning path/growth) of his/her competence development in a certain time and for
certain purposes. The person in question has independently selected the artifacts and
organized them about the learning objective. She (he), as the owner, has complete control
over who, when, and how much information from the portfolio may be viewed."
Based on this definition, the following section describes aspects of portfolio work and
contrasts them with the current conditions of portfolio examinations. The result is a collection
of aspects that – as the diversity of portfolio use suggests – are not all addressed in each of
the individual publications but are derived from a summative analysis (Barrett, 2011; Bauer
& Baumgartner, 2012; Bräuer, 2016).
To contrast portfolio work and portfolio examinations we need also a description of portfolio
examinations. In terms of the corresponding examination regulations, portfolio examinations
are defined in the examination regulation of the bachelor's program in psychology
(Technische Universität Dresden, 2021):
Portfolios serve, using a compilation of similar or dissimilar individual performances, to
demonstrate the ability to place the aspects of professional, scientific action determined by
the respective task in a larger context. This includes the ability to work in a team if the
respective task requires it. Portfolios can include both face-to-face and distant work, and the
result is an objective, e.g. written work. The time required for portfolios is specified in each
case in the module descriptions and may not exceed 300 hours. Derived from this, the
deadline for the submission of individual performances, the duration of individual
performances, and the deadline for the submission of the entire portfolio are to be determined
within the framework of the respective assignment.
29
The aspects of portfolio work and portfolio examination are contrasted in Table 1.
Table 1. Comparison of portfolio work and portfolio examinations.
Aspect Portfolio Work Portfolio Examination

× Strengthening the ability to reflect
× Support of self-directed learning × Verification of the achievement of
× Promotion of personal competencies the learning objectives of the module
Objectives (action competence, self- × Flexibility for students and teachers
competence) × Relieving the pressure of exams from
× new form of performance the end of the semester
assessment
Dimensions Process and product dimension Product dimension
× Documentation of competencies
× Mapping of learning process and
Content development × processed tasks
× reflection on learning experiences
and learning strategies
coherent (digital) portfolio, which
individual achievements, the results of
Structure contains individual artifacts and is
which are managed by the teachers
created by the students
"subtasks", e.g.
"artifacts", e.g.
× Essays
× Learning materials
× Tests
× Observation reports
× Excerpt
Components × Case studies
× Literature
× Reflections
× Research
× Learning diary entries
× Presentation
× Versions of text contributions
× Poster
recommended rather for small groups required also for lectures with 120
Group size
up to approx. 30 students students
Supervision
high low
effort
Feedback is obligatory, grading only in Grading obligatory points for individual
Assessment
some scenarios performances and grade
Students decide criterion-oriented Teachers decide on the content, form,
Who decides which components and contents their and number of individual
portfolio contains. performances.
Date of as a rule, the completed portfolio is each performance is handed in
submission handed in separately
The presentation in Table 1 reveals differences in almost all aspects between the basic ideas
of portfolio work and the portfolio examinations as they are currently used. The largest
degree of similarity can be found in the list of individual components, which labeled
30
differently as either artifacts or individual achievements but the components are similar in
terms of form and medium.
3. Didactic Development of Portfolio Examinations

Next we will identify the aspects of portfolio examinations that require further adaptation. It
is therefore necessary to characterize differences to portfolio work but also to take into
account to what extent the conditions given by the examination regulations described above
allow for an adaptation. Further developments are required and possible with regard to the
following aspects:
Objectives: Fundamental for an improvement of the teaching quality is that the strengthening
of the student’s ability to reflect and their self-competence as well as the support of self-
directed learning are added to the goals of the use of the portfolio examinations. While these
are the goals of the Bachelor's program in psychology at TUD they are not yet addressed by
the current practice of portfolio examinations.
Who decides: It also appears to be quite essential that students are given a greater opportunity
to influence the design of their learning and thus the examination process. In a first step, this
can concern the form, content, and/or number of individual performances. Therefore,
teaching staff should provide a framework – in accordance with the examination regulations
– that provide various opportunities for the students. However, the different opportunities
wihin a portfolio have to be comparable but also the individual and overall workload has to
be in line with the examination regulations that are defined for the respective module. In this
context, it could become critical that students are overwhelmed with the possibilities to
design their learning and examination process.
Dimensions and Content: To strengthen the student’s ability to reflect on their own level of
knowledge, they need to have opportunities to do so, which also includes to reflect on their
learning process, experiences and strategies. Therefore, the portfolio assessment should be
expanded to include a process dimension as well as reflective content. However, assessing
the reflective component in the context of portfolio exams seems difficult. Therefore, this
point should also be discussed with the students during the focus group. To ensure general
acceptance for a further developed version of the portfolio examinations also requires to
consider the supervision effort on the side of the teaching staff. The higher this effort will be,
the lower will be the general acceptance and the willingness for implementation.
Student-based Feedback: To evaluate current portfolio implementations a focus group
discussion was conducted with students in the Bachelor psychology program. Eleven
students discussed within the framework of a world café suggestions and ideas for the
didactic iteration of the portfolio examinations. The following results for further
31
consideration emerged: Students show dissatisfaction with the current practice of portfolio
examinations and expressed the need for a development towards a stronger consideration of
self-directed and reflective learning. Students do not see any danger of being overwhelmed
or unsettled by greater freedom to design their own learning and examination process. One
central idea suggests that the teaching staff provides a selection of portfolio assignments to
be completed (e.g., 10 individual assignments) and students select and complete a certain
number (e.g., 5 individual assignments). This suggestion was very much appreciated. It is
important, that students do not have to specify at the beginning of the semester which tasks
they want to complete, so that they have enough flexibility for trial and error. Further, they
see the importance of including the process dimension and reflection in the portfolio
assessment.
4. Creating the “Examination Kiosk”

The results of the comparison between portfolio work and the current practice of portfolio
examinations as well as the information from focus group discussions were used to further
develop the concept of portfolio examination. We call the concept the “examination kiosk”
and define it as follows:
The examination kiosk is a type of graded portfolio examinations. In this concept,
students compose their individual assignment combination and thereby design their
learning and examination process. Each student selects their own subtasks from the
available options. The sum of all subtasks creates the portfolio examination.
For a successful implementation, a smooth transition seems appropriate, therefore we will
describe a "start-up version" of the examination kiosk together with prospects for the further
development.
Figure 1. “Examination Kiosk”
32
Objectives: The examination kiosk is used as a form of examination to check the

achievement of the module objectives laid down in the study regulations. It also supports
self-directed learning and the promotion of the students' ability to reflect and their self-
(learning) and action competence. Lectures and students will benefit from a more equal
distribution of examinations across the semester as well as from greater flexibility in the
design of the module examination.
Dimensions: The exam kiosk primarily addresses the product dimension, but also considers
the process component. This is achieved through an ungraded reflection of the learning
process and outcome. In perspective, the reflection may be given bonus points and/or graded.
Structure: At the beginning of the semester, the procedure is transparently presented to
students with Video, FAQ, PDF and Miro Board as well as a weekly consultation hour
throughout the semester. Thereby, the teaching staff informs the students about the (i) number
of available subtasks, (ii) the required number of subtasks that need to be completed by the
students, and the scope in work units of each. In order to give students sufficient choice, we
recommend that at least six subtasks are offered, of which students may choose four. The
content, form, and latest submission date will also be specified by the teaching staff. Students
decide during the semester which subtasks they would like to work on and hand in. All
submitted individual performances will be evaluated with points. The points will be summed
up to a grade at the end of the semester. In an accompanying, ungraded reflection, students
answer predetermined questions that address, among other things, the choice of individual
performances and the learning process. In the future, students can be given even more
decision-making options, e.g. by allowing them to determine the content, number and form
of the individual assignments themselves on the basis of their own learning objectives.
Components: The kiosk consists of different subtasks, which are arranged by the students to
their personal Portfolio. The concrete tasks for the subtasks and the expected form are
specified by the teaching staff. The forms of the subtasks depend on the respective learning
objective that is to be examined. For example, tests, one-minute papers, essays, steps of a
research project, forum contributions, memes, videos or mind maps can be subtasks. Ideally,
these are products that are naturally created by students as part of their learning process.
5. Conclusion
The examination kiosk will be used for the first time in the summer semester of 2023. Its use
will be closely accompanied by a scientific study so that the knowledge gained can be
incorporated into a continuous development process. By using the examination kiosk, the
teaching/learning practice and thus the learning and teaching culture can develop further. In
the process, the teaching staff gradually change their role to that of learning facilitator. The
students are involved in a participatory way and are trained with regard to their self-learning
33
competence. Our long-term goal is to raise the quality of university teaching and its
assessment strategies and to train learners to become mature, independent and reflective
graduates through complex examination systems to create a participatory learning culture.
Furthermore, we aim to support teaching staff to implement complex assessment strategies
by educating them how to mix assessment formats, for example by using the Blended
Assessment Cube (Jantos and Langesee, 2023).
Acknowledgement
The work was funded by Stiftung Innovation in der Hochschullehre.
References
Ayaz, M., & Gök, B. (2022). The effect of e-portfolio application on reflective thinking and
learning motivation of primary school teacher candidates. Curr Psychol.
https://doi.org/10.1007/s12144-022-04135-2
Barrett, H. (2011). Balancing the two faces of ePortfolios. In S. Hirtz, & K. Kelly (Eds.),
Education for a Digital World: Innovations in Education (pp 289-307). Open School BC.
Bauer, R. & Baumgartner, P. (2012). Schaufenster des Lernens. Eine Sammlung von Mustern
zur Arbeit mit E-Portfolios. https://elibrary.utb.de/doi/book/10.31244/9783830976431
Bräuer, G. (2016). Das Portfolio als Reflexionsmedium für Lehrende und Studierende. (2nd
ed). Stuttgart: UTB GmbH https://elibrary.utb.de/doi/book/10.36198/9783838546322
Dede, C. (2010). Comparing frameworks for 21st centuryskills. In J. Bellanca & R. Brandt
(Eds.),21st century skills(pp. 51–76). Bloomington, IN: Solution Tree Press
Hornung-Prähauser, V., Geser, G., Hilzensauer, W., & Schaffert, S. (2007). Didaktische,
organisatorische und technologische Grundlagen von E-Portfolios und Analyse
internationaler Beispiele und Erfahrungen mit E-Portfolio-Implementierungen an
Hochschulen. https://www.e-teaching.org/materialien/literatur/fnm-austria-2007
Jantos, A., Langesee, L.-M. (2023). Blended Assessment in Higher Education Collaborative
Case Study Work – A Qualitative Study. In: Proceedings of the 25th International
Conference on Interactive Collaborative Learning 2022, pp. 1–17.
Keplinger, G. I. (2014). Auf dem Weg vom Lehrenden zum Lernbegleiter: Der Einsatz von
EPOSTL und Englisch-Assistent-WEB zur Professionalisierung von Englischlehrenden
für die Sekundarstufe I an Pädagogischen Hochschulen.
Knauf, B., Behrend, S., & Knutzen, S. (2020). E-Portfolio-Arbeit in der beruflichen
Erstausbildung - Das Ausbildungsportfolio als Instrument zur Lernreflexion und als
digitaler Ausbildungsnachweis. https://doi.org/10.15480/882.2880
Morris, T. H. (2019). Rolf Arnold's systemic-constructivist perspective on self-directed
learning. https://www.wbv.de/shop/Rolf-Arnold-s-systemic-constructivist-perspective-
on-self-directed-learning-6004599w022
Technische Universität (2021): Examination Regulation for Bachelor in Psychology. §10
https://tud.link/puf7
34
Pro-f-quiz: increasing the PROductivity of feedback through

activating QUIZzes
OVI@ACRO, Department Computer Science, KU Leuven Campus Diepenbeek, Belgium.
Abstract
Feedback beyond the grade is an important part of the learning process.
However, because of the large student groups, many teachers in higher
education are faced with practicalities such as the limited time to prepare and
communicate the feedback to individual students. We have set up an
experiment, titled Pro-f-quiz, in which over two years 236 students
participated and in which the feedback is communicated through an online
quiz that activates the students to reflect upon their solution. The system can
be set up in a very limited time compared to booking individual time slots. The
results show that approximately 85% of the students appreciate the approach
with 60% indicating that they reflect more intensively about their work than
when the feedback is transmitted traditionally. Moreover, the grade of the
students participating in the project was substantially higher than students not
participating.
Keywords: Grading; feedback; quiz; reflection; student appreciation.

Pro-f-quiz: increasing the productiveness of feedback through activating quizzes
1. Introduction
Feedback is important in the learning process of students. Providing proper feedback–not
overly negative but constructive, enthusing and with a focus on reflection–is of paramount
importance but comes with challenges such as tight time constraints and a perceived limited
return on investment. Time is a many-facetted problem. First, the teacher’s time: to grade, to
formulate and to communicate the feedback. Formulating feedback takes extra time over
grading because the match with the evaluation criteria must be made explicit and the required
improvement elucidated. Communicating can also take significant time, especially when
done orally. The second and often underestimated issue is the student’s time. Fitting the
schedule in both the teachers’ and the students’ agenda would imply an extended time span
between the first and the last student getting their feedback. In terms of equal treatment this
poses a major challenge when the idea of the feedback is to be formative, i.e., helping them
in performing better at consecutive tasks.
The limited return on investment is the other problem: Crisp (2007) reports a feeling of
wasted time providing feedback, as also confirmed by individual teachers in our environment.
Our conjecture is that students, unless triggered otherwise, act as mere receivers of feedback
and often prefer to scan the feedback superficially, only searching for the actual grade.
This paper presents Pro-f-quiz, a productive methodology for giving feedback through
means of a quiz. The quiz consists of concrete, multiple choice questions closely tied to the
task at hand. After each question, the student is provided feedback, not only the correct
answer but also about the imperfections of the incorrect options. Only after finishing the
entire quiz, the students can see their grade and score on each of the questions. We believe
this methodology improves the productivity of the feedback process because students are
triggered to reflect about their own solution before being able to read the answer, while only
requiring a limited time effort for the teacher. The experiment has been performed in the first
bachelor year of Engineering Technology for in total 374 students, of which 236 students,
divided in 146 project groups, voluntarily participated.
We defined the following research questions to evaluate the productivity of the methodology:
• RQ1: Do students appreciate the methodology?

• RQ2: Does it increase reflection?
• RQ3: Is the time needed to set up the feedback system limited in comparison
to traditional approaches?
The remainder of this paper is as follows: Section 2 provides related work; we describe the
context and setup of the experiment in Section 3 while presenting the results and the lessons
learnt in Section 4. After Section 5, exploring the threats to validity, Section 6 concludes with
a summary and suggestions to expand the experiment.
36
2. Related Work
The use of (automated) assessment and feedback systems to facilitate students’ learning has
been extensively researched in recent years. Barriocanal et al. (2002) found out that a
straightforward application of unit testing, a fully automated feedback mechanism for testing
software, does not improve student's engagement. Chatzopoulou (2010) proposes “adaptive
assessment”, where students with different abilities are served different sets of questions.
Matthews et al. (2012) introduced such a system and show that it is especially interesting for
a resource-based learning approach that focuses on process assessment. The framework
allows for custom comments by instructors. The quantity of feedback and response time was
improved, but the quality rates dropped slightly.
Sherman et al. (2013) found out that computer-assisted automated grading systems positively
affect student feedback and response time, allowing educators to quickly grade multiple
complex assignments while at the same time allowing students to increase their submission
per assignment rate. More recently, Thangaraj et al. (2022) compared six recent feedback
systems for introductory programming courses in higher education on five different axes.
They praise the immediate feedback and the opportunities for the student to initiate the
feedback by submitting a work, but also conclude that more development is needed to be
more effective in motivating the learning process.
When it comes to the format of feedback, Funk and van Diggelen (2014) claim that although
written feedback is still used a lot, students are dissatisfied with the way they are formulated
and the quality in general. A two-legged categorical system should be developed to mitigate
the fluctuating quality, they conclude. The first category, focusing on the content of learning,
should cover task, proces, vision, identity, etc. The second category determines the form of
giving feedback: positive-negative, specific-general, limited-elaborate, etc. A quiz-like
feedback format as presented in this work avoids most written feedback shortcomings
outlined by Funk and van Diggelen.
Online quiz systems have been successfully employed to support educators in providing
feedback in large-class courses. Furthermore, most of them allow for multiple attempts,
resulting in a significant increase in students’ quiz scores. Also, Mendoza and Lapinid (2022)
discovered that the positive attitude of students towards feedback increased compared to
conventional text-based feedback systems, as it allowed them to better manage their time and
help them understand the theoretical material at hand. Although Mendoza and Lapinid did
not use the system to effectively grade students, they do note that average results of online
quizzes is a good predictor of final exam grades, as also unveiled by Cohen and Sasson (2016)
and McDaniel et. al. (2007), who conclude that in the classroom testing can be used to
promote learning, not just to evaluate learning. Our approach aims to combine both.
37
3. Experimental Setup
The experiment took place during two consecutive years in the course Software design in
Java (4 ECTS) at the joint program ‘Engineering Technology’ at UHasselt and KU Leuven
Campus Diepenbeek (Belgium). The course, an “Objects First” course (Cooper (2003),
Barnes et. al. (2006)), is attended by roughly 200 students yearly. It is situated in the initial
general engineering phase of three semesters (1,5 year), after which students choose one of
seven options ranging from Chemistry or Construction to Electronics-ICT or Software
Systems. All students must pass the course. As students interested in majors outside IT often
demonstrate less interest, we use nudging techniques as stimulation. One form used to be an
early but non-committal task–no grades, only feedback–that helps them in succeeding in the
consecutive assignment (worth 33% of the grade). Because the number of students who took
up the challenge gradually decreased to less than 10, we changed the task from non-committal
to voluntary, and started grading the tasks, using a bonus system to ensure that students are
never punished for performing poorly in the voluntary task. Participation increased to 60-
70% but giving individual high-quality feedback would now require a lot of effort.
To minimize the effort and maximize the effect we set up of first version of the experiment
in 2021-’22. Instead of directly distributing the feedback and scores, we composed a quiz of
10 multiple-choice questions named the Pro-f-quiz. Some questions were basic: “Did you
declare your data members as public?”; some more introspective: “Wat representation did
you use for the quality of the snow?”; other questions listed options that could or could not
be applicable. Each question was followed by immediate feedback explaining the rationale,
e.g. “A String is not suited because any text can be used, not only the values we enumerated.”
We also provided an anonymized Excel-sheet with hashed student names containing
personalized feedback and grades. Student can only retrieve their hash by finishing the quiz.
We also added an additional question to the quiz “Did you learn from this feedback quiz?”
Because 87% answered positively, and because we later noticed that the grade of the students
who did the voluntary task was 3,63/20 higher than other students, we conducted a second
version of the experiment in 2022-’23 with additional questions regarding the feedback
mechanism and how much effort they put in the voluntary task. Results were collected
through the learning platform Toledo, a derivate of Blackboard and analyzed in Excel.

Table 1 shows a high overall participation rate of 63% of the students doing the voluntary
assignment. The participation rate, counted in number of quizzes completed, increased from
36% to 48,4% in the second year. This implies an average error rate of 7,53% with a
confidence level of 95%. If we consider the response as representative for the entire project
group, the error rate drops below 6%, thus considering the sample as highly reliable.
38
Table 1 Participation in the experiment.
Year Students Projects Students with Completed

project quizzes
2021-‘22 192 69 127 69
2022-‘23 182 77 109 88
On the overall question “Did you learn from this feedback quiz?” 87,0% of the students
responded positively in the first year and 83,7% in the second, which is consistently good.
Given an expected working time of approx. 4 hours and the responses on question 12 “How
much time did you spend in the task”, we divided the population as follows:
- Students working shorter than expected on the task (2 to 3 hours): 14 students
- The group that worked as expected (3,5 to 5 hours): 26 students
- The group that took slightly more time (5,5 to 8 hours): 24 students
- Students who worked a lot more (8,5 up to 20 hours): 20 students
100
75
50
25
0
2-3 hours 3,5-5 hours 5,5-8 hours 8,5-20 hours
Figure 1 Percentage of students who learnt from the quiz, versus time spent on the task.
Figure 1 shows a distinct difference between the groups that had to work the least versus the
most. The first group attributes to 16,7% of the population but contains almost half of the
students who did not learn from the feedback: their approval ratio is 57,1% as opposed to
90% in the last group, which is the more important target group. As they had to work much
harder to achieve a similar result, they can probably benefit more from high quality feedback.
Figure 2 shows the results classified according to their grade on the task. The percentage of
positive responses reaches 100% in the group who performed the least. Together with Figure
1, this results in a positive answer on RQ1 (Do students appreciate the methodology?).
39
100
75
50
25
0
3 - 11/20 12 - 14/20 15 - 16/20 > 16/20
Figure 2 Percentage of students who learned from the quiz, versus grade on the task.
The mere observation that students benefit from the feedback is insufficient to claim that this
methodology is superior. Therefore, we explicitly asked the following question: “To what
extent has this way of feedback triggered you to reflect more on the quality of your task?”
Table 2 shows that 60% of the students is convinced that the approach with the quiz helped
them in reflecting more about the quality of their task.
Table 2 Distribution of students reflecting more than when directly given the feedback.
Did you reflect more than when directly given feedback? Students Percentage
Very definitely yes 28 32%
Probably yes 24 28%
Undecided 28 32%
Probably not 4 5%
Very definitely not 3 3%
The final two questions look at the attitude of the students. Table 3 shows positive intentions
of the students that are not aligned with past observations: 61 students claim that they would
make the task irrespective of the promise of a reward, whereas this shrunk to no more than
10 students when there was no reward. Table 4 confirms to a certain degree the presumption
put forward in the introduction that many students are only interested in the grade, but on the
other hand more than 67% of the students in the sample value the feedback at least equally
important as the grade. The effort put in providing is thus highly appreciated by the students.
Table 2 and 4 confirm: RQ2 (Does it increase reflection?): 60% responded that they reflected
more than when receiving the feedback traditionally; and 67% point out that feedback is for
them at least equally important as the grade.
RQ3 (Is the time needed to set up the feedback system limited in comparison to traditional
approaches?) is more difficult to answer firmly, but an estimate is possible. Given 77 project
groups and being able to cater 5 sessions per hour would result in almost 16 hours nonstop.
40
Setting up the feedback quiz took about 2 hours, resulting in an 8-factor reduction. Equally
important is the fact that the feedback quiz ensures that all students receive their feedback at
the same time. Using oral feedback sessions that would need to be fit in the busy schedule of
both teachers and students, it would be nearly impossible to deliver the feedback of the
voluntary task to all students in time for the consecutive mandatory assignment.
Table 3 Doing the task without reward.
Would you have done Students

the voluntary task
without bonus
reward?
Very definitely yes 23
Probably yes 38
Undecided 13
Probably not 10
Very definitely not 3
Finally, Cohen and Sasson (2016) observed that results of online quizzes are a good predictor
of final exam grades, but in our case, participation sufficed: students doing the voluntary task
scored higher than students who did not, rising from 3,63/20 to 5,22/20 in the second year.
Table 4 Appreciation of feedback.
Feedback versus grade Students
I only remember the grade. 8
Grade is more important than feedback. 20
Grade & feedback are equally 39

important.
I need the grade, but feedback is more 16

important.
Grade without feedback is almost 3

worthless.
41
6. Threats to validity
When comparing the grade for the entire course, the control group consisted of students
without doing the task nor the quiz, but we had no control group of students who only missed
the quiz. Therefore, we couldn’t isolate the effects of the feedback quiz from merely
executing the task. Next, participation in the experiment was voluntary and not random. This
possibly implies a biased sample, consisting of motivated students who would have scored
higher in the course regardless of the feedback quiz, and wo appreciate feedback more than
undermotivated students. Nevertheless, the approach scored consistently high and the effect
on the grade was more than substantial, hinting at a significant effect of the methodology.
The reduction in time spent with a factor 8, without sacrificing the quality of the feedback,
was a very rough estimate and depends on the size of the group and the time needed to setup
the feedback quiz, but it is certainly realistic for large student groups.
7. Conclusion
We have presented an experiment in providing activating feedback to students through a
multiple-choice quiz in the context a voluntary assignment that precedes a larger mandatory
task. In total 236 students participated. Approx. 85% said they learned from the feedback and
60% reported reflecting more intensively because of the feedback system, with grades on
average 5,22/20 higher than who did not participate.
To be able to differentiate in the effect of performing the assignment and of the feedback
methodology, additional qualitative research needs to be done to identify the underlying
principles and expand their impact. Care should also be taken when composing the feedback
quiz to avoid equity issues related to automated feedback systems which is more present in
large classes (Xie et al. (2022)) where the time benefits of this approach are the most present.
References
Barnes, D. J., Kölling, M., & Gosling, J. (2006). Objects First with Java: A practical
introduction using BlueJ (p. 520). Pearson/Prentice Hall.
Barriocanal, E. G., Urbán, M. Á. S., Cuevas, I. A., & Pérez, P. D. (2002). An experience in
integrating automated unit testing practices in an introductory programming course. ACM
SIGCSE Bulletin, 34(4), 125-128.
Chatzopoulou, D. I., & Economides, A. (2010). Adaptive assessment of student's knowledge
in programming courses. Journal of Computer Assisted Learning, 26(4), 258-269.
Cohen, D., & Sasson, I. (2016). Online quizzes in a virtual learning environment as a tool for
formative assessment. JOTSE, 6(3), 188-208.
Cooper, S., Dann, W., & Pausch, R. (2003). Teaching objects-first in introductory computer
science. Acm Sigcse Bulletin, 35(1), 191-195.
42
Crisp, B. (2007). Is it worth the effort? How feedback influences students’ subsequent sub-
mission of assessable work. Assessment & evaluation in higher education, 32(5), 571-
581.
Funk, M., & van Diggelen, M. (2014, June). Feeding a monster or doing good? Mining
industrial design student feedback at large. In Proceedings of the 2014 Workshop on
Interaction Design in Educational Environments (pp. 59-63).
Matthews, K., Janicki, T., He, L., & Patterson, L. (2012). Implementation of an automated
grading system with an adaptive learning component to affect student feedback and
response time. Journal of Information Systems Education, 23(1), 71-84.
McDaniel, M. A., Anderson, J. L., Derbish, M. H., & Morrisette, N. (2007). Testing the
testing effect in the classroom. European journal of cognitive psychology, 19(4-5), 494-
513.
Laig Mendoza, M. J., & Caramoan Lapinid, M. R. (2022, January). Online Quizzes: The Use
of Multiple Attempts and Feedback System. In 2022 13th International Conference on E-
Education, E-Business, E-Management, and E-Learning (IC4E) (pp. 124-131).
Thangaraj, J., Ward, M., & O’Riordan, F. (2022). Use of assessment and feedback systems
for introductory computer programming modules of higher education: a comparative
study. 8th International Conference on Higher Education Advances (HEAd’ 22), 127-134.
Sherman, M., Bassil, S., Lipman, D., Tuck, N., Martin, F. (2013). Impact of auto-grading on
an introductory computing course. Journal of Computing Sciences in Colleges, 28(6), 69-
75.
Xie, B., Oleson, A., Everson, J., & Ko, A. J. (2022). Surfacing equity issues in large
computing courses with Peer-Ranked, Demographically-Labeled student feedback.
Proceedings of the ACM on Human-Computer Interaction, 6(CSCW1), 1-39.
43
Adverse effects of personalized automated feedback

J. Riezebos, N. Renting, R. van Ooijen, A.J. van der Vaart
University of Groningen, The Netherlands.
Abstract
In large classes with hundreds of students, it is rarely feasible to provide
students with individual feedback on their performance. Automatically
generated personalized feedback on students’ performance might help to
overcome this issue, but available empirical effect studies are inconclusive due
to lack of methodological rigor. This study uses a repetitive randomized
control experiment to explore whether automatically generated feedback is
effective and for which students. Our results indicate that feedback does not
have a positive effect on performance for all students. Some groups benefit
from receiving personalized feedback, while others do not perform better than
the control group. Students that perform average benefit most from receiving
personalized feedback. However, lower-scoring students who received
feedback tend to have lower attrition rates and if they participate at the final
exam, their performance is not higher than the control group. Therefore,
providing automated feedback is not something that should be undertaken
mindlessly.
Keywords: Automated feedback; summative assessment; randomized control
experiment; empirical study; adverse effects.

1. Introduction
Feedback can be one of the most powerful learning tools in education, but it is difficult to
use in some higher educational settings (Hattie & Timperley, 2007). Especially in large
undergraduate classes with hundreds of students, it is rarely feasible to provide students
halfway through the module with individual feedback on their performance and enable them
to act upon it when preparing for the final exam. Technological innovation holds the promise
to help overcome this issue, as it enables automatically generated personalized feedback on
students’ performance. Previous studies that have explored the effects of automatically
generated feedback on student performance have shown mixed results and often focused only
on smaller learning tasks. Moreover, they lack randomized controlled designs (Morris et al.,
2021). We provide a rigorous study that explores in greater detail whether automatically
generated feedback halfway through the module is effective and for which kind of student.
A commonly deployed technique to provide students with insight on their mastery of the
subject is a midterm some weeks before the final exam. This midterm can be formative or
low stake summative and poses an important opportunity for personalized feedback, as it
generates data on how each student performs on the different learning objectives, while there
is still time to improve before the end of the module. We experiment with a feedback system
that automatically generates personalized feedback for students on each of the learning
objectives of the module that are tested in both the midterm and final exam. The feedback
concerns students’ current performance, what is expected from them, and how they should
proceed from here on. Feedback is formulated for each learning objective at three levels, for
students scoring below standard, standard or above standard. Shortly after the midterms,
students receive an automatically generated e-mail with their grade and personalized
feedback. However, they do not know that the email is automatically generated, as it is send
from the lecturer’s account and formulated as a personal message.
We tested this feedback system in three large (n>300 students) first year bachelor modules
in business administration. These were all technical modules (Management Science (MS),
Supply Chain Operations (SCO), and Statistics (Stat)) that require deep learning strategies as
students really need to understand and apply the material rather than recall information. For
each module, students were randomly assigned to either the experimental group, receiving
an email with their grade and personalized feedback, or the control group, receiving an email
with only their grade. Our main outcome measures are attrition rate and performance on the
final exam (i.e., not the final grade, as that includes the midterm grade as well). We present
effects of receiving automated feedback for low, average and high performing students, for
which we used the 33% tertiles as boundaries.
46
2. Experimental setup
For the field study, we selected three first year modules of a single programme of study. The
modules were similar in their assessment plan, as they all included a final exam and a
midterm, i.e. an intermediate low stake exam 3-4 weeks before the final exam. All learning
objectives tested during the intermediate exam were retested during the final exam. The
purpose of the intermediate exams was therefore aimed at informing students on their mastery
of the subject and the type of exam they could expect for this module. The midterm is a low
stake summative test, as the weigth of the midterm is much lower than the weight of the final
exam. This allows students to treat it as a kind of formative assessment, since they could
make up a bad achievement on the midterm during the final exam. At the start of the year, all
318 freshmen were asked for consent in participating in an experimental study and to give
permission in using their outcomes. The students were not informed about the details and
purpose of the study to minimize the risk that the outcomes of the experiment are influenced,
because the students are aware that they take part in an experiment (Levitt & List, 2009) and
could learn from other’s individual feedback. In total 88% of the students gave consent to
participate in the study. Upon participation in a midterm, we randomly assigned them to
either the treatment or control group. Students who did not gave consent were left out of the
study. Note that not all students who participated in the midterm decided to participate in the
exam. Hence, the attrition rates of treatment and control group are of interest, as this decision
may have been affected by the treatment. After all three midterms had been offered, a within
subjects approach with repeated measurements has been applied to draw conclusions on the
effectiveness of the treatment, notwithstanding the differences between the repeated
measurements, as the modules are different.
2.1. Personalized automated feedback

The intermediate exams consisted of open (short answer) and/or closed questions (multiple
choice). For each graded element of the midterm (i.e., question or sub-question), we asked
the lecturers to identify the learning objective that was tested by that (sub)question. Per
module, 6-10 learning objectives were tested. As an intermediate exam could consist of 20-
40 (sub)questions, each learning objective was measured using on average almost 4
questions.
Students who would not receive feedback for this midterm (due their assignment to the
control group or their lack of consent) did receive a personal email with their name and the
grade on their exam and some general information (e.g., time left untill the final exam).
Students in the treatment group were sent a personal email at the same moment as the other
students, but in addition to their grade they received information that we denote as
personalized feedback. Feedback started postive by denoting all learning objectives that were
well-mastered (above standard). Next, the learning objectives that scored standard and might
47
need some more attention were addressed including suggestions per learning objective, and
finally the learning objectives that were below standard and for which substantive
improvements were required were addressed, including more extensive suggestions what to
do to achieve these improvements.
3. Results
3.1 Descriptive statistics

All students enrolled for the modules were invited to participate, although for this study we
were only interested in freshmen students. Their was no random selection from the population
involved, but only students that participated in a midterm and the final of a module could be
measured. Hence, per module we registered whether the student participated in a midterm
and could randomly be assigned to the treatment or control group based on the consent they
had provided. For module 2a Supply Chain operations (SCO) and 2b Statistics (Stat), only
freshmen who were assigned to a control or treatment group in module 1 Management
Science (MS) have been considered, as we use a repeated measurement study design.
Table 1. Partipant flow.
Module Total population Midterm Treatment Control group No consent

(#freshmen with participants group (#final exam) or midterm
consent) (#freshmen with (#final exam)
consent)
1. MS 342 (290) 335 (254) 143 (134) 146 (141) 53
2a. SCO 305 (255) 286 128 (118) 158 (147) 19
2b. Stat 319 (231) 270 128 (118) 142 (137) 49
3.2 Attrition effects

Table 1 shows percentage of freshmen with consent that participated in the midterm (i.e.,
assigned to either the treatment or control group) and attrited for the final exam. The attrition
differences that we found show a much larger attrition in the treatment group. Further
examination revelead that these differences cannot be attributed to confounding variables,
such as age, gender, high school GPA, et cetera. However, for students that scored low on
the midterm (lowest 33%) AND received feedback we found a significant effect in the last
Table 2. Attrition effect of feedback after midterm for each module.
Module Attrition in control group Attrition in treatment group

1. MS 3.7% 6.1%
2a. SCO 5.6% 8.9%
2b. Stat 3.7% 8.1%
48
two modules. Students with a similar low score who did not receive feedback had a higher
partcipation in the exam. Hence, feedback to lower scoring students has a significant attrition
effect, resulting in higher attrition as a result of personalized specific feedback.
The results are graphically depicted in the upper part of Figure 1, which shows dfferences
with the control group. If the boxplot crosses the 0-line, there is no significance treatment
effect at 0.05 level.
Figure 1 Treatment effects on Attrition level (upper part) and Grade of final exam (lower part).
3.3 Grade performance effects

We did not find significant effects of providing personalized feedback (treatment) on the final
exam grades in a module. The average grade received by students in the control group was
similar to the grade of students in the treatment group. We had expected that the group of
students who could benefit most from feedback, i.e. who received suggestions for
improvement for only a few learning objectives and hence scored average in the midterm,
would show a significant effect of the feedback received. But our data did not confirm this
hypothesis, for neither of the modules. Moreover, we did not found a significant performance
difference between the lower scoring treatment group students and their control-group
counterparts, notwithstanding the higher attrition rates in the treatment group. We denote this
as an adverse effect of feedback, as personalized feedback has had no clear impact on
performance, only on attrition rate.
49
4. Discussion and Conclusions

The randomized controlled experiment that we performed leads us to two conclusions. First,
we have not been able to demonstrate a positive or negative effect of personalized feedback
on the grade performance of the first year students. Next, we have been able to identify an
adverse effect of personalized feedback in terms of attrition level in two of the three modules,
both located after the students had already taken the first module in their study program.
These conclusions ask for discussion and reflection, as theory suggests mainly positive
effects of personalized feedback. Has the experiment correctly been designed? We believe it
is, as we followed a similar setup as has been used in other repeated measurements field
studies (Levitt & List, 2009). Has the treatment, i.e. the formulation of the feedback, been
provided correctly? We followed guidelines of Hattie & Timperley (2007) and Núñez-Peña
et al. (2015), but we made the choice to provide feedback based on the achievement of
module learning objectives, not on the detailed level of the question itself. Or did we identify
an adverse effect of feedback that should lead us to reconsider the power of feedback?
References
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research,
77(1), 81–112. https://doi.org/10.3102/003465430298487
Levitt, S. D., & List, J. A. (2009). Field experiments in economics: The past, the present, and
the future. European Economic Review, 53(1), 1–18.
https://doi.org/10.1016/j.euroecorev.2008.12.001
Morris, R., Perry, T., & Wardle, L. (2021). Formative assessment and feedback for learning
in higher education: A systematic review. Review of Education, 9(3), 1–26.
https://doi.org/10.1002/rev3.3292
Núñez-Peña, M. I., Bono, R., & Suárez-Pellicioni, M. (2015). Feedback on students’
performance: A possible way of reducing the negative effect of math anxiety in higher
education. International Journal of Educational Research, 70, 80–87.
https://doi.org/10.1016/j.ijer.2015.02.005
50
Authentic assessment in higher education: a collaborative

approach
Department of OPD, School of Business, Munster Technological University, Cork, Ireland.
Abstract
This research explores a collaborative learning experience using Authentic
Assessment between businesses in a European City and a Higher Education
(HE) Institution. Using the umbrella of Authentic Assessment, MBA students
solve a problem through Problem Based Learning (PBL) while working with
businesses and City Hall. Students explore the concepts of ‘best practice’ in
service delivery in the City.
A mixed methods research methodology was applied to gather the primary
data. The aim was to examine this collaborative approach to PBL using
Authentic Assessment for the students while enhancing their overall learning
experience and to gather their feedback and that of participating businesses.
The outcome resulted in the development of a Service Charter for the City,
through an interactive student assessment strategy. Further recommendations
for training business owners on how to deliver ‘best practice’ for service
delivery are also proposed.
Authentic Assessment through collaborative learning works and has many
benefits.
Keywords: Authentic assessment; collaborative learning; problem based
learning; experiential learning; city service charter.

Authentic assessment in higher education: a collaborative approach
1. Introduction and Research Context

Authentic Assessment (AA) is an engaging approach to assessment where students use and
apply knowledge in real-life settings. Many consider it the best approach to assessment for
reasons such as its authenticity, student engagement benefits, opportunities to work with
industries, prospect of observing ‘best practice’ in action, and, from an educator perspective,
it reduces the opportunities for contract cheating. Authentic Assessment in higher education
is concerned with working in conjunction with external organisations where students at all
levels are given the opportunity to explore real life and real world issues. Ultimately, the
emphasis is on better ways to assess work (McArthur, 2023). The Authentic Assessment
examined here is the work of an MBA cohort as they look for new ways to improve service
delivery and are tasked with writing a ‘Service Charter’ for a City.
1.1. Background to the Study:Developing a Service Delivery Experience through AA

A collective excellence in ‘service delivery’ creates an experience for the consumer which is
memorable, affirming, and, most importantly from the commercial perspective, it makes the
consumer eager to repeat it. While each business operation has its own style, personality, and
particular environment, there are certain constants in the provider-consumer interchange
which will, when taken all together in a city, paint a picture of either satisfaction or dislike,
contentment or discomfort, efficiency or sloppiness. Using Authentic Assessment and
collaborative learning while working with businesses to improve this service experience in a
City is the context here. The Collaborative learning process centred on a group of 22 MBA
students while they interacted with 20 businesses in a European City to research, develop and
write a Service Charter. Details of the development of the charter per se are not dealt with in
this paper, just the experience of its development by the students and business alike while
being authentically assessed.
Finding novel ways to assess third level students is always a challenge for Higher Education
Institutions; hence the value of placement at the fulcrum of learning and business
development through a collaborative learning challenge and experiential learning. An
experimental approach was afforded to MBA students when they were tasked with writing a
‘Service Charter’, while, in parallel, being assessed through ‘problem solving’ for 5 ECTS
credits with the third level partner. The dual aspect of learning between businesses and
college began when students, using Authentic Assessment, sought to solve a problem for City
businesses and present a solution, and the second, when a recommendation came from the
research that the businesses needed to undertake further training in order to implement the
plan of the Service Charter. Addressing inconsistencies in the service model and
synchronising efforts to grow commercial activity is best addressed through cooperation in
the competitive space, an approach known as ‘co-opetition’, coined in 1913 by the Sealshipt
Oyster System in the USA (Jansen van Nieuwenhuizen, 2016) to describe the idea of
52
cooperative competition, or cooperating with competitors. Its principles are that in

cooperating with one another you are creating more business, you are in ‘coopetition’ not
competition, working together to common advantage (Jansen van Nieuwenhuizen, 2016).
Co-opetition is a perspective on business relationships which highlights the ambivalence of
competition and cooperation (Stein, 2010). The City business community at the core of this
paper is now leading the way, joining together to establish a coordinated approach to the
quality of service in their City. Also, for educators there is a need to move beyond the
conceptualisation of education as a simple acquisition of knowledge to one which nurtures
and assesses innovation and expertise in the utilisation and application of knowledge,
(Boland, 2010), achieved here through Authentic Assessment.
1.2. The Taks: Develop A City Service Charter Using Authentic Assessment
A relatively new concept and not widely embraced (limited literature), the purpose of a
Service Charter is to outline and detail the standards a customer/visitor/client can expect
when engaging with service providers in a City. A City Service Charter is a collective
commitment by all those engaged in the daily service life of the City to focus on the needs
and preferences of their customers, motivated by values such as respect, integrity and
excellence. Among an extensive set of aspirations, some of the following desirable ones are:
1.Commitment to cooperation by all sectors of service providers in instilling a sense of ‘pride
of place’ and service excellence; 2.Enhancing the customer experience by being attentive to
all aspects of City life; 3.Constantly monitoring and communicating; 4.Ensuring staff are
trained to be thoughtful, courteous, motivated. When writing a Service Charter, it is necessary
to define the purpose, scope and standards of your business’s commitment to customer
service so that employees and customers know what to expect (smallbusiness.chron.com).
2. Literature
This section briefly examines some of the terms in the context of the literature that were
applied in this investigation; assessing in Higher Education; starting with Authentic
Assessment the main assessment approach that encompasses, Problem Based, Experiential
& Collaborative Learning.
2.1. Authentic Assessment, Problem Based, Experiential & Collaborative Learning

Authentic Assessment concerns students in higher level institutions deeply examining issues
in a real-life setting that are relevant to their field of study (Ashford-Rowe et al., 2014;
McArthur, 2023). This type of assessment allows the students to become immersed in the
real-life problem, transferring later to the future graduate’s skills that they bring to the
workplace (McArthur, 2023). Problem Based Learning (PBL) an element of Authentic
Assessment began in the the medical and health sciences, progressing into mainframe Higher
53
Education. A learner-centered approach, PBL is where students engage with a real-life

problem (Savery, 2006). PBL is complex in nature and not simply a teaching technique, but
a total educational strategy (Barrett, 2005). PBL is a very specific approach to education
learning and it affords students space and time to define their own learning and to be creative
(Barrett, 2005; Kahn & O’Rourke, 2005). Students are tasked with developing novel
solutions which facilitates interactivity with businesses and each other to enhance learning
(Barrett, 2005). Experiential learning also enhances student learning in higher education and
(Kolb & Kolb, 2107) developed a model of learning based on the notion that the best learning
is achieved though involvement, reflection, and action, placing action into the management
learning model.
Looking at collaboration, Roschelle & Teasley (1995) found it to be a coordinated,
synchronous activity that is the result of a continued attempt to construct and maintain a
shared conception of a problem. Dillenbourg (1999) found that it is challenging to agree on
what collaborative learning is and, in its broadest context, means that learning happens
together. Learning can involve two or more people, groups, or communities learning together,
and at all levels. Dillenbourg (1999) suggests that collaborative learning is an activity of joint
problem solving, and learning is expected to occur as a side-effect of problem-solving,
measured by the elicitation of new knowledge or by the improvement of problem-solving
performance. The pedagogical sense is prescriptive; one asks two or more people to
collaborate because it is expected that they will thereby learn efficiently. The psychological
sense is descriptive: one observes what two or more people have learned, and collaboration
is viewed as the mechanism which caused learning (Dillenbourg, 1999).
3. Methodology
The study explores Authentic Assessment practices and seeks to understand if there is a
benefit using this assessment strategy in collaborative learning experiences using PBL.
Twenty-two MBA students and 20 businesses were involved in this research as well as one
member from City Hall. A mixed methods approach was applied and both sets of data were
triangulated (Patton, 2012). To examine the experience of the learning of the students a
positivistic research methodology was applied as quantitative facts can be directive and
informative (Harvey, 1998). A quantitative survey instrument using a Likert scale was
developed after the learning, and in conjunction with colleagues involved in department
programme development. The questionnaire was tested to eliminate any errors and cleansed
prior to execution. All 22 MBA students were asked to fill out the survey in person in a class
setting to ensure 100% participation. The survey was completely confidential so that private
contributions and negative experiences where they arose could be provided in confidence
(Harvey, 1998). All ethical concerns were addressed including informed consent,
withdrawal, data used and storage in line with GDPR. Students were given the freedom to
54
make genuine comments about their experiences. Comment boxes were also included to
gather deeper insights and opinions which are informative. For the second element of the
research, the business participants were asked in person about their experience with working
with the students and their implementation of the Service Charter. A qualitative methodology
was applied as it was believed that a post-positivistic approach (Patton, 2012) would gather
in-depth answers from participants about their experiences. In all, 20 City centre businesses
participated in the interviews. An interview guide consisting of 10 questions which focused
on their overall experience, learnings, and suggestions for the future. The City Hall contact
was also asked to participate in the research to add to the data. A pilot interview was
conducted, and all cleansing and ethical considerations were applied. Due to the importance
of this Service Charter and the future of the project, all the businesses who participated in the
project completed the interview along with the City Hall; 21 participants in total gave their
full consent.
4. Findings & Discussion

This section sets out some of the pertinent findings; however, the restriction of the paper
limits the presentation of all findings.
4.1. AA Student Experience

Authentic Assessment, examining PBL at MBA level was found by the students to be
appealing and stimulating. Of the 22 students, 87% stated that is was a positive experience.
Stress was mentioned by 5% who maintained that it was a taxing experience overall. When
asked about solving problems together as a group to come up with a plan, 66% stated that
working in a class group was enjoyable overall. In terms of personal development, 39% stated
that it improved their professional development skills, with 30% stating that the process will
benefit their interpersonal and employability skills in the future due to the specific nature of
the task of dealing with City businesses and City Hall.
From a negative stance, students outlined that the real-life challenge was stressful as they
were tasked with delivering a working document that was to be implemented by City Hall
and participating businesses. 25% of the students worried initially that they may not be “up
to the task”, or, “the standard required”, as the work was of the level of a “professional
consultancy”. “I was gravely concerned initially; what if it all ‘fell flat’ and was not of the
standard expected by City Hall, and also, how could this impact on my overall MBA results?”
Another student stated that they were “worried that they may not be able to deliver the
required document in the given term time”.
Other positive encouraging comments included; “I felt excitement, joy and happiness when
we met the deadline and the City representatives loved our Service Charter”. Another student
55
stated that “I cannot explain the sheer delight at seeing the final colour version of our plan in
beautiful red, the colour of our City”. “It was such an honour to complete this work for City
Hall”. The final statistic showed that the students were very satisfied with this Authentic
Assessment for MBA level; 90% stated that they would love more collaborative learning
with industry in the future for other modules.
4.2. Experience of the City Businesses

Findings from the City businesses were extremely positive. They enjoyed working with the
students on developing the Charter. Positive comments included, “I enjoyed working with
younger people to determine how they viewed our City and our service delivery”. Another
participant said it “was a novel experience and I learned so much more about overall service
delivery”. The next retailer said, “we have to be competitive as a City and this project focused
our minds on how to be more competitive!”. Two contributors mentioned that businesses
need to work more closely together and not to dwell on competitiveness”. This of course is
the whole ethos behind co-opetition. “The three new principles for the City developed by the
MBA students from their research of ‘Service, Customer, and City Principles’ are something
that we can now strive to achieve as businesses and in a wider context as a City”. Other
participants greatly appreciated the training offered to learn how to implement the charter
that the students devised. “It was exciting returning to further training”. “We learned so much
from it”.
Some negative comments included, “the timing was bad for my business”. For me, “the
induction day was too long as I had to find staff cover”. Timing will always be an issue for
us. We are retailers always on the go”, but the “summertime was definitely the wrong time
due to staffing issues”. City Hall stated that one of the learnings was to run the training just
before the summer as many shops get temporary staff in over the summertime, so the mystery
shopper audits can be affected. For the next roll out, we will run it in February. The next
challenge will be to move beyond the 20 businesses that participated and to have a City-wide
implementation of the new City Service Charter.
The following Figure 1 outlines the cycle of learning from the process:
Students:
City Business City & HE
Learn
Led by City Institution:
Supervisor Experience Hall:
Problem: Lead: Reevaluate
Solve Problem Train Busines the Process
Develop a City HE Institution Develop Owners
Service Charter & City Business Further
Charter Use Charter Training for
Collaborate
Are Improve Business
Authentically Service Owners
Assessed
Figure 1. AA - Collaborative Educational Experiential Learning Cycle
56
5. Conclusion
Authentic Assessment is a challenging process for students and lecturer alike; however, to be
challenged is to learn and to learn one must be challenged. Setting a real problem through
AA and encouraging enquiry, is a perfect way to assess students in Higher Education settings
(Biggs, 1999). The findings of this research are very positive in the context of AA using PBL
and collaborative learning with a HE institution and local businesses. Theorists provide much
debate around the ideas & philosophies for the use of PBL and advocates of collaborative
and PBL provide compelling benefits in the literature for their use. This has been reflected in
this current research. Using AA and collaborating through problem solving can provide
original and exciting challenges for instructors and students. The benefits of this type of
learning experience and assessment for the students especially at MBA level are all-
encompassing as MBA students need and deserve to be challenged. PBL means the students
can become part of a ‘real life’ collaborative team in an organizational setting; in this case a
City scape setting. Authentic Assessment provides relevant student engagement and although
it is a more challenging process for lecturers to oversee, the realities are that it helps to
counteract any opportunities for contract cheating, now real concerns for Education. Overall,
solving the problem, developing the charter in this case and the continuation of the learning
was a very positive experience for all. The MBA students helped decipher the mindset of the
City business community by conducting surveys among regular shoppers, tourists, occasional
visitors and residents, and consulting broadly with many interested contributors. Students
became richly engaged in the process. The end of this AA resulted in a valuable Service
charter which will guide the consumer experience in the City into the future and set a
precedent for similar charters throughout Europe. The operationalization of this charter
continued the cycle of learning, as all the 20 service providers who signed up to the charter
undertook a short training courses to facilitate its implementation. This training will be
ongoing, with protagonists engaging with those who experience the City routinely. The
Service Charter, guided by values such as pride of place, integrity, accountability, respect
and excellence, will lead to improved service delivery, customer service and, ultimately,
increased profits in a positive cooperative community, while the benefits to the HE
community can be reassurance as to the reliability of this assessment approach that avoids
opportunities for assessment deception by students.
References
Ashford-Rowe, K., Herrington, J., & Brown, C. (2014). “Establishing the critical elements
that determine Authentic Assessment". Assessment and Evaluation in Higher Education,
39(2), 205–222.
Barrett, T. (2005). “Understanding Problem Based Learning”, Handbook of Enquiry and
Problem-based Learning: Irish Case Studies and International Perspectives, No 2., (eds),
57
Terry Barrett, Iain Mac Labhrainn & Helen Fallon, Galway, Centre for Excellence in
Learning and Teaching, AISHE Readings, pp 13-25.
Biggs, J. (1996). “Enhancing teaching through constructive alignment”. Higher Education,
Vol. 32, PP 347–364.
Biggs, J. (1999). “What the student does: teaching for enhanced learning”, Higher Education
Research and Development, Vol. 18, No. 1 PP 57-75.
Boland, T. (2010). “Joint Oireachtas Committee on Education and Skills Meeting on
Curriculum Issues”, https://studylib.net/doc/7365394/the-fundamental-challenges-of-
higher-education-are-teachi..., accessed 26/01/2019
Dillenbourg, P (1999). “What do you mean by collaborative learning?”In P. Dillenbourg (Ed)
Collaborative learning: Cognitive and Computational Approaches, Oxford: Elsevier.
Harvey, J. (1998). “Selecting your student sample”, in Evaluation Cookbook, ed., Jen
Harvey, Scottish Higher Education Funding Council, Edinburg: The Learning
Technology Dissemination Initiative.
Jansen van Nieuwenhuizen, P.J. (2016). “Partnering with the Enemy A Generic View of
Competitive Paradox for South African Financial Service Providers”, Johannesburg:
Growth Institute (PTY) Ltd, P1-23.
Kahn P. & O’Rourke, K. (2005). “Understanding Enquiry-Based Learning”, Handbook of
Enquiry and Problem-based Learning: Irish Case Studies and International Perspectives,
No 2. (eds), Terry Barrett, Iain Mac Labhrainn & Helen Fallon, Galway, Centre for
Excellence in Learning and Teaching, AISHE Readings.
Kolb, A., Kolb, D. (2017). “Learning Styles and Learning Spaces: Enhancing Experiential
Learning in Higher Education, Academy of Management Learning & Education, Vol. 4,
No. 2 , November 30.
McArthur, J. (2023).“Rethinking authentic assessment: work, well-being, and society.
Higher Education, Springer, 85, 85–101, available at https://doi.org/10.1007/s10734-
022-00822-y.
Patton, M.Q. (2012). Qualitative Research and Evaluation Methods, London: Sage.
Roschelle, J. & Teasley S.D. (1995). “The construction of shared knowledge in collaborative
problem solving”. In C.E. O'Malley (Ed), Computer-Supported Collaborative Learning.
Berlin: Springer-Verlag, pp. 69-197.
Savery, J.R. (2006). “Overview of Problem-Based Learning: Definitions and
Distinctions”,Interdisciplinary Journal of Problem-Based Learning, Vol 1., No. 1.
Smallbusiness.chron.com. Available at https://smallbusiness.chron.com/write-customer-
service-charter-66472.html, accessed 26/01/19.
Stein, H.D. (2010). “Literature Overview on the Field of Co-opetition”, Verslas: teorija ir
praktika, Vilnius Gediminas Technical University, NO. 3, PP 256-265.
58
Mind the gap: factors which inhibit supporting student teachers

to engage in action research while on school placement
Maria Moore, Dermot O’Donovan, Pauline Logue
Department of Creative Education, Atlantic Technological University, Ireland.
Abstract
In the Irish context, there is an expectation for research to be embedded across
the continuum of the teaching profession, including initial teacher education.
The Teaching Council has set out the requirement for student teachers to
engage in research on their own practice, while on school placement, which
links learning in their higher education institutions and the host school. This
paper aims to examine some of the factors which inhibit support for student
teachers to engage in action research while on school placement. This case
study uses a constructivist and interpretivist philosophy. The design and
analysis are underpinned by the triangulation of qualitative data collection
through questionnaires, focus groups, group interviews and semi-structured
interviews. Themes are developed using thematic analysis. Key findings
include challenges encountered in supporting student teachers to engage in
action research, not least conducting research ethically, and the power
dynamics at play in school placement.
Keywords: Initial teacher education; action research; student teachers; school
placement.

Mind the gap: factors which inhibit supporting student teachers to engage in action research
1. Introduction
Increasingly, knowledge-based societies require a consistent supply of quality teachers who
are equipped with the skills to actively engage in ongoing professional development and to
conduct school-based research to develop a culture of professional learning and sharing in
school settings (Darling-Hammond, 2017). A teaching practicum, or school placement is a
mandatory part of the preparation for teaching in most jurisdictions (OECD, 2022) and is
regarded by the Irish Teaching Council as the ‘fulcrum’ of teacher education (Teaching
Council, 2020, p. 9). By integrating new pedagogical knowledge and skills into initial teacher
education (ITE) and working in partnership with schools to develop reflective practice,
student teachers are facilitated to incorporate theory and practice (Hall et al., 2018; OECD,
2019).
The Department of Creative Education in Atlantic Technological University (ATU)
undertook funded1 research to develop a framework model to support student teachers of
technological subjects engage in action research while on school placement. This paper aims
to present initial findings that identify some of the challenges which inhibit student teachers
from engaging in action research on school placement. This paper contributes to the growing
field of study relating to teacher education in Ireland, with a specific focus on technical
education. It begins by examining literature pertinent to school placement and follows with
a description of the methodology employed to gather data. Findings from the research are
discussed and the paper concludes with some recommendations. The term ‘co-operating
teacher’ is used to describe a teacher who provides support for a student teacher on school
placement. Teaching Council documentation refers to this person as a Treoraí, the Irish word
for ‘guide’.
2. Literature
The quality of an education system depends on the high standards of its educators (OECD,
2019). Reflective practice and research-informed teaching are recognized internationally as
supporting quality teaching (Darling-Hammond, 2021). Significant developments have been
made in ITE in Ireland in the last 30 years, in response to changing social and economic
demands, as well as in response to a proliferation of academic research in this area (Hall et
al., 2018). Among these changes was the establishment of the Teaching Council in 2006,
which is tasked with establishing and increasing high standards for teaching and teachers
(Teaching Council, 2020). The Teaching Council aim to reconfigure ITE to meet the ever-
changing demands of society and learners in the 21st century. Their Research Strategy sets
out three pillars “Research, Reflective Practice and Relationships” to support research-
1
This study was funded by a Research & Innovation Strategic Endowment (RISE) Scholarship from ATU.
60
informed teaching (Teaching Council, 2016, p.1). As well as informing policy and decision
making in the Teaching Council itself, the strategy also aims to develop a “vibrant research
culture within the profession, whereby teachers as reflective practitioners and enquiry-
oriented learners are actively engaging in and with research” (Teaching Council, 2016, pp.1-
4). This strategy aligns with findings from international studies of high functioning education
systems, where there is a focus on research in teacher education, and where graduate teachers
are equipped to use research critically and reflectively (Sahlberg et al., 2012).
Of relevance to this paper is a requirement by the Teaching Council for student teachers on
school placement to “engage in research on their own practice”, that makes connections
between their learning in higher education institutions and practice in school (Teaching
Council, 2020, p. 13). Internationally, practitioner research is viewed as being particularly
suitable for student teachers in ITE (Cochran-Smith et al., 2009). It is premised on the belief
that they can develop professionally by examining and reflecting on their own learning, as
well as their teaching and learning (Qing-li et al., 2019). Action research is one approach. It
is based on a cyclical processs of self and critical reflection, evidence gathering, planning,
inplementation and review (Mc Niff, 2017).
3. Methodology
Using a constructivist and interpretivist philosophy, the research design for this case study
facilitates triangulation of qualitative data collection, through questionnaires, focus groups,
group interviews and semi-structured interviews. It has a single specific area of focus, which
is to gain an insight into some of the factors which inhibit support for student teachers to
engage in action research while on school placement. This gives a holistic view of the issues
from the perspectives of different stakeholders (Denscombe, 2017). Reflexive thematic
analysis was used to develop themes from the qualitative data (Braun & Clarke, 2006, 2019).
Ethical approval was obtained from the Research Ethics sub-committee of ATU, and this
study abided by the ethical standards set out in the ATU research ethics policy.
Participants were selected strategically, based on their expertise in the field of education and
their subject specialist knowledge. The population represented four groups of education
stakeholders. Group one consisted of post-primary school principals (n=15), and group two
of post-primary subject specialist teachers of Design and Communication Graphics and
Construction Studies (n=10). For these groups, stratified random sampling was used to create
a sampling frame and schools were selected proportionally from the Department of
Education’s online database of schools in Ireland (Department of Education, 2021). Group
three comprised of school placement tutors (n=6) from ATU, reflecting a mix of genders and
experience levels, who participated in a focus group. Group four consisted of education
experts (n=3), identified based on their own field of study in ITE, who took part in semi-
61
structured interviews with the researcher. While this paper focusses on the stakeholders
pertinent to school placement, due to time constraints, the scope of this study excluded
student teachers, their pupils, and the parents/guardians of pupils.
4. Findings
In the online questionnaire and interviews, participants were asked to consider from their
perspective, “What challenges might there be for student teachers to engage in action
research?” Power dynamics and the challenges of meeting requirements to research ethically
were identified as potential inhibiting factors, which are discussed further below. Participants
were anonymized using the following key: principal in questionnaire (PQ 1-15), subject
specialist in questionnaire (SSQ 1-10), principal in group interview (PGI, 1-3), subject
specialist in group interview (SSGI), school placement tutor (SPT 1-6), subject specialists
(SS1-3) and education experts (EE 1-3) in semi-structured interviews.
4.1. Power Dynamics

That tensions in professional relationships exist in schools between principals, teachers, and
student teachers, emerged during the analysis of data. Participants suggested that power
dynamics could negatively impact on the student teacher’s ability to fully engage in action
research while on school placement. An aspect of this was the predicament which might exist
if student teachers were pressurized by school management to engage in supervision of
classes, or other school activities, at the expense of time allocated for college attendance or
research practices. EE3 described how, in their experience, student teachers are faced with
the dilemma of “wanting to do the work for the principal, for financial reasons, but also to
keep the principal happy, but…they’re trying to also do their thesis.” EE1 suggested that the
solution was in empowering student teachers to advocate for themselves, to enable them to
address the power dynamic and to withstand the pressures which might be exerted on them
in schools. However, EE1 also acknowledged that the ad hoc nature of school placement
impeded structured supports: “we help the students advocate for themselves, but in practice
because of that lack of formalised structure, and the power dynamic...some students just do
not get the support for a range of reasons.”
Another aspect of the power dynamic related to the role of the co-operating teacher. Teaching
Council documentation states that “the student teacher shall discuss their research plans with
the Treoraí [co-operating teacher], as they have overall responsibility for the class” (Teaching
Council, 2020, p. 19). However, EE1 noted that in Ireland “teachers aren’t chosen for their
skills as co-operating teachers, they are chosen because they are the class teacher… so the
age-old issue is that co-operating teachers are not trained or recognized [as co-operating
teachers]”. This perception suggests that there may be a reticence, reluctance or even a lack
of skill, on the part of the teacher, in their role as co-operating teacher and researcher. This
62
reticence was evident in the online questionnaire from both principals and subject specialists,
who were asked to outline the challenges there might be for student teachers to engage in
action research. SSQ5 suggested that there may be a “reluctance of schools and staff to
participate”, while PQ8 intimated an “unwillingness of others to engage”. PQ6 implied that
there might be a challenge of “getting the co-operation from other staff who might be willing
but not see it as a priority”. A different aspect of the power dynamic was articulated by PQ5
who expressed this as “(how) to explain what is involved to students and how to ensure
teachers don't feel vulnerable”. This alludes to a consciousness on the part of teachers around
deficiencies in their knowledge about action research, which may challenge their own
perceptions of being an expert. For some, this may result in them feeling exposed
professionally. These and similar comments, allude to challenges for student teachers who
are required to engage in research, yet may face a lack of support from staff in schools.
4.2. The Challenges of Researching Ethically

Researching ethically is based on the principles of honesty, reliability, respect, and
accountability (All European Academies, 2017). A range of challenges to researching
ethically emerged in the analysis of the research data. PQ3 identified the principle of
informed consent: “I think it is important that all involved know the research is taking place”.
However, there did not appear to be alignment between respondents as to how this should be
achieved. PGI1 was prepared for student teachers to use the school’s consent policy,
however, they would be constrained by this and “you’d have to be very clear with the students
that they obviously are bound by the school policies and have to work within those
parameters”. This differed from PQ9 who wrote, “I would be happy that parents and students
are aware and that parents can sign for their students not to participate only”. While PQ8
noted that “without signed permission the school could find itself it a trick[y] situation.”
Others held differing views. SSQ9 felt that “depending on the nature of research and
assuming student work is anonymous, I don't see the need for informing parents/students”.
These comments highlight both the varied understanding of the ethical requirements for
consent and the range of differing approaches in place in schools.
Another aspect relating to informed consent emerged, that of problems which the requirement
for consent from participants might pose for the student teacher undertaking research. As
EE1 put it:
One of the other issues was parental consent…the age-old issue of handing out these
letters and getting them back in. You know that they were never coming back in.
So, it's… caused issues with the data collection...What do we do with kids who can't
be involved?
There are a range of potential challenges raised, including when data collection can begin
and how to navigate lack of permission with pupils in the class. Other concerns were raised
63
by SS3 who suggested that the refusal of parental consent might single a pupil out in class
“(t)he student themselves might feel excluded…everybody else in the classroom is handed
out a questionnaire and…they're not allowed to … take part in it”. This could result in
classroom management issues for student teachers, as they navigate how to deal sensitively
with the situation. The literature suggests that where consent is not obtained, the researcher
“should decide whether this was an active refusal of consent, in which case they would need
to respect this and find a practical solution.” (BERA, 2018, p. 12). As these are real issues, a
policy might be devised to support student teachers in dealing with these scenarios.
Another area on which there was significant consensus, was on the issues raised by the
General Data Protection Regulation (GDPR). Enacted in 2018, there is a high awareness of
this legislation in schools. Among the respondents in the online questionnaire this knowledge
was in evidence, as articulated by PQ1 “(t)he whole area of GDPR needs to be respected by
and complied with by everyone concerned”. However, there were few specifics as to what
this might include, apart from the legal requirements involved and here seemed to be some
confusion between GDPR and informed consent, as indicated by SS3 “GDPR would be
another issue…and you would find that…could be one of the most difficult parts of the work,
is actually getting permission from parents…whether it be a signed document or …getting
them to respond to an email.” EE1 summed up the impact of these difficulties “I think it's
getting more and more difficult involving the pupils in the schools”. However, once there is
an awareness of the issues, it may be possible to put measure in place to address them. In
relation to gathering informed consent, SSGI1 proposed that it might be of use “if there was
some sort of…a consent form, with…the header from [name of TU]” Therefore, a generic
form could be designed, and adapted for use by student teachers, to be shared with research
participants in schools.
5. Conclusions
The findings presented in this paper suggests the requirement for a national policy on
informed consent, which could be developed by the Teaching Council as part of their
strategic plan for 2022-27. Their goal for developing collaborative professional communities
for sustainability, includes developing greater collaboration between stakeholders in
education (Teaching Council, 2021). However, this is a complex issue, as the Teaching
Council, while responsible for developing effective policies, regulation, and research in
relation to education, are not responsible for the dissemination of this at a local level in
schools. The recommendations of the School Placement Working Group, in relation to the
training of co-operating teachers, awaits implementation (Teaching Council, 2019).
The aim of this paper was to examine some of the factors which inhibit support for student
teachers to engage in action research while on school placement. Pressure on student teachers
64
to engage in supervision and substitution of classes in schools, issues around effective support
from co-operating teachers, who may feel professionally vulnerable, a lack of consensus on
informed consent and difficulties posed by GDPR, have been identified as potential
inhibitors. In the absence of a national policy, one recommendation of this paper is that the
Department of Creative Education in ATU develop documentation and resources which
support action research, informed consent and GDPR for student teachers engaging in action
research while on school placement.
References
All European Academies [ALLEA]. (2023). The European code of conduct for research
integrity. Retrieved from ALLEA | All European Academies: https://allea.org/code- of-
conduct/
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative
Research in Psychology, 3(2), 77-101. Retrieved from
http://dx.doi.org/10.1191/1478088706qp063oa
Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative
Research in Sport, Exercise and Health, 11(4), 589-597.
https://doi.org/10.1080/2159676X.2019.1628806
British Educational Research Association [BERA]. (2018). Ethical Guidelines for
Educational Research (4th ed.). Retrieved from https://www.bera.ac.uk/researchers-
resources/publications/ethical-guidelines-for-educational-research-2018
Cochran-Smith, M., & Lytle, S. L. (2009). Inquiry as stance: Practitioner research for the
next generation. New York: Teachers College.
Cochran-Smith, M., Friedman, A., Barnatt, J., & Pine, G. (2009). Inquiry on inquiry:
Practitioner research and student learning. Action in Teacher Education, 31, 17-32.
https://doi.org/10.1080/01626620.2009.10463515
Darling-Hammond, L. (2021). Defining teaching quality around the world. European
Journal of Teacher Education, 295-308.
https://doi.org/10.1080/02619768.2021.1919080
Denscombe, M. (2017). The good research guide (6th ed.). London: Open University Press.
Department of Education. (2021, october 8). Find a school. Retrieved 2021, from gov.ie:
https://www.gov.ie/en/service/find-a-school/
Hall, K., Murphy, R., Rutherford, V., & Ní Áingléis, B. (2018). School placement in initial
teacher education. School of Education, University College Cork (UCC). Retrieved from
https://www.teachingcouncil.ie/website/en/publications/teacher-
education/documents/school-placement-in-initial-teacher-education.pdf
Mc Niff, J. (2017). Action research: All you need to know. SAGE.
OECD. (2019). A Flying Start: Improving initial teacher preparation systems. Paris: OECD
Publishing. Retrieved from https://doi.org/10.1787/cf74e549-en.
OECD. (2022). Education at a glance 2022: OECD indicators. Paris: OECD Publishing.
https://doi.org/10.1787/3197152b-en
65
Qing-li, H., Torres, M. N., & Shi-Ji, F. (2019). Collaborative action research for preparing
teachers as reflective practitioners. Systemic Practice & Action Research, 411-427.
https://doi.org/10.1007/s11213-018-9461-z
Sahlberg, P. (2018). The Structure of teacher education in Ireland: Review of progress in
implementing reform. Higher Education Authority. Retrieved from
https://hea.ie/resources/publications/the-structure-of-teacher-education-in-ireland-
review-of-progress-in-implementing-reform/
Sahlberg, P., Munn, P., & Furlong, J. (2012). Report of the international review panel on the
structure of initial teacher education provision in Ireland:Review conducted on behalf of
the Department of Education and Skills. Department of Education and Skills. Dublin:
Department of Education and Skills. Retrieved from
https://assets.gov.ie/25013/6e0e25a092e7409ea4212db97cdad1b0.pdf
Teaching Council. (2016). Research strategy. Retrieved from The Teaching Council:
https://www.teachingcouncil.ie/en/_fileupload/research/research-strategy-updated.pdf
Teaching Council. (2019). Report and action plan of the School Placement Working Group.
Maynooth. Retrieved from https://www.teachingcouncil.ie/en/teacher-education/initial-
teacher-education/school-placement/school-placement-working-group.pdf
Teaching Council. (2020). Céim: Standards for initial teacher education. Maynooth: The
Teaching Council. Retrieved from https://www.teachingcouncil.ie/en/news-events/latest-
news/ceim-standards-for-initial-teacher-education.pdf
Teaching Council. (2021). Teaching and learning for all. Maynooth: The Teaching Council.
Retrieved from https://www.teachingcouncil.ie/en/publications/strategic-plan/teaching-
council-strategic-plan-2022-to-2027.pdf
66
Ability of Spanish preservice teachers to differentiate between

creative and reproductive activities in the classroom
Eva Izquierdo-Sanchis1, Joan-Josep Solaz-Portolés1, Antonio Martín-Ezpeleta2,
Yolanda Echegoyen-Sanz1
1
Department of Experimental and Social Sciences Teaching, University of Valencia, Spain,
2
Department of Language and Literature Teaching, University of Valencia, Spain.
Abstract
Creativity is a key competency for the 21st century society and the development
of certain scientific aspects related to creativity should constitute a priority in
education. But to do so, teachers need to have the skills, confidence and
pedagogical knowledge necessary. This paper investigates the ability of 190
Spanish future teachers to differentiate between creative and reproductive
activities in the science classroom. Using the instrument prepared by Newton
and Newton (2010) they were asked to rate a set of activities according to their
capacity to develop scientific creativity. The results show that both the field of
knowledge and the scientific topic influenced their responses, although they
were able to differentiate between creative and reproductive incidents, with
statistically significant differences in both topics studied.
Keywords: Scientific creativity; classroom activities; preservice teachers.

Ability of Spanish preservice teachers to differentiate between creative and reproductive activities
1. Introduction
The importance of creativity and creative thinking is recognized in the OECD reports and it
has been included in the last PISA tests in 2022 (OECD, 2019). It is also considered a key
competence for the 21st century (Henriksen et al., 2016). Consequently, there are references
to creativity in the syllabus of many countries (Patston et al., 2021) and the recent Spanish
Educational Law states that “[...] artistic creation, audiovisual communication, digital
competence, the promotion of creativity and the scientific spirit will be worked on in all areas
of Primary Education” (LOMLOE, 2020, p. 122873).
Thus, creativity should be present in all school disciplines and teachers will be responsible
to training students with creative skills and competencies (Soh, 2017). However, the
difficulty in defining creativity and the poor teacher training regarding creativity causes
hesitation and doubts in its implementation (Mullet et al., 2016). Early childhood and primary
teachers have stated that they feel unprepared to effectively promote student creativity in the
classroom (Cheng, 2010). The study of Bereczki & Kárpáti (2018) show that inservice
teachers, despite holding positive beliefs towards creativity, did not propose activities that
foster creativity in their teaching practices. Recent studies with Spanish preservice teachers
(Echegoyen & Martín-Ezpeleta, 2021; Martín-Ezpeleta et al., 2022) show that, despite their
creative self-perception, their real creativity demonstrated in different academic works is not
sufficient to promote creativity in the classroom.
With respect to science, there is an international consensus that it is intrinsically creative, and
that the development of certain scientific aspects related to creativity should constitute a
priority objective of education (Hetherington et al., 2020). There are different factors that can
influence scientific creativity such as convergent and divergent thinking, general creativity
and domain specific creativity, inquiry skills, or knowledge about science, among others.
Newton and Newton (2010) proposed three types of activities to enhance scientific creativity
in the classroom: a) through the speculative description of situations with tentative,
hypothetical explanations and their possible alternatives; b) gathering knowledge and
assessing ideas (such as proposing a method to gather reliable and descriptive information or
an empirical test that allows contrasting a hypothesis); and c) applying scientific knowledge
to solve a daily problem.
There are, however, different studies that point out to a lack of teacher preparation to
adequately implement creativity in the science classroom (Ramnarain, 2018), despite its
importance. Newton and Newton (2010) found difficulties in some primary education
teachers to distinguish between creative and reproductive activities in the science classroom.
Cruz et al. (2020) found that primary preservice teachers were not completely able to pose
questions related to the primary education curricula for the development of scientific research
activities. Another study (Izquierdo & Solaz-Portolés, 2022) shows that the inquiry skills of
68
Eva Izquierdo-Sanchis et al.
Spanish primary preservice teachers are low. In this context, this study aims to evaluate the
ability of Spanish preservice teachers to differentiate those science classroom activities able
to promote creative thinking from others that were just reproductive activities were no
creative thinking was involved.
2. Methodology
An ex post facto descriptive design was used to analyze the ability of Spanish preservice
teachers to differentiate between creative and reproductive activities in the science classroom.
2.1. Participants
Participants were 190 students in their sophomore year of the Early Childhood Education
Teaching and Primary Education Teaching degrees of a large Spanish university. This was a
convenience sample pertaining to 6 different natural groups. 163 (85.8%) were female and
27 (14.2%) were male, wich corresponds to the population of study.
2.2. Instrument
The proposed classroom activities were selected from the instrument developed by Newton
and Newton (2010). Originally this instrument was comprised of 36 short classroom incidents
in three dissimilar science topics: “Earth, Space and Gravity”; “Electricity”; and “Plants and
Animals”. For the present study only 20 incidents related to the first two topics were used to
not overload the participants. There were eight incidents favouring creative thought in
science and eight incidents biased towards reproductive thought in science (four related with
descriptive science and four with explanatory science). These 16 items corresponded to Field
1 (constructing notional scientific knowledge such as speculative descriptions of situations,
tentative explanations, hypothesis, etc.) and Field 2 (constructing empirical ways of gathering
knowledge and evaluating ideas). Four additional items were related to incidents in Field 3
(applying scientific knowledge to solve a practical problem). The various categories of items
(descriptive-creative, descriptive-reproductive, explanatory-creative, explanatory-
reproductive, problem solving-creative, problem solving-reproductive) appeared equally for
each of the two topics. The degree of stimulation of creativity in each situation is asked to be
rated using a 5-level Likert scale, where the lowest level means “no opportunity to develop
scientific creativity” and the highest-level means “a very good or great opportunity to develop
scientific creativity”.
2.3. Data analysis

Descriptive statistical analysis was done using SPSS software version 28. Particularly mean
and standard deviation was calculated for each of the variables. To check the normality of
the distributions Kolmogorov-Smirnov test for one sample was used. Although not all of
69
them presented normal distributions, the fact that mean and median values were similar and
the analysis of Q-Q plots showed little deviations of normality (in the distribution tails)
suggested the use of parametric tests. Thus, Students’ t test for paired samples was used to
compare the ability to differentiate between creative and reproductive activities. The
significance level was set at .05.
3. Results
The mean scores (standard deviations in brackets) for each type of activity in topic 1 “Earth,
Space and Gravity” and topic 2 “Electricity” are shown in Tables 1 and 2, respectively. They
show that the creative activities in Fields 1 and 3 are generally scored higher than the
reproductive ones. They have more difficulties in Field 2 (empirical ways of gathering
knowledge) since the scores are similar between creative and reproductive activities in this
field and even they give a higher score to the reproductive explanatory activity in field 2 in
topic 1 (M = 3.64) than to the creative one (M = 3.58). It is worth pointing out that the items
in fields 2 and 3 tended to obtain higher scores than those in field 1, independently of their
nature. This indicates that these preservice teachers consider that practical and problem-
solving activities have a higher capacity to foster creative thinking in science than the non-
practical ones.
Table 1. Main scores awarded to the incidents in the different fields for topic 1: “Earth, Space
and Gravity”. (The higher the score, the greater the perceived opportunity for creativity).
Reproductive activities Creative activities

Field Descript Explanat Problem_solv Descript Explanat Problem_solv
1 3.11 (1.13) 2.57 (1.07) 3.64 (.94) 3.90 (.91)
2 3.69 (.97) 3.64 (.99) 4.23 (.87) 3.58 (.93)
3 3.56 (1.08) 3.83 (.96)
Comparing our results to those obtained by Newton and Newton (2010) with in-service
teachers, it is apparent that the difference in the scores obtained by creative and reproductive
activities is much lower in the case of preservice teachers. The mean scores also tend to be
higher in the case of preservice teachers: in the study of Newton and Newton (2010), the
highest score of 3.04 was awarded to descriptive creative incidents in field 1 in the topic
“Electricity”, while in our study there were mean values above 4 points in four cases, one in
the topic “Earth, space and gravity” and three in the topic “Electricity”. In this last topic a
mean score above 4 was awarded even to descriptive reproductive activities in field 2. This
could be due to the fact that the sample in this study is composed of preservice teachers at
their sophomore year, so they might not have yet assimilated what it means to be “creative
in a science classroom”.
70
Table 2. Main scores awarded to the incidents in the different fields for topic 2: “Electricity”.
(The higher the score, the greater the perceived opportunity for creativity).
Reproductive activities Creative activities

Field Descript Explanat Problem_solv Descript Explanat Problem_solv
1 2.71 (1.06) 2.82 (1.11) 3.74 (.94) 3.65 (.84)
2 4.27 (.94) 3.95 (1.06) 4.27 (.70) 3.93 (.87)
3 3.57 (1.11) 4.10 (0.87)
A more general view of the results can be found in Table 3, where the scores of all creative
and reproductive activities are grouped by topic. It is apparent that in all cases, the scores
awarded to creative incidents are higher than those awarded to reproductive incidents for
both topics. To assess if those differences were statistically significant, Students’ t test for
paired samples was carried out. The results (see Table 3) showed that it was statistically
significant in all cases, with a large size effect in both of them as considered by Cohen (1988)
for behavioral sciences.
Table 3. Descriptive statistics and results of the t test for paired samples for the scores awarded
to creative and reproductive incidents in both topics.
Min Max Mean SD t p d

Creat topic 1 2.67 5.00 3.79 .52
12.202 <.001 1.05
Repr topic 1 1.75 5.00 3.25 .68
Creat topic 2 2.60 5.00 3.94 .51
8.587 <.001 .78
Repr topic 2 1.20 5.00 3.47 .75
It is also worth pointing out that this group of preservice teachers gave higher scores for both
reproductive and creative activities in topic 2 than in topic 1. This can be due to the fact that,
as Jarvis and Pell (2004) stated, most primary teachers (and also preservice teachers) are not
science specialists and their knowledge in the different topics may differ, which will make
more difficult to assess the ability to promote creativity in one topic than the other. It is also
worth noting that some students gave high scores (even the maximum score of 5 points) to
reproductive activities in both topics, and that the mean score of reproductive activities was
also high, which might indicate that they do not understand what creativity in science is and
how to foster it.
4. Conclusions
The results of this study show that Spanish preservice teachers are in general capable of
differentiate between reproductive and creative activities in the science classroom. However,
the results are not completely satisfactory. The set of items considered creative obtained
similar results to those obtained in previous studies (Newton & Newton, 2010), so there was
71
a clear recognition of those activities as opportunities to develop creativity in students. On

the other hand, most of the participants also valued reproductive-type activities as good
opportunities to develop the creativity of the students, contrary to what was expected. This
could be related to the type of training received at the faculty of teacher training, with little
disciplinary specialization (in this case, in science) and a lot of general and transversal
didactic training, being both parts of the pedagogical content knowledge (Gudmundsdottir &
Shulman, 1987).
Didactic training received by these preservice teachers is generally based on preparing
activities whose value seems to be given to the procedure and materials (what is done and
how), but not always to its content (what must be learned). Therefore, when they are
presented with activities involving manipulation or involving innovative materials, they tend
to think that there is also an opportunity to develop creativity even if those activities are of a
reproductive type.
Thus, there is a need to improve the training of teachers in the development of creativity in
the scientific field for which a supportive learning environment encouraging curiosity,
inquiry and innovation is key. Teachers need to consider the learning objectives, level of
student engagement and autonomy, complexity, and assessment method, which focus is
different in creative and reproductive activities. By doing so, they can design activities that
promote creativity and critical thinking, and help students to apply their knowledge in new
and innovative ways.
References
Bereczki, E.O., & Kárpáti, A. (2018). Teachers’ beliefs about creativity and its nurture: A
systematic review of the recent research literature. Educational Research Review, 23, 25-
56. https://doi.org/10.1016/j.edurev.2017.10.003
Cheng, V. M. Y. (2010). Tensions and dilemmas of teachers in creativity reform in a Chinese
context. Thinking Skills and Creativity, 5(3), 120-137.
https://doi.org/10.1016/j.tsc.2010.09.005
Cohen, J. (1988). Statistical Power Analysis for Behavioral Sciences. Erlbaum.
Cruz, M., García, A., & Criado, A. M. (2020). Proposing questions for scientific inquiry and
the selection of science content in initial Elementary Teacher Training. Research in
Science Education, 50(5), 1689-1711. https://doi.org/10.1007/s11165-018-9749-0
Echegoyen Y., y Martín-Ezpeleta, A. (2021). Creatividad y ecofeminismo en la formación
de maestros. Análisis cualitativo de cuentos digitales. Profesorado, Revista de
Currículum y Formación del Profesorado, 25(1), 23-44.
https://doi.org/10.30827/profesorado.v25i1.15290
Gudmundsdottir, S., & Shulman, L. (1987). Pedagogical Content Knowledge in Social
Studies. Scandinavian Journal of Educational Research, 31(2), 59-70.
https://doi.org/10.1080/0031383870310201
72
Henriksen, D., Mishra, P., & Fisser, P. (2016). Infusing creativity and technology in 21st
century education: A systemic view for change. Educational Technology & Society,
19(3), 27-37. https://www.jstor.org/stable/10.2307/jeductechsoci.19.3.27
Izquierdo, E., & Solaz Portolés, J.J. (2022). Capacidad de indagación científica del
profesorado de primaria en formación: efectos del género y de la formación previa
(Indagatory skills of primary teachers in training. Effects of gender and previous studies).
Revista Universidad y Sociedad, 14(5), 109-120.
http://scielo.sld.cu/scielo.php?script=sci_arttext&pid=S2218-36202022000500109
Jarvis, T., & Pell, A. (2004). Primary teacher’ changing attitudes and cognition during a two-
year in-service science programme and their effects on pupils. International Journal of
Science Education, 26, 1787-1811. https://doi.org/10.1080/0950069042000243763
LOMLOE (2020). Ley Orgánica 3/2020, de 29 de diciembre, por la que se modifica la Ley
Orgánica 2/2006, de 3 de mayo, de Educación. Boletín Oficial del Estado, 340, de 30 de
diciembre de 2020, 122868-122953. https://www.boe.es/eli/es/lo/2020/12/29/3
Martín-Ezpeleta, A., Fuster García, C., Vila Carneiro, Z., & Echegoyen Sanz, Y. (2022). Leer
para pensar creativamente (el COVID-19). Relaciones entre lectura y creatividad en
maestros en formación. Revista Interuniversitaria de Formación del Profesorado,
97(36.3), 171-190. https://doi.org/10.47553/rifop.v97i36.3.96581
Mullet, D. R., Willerson, A., Lamb, K. N., & Kettler, T. (2016). Examining teacher
perceptions of creativity: A systematic review of the literature. Thinking Skills and
Creativity, 21, 9-30. https://doi.org/10.1016/j.tsc.2016.05.001
Newton, L. D., & Newton, D. P. (2010). What teachers see as creative incidents in elementary
science lessons. International Journal of Science Education, 32(15), 1989-2005.
https://doi.org/10.1080/09500690903233249
OCDE (2019). PISA 2021 creative thinking framework (third draft). OECD publishing.
Patston, T. J., Kaufman, J. C., Cropley, A. J., & Marrone, R. (2021). What is creativity in
Education? A qualitative study of international curricula. Journal of Advanced
Academics, 32(2), 207-230. https://doi.org/10.1177%2F1932202X20978356
Ramnarain, U. (2018). Scientific literacy in East Asia: Shifting toward an inquiry-informed
learning perspective. In Yew-Jin Lee & Jason Tan (Eds.). Primary Science Education in
East Asia (pp. 201-213). Springer. https://doi.org/10.1007/978-3-319-97167-4_10
Soh, K. (2017). Fostering student creativity through teacher behaviors. Thinking Skills and
Creativity, 23, 58-66. https://doi.org/10.1016/j.tsc.2016.11.002
73
Special needs pre-service teachers digital competencies: an

exploratory study at the University of Foggia
Learning Science hub (LSh), Humanities Department, University of Foggia, Italy.
Abstract
In Italy, K-12 teachers and school institutions’ digital comptencies were under
the average in the OECD Countries (Talis Survey, 2018), and they struggled
during the lockdowns due to the Covid-19 pandemic, which has forever
changed the way in which the didactic is provided.
The University of Foggia Learning Science hub (LSh), in Italy, works on
designing and planning new methodologies to teach digital skills to pre-service
teachers (especially special needs teachers), in order to enhance their
knowledge and use of digital skills in planning and providing interventions,
lessons and courses.
To guide the designing and planning activities, we have administered the
DigComp 2.2, the newest version of the self-report questionnaire of the Digital
Competence Framework for Citizens created by the European Commission, to
338 teachers attending a specialization course to become special needs
teachers at the University of Foggia in the current Academic Year
(2022/2023).
In the present study, the collected results of the survey will be analyzed and
discussed, becoming the cornerstone of LSh research in this field. At the end
of the specialization course, the survey will be proposed again, to compare
teachers’ digital skills at the beginning and the end of the course.
Keywords: Digital competencies; pre-service teachers; DigComp 2.2;
teaching methodology.

Special needs pre-service teachers digital competencies at University of Foggia
1. Introduction
In recent years, especially after the Covid-19 pandemic, educational research has mainly
focused on understanding how to improve the digital skills of K-12 teachers. Not only
“operational” skills, such as how to use a videocall software, but also a new knowledge about
the possibilities offered by the didactic digital software and tools in designing and planning
a lesson or a whole study course.
Even though the importance of implementing digital tools in didactic environments, from
digital softwares, personal computers and devices (e.g. smartphone or tablet) to the
implementation of digital methodologies and environments (e.g. Digital Game-Based
Learning, Gamification and Serious Games) has been enhanced by the Covid-19 pandemic,
teachers and researchers can’t ignor the impact that the “digital world” has had and is having
on the students (Nazempour et al., 2022).
The Digital Natives use technologies and digital tools everyday in every aspect of their lives
(Coggi & Emanuel, 2022). All the actors in the school environment – from pedagogists and
academics, who research everyday new methods and methodologies in order to help teachers
and students, to the teachers themselves, the ones who provide the didactic content and
knowledge to students – can no longer continue in providing didactic and pedagogical
contents in a tradition and formal way, expecting students to be happy, motivated and
committed in the teaching and learning processes.
The University of Foggia and the Learning Science hub (LSh), strongly believe in innovative
digital didactic methodologies. For these reasons, the LSh is involved in researching new and
more effective ways to teach innovative digital methodologies and technologies to pre-
service teachers, like the ones already mentioned: Game and Digital Game-Base Learning,
Gamification processes, Serious Games (of which the University of Foggia is also producer)
(Di Fuccio et al., 2021), Hybrid and Blended Learning Environoments etc.
Nevertheless, improving teachers’ digital competencies and the ways in which those
competencies are provided can’t take place without the opinions, thoughts and needs of
teachers themselves: the LSh involves pre-service teachers constantly asking for opinions
and needs they feel in order to plan new methodologies that can teach digital competencies
answering to the teachers’ needs.
2. The context and the sample

The University of Foggia holds the “Tirocinio Formativo Attivo” (active training internship,
abbreviated as TFA), a one-year university course aimed at qualifying to teach special needs
students in Italian schools. The Italian Ministry of Education plans the course and then it is
provided by the single Universities. The course is open to K-12 teachers, and they are divided
76
into groups according to the school grade they are entitled to teach in. They are divided into
Infanzia (Kindergarten, 3 to 5 years old), Primaria (Primary School, 6 to 10 years old), Scuola
Secondaria di Primo Grado (Middle School, from 11 to 13 years old) and Scuola Secondaria
di Secondo Grado (High School, from 14 to 18 years old).
The LSh teachers hold several courses about Psychology and Pedagogy. Among them, there
is the Information and Communication Technologies (ICT) course, which is a 150-hours
course that aims to provide digital competencies, knowledges and tools to train the teachers
in the use of technologies for didactic purposes.
The lectures of the course are provided mostly online. The University of Foggia wants to be
an example of good practice about Hybrid and Blended Learning, not only theoretically but
practically: even though the lockdowns of the Covid-19 pandemic are over, this doesn’t mean
that we have to come back to the usual frontal and formal lessons, giving the possibility to
attend the course even to people who have not the possibility to travel during holidays
(lectures are provided from Friday afternoon to Sunday afternoon, so working people can
have the possibility to attend the course as well).
The lessons are structured to be 8 hours long, from 9 am to 2 pm and then from 3 pm to 6
pm; the structure of the single lesson is:
• a first part shaped like a formal lesson, in which the Professor gives theory
references and knowledge to the participants (1 hour and 30 minutes);
• a second part in which an innovative didactic methodology, software or tool is
presented (30-45 minutes);
• a third part in which the participants must work individually and in groups. In this
way, they can not only listen about the digital methodologies, tools, software etc.,
but they can act like students, doing practical activities with the tools provided
(individual exercise and feedback, 2 hours and 30 minutes; group exercise and
feedback, 2 hours and 30 minutes);
• in the last part, participants deliver their work to the teaching staff, and there is a
moment of immediate feedback: some groups present their work in order to share it
with their colleagues and show it to the teaching staff. In this way, there can be a
dialogue and a debate among the teaching staff and the teachers attending the course,
providing feedback on the methodology of the course itself.
In this edition of the TFA, the ICT course is focused on presenting to the participants
innovative digital didactic methodologies. Every group attends to 7 lessons, which topics are:
1. Gamification
2. Serious Games
3. Differences between Gamification and Serious Games
77
4. Digital Game-Based Learning

5. Hybrid and Blended Learning
6. Elements of Instructional Design
7. Exam and Final Feedback
The macro goal of the ICT course in the framework of the TFA is to provide digital
knowledge and competencies to the training pre-service special needs teachers. In order to
understand and evaluate the efficacy of the course, the participants have been asked to answer
a questionnaire before and after the course. The questionnaire provided before the course and
data collection will be shown in the next paragraph.
3. The survey
In the first lesson, participants will be asked to answer a self-assessment questionnaire about
digital competencies. The same questionnaire will be administered during the last class, in
order to compare the results of the participants and detect if there are any changes in their
perceived levels of digital competencies.
3.1. The DigComp 2.2

The survey used in this study is the DigComp 2.2: The Digital Competence Framework for
Citizens, also known as DigComp. It provides a common language to identify and describe
the key areas of digital competence. It is an EU-wide tool to improve citizens’ digital
competencies, helping policy-makers to formulate policies that support digital competence
building and planning education and training initiatives to improve the digital competence of
specific target groups (Carretero et al., 2017). Digital competence is one of the key
Competencies for Lifelong Learning, and it was first defined in 2006 (Ala-Mutka et al.,
2008). The European Digital Framework (DigComp) was originally developed by the Joint
Research Centre, European Commission, in 2013 to identify and define the digital
competence that is relevant for all citizens who live and work in Europe today (Clifford et
al., 2020); with the 2.2 update in 2022, the aims of the Joint Research Centre are to include
tha latest discovers in the technology field, like Artificial Intelligence, Virtual and
Augmented Reality and Internet of Things and to keep the DigComp relevant in the learning
field (among other) (Vuorikari et al., 2022).
The DigComp is structured in five main areas, each one divided into competencies (Figure
1).
78
Figure 1. The DigComp conceptual reference model. Source DigComp 2.2, The Digital Competence Framework
for Citizens (2022).
Every competence is composed by 4 items, except for competencies 4.3 and 5.3, that are
composed by 3 items, for a total of 82 items. Every item is measured on a 4-point Likert
scale. Total scores that participants can obtain are: Low, Foundation, Intermediate,
Advanced.
3.2. Platform and data collection procedure

The participants were asked to answer the Italian version of the DigComp 2.2 on the
“mydigiskills” platform (https://mydigiskills.eu/it/). They obtain a .pdf file with their results
by email. Results showed to participants the total scores that they obtained for every
competence, and in every aggregate area. The results are graphic, as shown in Figure 2.
79
$&% '!&% %%&% "&# !%)% &"!'!&*%&''+"()+"(% ' " #'!)&%

$#$! $%% &"*!"*!''%+"(%)&"%' %&" ' " #'!"!'%'
%+"(%)&"%'!)( ' " #'!&"( +)%!'!)(
!!!! )&*'!"!" #'!%""(%"!"%)&&&"*!"*
" " *$ '####%##% #) %&%

%
"
*$ &%#% #) %&%%
*$ $%#% #) %&%%
# !# - )! %"*&!&%!'%!'!"% '"!!'
"!'!'
- )! )('!'!"% '"!!'"!'!'
!!
- )! !!'!"% '"!!'"!'!'
" " *$ %## %#%%#'#$ %
*$ '# #) %%#'#$ % %
*$ # !# - )! !'%'!'%"(''!""&
$#%#%%)%%#'#$ % %
- )! %!'%"('''!""&
*$ ##%%#'#$ % %
- )! !!!',!&#'%"(''!""&
*$ %"&%% - )! ""%'!'%"(''!""&
*$ $%#,%%+% - )! '$(''
-!'% ' !!'!''+
!
" " *$ '&!!# %&%%

* %### ## %&%% # !# - )! )"#!'"!'!'
*$ !(#%) - )! !'%'!!%"%'!'"!'!'
* # #) - )! "#+%'!!&
- )! %"% !
!!
"
" " * %# # %#$! $%'
*$ # %#%!#$ !#'( # !# - )! %"''!)&
*$ - )! %"''!#%&"!'!#%)+

# %#$&%$$#
* %# - )! %"''!'!*!
# %#,%
- )! %"''!'!)%"! !'

!
" " *$ $ '#!# %
* %# # !# - )! ")!'!#%" &
'&#$ #$! $%%
- )! !'+!!&!'!""%&#"!&&
* %))# #%' % %
*$ - )! %')+(&!''!""&
'&#'# !%)%
- )! !'+!'" #'!#&
Figure 2. Two DigComp 2.2 results received from the on mydigiskill platform (Italian and English).
Participants were than asked to upload anonymously the .pdf file of their results in a dedicated
Google Form. All participants signed informed consent, and they were secured about
voluntary participation and anonymity. This study was accepted by the Ethics Committee of
the University of Foggia, Italy, and conducted in line with the Decalaration of Helnsiki.
In the Google Form were uploaded 397 questionnaires. The final sample was composed by
338 teachers (after deleting duplicates). The total scores when then recoded as numers in
order to create and ordinal variable: Low = 0; Foundation = 1; Intermediate = 2; Advanced =
3.
Concerning data analysis, descriptive statistics in terms of means, frequencies and
percentages were performed using SPSS 27 (IB Corp., 2020)
4. Results
This work focuses on the results of the area 3, Digital Content Creation. The reason of this
decision is that the competencies of area 3 (Figure 1) are those that relate most to teachers'
use of digital skills.
80
Table 1. Results for area 3, mean and standard deviation. N=338, range = 0-3
Item M SD
3.1 Developing digital content 2.15 0.78
3.2 Integrating and re-elaborating digital content 1.71 0.79
3.3 Copyright and licenses 1.75 0.69
3.4 Programming 1.37 0.59
TOTAL 1.74 0.59

As Table 1 shows, in the area 3, Digital Content Creation, pre-service teachers don’t feel very
much confident. If item 3.1 “Developong Digital Content” has M=2.15, which is quite a good
result on a 0-3 range, the item 3.4, “Programming” with M=1.37 is actually very low.
Nevertheless, the item 3.4 result is not unexpected, since it refers to advanced digital
competencies. Looking at the answer for every competence, the highest percentage detected
certify the generally average result:
• 3.3 Copyright and licence, Intermediate (53.9%);

• 3.4 Programming, Foundation (49.7%);
• 3.2 Integrating and re-elaborating digital content, Intermediate (44.9%).
These results are showing that pre-service Italian special needs teachers should develop more
their digital competencies regarding digital content creation, a competence that can really
help in planning and design teaching and learning processes for Digital Natives.
There must be said that these results refers to Infanzia, Primaria and Secondaria di Primo
Grado schoolgrade: the study is currently going on with schoolgrade Secondaria di Secondo
Grado (High School). This survey will be developed in the next months, in order to collect a
wide overview about Italian special needs pre-service teachers. At the end fo the TFA course,
the DigComp 2.2 will be administered again, during the last lesson for every schoolgrade, in
order to analyze if there will be any changes in special needs pre-service teachers digital
competencies self-assessment, thanks to the ICT course. After the analysis, it will be clear if
the structure and topics of the ICT course were useful and efficient in enhancing pre-
servicespecial needs teachers digital competenciese.
In conclusion, there can be not mentioned that in previous surveys about Italian pre-service
teachers digital competencies, Italian pre-service teachers rate their skills higher than other
European colleagues, while “they receive slightly lower scores on tests of knowledge about
the possibility of using ICT in teaching processes and knowledge” (Tomczyk et al., 2022).
81
The resultsof this investigations will be used to reflect and develop new ways to provide ICT
and digital competencies courses to Italian teachers and pre-service teachers, in order to allow
the to develop and enhance their digital competencies in educational field.
Further help to this research will be brought by the next OSCE Teaching and Learning
International Survey (TALIS), which will be performed during 2023. In the 2018 edition
Italian teachers and school institutions digital competencies were rated under the average of
the OSCE Countries. The very goal of this research is to provide new methodologies to
enhance Italian pre-service teachers digital competencies.
References
Ala-Mutka, K., Punie, Y., & Redecker, C. (2008). Digital competence for lifelong learning.
Institute for Prospective Technological Studies (IPTS), European Commission, Joint
Research Centre. Technical Note: JRC, 48708, 271-282.
Carretero, Stephanie–Riina Vuorikari–Yves Punie. 2017. DigComp 2 1: The Digital
Competence Framework for Citizens with Eight Proficiency Levels and Examples of Use,
European Commission, Joint Research Centre. https://publications jrc ec europa
eu/repository/handle/JRC106281
Clifford, I., Kluzer, S., Troia, S., Jakobsone, M., & Zandbergs, U. (2020). DigCompSat. A
Self-reflection Tool for the European Digital Framework for Citizens (No. JRC123226).
Joint Research Centre (Seville site).
Coggi, C., & Emanuel, F. (2022). Formare i docenti Universitari alla didattica e alla
valutazione. SIRD, 45-76.
Di Fuccio, R., Toto, G. A., Ponticorvo, M., Parola, A., Marocco, D., & Miglino, O. (2021).
An innovative tool for olfactive learning with tangible user interfaces in MOOC learning
for viticulture and enology learning. In teleXbe (2).
Nazempour, R., Darabi, H., & Nelson, P. C. (2022). Impacts on students’ academic
performance due to emergency transition to remote teaching during the COVID-19
pandemic: A financial engineering course case study. Education Sciences, 12(3), 202.
Tomczyk, Ł., Fedeli, L., Włoch, A., Limone, P., Frania, M., Guarini, P., Szyszka, M., Mascia,
M.L., & Falkowska, J. (2022). Digital Competencies of Pre-service Teachers in Italy and
Poland. Technology, Knowledge and Learning, 1-31.
Vuorikari Rina, R., Kluzer, S., & Punie, Y. (2022). DigComp 2.2: The Digital Competence
Framework for Citizens-With new examples of knowledge, skills and attitudes (No.
JRC128415). Joint Research Centre (Seville site).
82
Advancing internationalization agenda amidst the war in

Ukraine: kindness and trauma-informed teaching project in
teacher education
Svitlana A. Kuzmina1, Sue F. Foo2, Olha V. Matviienko3, Tamara V. Glazunova4
1
Department of Foreign Languages, Vinnytsia Mykhailo Kotsiubynskyi State Pedagogical
University, Ukraine, 2Department of Special Education, Worcester State University, USA,
3
Pedagogy, Psychology Department, Kyiv National Linguistic University, 4Department of
Foreign Languages, Vinnytsia Mykhailo Kotsiubynskyi State Pedagogical University,
Ukraine.
Abstract
The paper aims to describe and analyze graduate students' collaborative
online learning project between Vinnytsia Mykhailo Kotsiubynskyi State
Pedagogical University, Vinnytsia [VSPU] Ukraine, and Worcester State
University [WSU], Massachusetts, USA, as one of the outcomes of the
internationalization agenda in Ukraine. Since the war in Ukraine and the
increased acts of violence in the U.S. require immediate action, the “Kindness
and Trauma-Informed Teaching” project is essential. Based on a one-semester
interaction for the case study, research, and project work, it is a novel practice
at both educational institutions. Graduate students engaged in asynchronous
and synchronous activities utilizing Zoom, Google Classroom, and Google
Documents. The project was carried out under the auspices of Education USA,
the American Council in Ukraine, and SUNY COIL Foundation in the USA.
The findings highlight the significance of such partnerships and persuade that
this project promotes further collaborative research in teacher education.
Keywords: Collaborative online international learning; international project;
internationalization agenda; kindness and trauma-informed teaching;
Ukrainian and American universities; graduate students.

Advancing internationalization agenda amidst the war in Ukraine
1. Introduction
The traditional understanding of the internationalization of education is associated with
concepts such as globalization, cross-cultural awareness, mutual understanding, research
integration, and student /faculty mobility (de Wit, 2019, p. 12; Ge, 2022, p. 229; Gromov,
2022, p. 58; Kuzmina & Glazunova, 2018, p. 294; Lisnychenko et al., 2022, p. 430; Malcolm,
2021, p. 4; Matvienko et al., 2022, p. 344). By definition, these terms entail a significant
increase in scope, novelty, change, and challenge. Nevertheless, internationalization is
primarily viewed as positive, where evident benefits outweigh shortcomings, although some
apprehensions might also arise (Kuzmina & Glazunova, 2018, p. 295). Projecting
internationalization on Ukrainian education makes this process a “necessary condition” for
higher educational establishments' competitiveness and financial security (Gromov et al.,
2022, p. 57; Matvienko et al., 2021, p. 142). Due to global developments, Ukrainian higher
education has committed to robust international collaboration in line with the country’s
political aspirations aimed at European integration (Gromov, 2022, p. 56; Tripses et al., 2018,
p. 214). Therefore, Ukraine’s internationalization agenda has confidently and steadily
become an integral part of higher education policies. Importantly, collaboration has got more
versatile, including joining international non-governmental educational organizations as their
affiliates, signing partnership agreements with universities, ensuring dual-degree education,
guest-lecturing, developing curricula and courses with co-teaching opportunities, and writing
research papers (Powell et al., 2014, p. 34; Gromov et al., 2022, p. 59; Lisnychenko et al.,
2022, p. 430; Tripses et al., 2018, p.216). Similarly, international collaboration is viewed
positively for student motivation to learn foreign languages, intercultural and values
awareness (Lisnychenko et al., 2022; p. 440; Matvienko et al., 2022, p. 344)
However, the war in Ukraine endangered the sustainability of Ukrainian education. The air
raids, shelling, internally and internationally displaced students and scholars, destroyed
infrastructure, and power outages became severe obstacles to educational services. The
regular academic communication and interaction paradigms were ruined and seemed
irreparable. Nevertheless, progress in technology integration in Ukrainian higher education
as a response to the COVID-19 pandemic (Matvienko et al., 2021, p. 141) and advances in
the internationalization agenda achieved before the war helped resume schooling and
internationalization agenda pursuits.
2. Methodology
The research describes the collaborative online learning of kindness and trauma-informed
teaching by American and Ukrainian graduate students. The authors intended to design the
project first, to provide students with an understanding of the significance of teaching
kindness, trauma, and stress responsiveness in educational practices because of the war in
84
Svitlana A. Kuzmina, Sue F. Foo, Olha V. Matviienko, Tamara V. Glazunova
Ukraine and increasing violence in the USA. Second, to promote international collaboration
in teacher education.
The project was integrated into the Traditions of Academic Collaboration: International
Experience class offered at VSPU graduate school in the fall semester (October – December
2022) and the Trauma-Informed Learning class taught to WSU graduate students. This
empirical research examines graduate students' behavioral patterns and attitudes to
designated activities, consisting of F2F lectures, thematic scientific reading interventions,
weekly assigned tasks on the materials, and discussions in international groups in real time.
The commitment to online collaboration resulted in a welcoming environment for the three
interaction modes: case study, research, and project work in asynchronous and synchronous
formats. The Google Classroom platform, Google Documents, and Zoom were employed to
unite forty-three students (17 American and 26 Ukrainian). Google Forms were used to
survey students’ experiences. The authors applied descriptive, qualitative, and quantitative
methods to explore opinions about online international collaborative learning and the
relevance of kindness and trauma-informed teaching. The literature under review is scientific
papers about views on and internationalization experiences (de Wit, 2019; Ge, 2022; Gromov
et al., 2022; Kuzmina & Glazunova, 2018; Powell et al., 2014; Tripses et al., 2018).
3. Collaborative Online International Learning
3.1. Internationalization Leadership Academy and SUNY COIL initiatives

The Internationalization Leadership Academy [ILA] All-Ukrainian program was launched in
January 2022 by Education USA and the American Council in Ukraine in partnership with
the State University of New York Collaborative Online International Learning Foundation
[SUNY COIL] to foster leadership in internationalization. The SUNY COIL Foundation has
been “empowering” leadership in collaborative learning since 2000 (SUNY COIL, n.d.). The
selection criteria prioritized candidates’ professional and personal skills and competencies
for international collaboration: a desire to step out of the box, have prior experience in
international cooperation, and the ability to connect with a U.S. partner-in-education and
develop a joint project for integrating it in the class of the home university. The Education
USA and SUNY COIL initiative resulted in fourteen international educational tandems from
Ukrainian and American universities, engaging more than 250 students in collaborative
learning in the fall semester of 2022-2023 (Education USA, n.d.). A three-week training for
American and Ukrainian professors-collaborators gave good insight into planning/designing
and involving students-collaborators in discussions and projects (Figure 1).
85
Figure 1. The layout of student/professor engagement in the COIL. (Source: website https://coil.suny.edu/).
3.2. The Kindness and Trauma-Informed Teaching Project

The Kindness and Trauma-Informed Teaching Project was preconditioned by the war in
Ukraine and increased violence in America (Statista, n.d.). Besides, the authors collaborated
on kindness as NGO Global Community Uniting for Equity members. The project design
required students to develop teacher competencies in the case study on kindness, research
and analyze presidential speeches, and develop project work on the effect of stress and
emotional trauma on a child’s brain development. Guided by the central questions about how
to make acts of kindness universal and the primary focus for teaching relationships, students
considered the case study through group skills, pro-social attitudes, and influencing societal
values positively. In conclusion, a plan was created to create a culture of kindness in the
classroom. Resulting in ranking the importance of teaching kindness as important by 20,5%
of students and very important by 82,1%, as the survey shows (Figure 2).
Figure 2. Display of ranking the importance of teaching kindness.
86
The “Presidential Speeches and Their Effect” research was performed in international teams
in joint Google Docs. Such leaders’ speeches as President Obama, Bush, Zelensky, Trump,
and Reagan were considered powerful for positive or adverse impact. Interestingly, both
American and Ukrainian students noted that the emotional tone, choice of vocabulary, and
stylistic figures are essential. The five created teams identified 33 to 53 words of kindness,
42 to 46 of their opposites, and 10-15 phrases of positive or negative character in the
speeches under scrutiny. The identified common themes in the studied addresses are
“elements of kindness,” “confronting barriers,” and “threats to unity.” Relating the speeches
to the events that generated public appeals proved the political context to be impactful. The
“Stress and Trauma Effect on Child’s Brain Development” project work addressed adverse
childhood experiences resulting in apathy to learning and behavioral disruptions. Students
identified the types of trauma and mechanisms of impact on a child’s brain development by
analyzing scientific papers, YOUTUBE materials and reflecting on lectures delivered to
project participants. Based on the acquired knowledge, 87,2% of students think trauma-
informed learning is very important, and 12,8 % agree it is important (Figure 3), admitting
practical recommendations on handling trauma in the classroom are timely provided and a
valuable tool under current circumstances.
Figure 3. Display of opinions of trauma-informed teaching importance.
While students were reflecting on integrating kindness and trauma-informed teaching into
professional activity, 82,1% of students confirmed the attitude as I will undoubtedly, 10,3%-
if they have like-minded colleagues; if my students are interested – 5,1%, and only 2,6% - I
don’t know (Figure 4).
87
Figure 4. Commitment to integrating into work.
Interestingly, the synchronous discussions in ZOOM breakout rooms and the presentations
of the teamwork outcomes at a general session concluded case study, research, and project
work with the most desirable interaction format: live communication and vibrant cultural
exchange. Decidedly, those F2F meetings became crucial for ruining stereotypes, developing
mutual respect, and professional and personal exchange.
4. Professional and Personal Gains for Practical Applications

Since the authors were genuinely interested in student attitudes to and benefits of the project,
they completed the work with the questionnaire. The Google Forms surveyed the graduate
students’ experience in kindness and trauma-informed teaching. The responders’ emphasized
the project’s relevance and value for practical application and successful career pursuit. The
highlighted benefits can be distinguished as professional (87%), personal (55%), or both –
(45%). Concerning the professional gains from the project, students highlighted: helpful in
creating a toolbox to have as teachers; giving new perspectives for professional development;
learning how to be a professional and kind person; getting an insight into what it is to teach
in wartime; learning approaches to teaching kindness in the classroom; pursuing a
successful career in education; solving similar problems in education; learning about
trauma-informed teaching in an applicable way, learning to feel as an educator when
students are stressed and how students may respond to the stress of war; a new experience I
am eager to implement in my work; kindness can be developed in every person; we need
more of such teaching for schools; exploring aspects Ukrainian education is lacking and
adopting them.
Considering personal gains, the students singled out: learning new myself; the most important
learning is to be kind and communicate with people of different cultures; taking such classes
refreshes your passion for teaching; it was inspirational and emotional; a humbling
88
experience to meet colleagues whose country is at war; learning the power of human spirit
and connection; eye-opening experience for me; helping see things from other perspectives;
learning kindness is vital to the well-being.
In addition, student feedback contained opinions of the benefits of English proficiency (15%),
cross-cultural awareness (86%), live interaction (88%), character traits relating to national
identity (17%), and evident intention to keep in touch (29%). The emotional component is
high (96%) and expressed as a once-in-life experience; I learned for this project more than
during all years of study; complete fun of interactive collaboration; fantastic, inspirational,
amazing experience; incredible exchange of ideas, cultures, and experience; like no other
experience. Concerning the drawbacks, American and Ukrainian graduate students are
unanimous here: there is not enough time for f2f sessions, forwarding suggestions to organize
more international collaborative projects.
5. Conclusions
The authors conclude that despite the war, Ukrainian higher education is determined to foster
the internationalization agenda in teacher education, considering it a professional and
personal enrichment resource and a powerful way to stay connected to educators worldwide
for solving common problems. Reflecting on the VSPU and WSU graduates’ feedback, we
assume that international collaboration on kindness and trauma-informed teaching is
essential for Ukrainian and American teacher education to help the traumatized and the
stressed. It is evident that kindness and trauma-informed learning is fundamental due to the
current situation and should be included in the teacher training curriculum in Ukraine to cope
with the aftermath of war. More emphasis on kindness and trauma-informed responsiveness
should be made in American schools to stop the violence eventually. In parallel, the research
findings testify that estimates of the significance of intercultural exchange and collaborative
research in teacher education are high. This paper’s researchers fully agree with a student’s
opinion that “it is essential to build a community of kindness in the classroom, school, and,
consequently, in society in general, and changing society means to change thinking.” When
young minds generate such ideas, there appears to be a conviction that creating a world of
kindness is existential and achievable, signifying education’s focal role in this endeavor.
References
de Wit, H. (2019). Internationalization in higher education, a critical review. SFU ED Review.
Special Issue: Internationalization of Higher Education. 12(3), 9-17. DOI
https://doi.org/10.21810/sfuer.v12i3.1036 Retrieved from
https://journals.lib.sfu.ca/index.php/sfuer/article/view/1036/696
Education USA. (n.d.). Internationalization Leadership Academy. Retrieved from
https://www.facebook.com/104514736246062/posts/pfbid03DHL1z4N5ozwK31Lp8Q
wNSkCTFrcSfzLLypQhZWvPeMPfee8ZxapGPvCbQL7eTVul/
89
Ge, Y. (2022). Internationalization of higher education: new players in a changing scene.

Educational Research and Evaluation. International Journal on Theory and Practice 27
(3-4), 229-238 DOI https://doi.org/10.1080/13803611.2022.2041850 Retrieved from
https://www.tandfonline.com/doi/full/10.1080/13803611.2022.2041850
Gromov, I., Kolomiiets A., Suprun M., Kolomiiets T., Hordiienko Y. I. (2022).
Internationalization of higher education in Ukraine as an integral component of the
globalization process. Scientific Journal of Dragomanov NPU. Pedagogical Sciences:
realia and perspectives. Special Issue 2022. 56-61. DOI https://doi.org/10.31392/NPU-
nc.series5.2022.spec.1.11
Kuzmina, S., Glazunova, T. (2018). Globalization and internationalization of education: pros
and cons. 2nd International Scientific Conference on Internationalization of Higher
Education in Ukraine in Global Multicultural Space: State-of-Arts, Challenges, and
Prospect. 294 – 296, УДК 378.658.5 (458). Retrieved from
https://mdu.in.ua/Nauch/Konf/2018/mizhnar_konf/materialy_konferentcii-
2018.pdf#page=294
Lisnychenko, A., Glazunova, T., Dovhaliuk, T., Kuzmina. S. , & Podosynnikova, H. (2022).
Facebook movie-based discussions: Bringing down intercultural barriers in English
language education. Arab World English Journal, 13(3), 429-444. DOI:
https://dx.doi.org/10.24093/awej/vol13no3.28
Malcolm, T. (2021). Globalization and internationalization as frameworks for higher
education research. Research Papers in Education.
https://doi.org/10.1080/02671522.2019.1633560
Matvienko, O., Kuzmina, S., Yamchynska, T., Kuzmin, Y.,& Glazunova, T. (2021). New
challenges imposed by the pandemic on a Ukrainian university. Arab World English
Journal (AWEJ) Special Issue on Covid-19 Challenges (1), 136-146. DOI:
https://dx.doi.org/10.24093/awej/covid.10
Matvienko, O., Kudina, V., Kuzmina, S. (2022). Values education and teaching zest for Life:
Japanese experience and New Ukrainian School Reform. 8th International Conference
on Higher Education Advances (HEAd’22), 339-346. DOI:
http://dx.doi.org/10.4995/HEAd22.2022.14626
Powell, D., Kuzmina, S., Kuzmin, Y., Yamchynska, T. & Shestopalyuk, O. (2014). Using
web-blended learning in Ukraine to facilitate engagement and globalize horizons: A pilot
study. TOJDEL: The Online Journal of Distance Education and e-Learning, 2 (2), 44–
51. Retrieved from http//www.tojned.net/journals/tojdel/articles/v02i02/v02i02-07.pdf
Statista. (n.d.). Crime & Law Enforcement.Violent Crime Statistics in the USA. Retrieved
from https://www.statista.com/topics/1750/violent-crime-in-the-us/#topicOverview
SUNY COIL. (n.d.). Connect. Engage. Collaborate. Retrieved from https://coil.suny.edu/
Tripses, J., Noe, M., Kuzmina, S., & Yamchynska, T. (2018). International social justice:
American-Ukrainian partnerships on school leadership. Modern Informational
Technologies and Innovative Methodologies in the Preparation of Specialists, (50), 214-
218. DOI: 10.31652/2412-1142-2018-50-214-218. Retrieved from
https://lib.iitta.gov.ua/715454/1/z_50.pdf#page=214
90
How university teachers can support the linking of theory and

practice in lesson debriefings with student teachers
Technical University of Berlin, Germany.
Abstract
Considering professionalism as a paradigm of higher education in university
settings, it seems crucial to establish supportive spaces in which reflection,
regarded as a mediating process between theory and practice, can be initiated.
Within the conceptual framework of UntAdFoKo supervision, developed and
tested in pre-service teacher education, N=10 lesson debriefings of university
teachers and student teachers in the practical semester of teacher training
were qualitatively analyzed to find out how and to what extent reflection can
be initiated by connecting practical teaching situations with theory-based
principles of learning-effectiveness and dimensions of teaching quality. First
results indicate that in lesson debriefings using the conceptual UntAdFoKo
supervision, linking of theory and practice occurred more as twice as often as
the description of teaching situations and that university teachers are the main
initiators of the linking process. These findings give insights into the effect of
conceptually linking theory and practice on reflection processes.
Keywords: Linking theory and practice; reflective practice; teacher education;
lesson debriefings; practical semester; teacher training.

How university teachers can support the linking of theory and practice in lesson debriefings
1. Theoretical Framework
1.1. The Paradigm of Professional Development in Higher Education

Higher education faces the challenge that faculty members “typically have been trained in
their subject matter but not in pedagogy” (Pesce, 2015, p. 1). However, assuming that tertiary
education pursues the objective of professionalism, the development of a "critical reflective
professional practice" (Davis & Moon, 2015, p. 30) seems inevitable to pursue a professional
development among university teachers. The call for professionalism is considered as a
university paradigm in the context of a “discourse of excellence” (Readings, 1996), but this,
on the other hand, is also a “challenge of excellence” (Light & Roy, 2001, p. 8). In order to
comply with the idea of excellence as a predominant feature of higher education (Light &
Roy, 2001), the call for conceptual frameworks in a time of supercomplexity (Barnett, 2000)
seems reasonable. The expectations set on higher education have a particular impact on the
university-based learning support of student teachers as well. What is the purpose of
university-based practice phases in teacher training? These university-based practice phases
are not about conveying teaching routines, but rather about bringing together theoretical-
conceptual analysis and practice (Baumert et al., 2007, p. 8), always having in mind the ideal
of the “reflective practitioner” (Schön, 1983). However, if universities want to train student
teachers to become reflective practitioners, ways must be found to make university teachers
“reflective professionals” (Light & Roy, 2001, p. 13 ff.) in the first place. Based on the
assumption that the ability to reflect is a way to support the development of a professional
teaching role (Korthagen, 2001, p. 53), this paper presents a concept on university-based
learning support for practical semester students in teacher training and empirical data on the
connection of theoretical dimensions of teaching quality to concrete teaching behavior and
lesson situations.
1.2. Lesson Debriefings as a Learning Opportunity for Student Teachers

First of all, it is important to specify that practical phases in university-based teacher training
cannot claim to implement a high degree of teaching experience, but should rather be
considered as an opportunity to learn how to reflect, thus to connect practical teaching
situations with scientific knowledge in the sense of reflexive knowledge (Herzog et al., 2001,
p. 19f.). The question, then, is how university teachers can design their lesson debriefings in
a way so that they become a space for initiating reflexive knowledge in student teachers in
order to pave their way for professional development in their future practice. In a German
study from 2005, the author concludes that “postconference feedback hardly ever lives up to
its frequent qualification as an ‘interface between theory and practice’” (Schüpbach, 2007).
Important learning processes in teacher education seem to take place uncontrolled and
unconsciously, leaving essential aspects of professional development to chance (Hascher,
2006, p. 145f.). But why is reflection of high relevance in the setting of lesson debriefings?
92
Reflection is not only significant to justify pedagogical action but also to deal with the
discrepancy between theory and practice or between knowledge and skill (Neuweg, 2011, p.
33ff.). Considering reflection as a mediating process between theory and practice (Leonhard
& Rihm, 2011), it seems crucial to create spaces in which reflective practice can be
habitualized (cf. Helsper, 2001, p. 13). The UntAdFoKo concept presented here, aims to
explicitly connect specific teaching situations to educational knowledge and thereby create
reflective spaces within university-based learning settings.
1.3. UntAdFoKo – A Concept for Lesson Debriefings to Foster Learning-Effectiveness

UntAdFoKo, i.e., lesson debriefings - adaptive, focused, criteria-oriented, is a
comprehensive, interdisciplinary concept for lesson debriefings between university teachers
and student teachers in the practical semester of teacher training. It has been used and pilot
tested in teacher training programs at the Technische Universität Berlin since 2018 in the
project "Digitalization in Vocational Teacher Education " (DiBeLe1) with the aim of
increasing the learning-effectiveness of lesson debriefings. In the following, the embedded
tools included in the UntAdFoKo concept are outlined: The process script (1) helps the
university teacher mentally plan and structure the course of the lesson debriefing, the
reflection sheet (2) is used by both university teacher and student to prepare and document
the debriefing session in a written form and it also helps to support the process of establishing
links between specific teaching situations to 17 basic principles of learning-effectiveness that
originate from the psychology of teaching and learning and are listed in the criteria tool (3).
The principles of learning-effectiveness are based on the three basic dimensions of teaching
quality (Gärtner et al., 2021) that can be defined as superordinate categories:
• Support of acquisition of competencies (includes principles such as: learning

activities aim clearly at a specific acquisition of skills; cognitive activation that was
achieved; task(s) adequately fit the performance level of school students)
• Motivation (includes principles such as: learning activities were suitable for
promoting self-efficacy; pupils had a scope of autonomy; learning activities aroused
interest)
• Classroom management (includes principles such as: clear program of action in
class; flow of instruction was well managed; all pupils actively involved in learning
activities)
Based on the outlined theory and former research, the current study addresses the following
research question: How and to what extent can university teachers initiate reflection
1
https://projekt.dibele.tu-berlin.de/wiki/doku.php?id=en:start
93
processes by conceptually linking theoretical principles of learning-effectiveness to practical

teaching situations in lesson debriefings with student teachers?
2. Methods
N = 10 lesson debriefings were audio recorded in the practical semester 2020/21 with a total
quantity of 3 university teachers and their assigned student teachers studying Prevocational
Education or vocational disciplines. The analysis focuses on a section of the debriefing
session in which successful and less successful teaching situations were named, explained
and each to be linked to one of the above mentioned theoretical basic dimensions of teaching
quality or to one of the principles of learning-effectiveness, and eventually associated with
an alternative course of action for the less successful situation. The conversations were
verbatim transcribed, deductively coded in Atlas.ti using qualitative content analysis
(Mayring, 2014). For the purpose of qualitative analysis, all the material was divided into
speaker turns (Sacks et al., 1974), each of which was identified on the basis of speaker
change. Based on the assumption about the above mentioned importance of habitualizing a
reflexive practice (Helsper, 2001, p. 13), the material was first examined to determine which
levels of reflection, according to a model by Fund et al. (2002), occur in the lesson
debriefings. The authors present a two-dimensional framework that contains the dimension
of content and also the dimension of form (Fund et al., 2002). Their evaluative tool is
developed on the basis of Hatton and Smith’s work on levels of reflection (Hatton & Smith,
1995). Four categories are focussed: Description, Personal Opinion, Linking and Critical
Bridging (Fund et al., 2002, S. 490) that are used to assess student teachers’ reflective
abilities. Using the tool by Fund et al. (2002) seemed reasonable to better evaluate the
gradation of reflection in "lower-level reflection" (referring to a specific lesson) and "higher-
level reflection" (going beyond a specific situation) (Fund et al., 2002, S. 491). Furthermore,
the reflection level "linking" seems particularly suitable as a category, not only to be able to
make statements about whether a connection between practical teaching situations and
theory-based principles of learning-effectiveness came about, but also to find out to which of
the theory-based basic principles a connection was made, i.e., which objects or contents
within the theoretical framework were addressed. Overall, the levels of reflection according
to Fund et al. (2002) - Description, Personal Opinion, Linking and Critical Bridging – were
coded and additionally, within the category linking, the 17 theory-based principles of
learning-effectiveness and the 3 basic dimensions of teaching quality from the criteria tool
were deductively coded as well. Although the basic dimensions function as a superordinate
category for the principles of learning-effectiveness, the distinction between both was made
in the coding system. This is because there are students who refer to both, the basic dimension
and the identified principle. In each case, the basic dimension was coded separately when
explicitly mentioned by the student teacher or the university teacher and was not coded
94
implicitly as a superordinate category of the mentioned principle(s). If one and the same
principle of learning-effectiveness or one and the same basic dimension of teaching quality
occurred twice within the same speaker turn, the corresponding code was only assigned once.
3. Findings
In the lesson debriefings that were conducted within the UntAdFoKo framework, the
percentage of the category Description (“what” and “how”) was only 7%, while the Linking
category that comprises the connection of practical teaching situations to theory-based
principles of learning-effectiveness and basic dimensions of teaching quality accounted for
17%. This means that Linking, considered as higher-level reflection (Fund et al., 2002, p.
491), occurred more than twice as often as the Description of teaching situations, considered
as lower-level reflection (ibid.). The results indicate that the majority of linking impulses was
given by university teachers, namely 61% (79 out of 129 times), while the linking impulses
given by students were only at a percentage of 39% (50 out of 129 times). The results can be
interpreted in a way that university teachers seem to play a key role in linking practical
teaching situations to theory-based principles of learning-effectiveness. Giving these linking
impulses does not seem to be a matter of course that happens automatically but should rather
be structurally supported on the part of the university teacher. On the other hand, the
percentage of linking impulses initiated by students (39% or 50 out of 129 times) shows that
applying the UntAdFoKo tools also seems to help students initiating linking processes by
themselves. Considering the percentage of linking impulses on both sides, students and
university teachers, the structural approach and the embedded tools within the UntAdFoKo
framework seem to promote the connection of practical teaching situations with theory-based
principles of learning-effectiveness instead of mainly describing teaching situations, which
can be considered as an increase in levels of reflection. The category Personal Opinion (Fund
et al., 2002, p. 492) accounted for the largest percentage of the lesson debriefings at 46%, in
which personal concerns of the “what” or “how”, relying on feelings or intuitions (ibid.)
about a lesson situation that is perceived as functional or dysfunctional, are expressed.
Although the Personal Opinion category can be considered as “lower-level reflection” (Fund
et al., 2002, S. 491), it still represents an increase within the levels of reflection on the side
of the students by moving from mere description to a student’s standpoint (Fund et al., 2002,
S. 490) which, in addition to describing, includes the reflection on “reservations, hesitation
or agreement of the student, in connection with the subjects in the first dimension” (ibid.).
The Critical Bridging category, in which there is a critical analysis of the "what" or “how”
and where the student “generalizes; reaches general conclusions […]. Suggesting alternatives
with explanations and reasons” (ibid.), only occurs at 0.53%. Looking at the contents of
reflection, it can be stated that almost half (49%) of the total amount of basic dimensions
relate to the dimension of motivation. The other half is distributed almost equally between
95
classroom management (27%) and support of acquisition of competencies (24%). Motivation

was the basic dimension favored by both student teachers and university teachers alike. There
are both overlaps and differences between the university teachers and the student teachers in
the frequency of the mentioned principles of learning-effectiveness. The 3 most frequently
mentioned principles of learning-effectiveness by university teachers are: Presence of a clear
program of action (11 codes); connection to prior knowledge (10 codes); cognitive activation
(10 codes). The 3 most frequently mentioned principles of learning-effectiveness by student
teachers are: students' sense of autonomy (7 codes); clear action program (7 codes); cognitive
activation of students (5 codes). The results indicate that university teachers can support the
linking of practical teaching situations to theory-based principles of learning-effectiveness
and basic dimensions of teaching quality by dialogically inviting students to establish this
connection by means of embedded tools within a conceptual framework. Knowing which
objects or contents are generally most common in the reflection process within the
UntAdFoKo framework, makes it possible for university teachers to adjust the selection of
contents so that the student teacher’s learning outcomes can be increased. However, the
results suggest that on the one hand the use of a concept seems helpful in establishing the
link between theory and practice, but on the other hand the role of the university teacher
seems to be a crucial for supporting the linking and thereby reflection processes. Overall, the
findings indicate that by preparing, structuring, and conducting the lesson debriefing by
means of the process script, reflection sheet and criteria tool, the university teacher is able to
provide a conceptual framework for student teachers within a higher level of reflection
through linking theory and practice can be achieved.
4. Discussion
Following Beckmann and Ehmke (2020), the findings of the current study suggest that the
conscious connection of theoretical knowledge and practical teaching situations seems to be
useful for student teachers’ learning. The concept presented here can contribute to
counteracting the many variants of structural implementation of lesson debriefings (Brack,
2019) by having university teachers follow a conceptually structured procedure for lesson
debriefings with the help of conceptually embedded tools. In addition to lesson debriefings
as a verbal and joint form of reflection (Christof et al., 2018), it would be interesting to find
out if theory-based frameworks such as the UntAdFoKo concept can also be applied in other
settings of higher education reflection. These could for example be e-portfolio work or
practical semester reports as a written matter of reflection or reflective talks among
colleagues as another verbal form of reflection (ibid.). The further application of a conceptual
framework like UntAdFoKo, which combines theory and practice, would need to be adapted
to the particular form of reflection and cannot be adopted unchanged. However, the tools
presented here could be a starting point to develop suitable tools for further reflection-
promoting settings within higher education. In order to draw conclusions about the learning
96
efficacy of lesson debriefings, it would be useful to conduct a contrastive comparison in

which the rather small sample of N=10 is expanded by analyzing lesson debriefings of a
comparison group of student teachers without UntAdFoKo concept supervision presented
here. This would help to gain insights into whether the establishment of theory references
occurs primarily through the application of the UntAdFoKo tools or whether the embedding
of theory references also occurs independently regardless of the use of conceptual tools. The
individual behavior of both, university teacher and students, should be taken into
consideration in order to be able to make statements about the establishment of linking
processes regardless of reflection-promoting concepts. In this case, it could be investigated
whether the embedding of theory references is basically individually dependent on the
university teacher or whether the conceptual embedding is causal and decisive for the
production of theory references made by student teachers. For further research, it would also
be helpful to analyze the extent to which students teachers’ individual dispositions, such as
self-efficacy or learning-goal orientation, influence the linking of theory and practice in
lesson debriefings. With regard to the university teacher, it would also be conceivable to
include his or her conversational role by using the MERID model (Crasborn et al., 2011) in
order to be able to assess the extent to which the university teacher’s role in the mentoring
dialogue (active or reactive, directive or non-directive) contributes to the establishment of
theory references.
References
Baumert, J. & Kunter, M. (2006). Stichwort: Professionelle Kompetenz von Lehrkräften.
Zeitschrift für Erziehungswissenschaft, 9 (2006) 4, S. 469-520.
https://doi.org/10.1007/s11618-006-0165-2
Barnett, R. (2000). University knowledge in an age of supercomplexity. Higher Education,
40(4), 409–422. https://doi.org/10.1023/A:1004159513741
Beckmann, T., & Ehmke, T. (2020). Unterrichtsbesprechungen im Praktikum –
Konstruktivistische und transmissive Lernbegleitung durch Lehrkräftebildner/innen
unterschiedlicher Institutionen. Zeitschrift für Bildungsforschung, 10(2), 191–209.
https://doi.org/10.1007/s35834-020-00275-2
Brack, L. (2019). Professionalisierung im Gespräch. Subjektivierungen in
Nachbesprechungen zum Grundschulunterricht im Rahmen des Semesterpraktikums.
Studien zur Professionsforschung und Lehrerbildung. Bad Heilbrunn: Klinkhardt.
Christof, E, Köhler, J., Rosenberger, K., Wyss, C. (2018). Mündliche, schriftliche und
theatrale Wege der Praxisreflexion. Beiträge zur Professionalisierung pädagogischen
Handelns. 1. Auflage. Bern: hep der bildungsverlag. ISBN: 3035509239;
9783035509236
Crasborn, F., Hennissen, P., Brouwer, N., Korthagen, F., & Bergen, T. (2011). Exploring a
two-dimensional model of mentor teacher roles in mentoring dialogues. Teaching and
Teacher Education, 27(2), 320–331. https://doi.org/10.1016/j.tate.2010.08.014
97
Davis & H. & Moon J. (2013). Making Room for New Traditions: Encouraging Critical
Reflective Professional Practice for Tertiary Education Management. Tertiary Education
Management Conference, Trends, Traditions, Technology: Refereed Papers, Pp. 23-33.
Fund, Z., Court, D., & Kramarski, B. (2002). Construction and Application of an Evaluative
Tool to Assess Reflection in Teacher-Training Courses. Assessment & Evaluation in
Higher Education, 27(6), 485–499. https://doi.org/10.1080/0260293022000020264
Gärtner, H., Thiel, F., Nachbauer, M., & Kellermann, C. (2021). Unterrichtsentwicklung
durch Unterrichtsfeedback – Erste Entwicklungsschritte eines Beobachtungsbogens für
Schulleitungen. Zeitschrift für Bildungsforschung, 11(3), 529–547.
https://doi.org/10.1007/s35834-021-00321-7
Hatton, N., & Smith, D. (1995). Reflection in teacher education: Towards definition and
implementation. Teaching and Teacher Education, 11(1), 33–49.
https://doi.org/10.1016/0742-051X(94)00012-U
Helsper, W. (2001). Praxis und Reflexion. Die Notwendigkeit einer „doppelten
Professionalisierung“ des Lehrers. In: Journal für LehrerInnenbildung 1 (3), S. 7-15
Herzog, W. & von Felten, R. (2001). Erfahrung und Reflexion. Zur Professionalisierung der
Praktikumsausbildung von Lehrerinnen und Lehrern. https://doi.org/10.25656/01:13447
Korthagen, F. (2001). A Reflection on Reflection. In Linking Practice and Theory.
Routledge.
Leonhard, T. & Rihm, T. (2011). Erhöhung der Reflexionskompetenz durch
Begleitveranstaltungen zum Schulpraktikum? Konzeption und Ergebnisse eines
Pilotprojekts mit Lehramtsstudierenden. Lehrerbildung auf dem Prüfstand 4 (2011) 2, S.
240-270. doi: 10.25656/01:14722
Light, G., & Roy, C. (2001). Learning & Teaching in Higher Education: The Reflective
Professional. 2nd ed, Sage Publishing, Thousand Oaks, CA.
Mayring, P. (2014). Qualitative Content Analysis. Theoretical Foundation, Basic Procedures
and Software Solution. Klagenfurt. https://nbn-resolving.org/urn:nbn:de:0168-ssoar-
395173
Neuweg, G. H. (2011). Distanz und Einlassung. Skeptische Anmerkungen zum Ideal einer
"Theorie-Praxis-Integration" in der Lehrerbildung. Erziehungswissenschaft 22 (2011)
43, S. 33-45. doi: 10.25656/01:5430
Pesce, J. R. (2015). Professional development for teaching in higher education: Faculty
perceptions and attitudes.
Readings, B. (1996). The University in Ruins. Harvard University Press, 79 Cambridge St.
Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the
organization of turn-taking for conversation. Language, 50(4), 696–735.
https://doi.org/10.1353/lan.1974.0010
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Basic
Books.
Schüpbach, J. (2007). Über das Unterrichten reden: Die Unterrichtsnachbesprechung in den
Lehrpraktika - eine „Nahtstelle von Theorie und Praxis“? (1. Aufl). Haupt.
98
An empirical survey of employment and MTI competence in

China
Xian Wang1, Wenxiang Si2, Linxin Ge3, Qiuhan Wang4
1
Foreign Language College, Shanghai Maritime University, China, 2Foreign Language
College, Shanghai Maritime University, China, 3Foreign Language College, Xi'an
Polytechnic University, China, 4Foreign Language College, Shanghai Jiao Tong University,
China.
Abstract
The Master of Translation and Interpreting (MTI) of China is controversial as
less than 10% of graduates take translation as their job, in contrast to over
89% employment rate. The striking disparity poses a question: why MTI
graduates are popular with the employment market? This research, uses
questionnaires and comparative empirical research to explore the impact of
employment requirement on MTI competence from the perspective of
‘professionalism’. We find that that a cluster of technology applications have
entered the employment requirement and becomes integral to the MTI
competence, framing a new type of professionalism that focus on information
acquisition and language conversion. Employment recruitment also put
emphasis on sector practice. The MTI education may improve the employment
rate via these enhancement measures relating to professionalism.
Keywords: MTI; employment; professionalism; translation technology cluster;
sector practice.

An empirical survey of employment and MTI competence in China
1. Introduction
From 2007 to 2022, China Higher Education approved 316 educational organizations to run
a major called Master of translation and interpreting (MTI) which aims to cultivate talents
with professional competences in translation and interpreting and meet the growing needs of
foreign exchange in terms of language. Over these years, some researchers claim that, in
terms of employment fulfillment, MTI is self-evidenced by the tremendous expansion in
scale (Zhang & Wang, 2020). Data show that the employment rate of MTI graduates has
reached 87.85% on average, indicating that cultivation in this regard can meet the needs of
employment (Zhang & Wang, 2020). Cui (2017) has conducted a series of survey on the MTI
major, finding that MTI competence is ‘strong’ adaptable to employment, considering the
fact that less than 30% of MTI graduates are engaged in translation and interpreting-related
jobs. However, previous research rarely discussed the adaption of MTI to the employment in
terms of competence. This present research explores the feature of the ‘strong’ adaptability
of MTI to jobs in light of recruitment requirements, using MTI graduates as empirical
samples.
In the context of relationship between employment and MTI education, we make contribution
to the literature in that with translation technology as the strong influencing factor, MTI is
newly labeled as being more adaptable to employment.
2. Study Design
In this study, a sample survey and information retrieval method was used to collect 70
questionnaires from 170 graduates from the grades 2018 and 2019 of Shanghai Maritime
University (Wang & Wang, 2020). The recovery rate of questionnaires was 41.3%. The valid
questionnaires were 70 with a 100% questionnaire efficiency rate. We distributed the
questionnaire to collect the data by means of the Questionnaire Star, Wechat and other
network channels. The questionnaires included (1) the employment satisfaction, industries
engaged, income status, and career development of MTI graduates; (2) the opinions of MTI
graduates on the curriculum after working for some time.
The questionnaire focused on the translation competence of MTI graduates, which mainly
includes: students’ initial goal of applying for MTI degree program, expected occupation,
and actual occupation, working industry, employment channels, factors affecting their
employment competence, employment satisfaction, salary, expected translation competence,
etc. Considering the validity, the questionnaire contains 18 questions, among which questions
1-5 involve basic questions such as students’ name, gender, working industry, age, and
income; questions 6-18 include expected employment, expected industry, expected
curriculum of employers, demand for practical courses of employers, etc. In addition, there
100
are 3 multiple-choice questions and 8 questions with fill-in-the-blank options within those 18
questions.
From the perspective of the market, we download 72 pieces of job information related to
translators from mainstream recruitment websites, such as Zhaopin, Liepin, HunterOn,
Qiankun Headhunting, and Sunsharer Headhunting. Those job requirements involve
translation competence, industry, major, salary, communication ability, and job scope, which
are similar to the design and content of the questionnaire, thus facilitating further comparison,
analysis, and categorization. The questionnaires collected and the job advertisements are
comparable for their number are the same.
3.1 Employment and Engagement of MTI

Enhancing competence in translation and interpreting is an important teaching goal for the
MTI major. Translation competence is essential for MTI students (Liu, 2011; Han, 2020).
Zhu (2019) equates professional knowledge with professionalism from the original purpose
of MTI training. Based on this, one prevailing view is that the specialization of polytechnics
be combined with specialized courses of each university (Li, 2020), and allocate more
training requirements to practice of translation for special purposes (Liu, 2017). Li (2020)
and Sun (2015) claim that the Higher Education Steering Committee stresses on
professionalism but ignores the necessity of practice in special fields. In summary,
professionalism needs to be oriented at employment. Professionalism is affected by
employment requirements (Yao, 2020) but is not detailed.
In the questionnaire of this research, we designed question 2 “What is your expected
occupation before employment?” and question 3 “What is your current occupation?” to
compare the employment expectation and actual engagement (See Table 1).
Table 1. Expected Employment and Actual Engagement of MTI Students.
Occupation Expected Employment Actual Employment Standard Deviation (SD)
Translators 36.3% 9.1% 0.19
Teachers 39.4% 33.3% 0.04
No expectations 6.1% 9.1% 0.02
Others 18.2% 48.5% 0.21
101
It can be seen the characteristics of the expected and actual engagement of MTI students. 1)
the gap between expected employment and actual engagement in being a translator is huge.
The expected proportion (36.3%) is much larger than the actual employment (9.1%). And the
Standard Deviation (SD=0.19) shows that expecting to be a translator, in the eyes of pre-
training students, is not sufficient for employment. 2) In terms of the occupation as a teacher,
the percentages of expected (39.4%) and actual employment (33.3%) are nearly the same
while the SD (0.04) is comparatively small, indicating that the language skills of MTI
students are still an important component for employment. 3) In the choice of “others”, the
expected percentage (18.2%) is lower than the actual percentage (48.5%), with SD at 0.21,
indicating that MTI students are employment ready with other competencies.
The above characteristics suggest that the actual demand for MTI students is hooked to the
basic language skills and new competencies (Hu & Tian, 2020).
3.2 The New MTI Competence Under Employment

The employment requirement for MTI job market post challenge to MTI education. Li (2019)
pointed out, the MTI curriculum should not only include basic language courses, but also
cultivate professional skills. So what professional competence required for job markets do
MTI students have?
Question 12 of the questionnaire is a multiple-choice question “Which aspects of the
specialized courses offered by the MTI program have contributed most to the development
of your professional competence?” The options include translation technology, translation
theory, economic and trade translation, science and technology translation, literary
translation, and others. We searched “translator and interpreter” on the job page and
randomly selected the job advertisements listed. After manual screening to remove duplicate
and invalid job offers, we obtained 72 job offers from mainstream headhunters and job sites.
Then, we sorted them by high-frequency words, removed words unrelated to competence,
and obtained keywords such as Computer-Aided Translation, Translation Certificate, English
Competence, Translation Practice, and Relevant Industry Experience. And the ratio of
demand was calculated based on the word frequency. The integration of student
questionnaires and job recruitment is shown in Figure 1.
From Figure 1 we can see that, first of all, the demands from employers (12%) and students
(55%) both focus on translation skills despite various percentages. Secondly, the demand for
industry experience (32%) of employers is greater than the maximum percentage of any
individual industry (18%) in translation from students’ viewpoint. These findings indicate
that students and markets both agree that translation technology is a distinctive feature that
distinguishes MTI majors from other foreign language majors. Therefore, we believe that
translation technology and industry experience should act as part of professional competence
in MTI job market.
102
Figure 1. Comparison of competence between MTI graduates’ expectations and employers’ demands.
3.3 Translation Technology as New Competence

In 2014, the World Translation Conference on Man vs. Machine? The Future of Translators,
Interpreters, and Terminologists was held in Dublin, highlighting the shift in translation
studies towards technology. Translation talents with technical ability can meet the needs of
the market (Miao & Wang, 2010; Yao, 2013). But what is translation technology? Does
translation technology promote employment?
According to the international standard for translation service ISO17100:2015, translation
technology contains three levels: technical tools, management, and communication.
Technical tools involve alignment software such as ABBYY Aligner, translation software
such as MemoQ and Trados, self-built terminology database, OCR recognition, format
conversion, and other over twenty technologies. Translation project management includes
complex missions such as forming teams, formulating programs, arranging schedules, and
assigning tasks. Communication includes instant messaging software such as Facebook, QQ,
WeChat, and email. Although Wang (2020) did not divide translation technologies into three
categories, his classification also mainly covers the first two parts. With a total number of
nearly thirty, all these diverse technologies involve translation corpus, format conversion,
and project management. Therefore, MTI students need to spend a lot of time learning as
many as 30 types of technology. And Fu (2015) points out that “relevant skills must be
acquired with special training”(p.81), which also illustrates the difficulty of translation
technology learning. In this regard, we believe that translation technologies are unique to
MTI students with the competence that effectively runs through the whole process of
translation.
103
3.4 Industry Experience in Specialized Field

The division of labor in the translation industry is becoming more and more detailed. In the
job market, MTI students should have much practice experience, so they can be competent
in different fields of translation work (Zhao & Yang, 2021). We analyzed 72 pieces of job
advertisements for translation positions that contained industry experience statements,
selected their high-frequency words about translation experience, and made them classified.
See Figure 2.
Figure 2. Industry experience for employers’ consideration.
It is shown that most employers (70%) require industry experience for MTI graduates, while
30% do not specify their industry requirements. To make it clear, the translation of documents
or papers accounts for 20%, the translation of scientific and technical documents accounts
for 16.67%, and the translation of economic and trade documents accounts for 3.33%,
revealing the proportion of employers’ demand for industry/profession. It can be seen that
employers measure the professionalism of translators by their experience and practice in
different translation industries. Professionalism should be reflected through industry practice.
As Xu (2010) pointed out that MTI education must be classified considering the market
factors, such as engineering, medicine, law, economy and trade, literature, and so on.
4. Conclusion
Against the background of technology trends such as CAT, intelligent translation, and deep
learning, the MTI competence represented by professionalism is insufficiently studied. This
may be caused by the traditional translation model based on basic language skills and
translation skills, has rooted in education as absolute soundness. But translation has entered
104
the machine translation age, thus simply focusing on students’ basic language skills and
translation skills, and ignoring such translation technologies as computational linguistics and
corpus linguistics will “obviously fail to meet the demand for talents in the society of
language intelligence development” (Hu & Tian, 2020).
Based on the questionnaires of MTI graduates from Shanghai Maritime University and the
feedback from employers, we can find that the employment market has redefined the MTI
competence by highlighting the role of translation technology. Such technology cluster used
in the translation process, in terms of the type of technology and the length of learning time,
has essentially distinguished MTI majors, and should be viewed as a key component of the
professionalism of MTI.
From the perspective of the teaching program, MTI curriculum should pay attention to the
training of translation technology in order to meet the needs of the market and improve the
efficiency of the translation. As Wang (2012) pointed out, the MTI course setting should
consider the market demand and the content should emphasize the combination of theory and
practice. So we propose that the curriculum design of MTI should open courses focusing on
translation technology (cluster) and translation professional practice to ensure the
effectiveness and employment rate, and further meet the market demand for translating
practitioners.
This study has some limitations. Participants in this survey from the same university may
result in homogenization. Also how translation technology can be included in the curriculum
remains to be designed. Therefore, a larger sample nationwide may present a deeper
understanding of the role of translation technology to employment in MTI education.
References
Cui, Q. L. (2017). National survey report on education and employment of postgraduates of
master of translation and interpreting. University of International Business and
Economics Press.
Fu, J. M. (2015). Research on translation competence: review and outlook. Foreign Language
Learning Theory and Practice, 4, 80-84.
Han, S. Q. (2020). An integrated system of teaching, learning, and evaluation for the
cultivation of MTI translation competence. Shanghai Journal of Translators, 2, 51-57.
Hu, K. B., & Tian, X. J. (2020). The cultivation of MTI talent in the background of language
intelligence: Challenges, countermeasures, and prospects. Foreign Language World, 2,
59-65.
Li, J. M (2019). Further research on the problems and their countermeasures in MTI
curriculum design. Journal of Shandong University of Science and Technology (Social
Science), 4, 107-112+119. doi:10.16452/j.cnki.sdkjsk.20190709.008.
Li, S. L. (2020). Study on the professionalism of ALF education in science and engineering
105
colleges. Shanghai Journal of Translators, 2, 57-63.

Liu, H. P. (2011). Study on the stages of translation competence development and its
pedagogy. Chinese Translators Journal, 4, 47-50.
Liu, Y. Q. (2017). Exploration of the “3 + 3 + 1” MTI teaching model: Case study of China
University of Petroleum. Shanghai Journal of Translators, 1, 52-57.
Miao, J., & Wang, S. H. (2010). MTI education in light of a survey on employment trend in
translation industry. Foreign Languages and Their Teaching, 3, 63-67.
Sun, H. (2015). Exploring the development of professionalism of MTI from the perspective
of localization. Journal of the Open University of Henan, 2, 104- 108.
Wang, H. S. (2012). Teaching practice of translation technology under the background of
information age. Chinese Translators Journal, 3, 57-62.
Wang, H. S. (2020). Research on translation technology in the era of artificial intelligence.
Intellectual Property Press.
Wang, X., & Wang, Q. Y. (2020). A discussion of professionalism of MTI: from the
perspective of graduate and employer needs. East Journal of Translation, 5, 26-29+68.
Xu, J. (2010). Some thoughts on the professional degree education of master of translation.
Chinese Translators Journal, 1, 52-54.
Yao, B. (2020). Pedagogic design and implementation of interpreting profession and
interpreter ethics. Shangdong Foreign Language Teaching, 3, 32-38
doi:10.16482/j.sdwy37-1026.2020-03-004.
Yao, J. Q. (2013). Professional translation teaching method (Chinese to English). Lecture on
"2013 Summer National Teacher Training for Translation Majors in Colleges and
Universities".
Zhang, M. L., & Wang, L. (2020). Core quality of master of translation and interpreting.
Educational Review, 3, 148-154.
Zhao, L. H., & Yang, D. (2021). Analysis on the dilemma and countermeasures of MTI
education with industry characteristics—taking civil aviation colleges and universities as
an example. Modern Communication, 4, 164-166.
Zhu, X. J. (2019). The regulation and model of master of translation and interpreting (MTI)
dissertation. Shanghai Journal of Translators, 2, 83-88.
106
Intercultural interaction modulating implicit attitudes towards

disability and cultural competence in higher education
Zijun Li1, Lyudmila Tokarskaya Valerievna2
1
Department of General and Social Psychology, Ural Federal University, Russia,
2
Department of General and Social Psychology, Ural Federal University, Russia.
Abstract
People with disabilities (PWD) are increasingly recognized as one of the most
overlooked potentials in the labor market. More and more disability inclusion
strategies appear and evolve in the workplaces. Nevertheless, employees with
disabilities in the work environment are still faced challenges of explicit and
implicit biases. Especially, comparison with positive explicit attitude due to
social expectation, implicit attitudes towards disability tend to be negatively
tied with charity, inability and distrust. This present article originally proposes
intercultural interaction as the solution to modulate implicit and emotional
attitudes towards disability. Besides, this article highlights that intercultural
interactions between people with and without disabilities could promote
individual cultural knowledge. It is underlined the importance to involve
schemes of cultural competences and intercultural interactions into higher
education to facilitate more inclusive work environment.
Keywords: Implicit attitudes; people with disability; intercultural interaction;
cultural competence; higher education.

Intercultural interaction modulating implicit attitudes towards disability and cultural competence
1. Introduction
According to the World Report on Disability (World Health Organization, 2022), more than
16% of global population live with disability and majority of people with disability (PWD)
potentially function effectively as workforce. Disability inclusion in workplace has been
growingly highlighted by government policies and organizational initiatives in the last
decade. However, stereotype and biases towards disability remaining as formidable obstacles
which hinder the effective cooperation and interaction between employees with and without
disabilities in the workplace (Bonaccio, Connelly and colleagues, 2019).
Explicit and implicit attitudes and towards disability predict the willingness and effectiveness
of intercultural interactions between these two social groups. Explicit attitudes refer to more
deliberative or intended response which vastly rely on motivationally and cognitively
controlled factors. Comparatively, implicit attitudes can be characterized as the automatic
association which individuals hold between an object/event with evaluation and it coincide
with automatic emotional attitudes (Rudman, 2004) . Implicit attitudes better than explicit
attitudes dominantly foretell individual behaviors, especially when it comes to the immoral
issues like the attitudes towards disability. Comparison with universally positive explicit
attitude towards disability, implicit attitudes towards employee with disabilities tend to be
negatively tied with charity, inability and distrust. Empirical evidence has presented that
traditional implicit attitudes towards disability negatively comprise pity, dependence and
marginalization and hinder the appropriate treatment and rehabilitation for people with
disabilities. This effectively explains the phenomenon that non-disabled employees feel
stressed and discomfort when actually interacting with disabled employees, even though they
identify not to discriminate against disabled employees. Extant study in Russia underlined
that in spite of three-quarters of the respondents holding positive attitudes towards disability,
only one quarter feel ready to interact with individuals with disabilities, and lack of
interactive experience as the main reason raises fear to cooperate (Volosnikovaa & Efimovab,
2016). It is crucial for scholars and practitioners to further investigate individual implicit
attitudes towards disability and its potential solutions.
It has been widely addressed that implicit attitudes primarily stem from affective experience
and cultural biases. For instance, pleasant interactive experience with partners with
disabilities leads to positive implicit attitudes towards disability. Evidence has emphasized
the two-way correlation between affective experience with implicit attitudes (Songa,
Slabbinck, Vermeir & Russo, 2019). Moreover, cultural milieu and biases towards
individuals from different cultural background are highly related to implicit attitudes. The
influence of cultural milieu on implicit attitudes offers the possibility that intercultural
interaction experience may foster the positive implicit attitudes towards another sub-culture
and culture. Besides, it has been shown that training regarding knowledge of disabilities
significantly promote the positive attitudes towards disability (Packer, Iwasiw and
108
Zijun Li, Lyudmila Tokarskaya Valerievna
colleagues, 2000). Cultural knowledge refers to the knowledge which could allow a better
grasp of the internal logic and modal behavior of the other culture, and the basic
understanding for the behavior. This type of knowledge helps for adapting to the new culture
(Thomas and colleagues, 2008). Cultural intelligence serves as the cognition facet of cultural
intelligence, which is considered as the crucial competence for adapting and functioning in
intercultural situations. It is reasonable to consider that individual with more cultural
knowledge will be more adaptive and tolerant with other cultures and diverse contexts.
Therefore, this article hypothesizes:
1. 2. Interaction with people with disabilities correlates with individual implicit attitudes
towards disability.
2. Intercultural interaction with people with disability correlates tot cultural knowledge
about disability.
3. Individual cultural knowledge about disability correlates to implicit attitudes towards
disability.
2. Method
It is arduous to collect implicit attitudes of individual by subjective self-report. Building on
literature, Implicit Association Test (IAT) have been applied to investigate implicit attitudes.
This article conducted the online survey via Google form among 112 university students (19-
27 y, 50 male, 62 female, all physically able) from Ural Federal University (Yekaterinburg,
Russia) and Institute of International Economic Relations (Moscow, Russia) in November
and December, 2022. The survey totally comprises 4 questions about intercultural
interactions, 6 questions about cultural knowledge and some questions about individual
demographics of participants such as age, gander, education, work experience and position.
4 questions about intercultural interactions embodying 1) “do you have abroad experience”;
2) “do you speak second language”; 3) “frequency of cross-cultural interaction”; 4)
“frequency of interaction with people with disabilities”. Questions about cultural knowledge
are based on Cultural Intelligence Scale (CQS) (Ang, Van Dyne and colleagues, 2007) with
seven-dimensional Likert form, including 1) “ I know the legal and economic system of the
other cultures”; 2) “I know the rules (e.g., vocabulary, grammar) of other languages”; 3) “I
know the cultural values and religious beliefs of other cultures”; 4) “I know the marriage
systems of other cultures”; 5) “I know the arts and crafts of other cultures”; 6) “I know the
rules for expressing nonverbal behaviors in other cultures”. Furthermore, this article applied
Harvard Implicit Association Test about Disability and collected results of every participants.
109
3. Result
Through analysis via IBM SPSS Statistics 26, this article lists the correlations among
intercultural interaction, cultural knowledge and implicit attitudes toward disability in the
Table 1 as follows. Results support hypotheses of the article, intercultural interactions with
people with disability positively correlate with implicit attitudes towards disability, as well,
individual cultural knowledge about disability positively correlates to implicit attitude
towards disability.
Table 1. Correlations among Intercultural interaction, cultural knowledge and implicit attitude.
Coefficient Intercultural Cultural Implicit attitude

interaction knowledge
Intercultural - 0.518*(0.031) 0.619**(0.007)

interaction
Cultural knowledge 0.518*(0.031) - 0.213*(0.046)
Implicit attitude 0.619**(0.007) 0.213*(0.046) -

Note: * indicates significance at 0.05 level; ** indicates significance at 0.01 level.
4. Discussion
According to results, negative implicit attitudes towards disability still commonly exist.
While, this article proposes the new solution that intercultural interactions with disabled
people and cultural knowledge about disability could improve implicit attitudes towards
disability among students in the universities. The stereotypes towards people with disabilities
represent the misunderstanding between different social and cultural groups. People with and
without disabilities could be considered as two different cultural groups, which are provided
with distinct traits and behavioral patterns. Thus, effective interaction between these two
groups requires cultural competence. Both practical intercultural interaction and theoretical
cultural knowledge could mitigate the stereotype and bias towards people with disabilities,
and hold more positive attitudes towards the disabled in the daily study and work. The
positive attitudes will let individuals more recognize and appreciate competencies of people
with disabilities, which will lead real disability inclusion and productive cooperation between
these two groups.
Current findings suggest the inclusion of cultural competencies into higher education. It has
been highlighted that interactions with people with disabilities could ease negative implicit
attitudes of individuals. In the university, it is crucial to provide opportunities of inclusive
programs or events to enhance interactions between people with and without disabilities. As
well, cultural competency-based schemes and programs should also be paid more attention
110
Zijun Li, Lyudmila Tokarskaya Valerievna
to develop students’ cultural knowledge of disability and accordingly foster the better
cooperation between employees with and without disabilities in the later work environment.
More future research are needed to investigate detailed interculturally interactive and cultural
competency-based schemes in the higher education.
Furthermore, this present study proposes the professional training program about cultural
knowledge into inclusive employment, which will help the two cultural groups understand
better about each other and ready for co-work. Except training program, events or activities
involving intercultural interactions could also be concerned as the method for inclusive
employment. Different kinds of interactions like work-related interaction and social
interaction could be offered by workplaces in the way of training, regular workshop, team
game, group travel, amusement event and so on. But it may be influenced by types of
disabilities and degree of disabilities of individuals. For instance, for employees with
blindness, organizational weekly reading day could be considered to involve employees
without disabilities to learn books for the blind. More additional studies are needed to
investigate the effective invention method for inclusive employment and education.
Lastly, this present article has several limitations. First, small sample has been collected by
this article. Investigations for larger sample are required. Second, the implicit association
test- attitudes towards disability is not adapted enough in Russia. More future studies are
needed for checking its adaptation in Russia and with different social groups. Third, the
questionnaire for intercultural interaction are self-developed, and it may be with more risks
for reliability than standard scales. Forth, this study did not conduct the correlation with
demographics factors, such as gander, education and work experience. These factors may
impact for the correlations among intercultural interactions, cultural knowledge and
individual implicit attitudes towards disabilities.
References
Ang, S., Van Dyne, L., Koh, C., Ng, K. Y., Templer, K. J., Tay, C., & Chandrasekar, N. A.
(2007). Cultural intelligence: Its measurement and effects on cultural judgment and
decision making, cultural adaptation and task performance. Management and
organization review, 3(3), 335-371.
Bonaccio, S., Connelly, C. E., Gellatly, I. R., Jetha, A., & Ginis, K. A. M. (2019). The
participation of people with disabilities in the workplace across the employment cycle:
employer concerns and research evidence. Journal of Business and Psychology, 35, 135-
158. https://doi.org/10.1007/s10869-018-9602-5
Packer, T. L., Iwasiw, C., Theben, J., Sheveleva, P., & Metrofanova, N. (2000). Attitudes to
disability of Russian occupational therapy and nursing students. International journal of
rehabilitation research. Internationale Zeitschrift fur Rehabilitationsforschung. Revue
internationale de recherches de readaptation, 23(1), 39-47.doi: 10.1097/00004356-
200023010-00005.
111
Rudman, L. A. (2004). Sources of implicit attitudes. Current Directions in Psychological

Science, 13(2), 79-82. https://doi.org/10.1111/j.0963-7214.2004.00279.x
Songa, G., Slabbinck, H., Vermeir, I., & Russo, V. (2019). How do implicit/explicit attitudes
and emotional reactions to sustainable logo relate? A neurophysiological study. Food
Quality and Preference, 71, 485-496. https://doi.org/10.1016/j.foodqual.2018.04.008
Thomas, D. C., Elron, E., Stahl, G., Ekelund, B. Z., Ravlin, E. C., Cerdin, J. L., ... &
Lazarova, M. B. (2008). Cultural intelligence: Domain and assessment. International
Journal of Cross Cultural Management, 8(2), 123-143.
Volosnikovaa, L. M., & Efimovab, G. Z. (2016). Faculty attitudes towards students with
disabilities in Russian universities: A glance at Western Siberia. In IFTE 2016: 2nd
International Forum on Teacher Education.http://dx.doi.org/10.15405/epsbs.2016.07.68
112
Developing supply chain competencies through experiential

learning and games
Heriberto García-Reyes1, Sandra Maycotte-Felkel2, Elena Isabel Victoria Quijano-
Dominguez3
1
Department of Industrial Engineering, Tecnologico de Monterrey, Mexico, 2Department of
International Business, Tecnologico de Monterrey, Mexico, 3Department of International
Business, Tecnologico de Monterrey, Mexico.
Abstract
The Supply Chain Management field has a shortage of talent to deal with the
complex problems organizations face nowadays. Higher education institutions
have worked to close the gap in knowledge and competencies with various
results. This paper presents the results of implementing a Supply Chain
Management course, which includes experiential learning, gaming, and
industry-academia collaboration to develop four competencies in this field.
The main results show how the relevance of hands-on learning, the variety of
learning environments -such as virtual, simulated, and real- and tutoring from
both practitioners and professors from different disciplines become key
elements in developing the required skills to perform a role in a Supply Chain
related position.
Keywords: Experiential learning; role-playing game; learning environments;
supply chain competencies; competencies development.

Developing supply chain competencies through experiential learning and games
1. Introduction
Nowadays, market trends’ dynamics are affecting the organizations’ configuration and
operation of supply chains. Constant and ever-faster changes in markets, economy, finance,
and technology encourage supply chain evolution in collaboration, configurations, and
management. Moreover, including Internet of Things (IoT) technologies or adopting cyber-
physical systems has accelerated the innovation in supply chain operations to meet customer
needs (Garay-Rondero et al., 2020). Similarly, globalization has changed how the industry
competes due to global sourcing, cultural challenges, the interactions among and between
nations, and how all these factors affect the production of goods and services (Fernando and
Wulansari, 2021).
On the other hand, graduates that enter a related supply chain position in the industry have
been showing gaps between academic training and the industry requirements. For these
people, the required skillset includes the capability of dealing with uncertainty, dealing with
non-routine situations, adapting to rapid changes in the work environment, flexibility, and
adapting to dynamic work requirements, among others (Al-Shammari, 2022). Additionally,
companies face significant challenges in recruitment, retention, and succession of the few
able talents, mainly due to the combination of competencies required and the level of
interaction needed within and across the organization to create value for all the stakeholders
(Flöthmann et al., 2018).
Considering this scenario, this paper presents the findings after implementing a supply chain
management (SCM) course involving active learning didactic strategies, information
technology aids, and dual learning to develop SCM competencies at the undergraduate level.
The remaining of the paper is as follows: section two briefly presents a theoretical
background on supply chain talent requirements and current teaching innovations; section
three describes the methodology followed to develop supply chain competencies in the
students; section four shows the results obtained during the implementation of the course;
section five includes the discussion results, and section six summarizes the conclusions and
recommendations for future work.
2. Theoretical Background
Developing supply chain competencies has been relevant for over twenty years. However,
the supply chain operations’ complexity emanating from a changing business environment,
globalization, and technological advances has raised hurdles in meeting the industry
requirement for SCM professionals, who additionally need to develop skills to perform
planning and strategic functions and not just carry on transactional or office tasks (Prajogo
and Sohal, 2013). Recent studies indicate a factor of one to six between the SCM
professionals supplied by higher education institutions and industry needs. This proportion
114
Heriberto García-Reyes, Sandra Maycotte-Felkel, Elena Isabel Victoria Quijano-Dominguez
does not consider the effectiveness of the training nor its alignment with the competencies
required for current SCM-related positions (Birou et al., 2022). Furthermore, supply chain
professionals have become a source of competitive advantage to organizations, so a lack of
talent is considered a relevant cause of supply chain risk (Dubey and Gunasekaran, 2015).
Thus, focusing on developing SCM competencies at the undergraduate level becomes critical
for an organization. The following subsections briefly describe previous research on the need
for SCM competencies in the current related positions and preceding academic efforts to
develop them.
2.1. Supply Chain Competencies

There is no unique classification or consensus about the skills required for SCM
professionals. For instance, Sun and Song (2018) recognized that the SCM field is
continuously evolving, such that it is becoming more interdisciplinary as the business
environment is more complex every day. After a thorough literature review, they classified
the SCM skills required by the professional field. They included technical knowledge and
soft skills requirements such as communication and teamwork, self-management and
improvement, and social responsibility. Conversely, Birou et al. (2022), based on a review
of 109 courses, found 120 related topics, of which 18% appeared once throughout these
courses, concluding that there is no clear consensus about the SCM field coverage. Finally,
Sweeney et al. (2010) remark on preparing future professionals to use sophisticated SCM
software to deal with problem complexity, efficiently use the information, and support the
decision-making processes. This broad scope makes it difficult to define a clear set of skills
to be covered in a course. Moreover, the contents may change between an engineering
program course and a business course.
2.2. Academic Efforts in Developing SCM Competencies.

Regarding academic efforts to provide an adequate environment for developing supply chain
competencies, the literature shows a variety of scopes. For instance, Pepper and Clements
(2008) used a role-playing approach to provide the “learner” with a structured methodology,
allowing individual learning outcomes to be accomplished through scenario-based activities.
The applied simulation exercise provided an environment for experiential learning and
problem-solving at a pace defined by the learner. Results showed benefits related to
complexity understanding, communication, and adaptation to changing scenarios. Similarly,
Sweeney et al. (2010) used specialized software to provide experiential learning in a more
realistic and significant context for understanding key management concepts. The authors
highlighted the benefits and challenges of designing courses based on specialized supply
chain software. Likewise, Gámez-Pérez et al. (2020) proposed a collaboration model to
develop supply chain competencies through experiential learning, international experiences,
and collaboration between two universities in different countries and industries. Their
115
findings include the need to promote closer collaboration between academics and
practitioners to generate an understanding of the needs, competencies, and skills required in
the future SCM workforce.
3. Methodology
As presented, the previous literature review emphasizes the importance of collaboration
between industry and universities in identifying, assessing, and developing relevant SCM
competencies in students. Regardless of the difficulty in identifying the key SCM
competencies to be developed in a course based on industry requirements, extant literature
shows some SCM competencies commonly identified as critical, presented here in Table 1.
Table 1. SCM competencies commonly identified as critical.
Competency Description Source
SC Diagnosis the ability to define and understand Birou et al., 2022; Gámez-Pérez et
supply chain complex problems al., 2020; Prajogo and Sohal, 2013;
Dubey and Gunasekaran, 2015
Technology Adoption the ability to be flexible and capable Al-Shammari, 2022; Fernando &
of adapting to new and changing Wulansari, 2021; Flöthmann et al.,
situations, including the use of 2018; Gámez-Pérez et al., 2020
technology to support SCM
operations
Data analysis and the ability to understand the Al-Shammari, 2022; Birou et al.,
communication business context and effectively 2022; Flöthmann et al., 2018
communicate findings
Improvement the ability to solve complex Prajogo and Sohal, 2013; Gámez-
proposals problems, proposing improvements Pérez et al., 2020; Flöthmann et al.,
or innovations based on creating 2018
value for all the stakeholders
Source: Authors’ compilation based on references.
Even though more competencies appear in the literature, the authors selected these four to be
evaluated during their SCM course. A series of formative evaluations were conducted, which
led to a summative assessment of each of the competencies, considering an intervention
process from diagnosis to improvement proposals for the supply chain of the collaborating
industry partners. In addressing the gap between academic training and industry
requirements, the authors included experiential learning, role-play gaming activities, and
industry-academia collaboration to enhance the development of the selected competencies.
116
Seeking to enhance the students’ experiential learning, the authors implemented a project-
oriented learning didactic strategy via the challenges posed by the collaborating
organizations. Industry professionals were involved before the start of the semester in
designing the challenge posed to the students and throughout it in assessing the presented
proposals. In addition, intending to provide different points of view to analyze the problems
and define feasible solutions, SCM experts from other industries and industrial chambers
were invited to share their experiences and opinions on current disruptions and challenges.
Also, a LEGO supply chain scenario simulator was applied twice in the semester to enhance
the students’ learning experience, helping them to understand key SCM concepts, identify
opportunities, and propose improvement actions. The first simulation was run at the
beginning of the course to clarify SCM concepts and provide a systemic scope and a role-
play gaming learning environment. The second one was run previously to elaborating the
improvement proposals.
Being sensitive to the increasing interdisciplinarity of the SCM field in the constantly
changing business environment, the authors considered that offering an integral approach
from different disciplines will add value to the student’s learning experience. Therefore, the
professors implementing the course pertained to various fields -industrial engineering and
international business- providing a multidisciplinary approach.
Concerning industry-academia collaboration, the organizations presented a specific problem
that required using the selected competencies and assigned a mentor for each. Each mentor
defined the number of follow-up meetings, whether in situ or virtual, the feedback procedure,
and provided the information needed. In addition, sessions required by the professors in
charge of tutoring the students complimented the students’ learning experience. The chosen
problems involved students analyzing and proposing improvements for previously defined
supply chain operations. The industry partners were a Mexican-based organization in the
meat industry during the Spring 2022 semester and a US-based organization in the heavy
agricultural machinery industry during the Fall 2022 semester. Both organizations have
worldwide operations and similar operative challenges. The results of the implementations
are shown in the next section.
4. Results
The pilot implementation for the Spring 2022 semester was successful in that students,
professors, and industry partners did perceive the development of the specified SCM
competencies. Nevertheless, the professors and industry partners evaluating the student
performance in such competencies detected room for refinement, particularly in the
competence related to the improvement proposal. Even though the industry partner received
the submitted proposals well, several issues were detected regarding the time scope limitation
117
confined to one semester, which limited the implementation and benefit measurement of the
proposed improvement proposals.
Under such conditions, students cannot learn about the consequences of their decisions, and
submissions remain at the proposal level. Therefore, the authors decided to implement The
Fresh Connection™ (TFC) simulator during the Fall 2022 semester, in addition to the
previously described conditions implemented during the Spring 2022 semester. The authors
intended to overcome some of the mentioned shortcomings since TFC requires analysis and
decision-making processes from participants, assuming the perspective of various roles. TFC
simulates the students’ decisions during six rounds in a SC scenario, each representing six
months, allowing the participants to face the consequences of their improvement decisions
and acknowledge their errors for three years. The following figure compares the average
grades obtained by students on the individual submissions designed to demonstrate the
development of the specified four SC competencies.
SCM Competencies Development Comparison

99,4
100
98 97,3
Average Grades
95,5 95,6 95,2

96
93,8 93,4
94 92,4
92
90
88
Diagnosis Technology Data Analysis Improvement
Adoption and Proposals
Communications
Spring 22 Fall 22
Figure 1. Average grading comparison of individual student deliverables on competencies development.
Even though the average grades increased during the Fall 2022 semester, the main concern
was identifying the benefit of a physical role-playing simulation using LEGOs and the
specialized SC software TFC to refine the students’ deliverables. In this regard, a Student’s
t-test analysis was conducted to compare the grades of the student proposals presented during
both implementations. The null hypothesis assuming group similarity was rejected,
demonstrating a significant upgrade of the improvement proposal competency in students
after experiencing TFC simulator in combination with various learning environments. The
grading process includes the comments from the mentors and the professors’ assessment of
the competence based on a rubric. Results are shown in Table 2.
118
Table 2. T-Test two samples assuming equal variance.
5. Discussion
A combination of learning didactic strategies, information technology aids, and dual learning
during the implementation of a SCM course enhances the development of the professional
competencies required to thrive in a changing business environment. However, there are
limitations to developing SCM competencies in the classroom, even when innovative
learning techniques are implemented, as was detailed in the literature review. The authors
tested a combination of several learning environments and were able to compare the results
from two consecutive implementations under similar conditions. There are limitations to the
development of competencies in students confined to one semester in the classroom. Hence,
it is necessary to provide experiential learning as close as possible to real-life challenges to
overcome such shortcomings and strengthen the development of the nuch-sougth SCM
competencies, which have become a competitive advantage for organizations. Although the
inclusion of several learning environments, such as the physical LEGO simulator and the use
of TFC created a positive benefit in terms of competency development, this was only possible
in combination with the guidance and tutoring of professors, the interaction with industry
experts, a multidisciplinary approach, and a hands-on experience.
6. Conclusion and Future Work

After two implementations of the course, the SCM competencies were developed
satisfactorily, based on evaluations of the students’ deliverables by the industrial partner and
the team of professors. The multidisciplinary work, the simulators, the organizations’ supply
chain problems, and the evaluation from industry experts and professors allowed the students
to experience similar real-life challenges. Finally, designing a SCM course is a dynamic,
iterative process if it is to be aligned with the changing environment and disruptions faced
119
by supply chains nowadays. Hence, this course’s didactic design and improvement processes
must be continuously adapted to current and future disruptions influencing a supply chain,
the industry's needs, and other collaboration schemes.
References
Al-Shammari, M.M. (2022). An exploratory study of experiential learning in teaching a
supply chain management course in an emerging market economy. Journal of
International Education in Business, 15(2), 184-201. doi.org/10.1108/JIEB-09-2020-
0074.
Birou, L., Lutz, H. and Walden, J.L. (2022). Undergraduate supply chain management
courses: content, coverage, assessment and gaps. Supply Chain Management, 27(1), 1-
11. doi.org/10.1108/SCM-07-2020-0309
Dubey, R. and Gunasekaran, A. (2015). Supply chain talent: the missing link in supply chain
strategy. Industrial and Commercial Training, 47(5), 257-264. doi.org/10.1108/ICT-11-
2014-0073
Fernando, Y. and Wulansari, P. (2021). Perceived understanding of supply chain integration,
communication and teamwork competency in the global manufacturing companies.
European Journal of Management and Business Economics, 30(2), 191-210.
doi.org/10.1108/EJMBE-06-2020-0157.
Flöthmann, C., Hoberg, K. and Wieland, A. (2018). Competency requirements of supply
chain planners & analysts and personal preferences of hiring managers. Supply Chain
Management, 23(6), 480-499. doi.org/10.1108/SCM-03-2018-0101.
Gámez-Pérez, K.M., Sarmiento, A.M., Garcia-Reyes, H. and Velázquez-Martínez, J.C.
(2020). An international university-industry collaboration model to develop supply chain
competences. Supply Chain Management, 25(4), 475-487. doi.org/10.1108/SCM-08-
2019-0317
Garay-Rondero, C.L., Martinez-Flores, J.L., Smith, N.R., Caballero-Morales, S.O. and
Aldrette-Malacara, A. (2020). Digital supply chain model in Industry 4.0. Journal of
Manufacturing Technology Management, 31(5), 887-933. doi.org/10.1108/JMTM-08-
2018-0280.
Pepper, M.P. and Clements, M.D. (2008). Extended scenario role‐playing: cumulative
learning for supply chain participants. Development and Learning in Organizations,
22(3), 21-24. doi.org/10.1108/14777280810861794
Prajogo, D. and Sohal, A. (2013). Supply chain professionals: A study of competencies, use
of technologies, and future challenges. International Journal of Operations & Production
Management, 33(11/12), 1532-1554. doi.org/10.1108/IJOPM-08-2010-0228
Sun, L. and Song, G. (2018). Current state and future potential of logistics and supply chain
education: a literature review. Journal of International Education in Business, 11(2), 124-
143. doi.org/10.1108/JIEB-10-2017-0039.
Sweeney, D., Campbell, J. and Mundy, R. (2010). Teaching supply chain and logistics
management through commercial software. The International Journal of Logistics
Management, 21(2), 293-308. doi.org/10.1108/09574091011071960.
120
Students’ experience of Double Degree programmes: motivations

and perceptions of skill acquisition
Elena Borsetto1, Chiara Saccon2
1
Department of Linguistics and Comparative Cultural Studies, Ca’ Foscari University of
Venice, Italy, 2Department of Management, Ca’ Foscari University of Venice, Italy.
Abstract
The implementation of collaborative and double degree programmes has been
promoted in higher education since the 1970s as an internationalisation
strategy for networking between universities. The value of the Double Degree
experience is unanimously recognised by European and international
institutions, but data on the evaluation of this experience from the students'
perspective, their motivation to take part in these programmes and the
perceived skills that can be acquired through this experience seem to be
scarce. Through a pilot questionnaire designed to collect quantitative and
qualitative data, this study aims to investigate these aspects - students'
motivations and perceptions of the skills acquired - in relation to their
participation in a Double Degree programme. The results of this survey can
contribute to increasing students’ and stakeholders’ awareness of the benefits
of Double Degrees and to improving the quality of teaching and services in
these programmes.
Keywords: Internationalisation; higher education; Double Degree
programmes; mobility experience; employability skills; soft skills.

Students’ experience of Double Degree programmes: motivations and perceptions of skill acquisition
1. Introduction
Higher education institutions are among the main actors involved in the internationalisation
process, as they determine strategies and activities to promote their international dimension
not only locally, but also globally, thanks to globalisation dynamics. At the regional and
national level, collaborative programmes are seen as means to contribute to an increase in a
country’s status, competitiveness, and capacity building (cf. Knight, 2008). The
implementation of collaborative and double degree programmes has been promoted in higher
education since the 1970’s as an internationalisation strategy for networking among
European universities in the EHEA (European Higher Education Area) and transatlantic
relations between Europe and USA. Double degrees can bring a number of benefits, not only
because they foster student mobility and institutional networking, but also because the
knowledge transfer created by partner institutions becomes a bridge between different
content materials, teaching methods, and ultimately, different cultures.
From an institutional perspective, universities can enhance their reputation through
partnerships with other renowned institutions. For students participating in this experience,
one of the perceived benefits of these programmes is the acquisition of a range of skills that
can make graduates more likely to be employed in the future (Jones, 2014). Double degree
programmes also represent an investment in terms of time and funds for the institutions
involved in the process, and a challenge for students who have to change their learning
environment and step out of their comfort zone. The value of the double degree experience
is unanimously recognised by European and international organisations (cf. European
Commission 2022), but data on the evaluation of this experience from the students’ point of
view, their motivation to participate in these programmes and the perceived skills that can be
acquired through this experience seem to be scarce. With a view to improving the quality of
the teaching and services provided in a Double Degree programme, this study aims to
investigate these aspects – students’ motivations and perceptions of skills acquisition – in
relation to their participation in such a programme, through a pilot questionnaire designed to
collect both quantitative and qualitative data. The results of this survey may contribute to
improving the students’ and practitioners’ awareness of the existing programmes, and better
inform other stakeholders about the benefits of double degrees.
The first double degree programmes in Europe were launched in the late 1970s and probably
originated from the pre-Erasmus Joint Study Programme, which ran from 1976 to 1984
(European Commission, 2020). Initially, double degrees were developed to open labour
markets in other countries to graduates (European Commission, 2020). According to the
literature, the terminology used to define double degrees is similar, but not always
122
unanimous. In Knight’s book on higher education, joint and double degrees are given the
same definition: “An arrangement whereby providers in different countries collaborate to
offer a program for which a student receives a qualification from each provider or a joint
award from the collaborating providers” (2008, p. 105). In a later publication, the scholar
differentiates the two types of programmes, indicating that a double degree programme
awards “two individual qualifications of an equivalent level upon completion of the
collaborative programme requirements established by the two partner institutions” (Knight,
2011, p. 301). Whereas, in a joint degree programme, a joint diploma is conferred upon
completion of the collaborative programme requirements set by the partner institutions
(Knight, 2011, p. 300). International double degrees are offered by two higher education
institutions, located in different countries, which have collaborated to develop an integrated
curriculum where students’ credits are recognised in the administrative system of both
institutions. Students attend classes and study at the two (or more) partner institutions (i.e. 1
home institution + 1 institution abroad). At the end of the study programme, each of the
institutions issues a degree certificate to the students.
The opinions of the main stakeholders involved in double degree programmes were analysed
in a couple of cross-countries studies (Culver et al. 2011, 2012). Culver et al. (2011)
examined the strengths and weaknesses of a sample of dual- and joint-degree programmes at
the graduate level in the United States, Sweden, Italy, and Germany. Their findings suggest
that the primary reasons for implementing such programmes are the possible benefits for both
students and their institutions: while students can add an international element to their
education through direct exposure to another culture and educational system, their institution
can increase its visibility and reputation through partnerships with other universities. In a
follow-up study (Culver et al. 2012), four different stakeholder groups provided insights into
their own perceptions of double-degree experience. According to the students and alumni
interviewed, the double degree could make them more employable, representing a way of
widening their pool of job opportunities, in both the countries in which they studied, and as
proof of their proactiveness and willingness to do more than their peers. In contrast, the group
of teachers was less convinced that the double degree could improve students’ employability.
In another study (Wiers-Jenssen, 2012) on the employability of students who have had a
mobility experience compared to those who have not, it was pointed out that a degree
obtained abroad is neither an advantage nor a major disadvantage. “This also indicates that
the ‘added value’ of a full degree abroad is not fully convertible to a domestic setting and
that some employers may be sceptical toward foreign education” (Wiers-Jenssen, 2012, p.
485). From the students’ perspective, double degrees present the advantages of boosting their
career opportunities, having international study and life experience, and the impression of
receiving “two degrees for one”, as the workload and tuition fees may be lower than in a
traditional degree (Knight, 2013).
123
3. Method and data collection

The data collection technique used in this pilot study was an exploratory questionnaire (cf.
Creswell, 2014), as it was the most suitable tool to investigate the opinions of Double Degree
students. The main research questions that guided the study were meant to uncover students’
motivations for participating in a Double Degree programme, and their perceptions of the
value of the experience, in terms of the acquisition of mobility skills. Of these, the present
paper will consider the following questions:
RQ: What are the main motivations for students to enrol in a Double Degree programme?
RQ: What are the students’ perceptions of the employability and mobility skills they
developed the most during this experience?
The questionnaire consisted of both open and closed questions, and was structured in a series
of sections based on the main aspects to be explored. The first section started with an open-
ended question on the students’ motivations to enrol in the Double Degree programme. This
required students to state the reasons for their choice, with no predetermined category from
which to choose. Another section of the questionnaire aimed at discovering students’
perceptions of the skills that can be developed during a Double Degree programme. The
available literature on employability and mobility skills – conceived as a mix of soft skills
and intercultural competence – was helpful in formulating the close-ended questions. The list
of items was adapted from a European study on the skills that students should gain through
mobility. The study is the result of an Erasmus+ Programme co-funded project called
‘Erasmus Skill Project’, which took place between 2018 and 2020, with the aim “to prepare
students before their mobility abroad for pursuing their studies under the Erasmus+ Exchange
Program and to help them assess their learning curve before, during and after this mobility
experience as well as to better understand the skills developed from this experience”.1
Participants were asked to rate a list of items according to a Likert-type scale from 1 to 5,
where 1 was ‘not important’ and 5 was ‘highly important’.
The anonymous, Internet-based questionnaire was first sent at the end of March 2022 to
students currently enrolled in Double Degree programmes at four selected universities in
Italy, Germany and France, which have established such programmes among themselves. A
total of 90 e-mails with the questionnaire invitation were sent to the students' institutional
addresses, using the e-mail address received when they enrolled in the programmes. Of these
1
Intellectual Output 2 of the Erasmus Skills Project (2018-2020). ERASMUS + Project number: 2018-
1-ES01-KA203-050439. Erasmus+ Key Action 203 - Cooperation for innovation and the Exchange of
good practices; Strategic Partnerships for higher Education. Retrieved from:
https://www.erasmusskills.eu/eskills/tools/self-assessment-
questionnaires/_intro/ES_IO2_Final_Report.pdf
124
e-mails, 20 were returned by the system because the e-mail addresses were no longer valid,
resulting in 70 e-mails actually delivered to the students. To ensure that all students received
the invitation, a second set of 42 e-mails was also sent to the e-mail addresses used at the
home institution, of which 7 were returned by the system and 35 were valid. After a few
weeks, a reminder message was sent to complete the survey. Thirty-five students, who were
either in the outgoing or incoming mobility phase at the time, answered the questionnaire.
The respondents were 20 males (57.1%) and 15 females (42.9%), the majority of whom were
between 23 and 25 years old (26 students), 6 between 26 and 28 years old and 3 between 20
and 22 years old. Some students attended their Double Degree programme between the 2020-
22 academic years, so they were close to the end of the programme when they filled out the
questionnaire, while others enrolled in the 2021-22 academic year and will conclude it in the
2022-23 academic year, thus providing very up-to-date answers about their experience.
Below, the analysis of the results presents the students’ views.
4. Results
One of the initial sections of the questionnaire aimed to discover why students are motivated
to participate in double degree programmes, through an open-ended question. The students’
responses were analysed through content analysis (Schreier, 2012) and classified according
to thematic labels, and the main motivations that emerged were: “living abroad and seeing
other cultures” (11 answers), “increasing employability and career opportunities” (9),
“enriching the curriculum vitae” (6). These themes can associated both with intrinsic
motivation, such as curiosity and the challenge of discovering another country and another
culture, and with extrinsic motivation, given by external rewards (cf. Ryan & Deci, 2000).
Other themes found were related to the type of extrinsic motivation, driven by the acquisition
of skills or the conscious improvement of an activity: “to broaden and improve my
knowledge” (5 answers), “to acquire new skills” (5) and “for the structure of the programme”
(5). Others were related to the benefits and final results of the programme (extrinsic
motivation), such as “to gain international experience” (4 answers), “to obtain a double
degree” (4), “to network (meet new people)” (4). Other extrinsic motivations cited less often
were “reputation of the business school” (2), “to improve English proficiency” (2), “the
opportunity to work in the host university's country” (1); while the fact that “it was the only
available double degree programme to choose from” (1) shows an amotivation (cf. Ryan &
Deci, 2000). The students’ responses seem to indicate a mix of intrinsic motivations, such as
experience abroad and the acquisition of skills and knowledge, and extrinsic motivations,
such as future career opportunities and the attractiveness of one’s CV if one holds two
degrees from different universities. In the study, these views were confirmed by a subsequent
close-ended question, in which students were asked to rate the importance of these
motivations in their decision to participate in a Double Degree programme.
125
One of the closed-ended questions on students’ perceptions of skill acquisition contained a

set of seven items adapted from the ‘Erasmus Skills Project’ (cf. par. 3): intercultural
awareness, global citizenship, resilience, openness and curiosity, empathy and tolerance,
creativity (i.e. thinking outside the box), and academic knowledge. Students were asked to
evaluate which competences they felt could be most acquired in a mobility experience,
answering the question: “Which of these ‘mobility skills’ can be developed during a Double
Degree experience?”. The large majority of the respondents (77%) indicated that
‘intercultural awareness and knowledge’ can be highly developed, while the possibility of
developing ‘global citizenship’ was rated as ‘highly’ by 23 students (66%) and as ‘very’ by
10 (28,6%). The students also expressed similar ratings for ‘resilience’, which 66% rated as
‘highly’ and 28,5% as ‘very’ likely to be developed, followed by ‘openness and curiosity’
(62% as ‘highly’ and 26% as ‘very’ likely). Other skills, instead, received mixed ratings as
‘empathy and tolerance’ was rated as ‘highly’ likely to be developed by 51% of respondents,
while 23% rated it as ‘very’ and 20% ‘moderately’ likely. The acquisition of creative skills,
meant as a way of ‘thinking outside the box’, was considered as ‘highly’ likely developed by
16 students (46%), and ‘very’ and ‘moderately’ by 8 students each. Finally, only 43% of the
students felt that ‘academic knowledge’ could be ‘highly’ developed, while 7 ‘fairly’ and 12
‘moderately’, making it the competence least associated with mobility, in the list provided to
the students. The results of their responses can be seen in the figure below (Figure 1).
Which of these 'mobility skills' can be developed during a Double

Degree experience?
Academic knowledge
Creativity (think outside the box)
Empathy & tolerance
Openness & Curiosity
Resilience
Global citizenship
Intercultural awareness/knowledge
0 5 10 15 20 25 30 35 40
Don't know 1 - Not al all 2 - Slightly 3 - Moderately 4 - Very 5 - Highly
Figure 1. Students’ perception of the acquisition of ‘mobility skill’ during a Double Degree programme.
126
The analysis of the findings, part of which has been presented here, revealed a number of
aspects related to the double degree experience that open the way for further reflection on the
impact of these programmes for the students and institutions involved.
5. Discussion
As the results from this study show, students are motivated to participate in Double Degree
programmes because they feel that they can acquire both knowledge and skills, as well as
intercultural and soft skills, considered as ‘highly important’ by half of the participants. Their
responses are in line with research that states that exposing students to different cultural
contexts through mobility experiences can help them question their identities, values and
beliefs, and help them improve their personal growth and self-efficacy, along with their
intercultural competence (Jones, 2014).
It is not unexpected that students who enrol in international mobility programmes are
motivated by the possible acquisition of these skills, as it is equally true that these
programmes attract students who already possess such skills and are drawn to mobility as a
way to further develop them (Jones, 2013, p. 102). In fact, it has been shown that mobile
degree students constitute a select group in terms of their prior exposure to international
experiences, because they are more likely to have lived abroad and to have parents who have
lived abroad (Wiers-Jenssen, 2012). Furthermore, it appears that the skills acquired through
students’ international mobility initiatives are the type of general employability skills sought
by employers (Jones, 2013). To support students in becoming more aware of their skills, it
might be beneficial to involve them in some preparatory activities before their departure for
their mobility, and after their return. An example of such activities is provided in the
‘Erasmus Skills Project’ mentioned before (cf. par. 3), where pre and post self-assessment
questionnaires were created to help students assess how and which mobility skills they
improved during their experience abroad. A similar assessment activity could be created to
meet the needs of Double Degree students, who should be able to emphasise the value of
their international experience, and make their CV’s stand out not only because of the presence
of two degrees awarded, but also because it indicates a goal-oriented, highly motivated and
competent candidate.
To bridge the gap between the skills sought by employers and those acquired by students, it
is crucial to raise awareness among the main stakeholders involved in the process – i.e.
students, universities, graduates and employers – so as to reflect on the pathway leading to
the development of students’ employability skills, and to better understand employers’
expectations (Succi & Canovi, 2020). This is especially important when students have
participated in an international mobility programme, because graduates need to be able to
adequately present the broad range of their competences, through relevant examples from
127
their own experience, during the recruiting phase, and then be able to convert their skills into
observable and value-adding behaviour once they are hired.
References
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods
approaches, (4th ed.). Thousand Oaks, CA: Sage.
Culver, S. M., Puri, I. K., Spinelli, G., Depauw, K. P., and Dooley, J. E. (2012). Collaborative
Dual-Degree Programs and Value Added for Students. Journal of Studies in International
Education, 16, 40 - 61.
Culver, S.M., Warfvinge, P., Grossmann, C. and Puri I. K. (2011). Evaluation of engineering
double-degree programs in Sweden: results of the Lund Focus Groups, European Journal
of Higher Education, 1:2-3, 220-232, DOI: 10.1080/21568235.2011.637678
European Commission (2022). Proposal for a council recommendation on building bridges
for effective European higher education cooperation. Strasbourg, 18.1.2022, COM(2022)
17 final 2022/0008 (NLE).
European Commission (2020). Implementing Joint Degrees in the Erasmus Mundus action
of the Erasmus+ programme, report by the Education, Audiovisual and Culture Executive
Agency (EACEA) of the European Commission Unit A3 – Erasmus Mundus, Sport.
Jones, E. (2014). Graduate Employability and Internationalization of the curriculum at Home.
International Higher Education, Special Issue, 78, 6-8.
Jones, E. (2013) Internationalization and employability: the role of intercultural experiences
in the development of transferable skills, Public Money & Management, 33:2, 95-104,
DOI: 10.1080/09540962.2013.763416
Knight, J. (2013) Joint, Double and Consecutive Degree Programs: Definitions, Doubts and
Dilemmas. In M. Kuder, N. Lemmens and D. Obst (eds). Global Perspectives on
International Joint and Double Degree Programs. New York: Institute of International
Education (IIE), pp. 1-15.
Knight, J. (2011). Doubts and dilemmas with double degree programs. In Revista de
Universidad y Sociedad del Conocimiento [Review of the University and the Knowledge
Society], 8(2), 235-248.
Knight, J. (2008). Higher education in turmoil: The changing world of internationalization.
Rotterdam: Sense Publishers.
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic
motivation, social development, and well-being. American Psychologist, 55(1), 68–78.
https://doi.org/10.1037/0003-066X.55.1.68
Schreier, M. (2012). Qualitative content analysis in practice. London: Sage.
Succi, C., Canovi, M. (2020) Soft skills to enhance graduate employability: comparing
students and employers’ perceptions, Studies in Higher Education, 45(9), 1834-1847,
DOI: 10.1080/03075079.2019.1585420
Wiers-Jenssen, J. (2012). Degree Mobility from the Nordic Countries: Background and
Employability. Journal of Studies in International Education, 17 (4), 471-491.
128
A suggested program for developing first-year university

Students’ agency
Edupedia for Educational Consultancy, Traning, and Curriculum Designing, Egypt.
Abstract
Students joining universities sometimes lack the agency to lead their lives and
succeed in higher education. In the educational context, agency refers to the
more empowered positioning of students to be active agents in their own
learning lives. Enhancing life skills/competencies can facilitate students’
agency when they instill positive set beliefs and personal competencies,
creating agentic capacity. Action Learning is one of the methods used to
enhance life skills/competencies by engaging students in real life problems,
taking action, and continuously reflecting on their actions. This paper presents
a suggested program for building firs-year university students’ agency through
a balanced approach of Competency-based Learning, Action Learning, and
Reflective Learning.
Keywords: Action learning; agency; competency-based learning; reflective
learning; university students.

A suggested program for developing first-year university students’ agency
1. Introduction
Student agency refers to the quality of students’ self-reflective and intentional actions and
interactions with their environment. “It encompasses variable notions of agentic possibility
(“power”) and agentic orientation (“will”)” (Klemenčič, 2015, p.11). Bandura (2006)
identified four core properties of human agency; intentionality, forethought (set themselves
goals and anticipate likely outcomes of prospective actions to guide and motivate their
efforts), self-reactiveness (self-regulation), and self-reflectiveness (metacognitive
capabilities). According to Heckman and Kautz (2013), skills give agency to people to shape
their lives in the present and to create future skills. Murphy-Graham and Cohen (2022)
offered a general definition of life skills. For them, life skills are skills “to be able to do life
well” (p. 37), opening the answer to be ‘it depends” (Schmidt, 2022, p.268). Research has
informed various intervention programs to enhance youth life skills according to the context
where they are applied. Due to life skills nature vary according to the culture and the
environment (UNICEF, 2003), each program differs in its goals and approaches (Bender,
2002).
One of the goals of teaching life skills is to support students’ transition from one educational
level to the other, raising their awareness (Akfırat & Kezer, 2016). The program suggested
was designed for first-year university students in Egypt to enhance students’ agency. The
program uses three approaches to learning to enhance students’ agency; Concept-based
Learning, Active Learning, and Reflective Learning. This program was implemented with
twenty first-year university students from different disciplines. There were clear signs of
skills/competencies development and agency enhancement in the final presentation, students’
reflections, and mentors’ notes.
2. The Suggested Program

The program used the backward design model (Wiggins & McTighe, 2005) to plan the
curriculum design sequence. The following are the details of the program under the three
aspects of the backward design; identifying desired outcomes, determining acceptable
evidence, and planning learning experiences.
2.1. Identifying Desired Outcomes

The first step is articulating desired outcomes needed for first-year university students. In this
step, Competency-based Learning was used to identify and define precise measurable
descriptions of knowledge, skills, and behaviors students will possess at the end of the
program. The Competency-based curriculum emphasizes life skills and evaluates mastery of
skills necessary for an individual to function proficiently in a given society (Savage, 1993).
The following are the four main competencies, each with sub-indicators that sets curriculum
skills focus and objectives.
130
Table 1: The Program Desired Outcomes of Main Competencies and Skills Indicators.
Main Skills indicators

Competencies By the end of the course, participants will be able to
Reliability S-1-1: Self-awareness
On S-1-1-a: identify the values and indicators of self-awareness
one self S-1-1-b: analyze preferences and style
S-1-1-c: develop self-awareness
the quality of S-1-3: Self-reflectiveness
being S-1-3-a: explore past experiences in detail
trustworthy of S-1-3-b: point out reasons behind preferable and unpreferable actions
oneself S-1-3-c: reconsider alternatives and identify the next steps
performing S-1-3-d: take reflective actions and develop new skills and strategies
consistently - S-1-4: Self-regulation
well. - S-1-4-a: practice self-control and resilience strategies
- S-1-4-b: practice self-motivation and positive thinking
- S-1-4-c: practice strategies of mental focus and concentration
- S-1-4-d: practice strategies to manage negative feelings
S-2-1: Communication
S-2-1-a: interpret received non-verbal messages and give appropriate answers
Connectability S-2-1-b: use a variety of speaking and writing techniques to communicate with
With a variety of audiences
others S-2-1-c: read critically from a variety of sources and understand the hidden
meaning
S-2-2: Social presence
the quality of S-2-2-a: collaborate with others to accomplish tasks
being able to S-2-2-b: build meaningful relationships and avoid negative peer influence
connect with S-2-2-c: accept and understand others’ differences
different people S-2-4: Civic engagement and leadership
for different S-2-4-a: recognize the relationship between community challenges and
purposes personal skills and contributions
S-2-4-b: manage peoples’ skills and monitor achievements
S-2-4-c: adopt passionate ideas and use influence strategy to form team
motivation
S-3-1: Intentionality
S-3-1-a: practice self-direction and clear vision
S-3-1-b: initiate and resume actions to reach desired results
Ingenuity S-3-2: Forethought
In S-3-2-a: set clear, challenging, and realistic long-term and short-term goals
life roles S-3-2-b: plan specific actions to achieve goals
S-3-2-c: follow-through with plans taking corrective actions throughout
the quality of S-3-3: Creativity
being clever, S-3-3-a: find, interpret, judge, and create information
original, and S-3-3-b: process and analyze information by sifting, sorting, comparing,
inventive. verifying, and trying out different ways to make sense of the information
S-3-3-c: practice observing, gathering, organizing, and testing information to
formulate and support critical arguments
S-3-3-d: consider multiple alternatives and create novel and innovative ideas
Main Skills indicators

Competencies By the end of the course, participants to be able to ………………….
S-4-1: Agility
131
S-4-1-a: think and draw conclusions using intellectual acuity

Adaptability S-4-1-b: react quickly to precarious situations with clear priority measures
To S-4-2: Decision-making and problem-solving
one’s S-4-2-a: combine knowledge, understanding, and skills to create solutions
environment S-4-2-b: consider unlikely or impossible alternatives
S-4-2-c: examine alternatives from different perspectives
S-4-3: Coping with unexpected situations
the quality of S-4-3-a: practice flexible thinking by developing multiple opposing,
being able to contradictory, and complementary arguments
adjust to new S-4-3-b: apply skills and knowledge in unfamiliar situations
conditions S-4-3-c: manage change by adopting alternation and practicing adaptive skills
2.2. Determining Acceptable Evidence

According to Mkonongwa (2018), competencies can be used as criteria for assessing the
competency attainment level required for success. The program uses the competencies set as
criteria of assessment through the program. Mentors keep track to produce their final
judgment of each sub-skills level for each student (skills are scored in four levels).
2.2.1. Students Journals
Students keep a journal throughout the program that they must use daily to record all actions
or problems encountered.
2.2.2. Ongoing Observations
Mentors write anecdotal notes for each group member after each reflective session describing
how they reacted and any significate signs showing the competencies.
2.2.3. Final Presentation
At the end of the program, every group presents the project journey, and each member
mentions their perspective of competencies enhanced in this project.
2.3. Planning Strategies for Implementation

2.3.1. Planned Interactive Workshops
As much as identifying which skills to teach in a program, how to teach them is more
important (Schmidt, 2022). The program starts with some interactive workshops on basic
competencies/skills. The interactive workshops through the implementation of the project
clarify some concepts students need at this specific phase/step of the project. Reflective tasks
through the implementation of the project are designed to consolidate students’ self-
discoveries and support skills’ development.
132
Table 2: Alignment of Project Steps/Phases with Action Leaning process, Interactive

Workshops, and Reflective Tasks.
Project Action Learning Interactive Skills Reflective

Steps/Phases Process Workshops Tasks
- Know about Task 1: Individual self-
Before yourself assessment of
introducing the - The growth competencies using
project mindset
program outcomes
before the beginning of
the project and putting
individual aims.
a. Explore - Present - Communication Task 2: Individual
their challenge/problem/ skills Reflective questionnaires
community. issue/question - Self-motivation focusing on self-
discoveries and setting
the next steps.
b. Creating an - Insight? New ideas - Collaboration Task 3: Group reflective
idea to and understanding, skills questionnaire of how the
develop perceptions and - Designing an idea was developed.
their assumptions. action plan Task 4: Individual
community. Reflective questionnaires
focusing on self-
the next steps.
c. Implement - Test actions in the - Time Task 5: Regular group
this idea. workplace management reflective sessions
- Bring results back - Self- supported by the mentor.
to set. What management Task 6: Individual
worked, what did - Perseverance Reflective questionnaires
not work focusing on self-
the next steps.
d. Collect - Drawing Task 7: Review of self-
evidence of conclusions and assessment in Task 1
learning. marking learning
e. Present the Task 8: Individual
outcomes. Reflective journal of
what was learned in the
program.
2.3.2. Designing the Project Procedures

The most effective learning is based on the need to solve problems (Welskop, 2013). Action
Learning is a structured method that enables small groups to engage in real-life problems,
take action, and learn as individuals and as a team while doing so (Serrat, 2017). Revans
(1983), the originator of Action Learning, believes that people must be aware of their
133
shortage of knowledge to be motivated to complete it. They then start to ask questions and
collaborate with others to find solutions, try them out, reflect again, and so on. In the process
of reflection, experience theory is transformed into knowledge, which undoubtedly increases
learning effectiveness (Welskop, 2013). Figure 1 clarifies the Action Learning Process. In
this process, they develop agency components of; intentionality, forethought, self-regulation,
and self-reflectiveness.
Figure 1. The Action Learning Process (Serrat, 2017).
The project takes students into different phases to implement an idea they create. The project
goes through the following steps in groups of five supported by a mentor:
a. Explore their community.
b. Creating an idea to develop their community.
c. Implement this idea.
d. Collect evidence of learning.
e. Present the outcomes.
2.3.3. The Reflective Spiral
According to Powell (2004), “Reflective practice is a hallmark of quality instruction” (P.2).
Moon (2005) mentions that "reflective learning emphasizes the intention to learn from
current or prior experience" (p.80). For Kember et al. (2008), "Reflection operates through a
134
careful re-examination and evaluation of experience, beliefs and knowledge" (p. 370). There
are different kinds of reflective approaches, for example, the use of journals and group
discussions following practicum experiences that are not directed to a solution of a specific
problem. Reaching the competencies set in the outcomes are achieved by students frequently
reflecting throughout the project. They go through the following: (see Table 2)
a. Individual self-assessment of competencies using program outcomes before the
beginning of the project and putting individual aims.
b. Individual Reflective questionnaires focusing on self-discoveries and setting
the next steps.
c. Group reflective questionnaire of how the idea was developed.
d. Regular group reflective sessions supported by the mentor.
e. Individual Reflective journal of what was learned in the program.
Frequent reflection helps them create their own cycle of plan, do, reflect, plan, do, reflect,
and so on. With the support of a mentor, this cycle consolidates the sources of agency of
intentionality, forethought, self-reactiveness, and self-reflectiveness.
3. Conclusion
The knowledge students learn from reading books about different theories and concepts is
important but insufficient (Weshop, 2013). Learning skills through action focuses on
application, giving students opportunities to try and refine skills. The accumulation of
skills/competencies learned through action and reflection build students’ agency and enhance
their ability to lead their lives. The program suggested takes students into a cyclic process of
learning, applying, and reflecting, then learning and applying again. It carefully identifies
what competencies first-year university students need and develops a path of application and
reflection supported by mentors to monitor and assess learning.
References
Akfırat, O. and Kezer, F. (2016). A Program Implementation for the Development of Life
Skills of Primary School Students. Journal of Education and Practice www.iiste.org ISSN
2222-1735 (Paper) ISSN 2222-288X (Online). Vol.7, No.35, 2016
Bandura, A. (2006). Towards a Psychology of Human Agency. Perspectives on
Psychological Science, Stanford University Volume 1—Number 2
Bender, C. (2002). A Life Skills Programme for Learners in the Senior Phase: A Social Work
perspective [Master Thesis, University of Pretoria]
https://repository.up.ac.za/bitstream/handle/2263/29679/00front.pdf?sequence=1&isAll
owed=y
135
Heckman, J. J., & Kautz, T. (2013). Fostering and measuring skills: Interventions that
improve character and cognition (Working paper no. 19656). National Bureau of
Economic Research.
Kember, D., McKay, J., Sinclair, K., and Wong, F. K. Y. (2008). A Four-category Scheme
for Coding and Assessing the Level of Reflection in Written Work. Assessment and
Evaluation in Higher Education, 33(4), 369–378.
Klemenčič, M. (2015). What is student agency? An ontological exploration in the context of
research on student engagement. PRIMOŽIČ, R. (eds.) (in press). Student engagement in
Europe: society, higher education and student governance (pp. 11-29). Council of Europe
Higher Education Series No. 20. Strasbourg:
Mkonongwa, L. (2018). Competency-based teaching and learning approach towards quality
education. Dar es salaam University College of Education (DUCE).
https://tenmet.org/wp-content/uploads/2018/12/Competency-based-teaching-and-
learning-approach-towards-quality-education.pdf
Moon, J. (2005). A Handbook of Reflective and Experiential Learning: Theory and Practice.
Routledge
Murphy-Graham, E., and Cohen, A. (2022). Life Skills Education for Youth in Developing
Countries: What Are They and Why Do They Matter? In Erin, J. & Graham, M. Life Skills
Education for Youth (p.13-42). Springer.
Powell, W. (2004). Count Me in. Developing Inclusive International Schools. Overseas
Schools Advisory Council Department of State Washington, D.C. 20522-0132 Fourth
Printing.
Revans, R.W. The ABC of Action Learning. Bromley-Kent: Chartwell-Bratt, 1983. 84 p.
ISBN 0862380448.
Savage, L. (1993). Literacy through a competency-based education approach. Washington
DC: Center for Applied Linguistics.
Schmidt, D. (2022). Concluding Thoughts on Life Skills Education for Youth. In Erin, J. &
Graham, M. Life Skills Education for Youth (p. 267- 276). Springer.
Serrat, O. (2017). Action Learning. In book: Knowledge Solutions. LicenseCC BY-NC 3.0
Project: Knowledge Solutions: Tools, Methods, and Approaches to Drive Organizational
Performance. https://www.researchgate.net/project/Knowledge-Solutions-Tools-
Methods-and-Approaches-to-Drive-Organizational-Performance
UNICEF (2003). Life Skills: Definition of terms. Available:
www.unicef.org/lifeskills/index_7308.html
Welskop, W. ( 2013). Action learning in education. 10.13140/RG.2.1.4513.8006. In book:
Evropské Pedagogické Fórum 2012, Pedagogical and psychological aspects of education,
vol. II
Wiggins, G., & McTighe, J. (2005). Understanding by Design. Alexandria: ASCD.
136
Information skills instruction in higher education students using

the 4C/ID model
Consuelo García1, Esther Argelagós2, Jesús Privado3
1
Facultad de Educación, Universidad Internacional de Valencia, Spain, 2Facultad de
Educación, Universidad Internacional de La Rioja, Spain, 3Facultad de Psicología,
Universidad Complutense de Madrid, Spain.
Abstract
Information skills instruction continues to be a pending subject among
university students despite the fact that they handle the Internet with ease, since
the ability to define what is needed, to be strategic when deciding where to
search and how, to select the information most appropriate and relevant and
to use it properly is not routinely taught. The purpose of this article is to show
how the 4C/ID model is applied to the design of this skill, the challenges
encountered in its adaptation and the improvement in self-perceived efficacy
by students. The results show that the model is successful and the students felt
more prepared to face this type of tasks in the future.
Keywords: 4C/ID model; information skills instruction; online; higher
education.

Information skills instruction in higher education students using the 4C/ID model
1. Introduction
University students regularly use the Internet to find information for their academic work.
Traditional libraries have been largely replaced by massive scientific databases and academic
search engines to provide the information that students require in their work at the University.
However, many students lack the ability to find the information they need and use it
competently in their academic texts (Zhou & Lam, 2019). Although many students claim to
be digital natives, using the Internet for academic purposes is not the same as using it for
leisure purposes (Hinchliffe et al., 2018). This deficiency in their informational skills implies
a great effort for their teachers, who must invest a lot of time and resources in training the
students (Lanning & Mallek, 2017). And although this training has traditionally fallen on
librarians and information management specialists, the information skill required is
increasingly specific (Taylor, 2012) and less generic, so many professionals from other
academic fields (engineering, medicine, law) are nowadays responsible for working on this
skill with their students.
This complex cognitive skill is often referred to as information literacy (IL) or information
problem solving (IPS) and has been widely explained by different theoretical perspectives
and approaches (ACRL, 2016; Brand-Gruwel et al., 2005; Brand-Gruwel et al., 2009; Catts
& Lau, 2008). IL involves much more than just searching the Internet. It is a complex process
that always points to a specific goal (Garcia & Badia, 2017). It begins with the ability to
clearly identify what you want to search for (the definition phase), the selection of the best
ways to search for information (which tools to use, which search engines, databases), the
strategic application of searches (use of Booleans, keywords, key authors), the selection and
organization of the information from the results (what to choose, for what purpose) obtained
and the integration of that information in a coherent and orderly manner with the personal
texts of the students. It is a more complex process than it seems at first sight and it cannot be
resolved simply by browsing the Internet or using social networks (Argelagós & Pifarré,
2016). To carry out this process properly, instruction and a critical attitude are needed
(Frerejean et al., 2019).
The 4C/ID model is an instructional approach whose primary objective is to help to teach
complex skills or professional competencies (Van Merriënboer et al., 2002). It is a task-
centred instructional model (Francom, 2016) that aims to avoid the fragmentation of
knowledge in favour of a flexible and interconnected integration of knowledge, skills and
attitudes (Van Merriënboer & Kirschner, 2018). It is based on four main components: (1)
learning tasks, (2) supportive information, (3) procedural information, and (4) part-task
practice.
1. Learning tasks are the backbone of the educational program, as, for instance, cases,
projects, professional tasks, problems, or assignments that students work on.
138
Consuelo García, Esther Argelagós, Jesús Privado
Learning tasks encourage inductive learning: students learn by doing and

confronting particular experiences. The learning tasks in the 4C/ID model must meet
a series of characteristics: their variability, that is, they must be different so that they
represent all the dimensions that are found in the professional environment, and also
the level of complexity of the tasks must be different and progressive, like the
support received from the instructor, which should be decreased gradually by the
process known as ‘scaffolding”. Finally, upon completion of the last tasks, the
student should be able to perform them correctly on his own.
2. The second component of the model is the supportive information (sometimes called
“theoretical information”), which helps students to perform the non-routine aspects
of learning tasks, which often require problem solving, reasoning, and/or decision
making. Supporting information provides the link between what students already
know (i.e. prior knowledge) and what they need to know in order to perform the
learning tasks.
3. The procedural information helps students to execute routine aspects of the learning
tasks, that is, those aspects that are always performed the same way. This
information must be presented the first time the student performs that aspect as part
of a learning task. For subsequent tasks, the presentation of the procedural
information has to be gradually removed since its need decreases as the student
gradually masters the routine.
4. Part-task practice aims at strengthening cognitive rules through extensive repetitive
practice. It is important to begin the practice of a part of the task within a fruitful
cognitive context, that is, after students have faced the routine aspect as part of a
complete and meaningful learning task. Then, students will understand that practice
can help them improve their performance in all tasks.
The objective of this article is to show the adaptation process of the 4C/ID model for
information skills instruction of university students in a completely online environment. The
reason for selecting the 4C/ID model to train this skill is precisely that it is a complex
cognitive skill, which is usually taught following classical instructional methods or its design
focuses only on part of the skill, on the search for information (Koufogiannakis & Wiebe,
2006). As well, we wanted to find out if students felt more prepared to this type of task after
completing a 4C/ID course on information skills.
2. Methodology
2.1. Context of the study

The context of the study was the Faculty of Education of a Spanish university whose
academic model is completely online. Both 4-year degrees and one-year university master's
139
degrees are studied at this Faculty. At the end of their academic programme, students have
to present a final study manuscript (between 40 and 80 pages) that always incorporates a
theoretical framework and that requires the application of information skills by the students.
The students who were going to be part of the voluntary instructional process belonged to the
degree in early childhood education and in primary education, the master's degree in special
education, in neuropsychology and education, and in psych pedagogy.
2.2. Design process

Since students belonged to a virtual university, it was decided to generate an instruction that
combined synchrony (adobe connect) and asynchrony (pdf, videos and mind maps) tools. So,
for the instruction, we decided to land the 4 components of the 4C/ID model (see figue 1):
Figure 1. Example of synchronous session. Source: own
• Learning Tasks. They had to be relevant for the student's real life and also had to
have different levels of complexity so that the students could feel comfortable and
gain more confidence in themselves. We chose the following topics: (1)
gamification and learning, (2) cyberbullying and childhood, (3) metacognitive
strategies in primary education, (4) cooperative learning and (5) personal choice of
each student.
• Supportive information. A pdf content was elaborated to explain the student the
main components of the instructional model, which help them to understand and
follow step by step the complete process to write the contents needed for a
theoretical framework of an academic text, based on the results obtained.
• Procedural information. Modelling examples video recorded were key to show
students how to proceed to apply the instruction by themselves. Having an expert
140
doing and reflecting on what to do, even incorporating mistakes and failures was
fundamental to include the routine aspects of the information process.
• Part-task practice. This was approached in two ways: firstly, the teachers taught in
synchronous classes how each part of the information process was applied, and
secondly, the students practiced that part of the process autonomously but with the
support of the written and audio-visual material. In such a way that at the beginning
the students received a lot of guidance from their teachers, but in the end, the task
had to be carried out completely autonomously.
2.3. Course instruction

In order to carry out the complete instructional process, it was decided to structure the course
contents, the modelling examples and the synchronous classes according to the step-by-step
model (see figure 2) designed by Brand-Gruwel et al., (2005).
Figure 2. Descriptive model for information problem solving. Source: Brand-Gruwel et al., (2005).
First, the ability to define research questions was deployed by the teachers, explaining all the
steps involved in the development of this ability: topic analysis, activation of prior
knowledge, how to specify a topic and how to establish research questions. Secondly, the
focus was on the selection of academic sources and search strategies, how to carry them out
through google scholar and the databases of the university, how to use the thesaurus,
keywords and Booleans within search engines or metasearch engines and how to carry out
an initial evaluation of the results based on a checklist to control the reliability, veracity and
timeliness of the source. In the third place, the analysis of the results, the selection and the
summary of the publications found for its later consultation was worked. And, eventually,
the teachers showed how to analyse and extract the relevant information from the relevant
sources and how to integrate the information. Subsequently, the last section of contents was
oriented to how to organize and present the information, plan the text and the fundamentals
to paraphrase, quote and correctly incorporate the information found in the content itself.
At the beginning of the course, the students received the following action guideline (which
was repeated in each phase of the instructional process):
1. Read the .pdf that explains the step to be carried out.
141
2. Watch the video (as many times as necessary) about the step to be taken.
3. Run the task (see the video while it is being done).
4. Reflect on the results.
5. Share doubts with the teacher in the Connect session
6. Attend the next class.
7. Perform the following task.
8. Start with step 1 again.
The resolution of the task allows to reflect on the results and problems encountered. During
the connect session with the teacher, you will see the results that you should have obtained
and the teachers will explain in detail the steps that should have followed, and they will
resolve the doubts that you may have in relation to the phase worked on. To begin with,
remember to watch the first video that explains all the steps to carry out the search and use
of the information needed to write a theoretical framework.
In task 1 the complete instruction of the process was shown and shaped by the teachers. In
tasks 2 and 3 the part-task practice was applied so that the students internalized the routines
supported by the procedural information. In task 4 the students carried out the activity
autonomously with prompts and occasional help from the teachers. Finally, the students
performed the last task without help. The course lasted two months, the first four tasks were
developed during 3 weeks, and the last task (autonomous) was done one month after the
instruction.
2.4. Participants
The call for the voluntary course and subsequent research was made to 120 students, and 80
(28.8 % female, 36.86 average age) accepted to participate. Finally, 25 completed the entire
course and 55 did not. The Self-Efficacy Scale for Information Searching Behaviour (SES-
IB-16; Rosman et al., 2015) is a questionnaire to measure self-reported information literacy.
It was used to measure students’ assessment about their possibilities of successfully
approaching the task of preparing the theoretical framework of their final project. It was filled
out by the students before and after completing the course.
2.5. Data anaylsis

Improvement in students' self-perceived efficacy was analysed by means of a mixed analysis
of variance (ANOVA) using group (experimental and control) as an intergroup factor, and
the pre-test and post-test results of the SES-IB-16 as an intragroup factor, applying SPSS
v18.
142
3. Results and conclusions

The application of 4C/ID model to information skills instruction was successful. On the one
hand, the main challenges we had to address in the adaptation were of a different nature. The
first was to start from a global vision of the entire training process from an expert perspective,
in order to break it down into its parts. The conceptual framework of the IPS skills by Brand-
Gruwel et al., (2005) was of great help, when comparing the skill breakdown of the model
with the steps carried out by the expert. Secondly, we had to decide what each task would
consist of, not only the different content but also their different level of complexity. And how
to scaffold during the instruction period was another of the issues we faced, since the
objective was for students to internalize the most routine mechanisms to address more
complex aspects, as they progressed in their training. Another challenge was deciding the
timing of the classes to give students time to complete the parts of the tasks, attend classes,
share the results and reflect on their mistakes and successes. We made the decision to leave
a long period of time (one month) between the instruction and the delivery of the last task
(free theme developed autonomously) so that the students had time to reflect on everything
they practiced and learned.
On the other hand, from the teachers’ point of view, the process was enriching when
observing how the students progressively not only understood what they had to do to solve
the tasks, but also became more and more autonomous in the process of defining, searching,
selecting and elaborating the information for the theoretical frameworks of their academic
thesis. It was a little more difficult to personally manage each student during the synchronous
classes, but the decision to expose the "best" results and share the most common mistakes
was very helpful for the students.
Finally, on the part of the students, the feeling of self-efficacy of the group that took the
course was significantly higher than the group that did not take it. In the case of the self-
efficacy expectations measured by SES-IB-16, we observe that there is an intra-subject effect
[F1,78 = 25.27, p < .001, η2partial = .245] and interaction between this factor and the group [F1,78
= 16.08, p < .001, η2partial = .171] with a high effect size, so the training was considered very
effective. So, we consider the task centred approach of the 4C/ID model, based on an
integration of knowledge, skills and attitudes (Van Merriënboer & Kirschner, 2018) helpful
for information skills instruction and to increase students’s sense of future success concerning
their academic challenge at the end of their university studies.
References
ACRL - Association of College and Research Libraries (2016). Framework for Information
Literacy for Higher Education. American Library Association,
143
Argelagós, E., & Pifarré, M. (2016). Key Information-Problem Solving Skills to Learn in
Secondary Education: A Qualitative, Multi-Case Study. Journal of Education and
Learning, 5(4), 1-14.
Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by
experts and novices: Analysis of a complex cognitive skill. Computers in Human
Behavior, 21(3), 487-508.
Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of information
problem solving while using internet. Computers & Education, 53(4), 1207-1217.
Catts, R. & Lau, J. (2008) Towards information literacy indicators. UNESCO. Available at:
hdl.handle.net/1893/2119
Francom, G. M. (2016). Principles for task-centered instruction. In Instructional-Design
Theories and Models, Volume IV (pp. 81-108). Routledge.
Frerejean, J., Velthorst, G. J., van Strien, J. L., Kirschner, P. A., & Brand-Gruwel, S. (2019).
Embedded instruction to learn information problem solving: Effects of a whole task
approach. Computers in Human Behavior, 90, 117-130.
Garcia, C., & Badia, A. (2017). Information problem‐solving skills in small virtual groups
and learning outcomes. Journal of Computer Assisted Learning, 33(4), 382-392.
Hinchliffe, L. J., Rand, A., & Collier, J. (2018). Predictable information literacy
misconceptions of first-year college students. Communications in Information
Literacy, 12(1), 4-18.
Koufogiannakis, D., & Wiebe, N. (2006). Effective methods for teaching information literacy
skills to undergraduate students: A systematic review and meta-analysis. Library and
Information Science: Parameters and Perspectives, 3(3), 3-43.
Lanning, S., & Mallek, J. (2017). Factors influencing information literacy competency of
college students. The Journal of Academic Librarianship, 43(5), 443-450.
Rosman, T., Mayer, A.-K., & Krampen, G. (2015). Combining self-assessments and
achievement tests in information literacy assessment: Empirical results and
recommendations for practice. Assessment & Evaluation in Higher Education, 40(5),
740–754.
Taylor, A. (2012). A study of the information search behaviour of the millennial
generation. Information research: an international electronic journal 17(1), 1-20.
Van Merriënboer, J. J., Clark, R. E., & De Croock, M. B. (2002). Blueprints for complex
learning: The 4C/ID-model. Educational technology research and development, 50(2),
39-61.
Van Merriënboer, J. J., & Kirschner, P. A. (2018). 4C/ID in the context of instructional design
and the learning sciences. In International handbook of the learning sciences (pp. 169-
179). Routledge.
Zhou, M., & Lam, K. K. L. (2019). Metacognitive scaffolding for online information search
in K-12 and higher education settings: a systematic review. Educational Technology
Research and Development, 67(6), 1353-1384.
144
A systematic review: foundations for online course design in

higher education
University of Puerto Rico, Bayamón, Puerto Rico.
Abstract
The purpose of this study was to select specific best practice indicators for
improving online course design based on the recent research findings shared
during and post COVID-19 pandemic. PRISMA guidelines were used during
the search process to identify qualifying studies. EbscoHost, Academic Search
Ultimate, Educational Research Information Center (ERIC), Google Scholar
and JSTOR were the six databases searched for selecting articles that met the
following criteria: a) written in Spanish or English, b) peer-reviewed studies
and indexed journals c) higher education level and d) online courses design
best practices. The database searches identified a total of fifteen relevant
studies. The findings of this study demonstrate four core indicators of online
course design: planning course alignment, using templates for framework
design, developing articulated connections and continuous cyclical review.
These essential indicators are critical for online course planning as they have
shown to be vital to student achievement, retention, and satisfaction.
Keywords: Online course; course design; best practices; higher education.

A systematic review: foundations for online course design in higher education
1. Introduction
Trends in higher education suggest that distance learning is the most rapidly growing sector,
in great part due to the COVID-19 pandemic. For this reason, online course design is a topic
of growing importance since inefficient online course design has been shown to result in poor
student achievement, low retention rates, and dissatisfied students (McMahon, 2021, Joosten
& Cusatis, 2019). While online course models are not a new phenomenon (Abdous, 2020),
the growth of online teaching experience over the past three years has broadened the amount
of research regarding effective online courses and positive student outcomes around the
world; therefore, creating a need for an updated review of the literature.
Distance education refers to learning where the participants are not required to be in the same
location but are dependent on either technology to engage in educational activities
synchronously or asynchronously (Traboulsi, 2008). Online course design is fundamental to
student success as it is the framework that connects the student at a distance to course
materials, recorded lectures, assignments, assessments, and collaboration with the instructor
and classmates (Fischer, et al., 2022). For this reason, educators creating online courses are
tasked to approach online course design from a different paradigm than traditional pedagogy.
The purpose of this study was to select specific best practice indicators for improving online
course design based on the recent research findings shared during and post-pandemic.
2. Methodology
PRISMA guidelines include a 27-item checklist to facilitate the clarity of systematic reviews
(UNC University Libraries, 2020). Following the PRISMA guidelines, six databases were
selected in this study. These databases were chosen because of their international scope and
alignment with topics in education and technology to answer the research question of what
current best practices in online course design are as identified during and after the pandemic.
Different keywords with combinations including higher education and online courses, best
practices, and distance learning were used during the search process.
Articles were selected using the following criteria: a) written in Spanish or English, b) peer-
reviewed studies and indexed journals c) higher education level and d) online courses design
best practices. The initial search yielded 928 matches. Through the identification process 274
studies were duplicated and therefore eliminated, 539 studies were ineligible due to not
meeting eligibility criteria, 67 studies were removed for other reasons such as not being a
research article. This resulted in 28 studies that were screened for eligibility, resulting in a
total of 15 international studies that met the selection criteria. Articles were initially
categorized according to the following dimensions: country of origin, purpose of the study,
data collection methodology and key findings. Studies referenced in the journal articles used
qualitative, quantitative, or mixed methods research methodology. This international sample
146
helps to minimize bias, as the results are not likely to be attributed to a specific geographic
location. The applied searches and results from each database are shown in Figure 1.
3. Results
Participating studies included asynchronous, synchronous learning and/or hybrid model
courses. Journal articles were chosen internationally and included Spain, Malaysia, Taiwan,
India, USA, Turkey, Jordan, and Sri Lanka. Eligible studies focused on the challenges with
transitioning from face to face learning to distance education, online course design
characteristics, online course instructional practices, student and faculty experiences and
perceptions of the online experience. The results revealed that effective online course design
frameworks are the principle requisite for positive student learning outcomes, which also
encouraged improved concentration and independence.
An emergent theme from this study has been the articulation that the shift to online learning
in higher education has been uncertain due to the lack of adequate design, planning &
development (Adedoyin & Soykan, 2020; Oliviera, et al. 2021). These deficiencies were
attributed to a lack of clear institutional policies that clearly address their vision of the online
classroom (Haider & Al-Salman, 2020). Without a structure to follow, courses within the
same institution could be presented in a variety of organization patterns leading to confusion
among students. Student frustration with course design was correlated with poor learning
outcomes, and perceived lack of course quality, both of which lead to the reduction in course
effectiveness (Nambia, 2020; Bradley & Vigma, 2016).
Online course design is the organization of course content so that it aligns with student
learning outcomes resulting in effective learning. (Ceballos, 2018; McMahon, 2021; Sanga,
2019, Sandanayake, et al., 2021; Tamilarasan, et al., 2021). The contribution of this study is
to inform institutional policies, positively affecting future online course creation and
providing clearer guidelines for course creators to follow.
The first essential indicator in best practices for online course design was planning. Planning
begins with a needs analysis to choose a delivery model (synchronous, asynchronous, or
blended) suited to the content of the course & to the audience. Once established, the
determination of learning objectives can take place, which are then aligned with assessments
(McMahon, 2021). Objectives are defined to mark specific learning goals to be met as the
course progresses; course articulation refers to the transparency regarding the relationship
these individual objectives have among them and how they contribute to achievement of the
learning outcomes expressed at the onset of the course. LaForge (2022) found that improving
course organization, transparency, objective-assignment alignment and developing instructor
feedback opportunities, resulted in improved student retention, better learning outcomes and
higher student satisfaction. Providing clearly stated learning objectives and their
147
interrelationship, facilitates students' understanding of what they should be learning is often

overlooked.
The lack of articulated course was another emergent theme uncovered within the literature.
The benefits of course articulation is that it results in manageable targets for students to
achieve (Hsu, et al., 2017). Nave (2020) found that student effort to persevere is aided by
dividing longer-term goals into short-term objectives. When class content is presented in
logical chunks of information, it promotes greater student concentration (Bao, 2020). Using
themes, lessons or modules presents learning outcomes by segmenting or chunking course
content, which aids in the next foundational indicator: course framework design.
Course framework design has been defined as the organization and presentation of course
materials. This type of organization requires the course structure to be clear and consistent
(Fischer, et al. 2022). At the institutional level, having a model further supports students'
abilities to navigate in different courses due to the use of a uniform model (Garcia-Penalvo,
2021, p.12; Oliveira, et al., 2021). Consistency can be implemented using a standard
framework in the form of module templates (Ralston-Berg & Braatz, 2021). Module
templates would support lesson structure transparency by having a clear and logical sequence
that is student friendly, consistent and intuitive for each module that is repeated throughout
the course. For example, each module would begin with the presentation of the objective and
how it fits within the course objectives, followed by required content materials, videos, texts,
practice activities, assignments, and end with a module evaluation. Intuitive design supports
the learner by predicting potential problems or needs a learner may encounter and designing
solutions to minimize these problems. In a study conducted by Fischer, et al. (2022), they
found course transparency in organization had significant positive effects on students’
independence, which is a required skill for student success in online courses.
Once planning and a design framework are in place, instructors should articulate content
connections to facilitate students' recognition of the importance of that unit or module in
meeting course objectives (Martin, et al, 2019). The connections between objectives and
learning outcomes should be explicitly expressed to students as a means of facilitating the
big picture image that students are creating as they progress (Sandanayake, et al, 2021;
Kharagpur, 2016). Nave (2020) found that students were more engaged when they
understood why they were learning the content, understanding its relevance in real-life
contexts. This was a motivator of efficient learning which reduced course abandonment rates.
Development of methods & materials will direct the student from the course objectives to
learning necessary content to achieve learning outcomes. Learner-content interaction,
learner-learner interaction and learner-teacher interactions are three areas of interaction to be
considered when developing course materials (Moore, 1989). Course materials can be in the
form of videos [screencasts & lecture recordings], audios, texts, animations, drawings &
graphics and more as new platforms and interactive technologies are being developed to
148
support interaction with the content, instructor, and classmates. In distance education,
environments ideally there would be a balance among tasks that are done independently and
those that encourage collaborative learning for greater student interaction and persistence in
the course (Bilgiç, et al., 2020; García-Peñalvo, 2021).
Formative assessments can support students' self-reflection by acting as checkpoints for the
students' understanding of their progress. In the study conducted by Holzmann-Littig, et al.
(2023), students expressed that assessments motivated them to monitor their understanding
and maintain focus on the material. In terms of summative evaluations, rubrics provided
ahead of time add clarity of the instructors’ expectations. Additionally, Lorente-Ruiz, et al.
(2021) recommend identifying standard feedback strategies to increase student-teacher
communication that may increase participation. Students that receive frequent and specific
feedback tend to stay connected to the course (Nave, 2020). Communication, despite
distance, is facilitated when communication channels are explicitly expressed for students to
initiate contact with the instructor (LaForge, 2022; Hsu, Baldwin, & Ching, 2017).
Course review in its entirety is the final indicator found in the research for best practices in
online course design. During this stage, best practice includes identifying opportunities for
student feedback, guides for self-reflection and evaluation of the course after completion
(Nave, 2020). McMahon (2021) recommends creating a framework for self-assessment as
well as peer review processes to continue refining course quality. Once student and instructor
feedback have been reviewed, the instructor would return to the planning stage to enhance
identified areas of improvement. Ralston-Berg & Bratz (2021) recommend refining design
and course structure after each completion of the course. Prior experience has been the
greatest predictor of positive online teaching experience, thus supporting the importance of
a cycle of continuous improvement and growth (Marek, Chu & Wu, 2021). This final
indicator is crucial to the instructor’s professional development in online course design.
Ongoing evaluation and reflection could also inform institutional policies regarding the
production of quality online courses.
4. Conclusion
This systematic literature review has identified four essential indicators to improve online
course design, rooted in recent research of best practices in distance education. The following
indicators are (1) to plan aligned course objectives, (2) use module templates to facilitate
consistency and transparency within the design framework, (3) develop articulated
connections within the course to share with students regarding how the material is connected
within the course and its relevance to their future and (4) conduct continuous cyclical reviews
of the course after each completion to include feedback from the instructor and students. By
including these indicators, students are more likely to navigate efficiently for effective
149
learning, participate in activities, and successfully complete the course thereby improving
student retention rates. However, if not included in online course design potential
consequences are lack of achievement of learning outcomes, disengagement, and course
desertion. Institutions and educators can use these findings to inform formal policies
regarding online course design. Based on each institution's particular needs, resulting policies
could in turn provide direction for relevant professional development topics for online course
instructors. While the focus of this study was on higher education online course design,
further research may consider differences in public versus private institutions. An online
course design model is crucial to student success, and by incorporating these fundamental
indicators, instructors are better equipped to support student achievement at a distance. This
study provides evidence to support standard indicators to be included in online course design
models and in decision-making relating to online course policies.
Figure 1. Details of selection of studies for review.
150
References
Adedoyin, O. B., & Soykan, E. (2020). Covid-19 pandemic and online learning: The
challenges and opportunities. Interactive Learning Environments. Advance online
publication. https://doi.org/10.1080/10494820.2020.1813180
Abdous, M. H. (2020). Designing online courses as a team: a team-based approach model.
International Journal of Online Pedagogy and Course Design (IJOPCD), 10(1), 61-73.
Bilgiç, H. G. & Tuzun, H. (2020). Issues and Challenges with Web-Based Distance
Education Programs in Turkish Higher Education Institutes. Turkish Online Journal of
Distance Education, 21(1), 143-164. DOI: 10.17718/tojde.690385
Ceballos Van Grieken, Angel (2018). Diseño y Creación de un Curso en Línea Sobre
Licencias Creative Commons. Revista de Tecnología de Información y Comunicación en
Educación, Volumen 12, No. 1, Enero–junio 2018. https://tinyurl.com/mwwn4sh2
Haider, A. S., & Al-Salman, S. (2020). COVID-19’S impact on the higher education system
in Jordan: advantages, challenges, and suggestions. Humanities & Social Sciences
Reviews, 8(4), 1418-1428.
Holzmann-Littig, C.; Jedlicska, N.; Wijnen-Meijer, M.; Liesche-Starnecker, F.; Schmidt-
Bäse, K.; Renders, L.; Weimann, K.; Konukiewitz, B.; Schlegel, J. Design and Transition
of an Emergency E-Learning Pathology Course for Medical Students—Evaluation of a
Novel Course Concept. Eur. J. Investig. Health Psychol. Educ. 2023, 13, 112–129.
https://doi.org/10.3390/ ejihpe13010008.
Hsu, YC., Baldwin, S. & Ching, YH. Learning through Making and Maker Education.
TechTrends 61, 589–594 (2017). https://doi.org/10.1007/s11528-017-0172-6
Joosten, T., & Cusatis, R. (2019). A cross-institutional study of instructional characteristics
and student outcomes: Are quality indicators of online courses able to predict student
success? Online Learning, 23(4), 354–378. https://doi.org/10.24059/olj.v23i4.1432
Kharagpur, K. (2016). Basic concepts in user interface design. https://tinyurl.com/2y6nzw9k
LaForge, C. (2022). Confessions of a (Former) Killer Course Using the QM Review Process
to Improve an Undergraduate Research Course. The quarterly Review of Distance
Education, Volume 23(1), pps. 23-39.
Nave, L. (2020). Universal Design for Learning: UDL in Online Environments: The WHY
of Learning. Journal of Developmental Education, Vol. 44, No. 1, pp. 30-31. Reich
College of Education, Appalachian State University.
https://www.jstor.org/stable/45381097.
Marek, M., Chiou, S., Wen-Chi, V. (2021). Teacher Experiences in Converting Classes to
Distance Learning in the COVID-19 Pandemic. International Journal of Distance
Education Technologies Volume 19, Issue 1. DOI: 10.4018/IJDET.20210101.oa3
McMahon, E. (2021). Designing Effective Online Courses: Exploring the Relationships
Amongst Online Teaching Self-efficacy, Professional Development, Online Teaching
Experience, and Reported Implementation of Effective Higher Education Online Course
Design Practices. The Interactive Journal of Global Leadership and Learning, 2(1).
https://doi.org/10.55354/2692-3394.1026.
Oliveira, G., Grenha Teixeira, J., Torres, A., & Morais, C. (2021). An exploratory study on
the emergency remote education experience of higher education students and teachers
151
during the COVID-19 pandemic. British Journal of Educational Technology, 52, 1357–
1376. https://doi.org/10.1111/bjet.13112
Ralston-Berg, P., & Braatz, H. (2021). Online Course Design Structure and Interface. New
Directions for Adult and Continuing Education, 15–33.
https://doi.org/10.1002/ace.20411.
Tamilarasan, P.; Vijayakumar, S.; Anupama, C. (2021). Addie Model of Instructional Design
in an Online Course At Higher Education. Turkish Online Journal of Qualitative Inquiry
(TOJQI) Volume 12, Issue 6, pgs.8394 -8401. UNC University Libraries (2023, April
20). Systematic Reviews: Step 8: Write the Review.
https://guides.lib.unc.edu/systematic-reviews/write#s-lg-box-28834069.
152
E-learning and economics: knowledge dissemination through

social networks
Roberto Ivan Fuentes Contreras, Moises Librado Gonzalez, Yadira Zulith Flores
Anaya
Universidad Autónoma de Baja California, México.
Abstract
Increasing digitalization has led to a shift from learning and knowledge
transfer based on reading texts, books and/or journals, to the use of virtual
platforms. These changes represent future challenges for the acquisition,
appropriation, and dissemination of knowledge. According to the Availability
and Use of Information Technologies in Households National Survey
(ENDUTIH, 2021), in Mexico there are just over 88 million Internet users and
more than 91 million cell phone users. This survey also shows that more than
94% of users use social networks. The objective of this work is to measure the
reach that the publication of infographics with scientific content through social
networks has had. Using an analysis of variance (ANOVA) and controlling for
characteristics of the publications, it was possible to estimate statistical
differences between groups. The results indicate that the classification by
control groups: people reached, and interactions; show differences in means
between groups and years of publication, demonstrating positive results for
the dissemination and acquisition of scientific knowledge through
infographics.
Keywords: Social networks; knowledge dissemination; infographics in
education.

E-learning and economics: knowledge dissemination through social networks
1. Introduction
In recent years changes in the learning process have encouraged many students to
complement their own process in the classroom with digital tools. These learning media
represent an opportunity for students to access knowledge faster and with better results. Much
of the change in the digitization process boomed during the COVID-19 pandemic. According
to the National Survey on Availability and Use of Information Technologies in Households
(ENDUTIH) 2021, in Mexico, just over 55.3% of homes with young people had internet
access, representing more than 11.8 million people. The same survey found that more than
24 million young people searched the internet for information on education, research, and
homework. Regarding internet use, just over three million people between 25 and 29 years
of age use this medium for training. Among the most used social networks are: WhatsApp,
Facebook, Instagram, Messenger, and YouTube.
The digitization process has given rise to new means of disseminating and spreading
knowledge. Significant changes have been observed in students, teachers, researchers and all
Internet users. However, a positive acceptance has been found in infographics, whose main
feature is to summarize extensive information in a template that can be consumed through
social networks. Infographics are considered part of information visualization. It is a field of
research that focuses on patterns and trends in abstract data sets and can help design content
with maximum effect (Naparin and Saad, 2017). In this sense, infographics become an
attractive medium to disseminate knowledge, becoming an ideal method of dissemination
and knowledge transfer for all audiences. This is the reason why infographics have greater
acceptance when they meet visualization requirements, such as colors, images and text.
In the understanding that didactic and fast learning strengthens soft and hard skills in
university students. The development and dissemination of infographics can reduce
knowledge gaps in adults and young users of social networks. The dissemination of
knowledge through digital media can contribute to the Sustainable Development Goals of the
United Nations 2030 Agenda, whose goal is to "leave no one behind"; and in goal 4, quality
education, efforts are sought to ensure inclusive, equitable and quality education that
provides opportunities for all.
With this philosophy of bringing knowledge to all, in 2019, through a call for professional
social service, a research project on Financial Inclusion for Development is created, this
initiative consists of participatory action research to promote financial education. The project
in addition to having a research agenda, consists of a professional social service program
where students put into practice their communication skills, design and dissemination of
science through infographics in a project called "Econographic". With the intention of
analyzing the reach and reactions that the infographics published during the period 2019-
154
Roberto Ivan Fuentes Contreras, Moises Librado Gonzalez, Yadira Zulith Flores Anaya
2022 have had. This document aims to measure the effect that the publication of infographics
with scientific content has had through the social networks of the "Econographic" project.
2. New ways of learning

Studies show that the use of modern technologies in the classroom offers them the
opportunity to learn faster, with better results and with greater satisfaction. The appeal of
infographics seems to be inherent in their nature, as users are attracted to visualization, colors
and images.An infographic can transfer knowledge about a subject quickly and massively,
however, this condition depends on the quality and presentation of the infographic.
The use of this medium to transfer knowledge has become increasingly popular among the
academic community, however, the speed of learning depends on multiple factors. The
process of acceptance, assimilation, appropriation and absorption of knowledge is
heterogeneous in each individual. Returning to the theoretical contributions on learning, from
Piaget's cognitive theory (1969), also known as evolutionary, learning focuses on a gradual
and progressive process, i.e., it is directly related to the individual's biological process and
the environment in which he/she develops, however, much of learning depends on adaptation.
On the other hand, from the approach of Watson's behaviorism (1913), the learning process
is obtained from behavior, in this sense, the observable and measurable behavior of an
individual is analyzed until it becomes repetitive. In this sense, the individual can, through
adaptation, generate his own learning. This is undoubtedly closely related to the theoretical
approach of cognitivism or Information Processing Theory, where one of its authors Neisser
(1967) states that the observation of new patterns of behavior leads to the construction of
new learning. This learning includes how information is interpreted, processed and stored in
memory. Furthermore, this approach is based on the philosophy that the human differential
processes information through symbols, logical rules and external reality that is perceived
and processed by individuals.
In addition, as mentioned by Stoyanova and Kommers (2002), the cognitivist approach states
that information is stored in memory through a pattern of nodes that creates a network, in
which the nodes are connected to each other through relationships. This process is nothing
but an interconnected learning process, in this sense, recently e-learning developers adopt the
constructivist approach, where knowledge is interpreted through individual perceptions
(Pange and Pange, 2011).
However, the learning process has been adjusting to the needs of users. Much of this new
approach alludes to the intensive use of the Internet. The concept of e-learning, beyond. One
of the main approaches that enable a teaching and learning process is Active Learning.
According to Bonwell and Eison (1991), it consists of an educational strategy that involves
155
students in the learning process. In this sense, this method requires the active participation of
individuals in the construction of learning, i.e., learning by doing.
However, the learning process has been adjusting to the needs of users. Much of this new
approach alludes to the intensive use of the Internet. The concept of e-learning, beyond
focusing on the individual's learning process, is focused on the means used to achieve that
learning. This approach is based on the intensive use of information and communication
technologies (ICT) to disseminate knowledge. In addition, learning design using digital
media, such as the use of infographics, can encompass the theoretical approaches mentioned
above.
3. Transferring knowledge through "Econographic"

Infographics assist people to make informed and quick decisions about a specific topic. Using
the constructivist approach and active learning, where university students can build and
disseminate knowledge, in 2019, a project was created to produce and disseminate
infographics on economic, social, political, and cultural topics regarding the Mexican
context. "Econographic" is a project devised by students and supervised by teachers of
Faculty of Economics and International Relations of the Autonomous University of Baja
California.
The main objective of this project is to disseminate scientific knowledge concerning social
sciences. The process of knowledge transfer is massive and fast. The project uses social
networks to publish infographics, the platforms used are Facebook, Twitter, and Instagram.
Regarding the reach of the infographics depends on multiple factors, such as the type of social
network where it is published, the time of publication of the infographic, the number of
followers, and the quality and visualization of the infographic.
In theory, an infographic is considered a means of visualizing information. It is a medium
used by research to connect patterns and trends in abstract data sets and can help design
templates for greater effect (Naparin and Saad, 2017). An infographic is considered a
collection of information (Krauss, 2012), this data is interconnected into a representation or
a set of ideas (Polman and Gebre, 2015). In this sense an infographic is integrated by a set of
texts, drawings, paintings, graphics and images (Taspolat, Kaya, Sapanca, Beheshti, and
Ozdamli, 2017). The use given to infographics is diverse, depending on the intended purpose.
The "Econographic" project is based on the 5E instructional model, which is based on the
cognitive principle. In this phase, the student engages in a topic of their own interest,
followed by an exploration stage of the different explanations and relationships that interact
with the topic, then explains a complex process in a synthesis process with which they
156
elaborate an original iconography. Finally, this effort is evaluated directly by the teacher, and
the success of the dissemination objective is measured through the number of people reached.
4. Methodology and results

The methodology used was chosen to analyze the following variables the reach of the
infographics published and to identify whether there are significant differences in the results
when controlling for the characteristics of the publications. For the first objective, a graphical
analysis was performed, and for the second objective, the analysis of variance technique
(ANOVA) was used to estimate whether the group averages were similar or statistically
different.
As a variable to approximate the dissemination of content, two metrics published by the
social network were used: interactions and people reached by each of the publications.
Interactions are understood as the actions that people perform with the publication, which
include reactions, comments, sharing the content, clicking on the photo or link. On the other
hand, the people reached are defined as those who see the publication, either because they
are followers or because a user shared the content. The difference between both metrics lies
in the fact that an interaction requires some type of participation by the users of the social
network to be counted that’s not the case to count a person reached. In the period analyzed,
210 infographics were published, 169,588 people were reached and obtained 14,786
reactions.
The variables of dissemination were grouped into three categories: lexicon, topic, and year
of publication. The lexicon of the publications was subdivided into positive, neutral, or
negative, iconographies with a positive approach are those that address content associated
with entrepreneurship, economic recovery or technological advances, the negative approach
focuses on phenomena related to the loss of welfare or natural disasters and the neutral
approach only presents information without the intention of generating a judgment on it. The
groups and categories were constructed after the publication of the content.
Table 1. Divulgation variables, all categories.
Topic Year Lexic

Economy 76 2019 33 Negative 62
Politics 88 2020 39 Neutral 100
Other 34 2021 103 Positive 48
Entrepreneurship 12 2022 35
n 210 210 210
When comparing the evolution by year, an increase in the number of people reached is
identified, but an opposite trend in interactions. This indicates the relationship between the
frequency of publication and the increase in average reach, even though users tend to interact
157
less with the publications. Graphically, no different average is identified when grouped by
type of lexicon or topic. The outliers observed are found in both criteria of disclosure
classification and all the groups constructed.
Figure 1. People reached and interaction by topic, lexicon and year. Source: Own elaboration.
The null hypothesis (𝐻! ) of the ANOVA is that there is no difference in the means, and the
alternative hypothesis (𝐻" ) is that the means are different from each other. Table 2 shows
that the differences between the means are significant when grouped by year of publication
for both individuals reached and interactions, something that was apparent from the graphical
analysis. When comparing the means of the 6 groups constructed using the criteria topic and
lexicon, statistically significant differences were also identified with 95% confidence.
The ANOVA technique identifies whether there are differences between group means, but
not the magnitude of the differences. A Tukey's Honestly Significant Difference (Tukey's
HSD) post-hoc test was performed to identify the difference between the groups. Table 3
shows that the average number of people reached in 2021 was higher than 2020 and 2019,
and the 2022 average was higher than 2020 and 2019. When performing Tukey's HSD using
158
interactions as the dependent variable and the between the subject and lexical categories with
no significant differences found.
Table 2. Divulgation variables, all categories.
Variables # of Df People Reach Interactions

Groups F value Pr(>F) F Pr(>F)
value
Topic 4 3 1.206 0.309 0.268 0.849
Year 4 3 13.967 3.32e-08* 10.08 3.65e-06*
7
Lexicon 3 2 2.283 0.105 0.609 0.545

Topic and Year 9 8 4.49E-01 0.89 1.026 0.418
Topic and 6 5 1.946 0.0891** 1.526 0.184
Lexicon
Year and 7 6 1.48 0.1874 1.188 0.315
Lexicon
Topic, Year and 8 7 0.396 0.9041 0.754 0.626
Lexicon
Residuals 175
*Significativo al 99%, **Significativo al 95%
Table 3. Tukey’s Honestly Significant Difference.
People Reach Interactions

Compariss diff lwr upr p diff lwr upr p
on
2020-2019 41 -590.08 672.13 0.99 -54.79 -106.43 -3.159 0.03*
2021-2019 1099.6 565.9 1633.3 0.000* -70.90 -114.57 -27.24 0.000
*
2022-2019 710.4 63 1357.8 0.02* -106.8 -159.86 -53.92 0.000
*
2021-2020 1058.6 556.9 1560.3 0.000* -16.11 -57.15 24.93 0.73
2022-2020 669.4 48.1 1290.6 0.029* -52.09 -102.93 -1.26 0.04*
2022-2021 -389.2 -911.2 132.8 0.21 -35.9 -78.70 6.72 0.13
* Statistically significant
Graphical analysis and ANOVA were performed, eliminating the data of persons reached
and interactions above the upper interval defined as third interquartile +(1.5 *interquartile
range), without modifying any of the results.
5. Conclusions
The results show that the "Econografico" social service project has fulfilled its objective of
popularizing science. Students designed and published 210 infographics, reached 169,588
people and obtained 14,786 reactions. Positive and significant differences were identified in
the average number of people reached in 2021 compared to 2019 and 2020. Negative and
159
significant differences were identified in the average number of interactions for 2022, 2021
and 2020 publications compared to 2019. No differences in average person reach or average
interactions per publication were identified if grouped by topic or lexicon.
Future analyses should consider as grouping factors a more specific time element, the profile
of the users reached or some approximation of the context to identify the elements that allow
publications to be more successful in their outreach objective.
References
Bonwell, C. C., & Eison, J. A. (1991). Active learning: Creating excitement in the classroom.
1991 ASHE- ERIC higher education reports. ERIC Clearinghouse on Higher Education,
The George Washington University, One Dupont Circle, Suite 630, Washington, DC
20036-1183
National Survey on the Availability and Use of Information Technologies in Households
(ENDUTIH, 2021). INEGI.
Jaleniauskiene, E., & Kasperiuniene, J. (2022). Infographics in higher education: A scoping
review. E-Learning and Digital Media. Doi: 20427530221107774.
Krauss, J. (2012). More Than Words Can Say Infographics. Learning & Leading with
Technology, 10- 14.
Naparin, H., & Saad, A. B. (2017). Infographics in education: Review on infographics design.
The International Journal of Multimedia & Its Applications (IJMA), 9(4), 5.
Neisser, U. (2014). Cognitive psychology: Classic edition. Psychology press.
Pange, A., & Pange, J. (2011). Is e-learning based on learning theories? A literature review.
International Journal of Educational and Pedagogical Sciences, 5(8), 932-936.
Piaget J. (1969) Psychology and Pedagogy. Barcelona: Ariel.
Polman, J. L., & Gebre, E. H. (2015). Towards Critical Appraisal of Infographics as
Scientific Inscriptions. Journal of Research in Science Teaching, 52 (6), 868-893.
http://dx.doi.org/10.1002/tea.21225
Ruiz-Martín, H., & Bybee, R. W. (2022). The cognitive principles of learning underlying the
5E Model of Instruction. International Journal of STEM Education, 9(1), 21.
Stoyanova, N., & Kommers, P. (2002). Concept mapping as a medium of shared cognition
in computer-supported collaborative problem solving. Journal of interactive learning
research, 13(1), 111-133.
Taspolat, A., Kaya, O. S., Sapanca, H. F., Beheshti, M., & Ozdamli, F. (2017). An
Investigation Toward Advantages, Design Principles and Steps on Infographics in
Education. International Journal of Sciences and Research, 73 (7), 157-166.
Watson, J. B. (1913). Psychology as the behaviorist views it. Psychological Review, 20, 158-
177.
160
The digital platform for the Unite! Alliance: The Metacampus

Department of Network Engineering, Universitat Politècnica de Catalunya, Spain.
Abstract
Nine European technology universities constitute the Unite! alliance, a
network that creates a new model for a virtual and physical European inter-
university campus through close cooperation, physical and virtual mobility of
members, joint programmes, communities that support teaching innovation,
and open and entrepreneurial innovation networks. The Metacampus virtual
campus platform is an important aspect of this framework. All participants in
the Unite! community can join at the Metacampus, which coexists alongside
other universities' virtual campuses. It must become a valuable and useful tool
for a wide range of users who are used to very different tools and processes,
which is only one of the many difficulties it faces. The Moodle platform was
chosen as the base for the work, and it was decided to maximise its features
while minimising the amount of new innovations required to achieve the goals.
After four years, the number of users who have registered and taken part in the
activities that have been suggested enables us to draw the conclusion that the
path is the right one for handling a challenge like the design and execution of
a multi-university virtual campus.
Keywords: Education; virtual campus; interuniversity network; e-learning;
engineering education.

The Digital Platform for the Unite! Alliance: The Metacampus
1. Introduction
Currently, one of the 42 European Universities in Europe is Unite! (University Network for
Innovation, Technology, and Engineering)1. It was chosen for the 2019 Erasmus+ call, which
had a three-year funding cycle that ended in 2022. It was chosen again for the 2022 Erasmus+
call as a European University. It is a network of universities from nine different countries
that creates a novel framework for a digital and physical inter-university campus in Europe.
The partners collectively have 280,000 students and 77,300 graduates each year. They
already work closely together on more than 80 EU initiatives, including ones involving
blended learning (Valderrama et al, 2018), and they have exchanged more than 2,000 students
in the previous five years. The nine universities considerably span Europe, as shown in Fig
1. Northern Europe has KTH Royal Institute of Technology, Aalto University, on the one
hand. Western Europe, on the other hand, is home to the universities of Wroclaw Tech,
Technical University of Darmstadt, Grenoble INP-UGA, and Graz University of Technology.
The Universidade de Lisboa, Universitat Politècnica de Catalunya, and Politecnico Di Torino
are the last three institutions in Southern Europe. This close cooperation is made possible by,
among other things, communities of innovation in education, virtual and physical mobility
for all students and staff, an academy for teachers to create cutting-edge models and
pedagogies, and, ultimately, an entrepreneurial open innovation network. The Metacampus2,
an inter university virtual campus that allows for digital mobility, virtual spaces, and online
resources, is necessary for this entire network. All of this while coexisting with the member
universities' virtual campuses and highlighting their complementary nature.
Four fundamental principles must be followed while creating a learning ecosystem (Tiwana,
2014), such as simplicity, resilience, sustainability and the ability to evolve. The integration,
interoperability, and evolution of its components must also be taken into account when
establishing a framework for technological ecosystems, in addition to a proper
characterization of the architecture that underpins it (García-Peñalvo, 2016). We satisfy these
demands as well as those of the nine partner universities by using Moodle as a platform and
reduced customised developments.
An interdisciplinary team from the Universitat Politècnica de Catalunya (UPC) created the
Metacampus under the lead of the Institute of Education Sciences (ICE), in collaboration
with the ICT Services Area and the company UPCnet, which provides IT services to UPC.
The team has substantial expertise with the UPC institutional virtual campus (Atenea), which
is built on the Moodle learning management system, and they have previous experience with
1
https://www.unite-university.eu/
2
https://metacampus.unite-university.eu/
162
Campus Digital, an earlier version of it that was based on Lotus Notes, and that was used up
to 2007 (Alcober, 2000).
Figure 1. Partners of Unite! alliance. Source: Unite! website (2022).
The validation of the Metacampus platform's best practices is described in this study. The
goals of the project are discussed in section 2, and the approach is presented in section 3. The
activities that were undertaken and their results are described in Section 4. And lastly, section
5 presents the conclusions and finally section 6 is acknowledgment.
2. Objectives
One of the goals of the Metacampus is to serve as a meeting point for all those involved in
the Unite! community, which includes the nine member universities' administrative and
support staff as well as students and faculty. On the other hand, this portal aims to achieve
the goals of the Unite! project, which include achieving high levels of collaboration in
teaching innovation, the field of mobility, in communities, and in joint programmes. These
goals were mentioned in the introduction section. As a result, the Metacampus must address
demands, be effective and efficient, as well as long-lasting and with benefits that are
transferable outside of the specific setting in which they emerged (García-Peñalvo, 2016).
3. Methodology
The methods used involves a number of time-ordered steps. The initial stage was to compile
the needs as seen from the user's perspective. The users were community members from
Unite! who contributed to a shared document that was divided into sections for the various
work packages. After completing this compilation, the Metacampus team and UPCnet
gathered, assessed, and prioritised a set of roughly 70 needs, forming the four pillars
mentioned in Section 4. The third step involved the work package participants, academics as
well as staff, suggesting additional project activities that allowed the Metacampus to
163
demonstrate its potential. The fourth phase involved validating the findings using various
measures, such as the quantity of users who had signed up and the volume of visitors to the
Metacampus website. In addition to the number of visits to the Metacampus website, this last
metric, which is displayed in subsection 4.6, should also include the average session length,
in order to measure the engagement of the user.
There were two working initiatives that deserve mention. On the one hand, there was a set
of requirements with a specified development, such as enabling information sharing between
universities and the Metacampus, allowing for the creation of a course catalogue, as well as
the integration of a search engine and Learning Tools Interoperability (LTI)3 technology,
which allows lecturers to offer courses in Metacampus from their home learning management
system, without the need of replicating contents in other systems. While the other task groups
worked together to analyse these needs and how the platform would be able to deliver a
solution, the utilisation of the Moodle functionalities was prioritised. The deployment of
corporate information systems (Parthasarathy, 2018), which suggests organisations adapt to
these reliable information systems rather than customising these information systems to
conform to the way businesses work, served as inspiration for the second line. This creates a
learning ecosystem with the characteristics indicated earlier.
4. Results
The projects completed within the two lines of work previously described are outlined in this
section, including the integrated search engine, the LTI integration, the course catalogue,
eduGAIN, and the Moodle functions. Finally, it is explained how these results were validated.
4.1. Integrated search engine

The ability to search and find information from the nine universities is one of the key features
that makes the Metacampus an entry portal for users of the Unite! ecosystem. Setting up a
Google search within the nine colleges' domains is an easy fix. The idea goes one step further
and enables results to be filtered by a collection of information, for example, the university
of that result, one of the nine languages, or the type of result, such as a course, a person, or
even a thesis proposal. This search engine's initial iteration, which is based on the Apache
Solr search server, is currently being finished (Smiley et al, 2015).
4.2. LTI Integration

LTI integration is a crucial feature that the team has experience with (Alier et al, 2021). This
technology enables students to access courses through Metacampus that are actually hosted
3
https://en.wikipedia.org/wiki/Learning_Tools_Interoperability
164
in the university where the professors are currently employed. As a result, the professors can
carry out their normal duties on their virtual campus of origin without having to duplicate
their materials in a different system. Using an upgraded version of Moodle (v.3.8 at least),
the development of a plugin by UPCnet, and the assistance of the University of Grenoble
INP-UGA. There have been issues with user management and security that have been
analysed and require further research to be implemented in the exploitation systems.
4.3. Course catalogue

The course catalogue is a feature that needs data to be shared across universities. Since all
institutions are actively upgrading their procedures for the Erasmus without Papers4 (EWP)
project, it was decided to use the API (Application Programming Interface) standards of the
courses. The course catalogue will not be implemented until an updated version of the API
is made available.
4.4. eduGAIN
A functionality that was considered important is the possibility to prevent users of the Unite!
ecosystem to register again in the Metacampus. On paper the right solution is to use eduGAIN
(Michael, 2019), although there have been technical difficulties. A temporary interim
solution has been chosen, and that has been that users with institutional mail from any of the
nine universities could register temporarily, pending the resolution of the technical obstacles.
Instead of re-registering in the Metacampus, users of the Unite community were able to
access using their home credentials, which was a feature that was much valued. The best
choice seems to be to use eduGAIN (Michael, 2019). As a result, the platform is aware of the
user's role within the home university and can take appropriate action, such as hiding things
that are not meant for that user.
4.5. Moodle functionalities

Another line of work attempted to maximise the use of Moodle functionalities to suit the
needs of the work packages in parallel with the creation of the aforementioned functionalities.
As a result, the Metacampus adheres to the General Data Protection Regulation (GDPR)
(Amo et al., 2019) and follows the web content accessibility guidelines, Web Content
Accessibility Guidelines (WCAG) 2.1, with an AA level (White, 2019). By combining the
Atto Multilanguage plugin with the MultiLanguage Content filter plugin, users can upload
content in any of the nine languages (Mangiatordi et al, 2019). The platform users must
upload accessible material in a variety of languages in order for the platform to be in
compliance with accessibility requirements or multilingualism. The outcome would not be
4
https://erasmus-plus.ec.europa.eu/european-student-card-initiative/ewp
165
accomplished without the users' active participation. It has been suggested that one of the
essential elements for achieving the platform's objectives is training.
In addition to English being the primary language of instruction, partner institutions' native
tongues and cultures are also included thanks to multilingualism. English is used for any
interuniversity course even if none of the nine universities in Unite! have it as their primary
language because it is the common language that all of them accept. Since Moodle has proven
to be useful for multilingual teaching (Qin et al., 2022), Metacampus team has installed the
nine various languages using the respective language packs.
Communities have been formed using the Moodle overflow plugin, which permits behaviour
similar to StackOverflow in addition to the qualities stated above (Vasilescu et al, 2013).
Additionally, OpenBadge credentialed course functionality (Myllymäki, 2014) has been used
to deliver courses as a standard concept to students, faculty, and administration staff.
Asynchronous collaboration spaces, such as project events (Boot Camp and Dialogue), or
student co-creation initiatives, have also been used. The publication of Master's thesis
proposals, which has been made available through the Moodle database activity, is one
example of how Moodle functionality can satisfy a necessity that seems out of reach.
4.6. Validation of the results

The number of spaces developed by the Metacampus last year 2022 was 55, with 3 spaces
having more than 150 people and 5 spaces having more than 50, in addition to 23 spaces with
more than 20 attendees, according to the activities of the Metacampus. These areas included
the Multicultural and Multilingual Training Center, the Teaching and Learning Academy, the
Hackathon (H@ckyour-COVID), and Student Co-Creation Unite! Future and Joint
Initiatives. The nine Unite member universities currently (April 23) have 5,450 registered
users on Metacampus.It should be noted that the average session length (Fig. 2) and user
count (Fig. 3) both demonstrate that there were 5200 users during this time and that the
average engagement time was 3 minutes and 28 seconds. Additionally, it was discovered that
21% of users used mobile devices to visit the website, while 78% used desktop computers.
The reason probably is because there is no specific mobile app to access the Metacampus.
166
.
Figure 2. Session duration time Source: Metacampus Google Analytics (2022).
Figure 3. Number of users Source: Metacampus Google Analytics (2022).
5. Conclusions and Discussion

Due to the uniqueness of the alliance's member universities, with their various dynamics and
expectations, and the engagement of distributed groups of people from each service,
designing and launching a virtual campus like the Metacampus for Unite! Is a challenging
endeavour. In the end, the platform must be beneficial to users and include educational tools
that enable participants to grow their abilities. As a result of the variety, virtue, knowledge,
and conviction that the project's criteria and objectives have finally been achieved, Unite! has
become, in our opinion, a model for other European alliances with comparable goals that
have been founded in Europe.
167
Acknowledgement
This work has been supported by the Unite! alliance that has received funding from the
European Union's Horizon 2020 research and innovation programme under grant agreement
No 101017408, and Co-funded by the Erasmus+ Programme of the European Union.
References
Alcober, J., & Rincón, D. (2000). The digital campus: an experience of an asynchronous
collaborative learning platform at the UPC. A: International Symposium on Telemedicine
and Teleeducation in Practice. "Symposium Proceedings of the International Symposium
on Telemedicine and Teleeducation in Practice - ISTEP 2000". Frantisek Jakab, Ladislav
Samuelis / elfa SRO, Kosice, Slovak Republic, 2000, p. 13-18. ISBN 80-88964-38-5.
https://upcommons.upc.edu/handle/2117/350330
Alier, M., Casany, MJ., Llorens, A., Alcober, J., & Prat, Jd. (2021). Atenea Exams, an IMS
LTI Application to Solve Scalability Problems: A Study Case. Applied Sciences, 11(1),
Article 1. https://doi.org/10.3390/app11010080
Amo, D., Alier, M., García-Peñalvo, F. J., Fonseca, D., & Casany, M. J. (2019). GDPR
Security and Confidentiality compliance in LMS’ a problem analysis and engineering
solution proposal. Proceedings of the Seventh International Conference on Technological
Ecosystems for Enhancing Multiculturality, 253–259.
https://doi.org/10.1145/3362789.3362823
García-Peñalvo, F. J. (2016). En clave de innovación educativa. Construyendo el nuevo
ecosistema de aprendizaje. Paper presented at the I Congreso Internacional de Tendencias
en Innovación Educativa, CITIE 2016, Arequipa, Perú,
https://repositorio.grial.eu/handle/grial/689
Mangiatordi, A., Pastori, G., Pagani, V., Sarcinelli, AS, & Menegola, L. (2019). Design For
Inclusion In A Linguistically And Culturally Diverse Europe: Challenges In The
Development Of A Virtual Learning Environment. EDULEARN19 Proceedings, 7472–
7481. https://doi.org/10.21125/edulearn.2019.1788
Michael, S., & Anna, ZJ (2019). An identity provider as a service platform for the educational
research and education community. 2019 IFIP/IEEE Symposium on Integrated Network
and Service Management, 739–740.
https://ieeexplore.ieee.org/abstract/document/8717796
Myllymäki, M., & Hakala, I. (2014). Open Badges in Higher Education. In LG Chova, AL
Martínez, & IC Torres, EDULEARN14 Proceedings. 6th International Conference on
Education and New Learning Technologies, 2027-2034.
https://jyx.jyu.fi/bitstream/handle/123456789/45506/myllymakihakalaopenbadgesinhig
hereducation.pdf
Parthasarathy, S., & Daneva, M. (2016). An approach to estimation of degree of
customization for ERP projects using prioritized requirements. Journal of Systems and
Software, 117, 471–487. https://doi.org/10.1016/J.JSS.2016.04.006
Qin, S., Orchakova, L., Liu, Z.-Y., Smirnova, Y., & Tokareva, E. (2022). Using the Learning
Management System “Modular Object-Oriented Dynamic Learning Environment” in
168
Multilingual Education. International Journal of Emerging Technologies in Learning

(IJET), 17(03), Article 03. https://doi.org/10.3991/ijet.v17i03.25851
Smiley, D., Pugh, E., Parise, K., & Mitchell, M. (2015). Apache Solr Enterprise Search
Server (Third Edition). Package https://www.packtpub.com/product/apache-solr-
enterprise-search-server-third-edition/9781782161363
Tiwana, A. (2014). Platform Ecosystems. In Platform Ecosystems: Aligning Architecture,
Governance, and Strategy. Elsevier. https://doi.org/10.1016/C2012-0-06625-2
Valderrama, C., Hagstrom, P., & Nordgreen, T. (2018). Shared curriculum at KTH and UPC
universities: Blended learning experience at the MSc SELECT program. IEEE Global
Engineering Education Conference, EDUCON, 669–676.
https://doi.org/10.1109/EDUCON.2018.8363295
Vasilescu, B., Filkov, V., & Serebrenik, A. (2013). StackOverflow and GitHub: Associations
between software development and crowdsourced knowledge. Proceedings -
SocialCom/PASSAT/BigData/EconCom/BioMedCom, 188–195.
https://doi.org/10.1109/SOCIALCOM.2013.35
White, J. (2019). WCAG 2.1 Meets STEM: Application, Interpretation, and Opportunities
for Further Standard Development. Journal of Science Education for Students with
Disabilities, 22(1), 1–7. https://doi.org/10.14448/jsesd.11.0008
169
Nurturing the human connection: increasing student engagement

and personal connection in an asynchronous language course
Scott Despain, Jennifer A Despain
North Carolina State University, United States.
Abstract
Based on course evaluations, student needs and expectations regarding
asynchronous courses have changed during and after the COVID pandemic.
With the goal of meeting, understanding and responding to the needs of the
post-COVID student, intentional modifications were made to an online,
asynchronous course to ensure attention to the student as a whole. While
respecting the need for asynchronous offerings, designing increased
opportunities for human face-to-face interaction in the form of guided personal
and language goal setting sessions, orientation sessions, chapter check-ins and
exam preparation sessions were all implemented. Initial feedback results show
a positive shift and satisfaction in learning, resilience and growth not only for
students, but also for instructors.
Keywords: Asynchronous; engagement; F2F interaction; goal-setting; needs
assessment; online.

Nurturing the human connection: increasing student engagement and personal connection
1. Introduction
The recent worldwide pandemic forced a shift to online learning in higher education for a
significant period of time. While instructors and students retooled courses and transformed
learning, it has been through experience in recent months that we acknowledge the need for
increased human connection not only within face-to-face courses, but also in traditionally
asynchronous online courses. During the fall 2022 semester, our university family suffered
an increased number of student deaths. This sparked additional, increased focus on student
and faculty psychological well-being, which led administrators to expand, at an accelerated
rate, access to mental health care, to initiate wellness days, as well as assess and adjust
associated administrative and crisis-response policies. As instructors, we have observed that
the post-COVID student population, while more adept at technology than ever before, has a
greater need for one-on-one, meaningful human engagement to facilitate success in life and
learning. As noted by Booker et al. (2022), the pandemic has been an “enduring, thoroughly
disruptive, and as yet unresolved, phenomenon—impacting social life, academic and
employment opportunities and demands, physical and mental health, and financial standing
for many young adults, their families, and their communities” (p. 1946).
2. Course Modifications
Having offered distance and online courses over the past 24 years, we recognize that gone
are the days of plug and play, grade and graduate. Over the years, our introductory Spanish
course has gone through many technological and instructional transformations. In this most
recent iteration, through months of assessment, trial and discovery, we have reassembled our
asynchronous course with several key features to enhance and celebrate the teacher-student
connection. Incorporating a needs assessment, increasing student choice through
gamification and meeting the needs of each student through one-on-one personal check-ins
with an instructor on a regular basis has led to a student/teacher relationship that is not only
beneficial to overall positive student outlook and personalized learning, but has also
increased instructor job satisfaction.
Regular and intentional modifications to the online course have been a constant in response
to a world of ever-changing needs and technological progress. The course was originally
taught by one instructor, streamed live, recorded for later access and testing was conducted
on campus, in person with the instructor (Despain, 2003). Full-class-length videos were later
replaced with brief video segments, the coursemate element was modified, testing became
remote or in-person at local testing centers, an additional instructor was added, graduate
student support was incorporated, language-engagement projects were added and removed.
Keeping delivery technologically up-to-date, relevant and generationally current has been a
characteristic of our course. While some may argue that the answer to increasing student
172
Scott Despain, Jennifer A. Despain
engagement is to limit asynchronous course offerings, registration statistics at our institution

show that students seeking higher education degrees have a continued interest and need for
online asynchronous courses.
3. Needs Analysis
Our most recent modifications have been based on results of an informal needs assessment
conducted through a course format survey. This survey has been pivotal in being able to
ensure that students who truly need a face-to-face course are directed to that format well in
advance of the start of the semester. Acknowledging that students may not fully understand
the value of face-to-face versus a virtual language learning experience, we have outlined the
benefits and challenges of virtual language courses in a very transparent manner. Another
informal needs assessment that we employ comes in the form of one of the most foundational
connections we make with students--the mandatory orientation. Not only is this gathering the
first time students meet instructors, students meet their potential coursemates as well. After
a few minutes of welcome and informal introduction, instructors guide students through each
element of the course, answering any questions posed along the way. This interaction gives
all in attendance an opportunity to connect on a personal level and helps make sure that the
course format placement is accurate. Instructors guide students through the technological
aspects of the course to ensure that technology will not be a hindrance to their language
learning. The orientation also provides the opportunity for instructors to be extremely clear
about the expectations of the course.
One of the first assignments for students is designed to activate students’ background
knowledge of language learning as well as to introduce them to the world of second language
acquisition along with researched learning strategies so that they might understand that
language learning can and should be personalized. Students then use this knowledge to
develop three SMART goals (goals which are specific, measurable, attainable, relevant and
time-based). In many ways, what we see in our classes is not unlike what we might see in
society at large. We have students who are, as Carroll et al. (2013) describe as being,
“delinquent” in not fully participating in the class or meeting student learning outcomes, as
well as students who are “at-risk” of not learning the material or passing the course, and we
have students who are “not at risk”. According to Carroll et al. (2013) intervention programs
should focus on assisting students to “develop clear self-set achievable goals and support
them through the process of attaining them, particularly if the trajectory towards delinquency
is to be addressed” (p. 431). Obviously, we are especially concerned for students who are
challenged or at-risk of not meeting the learning standards set for the course. Research
suggests that students who engage in active goal setting and self regulated learning and who
receive guidance from others in achieving their goals, tend to belong to the not at-risk group
(Carroll et al., 2013). Committing students to evaluate their purpose for taking the course,
173
personally designing action items and periodically reporting their progress is another effort
at assessing and reassessing students' needs on a regular basis.
4. Student Needs
Identifying students’ needs is pivotal in offering course content in a personalized way.
Respecting and offering student choice within a course is another significant element of
engaging students. As Wang et al. (2015) note, student choice in learning leads to increased
motivation to learn. Recognizing the connection that exists between student choice and
motivation, elements of gamification were added to the course to increase student interest
and involvement. Offering meaningful yet optional language application and culture
experiences with a reward upon completion, adds an element of friendly competition that
resonates with a significant number of students. With the successful completion of each
activity, students earn badges or “chapas” which have the potential to positively impact their
overall score by deleting low participation, homework or recording scores. Including this
element of fun, freedom and competition within a framework of discovery, target language
application and student choice has led to a positive shift in engagement.
Expanding on the concept and practice of respecting students and their choices, as well as
rededicating ourselves to student success as modeled by student affairs professional James
C. Hurst’s (personal communication, March 10, 2020) mantra, Each Student – A Person, we
were led to develop increased opportunities for instructor/student interaction, while
maintaining the asynchronous model. In previous years we had offered TA Café sessions -
essentially drop-in online office hours, during various times of the day, where students were
welcome to visit, practice their new language skills, ask questions or review performance
with the graduate teacher assistant. While modestly attended, the Café did not engage
students on a level that was making a positive difference for the majority of students.
Consequently, we retooled and developed the Chapter Check-in. During the Chapter Check-
in, a graded element of the course, students have the opportunity to visit one-on-one with an
instructor on a regular basis (roughly once every three weeks) throughout the semester. In
coordination with the chapter conversation project, the student signs up for an appointment
slot with an instructor and is prepared to ask and answer questions regarding their
performance in the course. The session begins with simple yet personal questions. How are
YOU doing? What is working for you as you study and learn Spanish? What are you finding
challenging? Students then take the initiative to evaluate their progress with their three
SMART goals, their progress in the course and they are guided as they review and modify
their language goals as well as their process of learning. Time is then designated for a student
Q&A about current content and then the instructor leads a look-ahead which briefly
introduces upcoming themes, grammar structures and associated vocabulary. The Chapter
Check-in culminates in a five-minute conversation wherein the student initiates a
174
conversation with the instructor in the target language, demonstrating their acquired and
practiced skills in the learned concepts and structures according to preassigned academic
parameters.
5. Student/Instructor Engagement
As Zhou (2020) asserts in research regarding online courses during the COVID pandemic,
online instructors should “communicate with students through multiple channels and ways”
(p. 1488). To encourage increased student/instructor engagement, use of the target language,
and to support recently learned concepts, we established virtual F2F exam preparation
sessions. Students benefit from attending the student-led sessions that last no longer than 1
hour. Time is spent on addressing student generated questions about course content, but more
importantly instructors have the opportunity to listen to students’ general feedback about the
course or life experiences. While this interaction is couched in the framework of language
learning, the extended and likely most important benefit is offering another human
connection with mentors who are working to guide, support, and foster resilience. This
ensures both personal and academic success on the challenging path of learning and growth.
Lastly, while the recent adjustments that were made in the course were initiated to address
the student learning experience, one of the unexpected benefits resulted in greater job
satisfaction for the instructors. Having taught an online, asynchronous course for decades we
were used to the rather impersonal nature of the format. As an instructor, while it was natural
to recognize a name and to be able to recall the academic challenges or progress of a current
student, having an opportunity to connect with students on a more personal level has resulted
in deeper and more meaningful relationships. A legitimate partnership has been forged in the
asynchronous realm. Early feedback to these course modifications indicate that both students
and instructors benefited from increased interaction and the resulting personal connection.
Results from a student survey administered at the end of the semester are included in the next
section.
6. Survey Results
An attitude survey regarding the Chapter Check-ins was conducted following the final
Check-in session of the course. Nineteen students were enrolled and 13 responded to the
brief, anonymous, volunteer survey (68% return rate). Quantitative results of the survey are
found in Table 1. A majority of students looked forward to the Chapter Check-ins and they
felt both seen and heard by their instructors. They also found it helpful to review upcoming
content during the Chapter Check-in. A majority also indicated that having regular check-ins
helped them feel connected and that the check-ins motivated them to be more engaged in the
asynchronous class. Students were asked to self-report their predicted final grade, with 69%
175
predicting a grade of A or B, 15% predicting a grade of C and 15% predicting a grade of D

or F.
Table 1. Student Survey Results.
Disagree
Agree or
or
Question Strongly
Strongly
Agree
Disagree
I looked forward to having a one-on-one personal conversation with 62% 38%

my instructors during the Chapter Check-ins.
Creating and regularly revising my 3 SMART goals had a positive 46% 54%
impact on my success in FLS 101-601.
I felt seen and heard - knowing that my instructors would be asking 85% 15%
about how I was doing, about my SMART goals and if I had any
questions.
I found it helpful to review the content of the upcoming chapter 77% 23%
Having regular Chapter Check-ins helped me feel more connected with 69% 31%
my instructors
Having the regular Chapter Check-in motivated me to be more 69% 31%

engaged in this online asynchronous class.
When scheduling the Chapter Check-in, I could easily find a day/time 62% 38%
that met my needs.
Regarding the SMART goals a slight majority of the students felt that the creation and regular
revision of goals did not positively impact their success in the course. Positive student
comments included, "very helpful," "great way to maintain progress," "helped me hold
myself accountable," "a good idea," "a helpful tool." Ambivalent or negative comments were,
"had no impact on my experience," "I didn't find them very helpful," and "weren't as helpful
as the projects." Student responses to the open-ended prompt, the best thing about the chapter
check-ins included, "seeing and being able to ask questions directly," "talking to an instructor
about my progress throughout the class," "being able to practice verbal skills," "interpersonal
interaction with my instructors," "encouraged interaction," "I can talk to my instructors and
that helps me feel more comfo[r]table," "checki[n]g in with your instructor is nice," and
"getting to know the professor." To the open-ended prompt of other thoughts or feedback
regarding the chapter check-ins, student comments included, "very informative if you are
confused," "the chapter check-ins did not make me feel any better about the exams," "keep
them going," "asking about mental health each time may be good, considering all that has
176
happened this past semester," "I have missed [some] chapter check ins because I have
anxiety."
7. Conclusion
Based on student feedback and instructor experience, we will continue to incorporate a
regular needs assessment, student choice through gamification and goal setting and chapter
check-ins within our asynchronous online language course. We have found that striking an
appropriate balance between the convenience of asynchronous online learning, and
developing and nurturing the human connection between student and instructor can be
achieved by responding to a thorough needs assessment and ensuring proper placement both
in proficiency level as well as course format. Additionally, thoughtfully crafted student-
driven goals facilitate the student’s responsibility and accountability for the learning process.
Furthermore, providing options for student choice in learning not only communicates respect
for the student as a unique individual but also engages student motivation and customizes
learning. And finally, for many learners early and regular student/instructor synergistic
interaction solidifies success on the scaffold of learning.
References
Booker, J. A., Ell, M., Fivush, R., Greenhoot, A. F., McLean, K. C., Wainryb, C., &
Pasupathi, M. (2022). Early Impacts of College, Interrupted: Considering First-Year
Students’ Narratives About COVID and Reports of Adjustment During College
Shutdowns. Psychological Science, 33(11), 1928–1946. https://doi-
org.prox.lib.ncsu.edu/10.1177/09567976221108941
Carroll, A., Gordon, K., Haynes, M. et al. Goal Setting and Self-Efficacy Among Delinquent,
At-Risk and Not At-Risk Adolescents. (2013). J Youth Adolescence 42, 431–443.
https://doi.org/10.1007/s10964-012-9799-y
Despain, J. S. (2003). Achievement and Attrition Rate Differences Between a Traditional and
Internet-based Beginning Spanish Course. Foreign Language Annals 36 (2): 243-257.
Wang, H., Huang, H., & Hsu, C. (2015). The impact of choice on EFL students' motivation
and engagement with L2 vocabulary learning. Taiwan Journal of TESOL, 12(2), 1.
Zhou, Z. (2020). On the Lesson Design of Online College English Class during the COVID-
19 Pandemic. Theory and Practice in Language Studies, 10 (11), 1484-1488,
http://dx.doi.org/10.17507/tpls.1011.21
177
Appendix
Student Survey
Thank you for being awesome students! We are interested in your feedback regarding the
new addition of the Chapter Check-in and its components (life check-in, 3 SMART goals,
class content review) for this semester. Usernames and emails are not being collected as part
of this survey. Your responses are anonymous.
1. I looked forward to having a one-on-one personal conversation with my instructors during
the Chapter Check-ins. (1-4, Strongly disagree ... Strongly Agree)
2. Creating and regularly revising my 3 SMART goals had a positive impact on my success
in FLS 101-601. (1-4, No impact ... Positive impact)
3. I felt seen and heard - knowing that my instructors would be asking about how I was doing,
about my SMART goals and if I had any questions. (1-4, Ignored ... Seen and Heard)
4. Feedback or thoughts on 3 SMART goals (Open-ended)
5. I found it helpful to review the content of the upcoming chapter. (1-4, Strongly disagree ...
Strongly Agree)
6. Having regular Chapter Check-ins helped me feel more connected with my instructors. (1-
4, Disconnected ... Very Connected)
7. Having the regular Chapter Check-in motivated me to be more engaged in this online,
asynchronous class. (1-4, Strongly disagree ... Strongly Agree)
8. When scheduling the Chapter Check-in, I could easily find a day/time that met my needs.
(1-4, Strongly disagree ... Strongly Agree)
9. The best thing about the Chapter Check-ins...(Open-ended)
10. Any other thoughts or feedback about the Chapter Check-ins? (Open-ended)
11. What grade do you expect to earn in this course? (A or B | C | D or F)
178
Critical analysis of the potential of social robotics in higher

education for the management of illness and bereavement
Universitat de les Illes Balears, Spain.
Abstract
Social robotics and the use of information and communication technologies
have a significant impact on education. This approach combines robotics and
artificial intelligence with a social and emotional perspective to improve the
quality of education while providing psychoeducational and emotional support
to students. This critical analysis aims to discern whether these technologies,
together with inclusive and quality teaching strategies and organisational
strategies adapted to each centre and classroom, can enable the presence in
class during periods of hospitalisation and/or convalescence of seriously ill
students, and facilitate their participation and progress, as well as allow for
better accompaniment during the bereavement of the class if this colleague
dies. The advantages of these technologies are highlighted as tools of great
value in the inclusion and integration of these students during higher
education, although it is true that it is essential to carefully assess the
advantages and disadvantages before implementing them.
Keywords: Benefits; drawback; artificial intelligence; telepresence; teacher;
classmate.

Critical analysis of the potential of social robotics in higher education for the management of illness
1. Introduction
Social robotics and the use of information and communication technologies (ICT) in higher
education are of great importance in the field of education. This approach combines robotics
and artificial intelligence (AI) with a social and emotional perspective to improve the quality
of education and support students (Belpaeme et al., 2018). In particular, for students facing
limiting illnesses, social robotics and ICT can be valuable tools to help them stay connected
and participate in their education effectively (Belpaeme & Tanaka, 2021).
According to Johal (2020), social robotics can enhance student engagement and learning in
higher education by offering a more personalised and enriching experience. Furthermore, the
combination of ICT with social robotics allows students to access online learning resources
and materials in real time and from anywhere (MAarell-Olsson et al., 2021). According to
the authors, these resources provide a unique opportunity to enhance education and provide
support for students with limiting illness, allowing them to remain present in the classroom
without being physically there. It is not just a computer screen, but the social robot gives a
physical and corporeal presence to the student who for health reasons cannot be in the
classroom. This innovative and emotional approach can help students stay connected and
actively engaged in their education, despite any challenges they may face. This research aims
to discern from already published contributions whether these technologies, accompanied by
inclusive and quality didactic strategies together with organisational strategies adapted to the
characteristics of each centre and classroom, can make possible the presence in classes during
periods of hospitalisation and/or convalescence, as well as facilitating the participation and
progress of students with serious health problems.
The use of social robotics in higher education for students with limiting illnesses and
difficulties in attending face-to-face classes is justified by its ability to enable classroom
presence, improve accessibility, encourage participation and, ultimately, provide an inclusive
context for students with serious health problems. In addition, in the case of a palliative
illness and possible death, the robot allows classmates to say a final farewell with an
appropriate farewell. The acceptance and gradual farewell of fellow students with the student
in a palliative illness situation through the presence in the classroom of the figure of the
student who no longer exists avoids the empty chair syndrome by peers and normalises the
end of life, eliminating taboos and associated fears (Riera & Ruiz, 2021). However, it is
necessary to evaluate all the elements involved in the use of this particular type of technology
in a classroom. To this end, a critical analysis article is carried out in order to evaluate and
analyse the existing contributions on the subject. The resulting research question is: What are
the uses, advantages and disadvantages of digital technologies and social bots in higher
education for students with a limiting illness?
180
With the great advances in technology, social robots have become a useful tool to help these
students overcome physical barriers and participate in school activities (Newhart & Olson,
2017). Social robots are electro-mechanical devices programmed to interact with people and
help them meet their needs. In the case of students with limiting conditions, social robots can
be used to provide a presence in the classroom and keep them connected to their peers and
teachers (MAarell-Olsson et al., 2021), which has positive implications in terms of
accessibility and inclusion (Belpaeme et al., 2018). For example, social robots can be
programmed to deliver classroom presentations, participate in group discussions and
activities, and provide ill students with the opportunity to interact with peers and teachers in
real time (MAarell-Olsson et al., 2021). In addition, social bots can also help students with
limiting illnesses to keep up with their school work and receive instructions and feedback
from their teachers. Not only do they provide educational advantages, but social robots can
also provide emotional support to students with limiting illnesses (Newhart & Olson, 2017).
However, it is important to note that the use of social bots in education should be a
complement to and not a substitute for the care and support of teachers and other educational
professionals (Belpaeme & Tanaka, 2021). The use of this technology for students with
limiting illnesses that prevent them from attending face-to-face classes can be a valuable tool
to help them overcome physical barriers and stay connected to their peers and teachers,
although there are a number of legal and ethical issues to consider (Powell et al., 2021).
Telepresence robots can also improve efficiency in classroom management, enabling better
coordination and communication between students and teachers. This can result in a better
learning experience for all students, not just those who are limited by health problems.
In addition, there are also some negative practical implications to consider, such as classroom
and home privacy (Wadley et al., 2014), data protection (Pusztahly & Stefán, 2022), the
possibility of technology malfunctioning or connection disruptions, and the high cost
associated with implementing and maintaining telepresence bots (Khaksar et al., 2020). Some
of the utilities through practical examples of the use of social bots and ICT in the classroom
for other purposes are as follows, although the use of these technologies continues to grow
in higher education:
1. Pedagogical robots: pedagogical robots, such as the NAO robot, are used to enhance
learning in the classroom (Robaczewski et al., 2021). The literature shows the use
of up to 26 types of robots (Dawe et al., 2019).
2. Virtual and augmented reality: Virtual and augmented reality is used to enhance the
learning experience in the classroom (Boyles, 2017).
3. Educational games: Educational games are an effective way to enhance learning in
the classroom (Amanatiadis et al., 2017).
181
In the case of students with a limiting illness with frequent hospitalisations or absences, this
technology improves collaboration between school and home and allows the student to learn
independently while maintaining social relationships (Zhu & Van Winkel, 2015). In this
sense, the potential of social robots for bereavement support arises in the event of the death
of a partner with a limiting or complex illness. Although there is no literature on the subject,
it is estimated that they allow the class to say a gradual and preventive farewell to the sick
classmate, which will result in a better emotional management of grief. It should be noted
that this is a first approach to the subject in the form of an analysis of possibilities, without
claiming to be a systematic review of the literature.
In summary, social robots can have both positive and negative implications for university
learning. However, the implications for accessibility and inclusion of health-limited students
and bereavement management for their peers, while improving the efficiency and quality of
education in general, need to be analysed.
3. Critical argument and conclusion

The use of social robotics and ICT in higher education appears to have the potential to
improve the quality of teaching and support for students with limiting illnesses. However, it
is important to keep in mind that social robotics and ICT are complementary tools and not a
single, definitive solution to all the challenges that students with limiting illnesses may face.
One criticism of social robotics in the educational context is that, while it can provide a
physical presence for students with limiting illnesses, it cannot replace human interaction and
the face-to-face teaching experience. Social robotics and ICT can improve accessibility and
participation, but cannot provide the same experience as a face-to-face classroom and
interaction with peers and the teacher. In addition, it is important to consider equity and
accessibility in the use of social robotics and ICT in education. Not all students have access
to the same technological resources and, in some cases, social robotics and ICT may
exacerbate inequalities rather than reduce them.
There is as yet no evidence of their usefulness and effectiveness in emotionally managing the
grief of peers who are left behind, although it is thought that they may be a useful resource
for dealing with empty chair syndrome in the classroom. The figure of the robot in the
classroom allows for a remembrance of the departed peer, while the teacher guides a gradual
farewell to the departed peer. Finally, it is important to highlight the need for proper and
ethical regulation in the use of social robotics and ICT in education. There is a need to ensure
the privacy and security of student data, as well as to consider the possible social and ethical
implications of the use of social robots in the classroom.
182
In conclusion, although social robotics and ICT can be valuable tools to support students with
limiting conditions in higher education, it is important to carefully evaluate the uses,
advantages and disadvantages before implementing them. A balanced and critical approach
is required to ensure that these technologies are used ethically and effectively to enhance
education and, above all, further research is needed on the potential of using this technology
beyond distance learning.
Based on the above, the following recommendations are made for future research on the use
of social robotics and ICT in higher education:
1. Detailed evaluation of the effectiveness of social robotics and ICT in improving
the quality of teaching and support for students with a limiting illness.
2. Comparative studies between the experience of students with illness in a face-
to-face classroom and a virtual classroom using social robotics and ICT.
3. Research on equity and accessibility in the use of social robotics and ICT in
education, including analysis of barriers to accessing the technology and the
possible exacerbation of inequalities.
4. Studies on ethical regulation in the use of social robotics and ICT in education,
including privacy and security of student data, as well as the social and ethical
impacts of their use in the classroom.
5. Cost-benefit analysis of the implementation of social robotics and ICT in
education, including teacher training and the infrastructure required for its use.
6. Studies on the training needs of teaching teams so that they can make
appropriate use of these technologies in the Teaching-Learning processes.
7. Analysis of the pedagogical, organisational and technological needs of
educational centres to adequately integrate social robotics in the classroom.
These recommendations can help future research to gain a more complete and critical
understanding of the use of social robotics and ICT in higher education and to make informed
decisions about their implementation. What is undeniable is that technology is the future of
education, so it is our responsibility to adapt and adapt this technology in the service of
educational quality.
References
Amanatiadis, A., Kaburlasos, V. G., Dardani, C., & Chatzichristofis, S. A. (2017,
September). Interactive social robots in special education. In 2017 IEEE 7th international
conference on consumer electronics-Berlin (ICCE-Berlin) (pp. 126-129). IEEE.
Belpaeme, T., & Tanaka, F. (2021). Social robots as educators. OECD Digital Education
Outlook 2021 Pushing the Frontiers with Artificial Intelligence, Blockchain and Robots:
Pushing the Frontiers with Artificial Intelligence, Blockchain and Robots, 143.
183
Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., & Tanaka, F. (2018). Social
robots for education: A review. Science robotics, 3(21), eaat5954.
Boyles, B. (2017). Virtual reality and augmented reality in education. Center For Teaching
Excellence, United States Military Academy, West Point, Ny, 67.
Dawe, J., Sutherland, C., Barco, A., & Broadbent, E. (2019). Can social robots help children
in healthcare contexts? A scoping review. BMJ paediatrics open, 3(1).
doi: 10.1136/bmjpo-2018-000371
Johal, W. (2020). Research trends in social robots for learning. Current Robotics Reports, 1,
75-83.
Khaksar, S. M. S., Slade, B., Wallace, J., & Gurinder, K. (2020). Critical success factors for
application of social robots in special developmental schools: Development, adoption and
implementation. International Journal of Educational Management, 34(4), 677-696.
MAarell-Olsson, E., Mejtoft, T., Tovedal, S., & Söderström, U. (2021). Opportunities and
challenges of using socially intelligent agents: increasing interaction and school
participation for children suffering from a long-term illness. The International Journal of
Information and Learning Technology. DOI 10.1108/IJILT-11-2020-0199
Newhart, V. A., & Olson, J. S. (2017) My student is a robot: How schools manage
telepresence experiences for students. In Proceedings of the 2017 CHI conference on
human factors in computing systems (pp. 342-347).
https://doi.org/10.1145/3025453.3025809
Powell, T., Cohen, J., & Patterson, P. (2021). Keeping connected with school: Implementing
telepresence robots to improve the wellbeing of adolescent cancer patients. Frontiers in
psychology, 5250. https://doi.org/10.3389/fpsyg.2021.749957
Pusztahelyi, R., & Stefán, I. (2022). Household Social Robots–Special Issues Relating to
Data Protection1. Acta Universitatis Sapientiae, 11(1), 95. DOI:
110.47745/AUSLEG.2022.11.1.06
Riera, L., & Ruiz, J. D. (2021). Diseño de un repositorio digital para la gestión de la
enfermedad grave y la muerte en pedagogía hospitalaria. Edutec. Revista Electrónica de
Tecnología Educativa, (77), 120-135. DOI: 10.21556/edutec.2021.77.2177
Robaczewski, A., Bouchard, J., Bouchard, K., & Gaboury, S. (2021). Socially assistive
robots: The specific case of the NAO. International Journal of Social Robotics, 13, 795-
831.
Wadley, G., Vetere, F., Hopkins, L., Green, J., & Kulik, L. (2014). Exploring ambient
technology for connecting hospitalised children with school and home. International
Journal of Human-Computer Studies, 72(8-9), 640-653.
https://doi.org/10.1016/j.ijhcs.2014.04.003
184
Enhancing personalization and experiential learning in higher

education through the VR mobile application, I’m IN–HKUST
Kasina Wong
The Hong Kong University of Science and Technology, Hong Kong.
Abstract
The present project aims to break the physical limitation of job interview
practices for students in LABU2060 Effective Communication in Business, the
last required CLE course of the Business School’s program at the Hong Kong
University of Science and Technology (HKUST). The pandemic has intensified
the need for incorporating emerging technologies, e.g., virtual reality (VR), in
Higher Education. A VR job interview mobile application was implemented in
LABU2060 in 2022. Surveys, field observations and focus group interviews
were conducted within HKUST. It was found most students were satisfied with
the usability and their learning experience gained through using the
application in the 2D mode. Students participated in the field tests, utilizing
the application in the VR mode (N=17). The findings revealed that the VR job
interview application successfully helped students practice and prepare for
their job interviews. Students preferred the VR mode to the 2D mode thanks to
the immersiveness and authenticity of the VR job interview practices.
Keywords: Virtual reality; job interview application; business
communication; higher education; personalization; experiential learning.

Enhancing personalization and experiential learning in higher education through I’m IN–HKUST
1. Introduction
This project was developed to meet the needs of business students in the LABU2060
Effective Communication in Business course, as part of the language requirement for the
Business School’s program at the Hong Kong University of Science and Technology. The
course, beginning with job-seeking, utilizes a simulation approach, but only one online lesson
on job interview practices is not sufficient for students. Finding partners to practice outside
of class has proved futile (with or without the pandemic), and teachers are unable to provide
additional support. To address this, the project team employed Virtual Reality technology as
a solution to provide an authentic and immersive learning experience for students. The VR
job interview trainer, enhancing learning effectiveness and administrative efficiency
(Fabrisa, Rathnera, Fonga, & Sevigny, 2019), was designed to fill the gap in teaching and
learning of business communication in a job interview setting. It allows learners to actively
engage with the application at their own pace and time without added stress, such as the
presence of a teacher or assessments. Through this process, learners are able to reflect on
their experience and partake in a meaningful learning process. There are a few online VR
training courses for job interview skills in the market. However, these commercial solutions
are not without costs. The free job interview software in the market might not be satisfactory
while some provide training contradicting what we is taught in our Business Communication
course. One existing job interview mobile application in higher education in Hong Kong does
not seem to be in full operation due to the limitations presented. Our VR job interview
practice application, I’m IN–HKUST, in the mobile application market (Play Store and Apple
Store) has served to offer comprehensive training for HKUST student users to improve their
communication and job interview skills.
2. Literature
The project is based on the principles of experiential learning and the learning model that
accounts for how the brain processes and registers information.
2.1. Experiential Learning Theory (ELT)

The creation of meaning is achieved through personal experiences. Kolb’s (1984)
experiential learning cycle consists of four steps that occur repeatedly during every
experience: Experience, Reflect, Think, and Act. The cycle begins with a tandem, a concrete
experience followed by reflection on that experience, leading to abstract thinking and
deriving conclusions – “learning [is] a function of experience” (Moore 1999, as cited in
Moore 2010, p.4). According to ELT (Kolb, 1984; Kolb & Kolb, 2005; Kolb & Kolb, 2009),
learning is best facilitated by a process that draws out the students’ beliefs and ideas about a
topic so that they can be examined, tested and integrated with new, more refined ideas.
Furthermore, ELT sees a holistic process as pivotal to learning that involves the integrated
186
Kasina Wong
functioning of the whole person, including thinking, feeling, perceiving and behaving. ELT
proposes that learning results from synergetic transactions between the learner and the
environment. Moreover, ELT views learning as the process of creating knowledge, as
opposed to the model which is based on the transmission of prescribed fixed ideas. The
experiential learning process is rooted in the process of brain functioning (sensing,
integrating and motor) (Zull, 2002). The two sets of experiential learning and brain-based
learning principles intersect and are juxtaposed (e.g., Duman, 2010). Learning naturally
involves the brain, and learning could be deeper if more parts of the brain are involved.
2.2. Brain-based Teaching and Learning

The Brain-based teaching and learning (BBTL) approach, based on research in cognitive
science and neuroscience, aims to align with the brain’s natural operations to promote full
understanding and memory retention. There are 12 principles established by Caine & Caine
(1990) which cover topics such as the brain as a computer, the importance of meaning and
experience in learning, which coincide with ELT, and the effects of emotions, attention, and
memory. One principle states that people understand and remember best when facts and skills
are embedded in natural, spatial memory, i.e., experience, in short. This approach emphasizes
a holistic approach to learning and uses the three instructional methods of Orchestrated
Immersion (OI), Relaxed Alertness (RA), and Active Processing (AP) to create a highly
engaging and meaningful learning experience. The mental model developed through the three
instructional techniques (Gülpinar, 2005) is particularly interesting in how BBLT and ELT
work together in a learning environment that could maximize the benefits for the learners.
Shifting from rote memorization to meaningful learning is essential in BBTL.
3. Project Description
The primary aim of the project was to enhance students' practice opportunities for job
interviews and their awareness of their communication skills when answering interview
questions. The project objectives were successfully met within the 1.5-year time frame. A
VR job interview application with instructional content on job seeking was implemented in
a course at HKUST in Spring and Fall 2022 and was made available in the mobile application
market. The VR app is linked to a website with a user menu and a database for storing users'
performance recordings. The immersive learning experience incorporated virtual reality,
personalization, reflective elements, and gamification. Unique features of the project include
a VR and 2D mode option, 3 different practice time frames and 3 challenge levels, a
collection of job interview questions for a suite of 15 most sought-after jobs among HKUST
students, and a replay of interview questions and a skip option during practice sessions.
Students can receive verbal feedback from the VR trainer based on their self-assessed
performance levels, and training resources, i.e., mini-lessons in the form of videos, are
187
provided. The mini-lessons feature input from HR directors and managers in the industry as
well as instructors of the course.
4. Methodology
Two Qualtrics surveys were conducted in the Fall of 2021 and Spring of 2022 to gather
feedback on the VR app. The surveys received positive feedback on the app's usability and
the learning experience provided, along with useful suggestions. However, the response rate
for both surveys was low, possibly due to the timing of the VR app's launch. Additionally,
many respondents used the 2D mode instead of the VR mode, likely because of the pandemic
and the difficulty accessing the HKUST library to borrow VR headsets. Hence, the project
team invited students from different disciplines to evaluate the project and assess the
effectiveness of the VR features in Summer 2022.
HKUST students were invited to partake in a field observation study from 15 June to 16 July
2022. A total of 17 students provided valid responses. The study consisted of three parts: (1)
VR app experience with short evaluations, where students were asked to use VR goggles
provided by the project team and practice an interview in the VR mode, (2) individual
interviews, and (3) focus group interviews. During the field tests, students were given a 15-
minute interview practice at a designated difficulty level, but were able to choose any
industry and job they were familiar with. After the practice interview, students completed a
short self-evaluation on the App and received feedback from the VR app based on their self-
evaluation. They were also asked to provide written ratings of their performance and an
observer also made notes on their performances. In the final part of the study, students were
asked to watch a mini-lesson (out of 13 in total), a 5-10 minute video in the VR mode with
any topic of their choosing.
An individual interview was held to gather information about students' experience with the
VR app and their engagement during the interview practice. The purpose of the interview
was to assess the usefulness and ease of use of the VR app. Afterwards, students participated
in a focus group discussion to explore potential uses for the VR app, such as job interview
training. The groups were composed of 2-3 students from the same field of study, for
example, students enrolled in the LABU2060 class were placed in the same group. The
breakdown of participants is listed in Table 1 below.
Table 1: Disciplines of student subjects.
Disciplines Number of groups Number of students
Business 4 10
Engineering 2 4
Science 1 3
Total 7 17
188
Kasina Wong
5. Findings
The results of the study on the users' experience with the interview practice are presented
below, including their level of engagement, level of stress, effectiveness of the VR practices,
and effectiveness of the feedback received.
5.1. Level of Engagement

The students' level of engagement during the interview practice was evaluated using a 3-point
Likert scale, with results indicating that the students were able to actively participate (average
score of 2.29 and a standard deviation of 0.686). The observations made by the teachers also
suggest that the students were able to fully engage in the interview practice (average score of
2.47 and a standard deviation of 0.624). These results are presented in Table 2 below. The
majority of students were reported to have engaged in the practice and interacted with the
VR app. One student was reported to have been initially nervous but eventually became more
comfortable as the practice progressed. Students felt more engaged in the VR mode as
compared to practising in 2D, as it allowed them to be fully immersed and not distracted by
their surroundings.
Table 2: Students’ engagement levels as reported by the students and the observers.
Std.
Engagement Level N Min Max Median Mean
Deviation
Students 17 1 3 2 2.29 0.686
Teacher-observers 17 1 3 3 2.47 0.624
Note: A 3-point Likert scale
5.2. Level of Stress

According to the students' feedback, the majority of them felt low stress levels during the
interview practice. They attributed this to app’s ability to repeat any questions and the fact
that the interviewer was not a real person and would not have any facial expressions affecting
their performances. They also highlighted that they felt they were practicing, as the VR
interviewer would “go easy on you”, and “mistakes are allowed”.
5.3. Effectiveness of the VR Practices

The VR interview practice was found to be effective in improving communication and
interview skills. The flexibility and convenience of the app, as well as the immersive
experience provided by the VR goggles, were cited as factors by students that contributed to
its effectiveness. They reported that they were able to engage more fully in the practice and
felt more prepared for real job interviews as a result. Additionally, the app provided an
authentic experience that was close to a real job interview and helped students focus better
and practice more effectively. A student claimed they would “try to use it again, especially
189
when I'm having a job interview soon. I can listen to some questions from the interviewers.
I will be able to practise more and also get more well prepared before the real job interview.”
Another student indicated they “would like to try 100% (perform at their best)” in the VR
interview. Students reported it was really difficult to have a job interview practice partner in
real life. Some students were able to make subsequent improvements based on the self-
reflections through the VR app. In contrast, students who had used a 2D version of the app
reported being easily distracted and abandoning the practice quickly.
5.4. Effectiveness of the Feedback

To assess the effectiveness of the feedback provided, students were asked to rate their own
and teachers the students’ performances using the same 3-point scale during the interview
practices with the VR app. The results showed that the teachers' ratings were slightly higher
than the students' self-evaluations (see Table 3). Related Sample Wilcoxon Signed Rank Test
("Wilcoxon-signed-Rank test," n.d.) was conducted to determine if significant differences
between the median scores of the teachers' and students' evaluations were presented. The
results showed that there were no significant differences in task fulfillment, fluency, stress
and pausing, and energy and enthusiasm level. However, there was a significant difference
in the use of language, with the teachers' ratings being higher than the students' self-
evaluations, indicating that the studnets’ language use was moer proficient than what they
reported and the feedback by the VR trainer might have been more critical than what the
students should receive in the end.
Students were further asked to provide their thoughts on the feedback they received. The
majority of students stated that the feedback provided by the VR app was useful, providing
specific and accurate suggestions. One student said the feedback on the VR app was very
close to the teacher comments on her job interview perfornances before.
190
Kasina Wong
Table 3: Related Sample Wilcoxon Signed Rank Test on the Observation.
Median Positive Negative p-value

N Differences Differences (2-
Observer - Students Observers Students tailed)
Task Fulfilment 17 3 2 10 3 0.052
Use of Language 17 2 2 9 2 0.029^
Fluency, Stress and
17 2 2 6 2 0.132
Pausing
Energy and
17 2 2 4 3 0.380
Enthusiasm Level
Notes: A 3-point Likert scale. ^ Lower than significant level 0.05
6. Discussion
The purpose of the paper was to analyse the findings in regard to the perspective of
personalization and experiential learning conceived through the VR app. The results will be
analyzed to explore the benefits of using VR technology and self-reflections in learning.
6.1. Personalization through VR Technology

Students’ need for more learning practices is often neglected. The VR job interview program
is based on the idea of continuous learning which was not feasible in a classroom setting due
to administrative and resource concerns and the curriculum design. It is not up to the students
if they hope to have practices on a given topic in the capacity and frequency each individual
would prefer to have. In reality, solely one lesson was assigned for students to practice the
communication skills they have acquired in teachers’ preferred format (in pairs or groups) in
class. The students in the study found that repeating the same tasks in the VR environment
helped them improve their responses, and they felt that this repetition helped them answer
job interview questions in a more organized way. In addition, the communication skills they
developed would be pivotal not only to the job interview assessments that will shortly be
conducted but also to real-life job or internship interviews that might occur at a much later
time. With the VR app, job interview practices are personalized – it has become a viable
option according to the time and space as needed by students.
6.2. Experiential Learning through VR Technology

The main focus of the study is to understand the effectiveness of using a virtual reality (VR)
job interview speaking program, in the form of a mobile application. The research
specifically examines the experiences of the student users and the benefits of using VR
technology for this type of communication skills training (see Table 2). The results suggest
that students preferred the VR mode as it provides an authentic, immersive and interactive
191
experience, which leads to better performance during the interview. The VR environment
creates a sense of presence and eliminates distractions, contributing to the overall
effectiveness of the program. All of these coincide with the intended pedagogy behind the
program which is based on Experiential Learning Theory and Brain-based Teaching and
Learning, focusing on the “integrated functioning of the whole person–thinking, feeling,
perceiving, and behaving,” Student users are engaged in a learning environment that fully
engages their many communication skills (OI), allowing them to process learning materials
by actively practicing them (AP) while removing the unnecessary fear (RA). The VR job
interview application aims to provide a safe and secure learning environment that promotes
meaning-making and counters ‘downshifting’ (when learners retreat to a defensive mode
being less flexible and open to new information). The study's findings indicate that the VR
app is a superior platform for experiential learning, which was not initially expected by the
researcher.
7. Conclusion
The VR job interview training environment creates a sense of presence, eliminates
distractions, and contributes to the overall effectiveness of the program. The VR app was
designed in a manner to allow students to reflect on their own learning and take charge of
their own development by providing personalized options (in terms of the length of practices,
challenge levels, industry and job, replaying or skipping any interview questions during the
practice) and flexibility (with reference to time and space), in line with the principles of
experiential learning theory (ELT). ELT posits that learning is most effective when it is an
active process, where students learn by doing and reflecting on their experiences. The study's
findings support the effectiveness of using VR technology for communication skills training,
yielding the benefits of personalization and meaningful learning experience.
References
Caine, R. N., & Caine, G. (1990). Understanding a Brain-Based Approach to Learning and
Teaching. Educational leadership, 48(2).
https://www.uvm.edu/~mjk/013%20Intro%20to%20Wildlife%20Tracking/Brain-
Based_Learning.pdf
Duman, B. (2010). The Effects of Brain-Based Learning on the Academic Achievement of
Students with Different Learning Styles. Educational Sciences: Theory and
Practice, 10(4).
Fabrisa, C. P., Rathnera, J. A., Fonga, A. Y., & Sevigny, C. P. (2019). Virtual Reality in
Higher Education. International Journal of Innovation in Science and Mathematics
Education, 27(8), 69-80. file:///C:/Users/WIN/Downloads/13983-85-40497-2-10-
20191022.pdf
192
Kasina Wong
Gülpinar, M. A. (2005). The Principles of Brain-Based Learning and Constructivist Models

in Education. Educational Sciences: Theory & Practice, 5(2).
Kolb, D. A. (1984). Experiential Learning: Experience as the Source of Learning and
Development. New Jersey: Prentice-Hall.
Kolb, A. Y., & Kolb, D. A. (2005). Learning styles and learning spaces: Enhancing
experiential learning in higher education. Academy of Management Learning &
Education, 4(2), 193-212. https://doi.org/10.5465/amle.2005.17268566
Kolb, A. Y., & Kolb, D. A. (2009). Experiential learning theory: A dynamic, holistic
approach to management learning, education and development. The SAGE Handbook of
Management Learning, Education and Development, 42-
68. https://doi.org/10.4135/9780857021038.n3
Moore, D. T. (2010). Forms and issues in experiential learning. In D. M. Qualters (Ed.) New
Directions for Teaching and Learning (pp. 3-13). New York City, NY: Wiley.
Wilcoxon-signed-Rank test. (n.d.). SpringerLink.
https://link.springer.com/referenceworkentry/10.1007/978-3-642-04898-2_616
Zull, J. E. (2002). The Art of Changing the Brain. Sterling, VA: Stylus.
193
Student response systems: enabler of active learning in a large

class
Willie Golden
J.E. Cairnes School of Business & Economics, University of Galway, Ireland.
Abstract
This paper presents students opinions on the benefits they gained from using
Vevox – a student response system (SRS), in a 2022-23 Semester I, final-year
undergraduate module with 56 students in an Irish University. The students
were surveyed to assess what benefits, if any, they believed they had obtained
from using the SRS in the module. The key benefit identified by 41% of the
students was that using the SRS contributed to them engaging with the lectures
– both in terms of the material being taught and with their classmates. The
second most important benefit identified was that the questions posed via the
SRS enabled the students to reflect on their learning while in the lecture. The
research contributes to the literature by showing that possing open-ended
questions via SRS leads to active engagement and learning by students.
Keywords: Student response system; audience response system; active
learning; large class teaching; Vevox.

Student response systems: enabler of active learning in a large class
1. Introduction
The ability to use computer technology to enable students to ‘vote’ emerged in 2005 when
physical clicking devices in the possession of students could be connected via a radio-
frequency to a receiver. These devices normally had 4 buttons from which students could
choose to answer a multiple-choice question posed by the lecturer. Even with such limited
capabilities research found that using them resulted in significantly increased student
engagement and improved satisfaction with their learning experience (Chou, Chang, & Lin,
2017; Nikou & Economides, 2016; Pimmer, Mateescu, & Gröhbiel, 2016).
In recent years, such ‘clickers’ have been replaced by mobile phone apps that allow student
responses. There are currently 80+ such app systems, with the most popular being Kahoot,
Polleverywhere, Vevox, QuestionMark and TurningPoint. These new student response
systems have significantly improved functionality – allowing different question types,
including open questions where the response is a typed answer. In addition, because they run
on mobile phones – which are ubiquitous among the student population – and can
communicate via either wifi or mobile network, they can be used by all students easily and
do not suffer the technical connection issues that was common among radio-frequency clicker
systems. (Bogdanović, Barac̈, Jovanic̈, Popovic̈, & Radenkovic̈, 2014).
The general availability of these technically reliable student response systems (SRS) which
are now available for purchase via Institutional software site licences, provides the possibility
of widespread adoption. In addition, these systems now facilitate various assessment
functionality via the variety of question formats that they allow. Which, makes them an ideal
technology to enable technologically based assessment as part of the live teaching
environment. While traditionally such assessments in the live teaching environment would
have used pen and paper, in large classes this was not practical due to the time required to
gather such paper-based responses alongside the need to manually match the individual
student responses against a class list. With SRS, the assessment questions are available to all
students at the same time and for an equal duration. Student responses are available
immediately electronically and the student response system has functionality to amalgamate
answers which can immediately be displayed to students – such immediate feedback has
consistently been demonstrated to improve learning. From an administrative point of view
each students answer is available in the system against their name – so marks awarded are
easily recorded.
Research to date on SRS has been predominantly on enabling student participation and
enhancing their learning, but little research has focused on using them as an assessment
method – even though they have been identified as a technology that could enhance formative
assessment in the 21st century (Spector et al., 2016). An early systematic review on student
response systems undertaken by (Kay & LeSage, 2009) suggested that research need to be
196
Willie Golden
carried out with respect to the impact of specific types of questions in enabling improved
learning environments. A recent systematic review on SRS (Wood & Shirazi, 2020) and the
student experience found that the most widely discussed theme within the articles identified
was student engagement. Echoing the research call of (Kay & LeSage, 2009) who also
identifed the area of assessment – specifically ‘question design’ as a critical element in the
use of SRS, and acknowledge that it is an area that requires further study, and more
specifically the use of SRS to pose open-ended questions to students. This research seeks to
answer this call for research.
2. Case Study
The SRS, Vevox, was used in the final year University under-graduate module: Business
Intelligence & Analytics in Semester I, 2022-23 by the author in the University of Galway.
117 students took the module. Vexox was used in each of the eleven 2-hour lectures to ask
open-ended questions which together were worth 15% of the overall assessment marks for
the module. Forty questions were asked over the duration of the module. The decision to part-
take in some or all of the 40 questions was a decision each student had to make and
importantly they were not deducted marks for not partaking, instead any marks not availed
of through answering the in-class questions were instead awarded for their final written exam.
2.1. Data Collection method

At the last lecture in the module, students were asked 3 questions via Vevox, about their
experience of using Vevox. In advance of putting up the questions, the lecturer made it clear
that these questions would not form part of their in-class assessment mark and so it was
entirely voluntary if they wanted to answer them. Also, they were assured that at no stage
would the answers they gave be attributed to them individually – i.e. their anonymity was
assured. There were 61 students present at the last lecture and 56 of those chose to answer
the questions about Vevox. This means that 47% of the students registered for the module
partook in the survey. This would have been higher had attendance at the final lecture been
higher.
A central part of the pedagogy for using in-class assignments in this module was to ask open-
ended questions that required students to think and provide their own answers based on their
own individual learning. This ethos was also used in the phrasing of the questions on their
experience of using Vevox.
The three questions asked were.
1. What benefits (if any) do you believe you have gotten from using Vevox polling
software in this module?
197
2. In what ways (if any) do you think using Vevox polling in this module has helped
your learning?
3. Write down your questions/suggestions as to how the lecturer could use Vevox
Polling to better improve your learning?
3. Findings
The answers given for each of the questions were analysed and categorised at the individual
question level and additionally also across the three questions.
Given that submitting the answers via Vexox was the means through which the 15% available
for in-class assessment was facilitated, it might reasonably have been expected that this was
a benefit that students would self-identify from using the Vevox software. To assess this, the
answers across all three questions were analysed. Across the three questions, only 6 students
(10% of respondents) mentioned anything about obtaining marks for their answers to
questions each week and 3 of these were with respect to suggestions on how to improve
things.
The lack of responses that even mentioned the availability of continuous assessment marks,
is surprising, especially given this module is a final-year module and as such the marks from
the module contribute to the calculation of each students overall degree result.
3.1. Benefits of using the student response system

The key benefit identified by 41% of the students was that using the SRS contributed to them
engaging with the lectures (Table 1) – both in terms of the material being taught and with
their classmates. Examples of responses identifying engagement, were:
• “Engagement in class and improving participation/attendance. Helped me stay

focussed. Feel like opinion listened to.”
• “From being forced to be engaged (at the beginning), to really getting engaged into
this module, serious thinking, etc.”
• “It has led to me being much more engaged with the lecture as it is more interactive.
Without any sort of engagement in lectures I tend to stop paying attention, so I have
really enjoyed engaging with the content in real time rather than just in assignments”
198
Willie Golden
Table 1. Benefits of using student response system.
Benefits identified by students from using Vevox? Responses % of Responses
Engagement in lectures 23 41.07%
Learning Reflection during lectures 16 28.57%
Anonymity 6 10.71%
Critical Thinking 3 5.36%
Active Participation 3 5.36%
Answer Comparison 3 5.36%
Class Participation Marks 2 3.57%
Total 56 100.00%
The second most important benefit identified was that the questions posed enabled the
students to reflect on their learning while in the lecture. Some of the answered classified
under this benefit are below:
• “It got me to think more deeply during the lectures, it helped me to start a discussion
in my own head and listening to other student’s mental models afterwards for me to
think differently”
• “It helped me to actually think about the lecture material rather than just listen. So,
I do think this type of engagement helped me learn because the info sticks better.”
Six of the respondents identified that the anonymity of the system – the fact that who gave
the answer was never identified when answers were shown on-screen in class – was an
important benefit for them. One example of a response that specifically cited the benefit of
anonymity was:
• “I enjoyed using Vevox, it allowed students to answer questions freely and without
having to shout out answers in class which many people would be uncomfortable
with.”
A benefit that was identified by 3 students was that the use of SRS to pose questions enabled
them to think critically, one example of a response is: “As explained by the lecturer, several
polls were given to challenge our thinking and critically discuss several topics.” Three
students stated that the SRS enabled them to actively participate in the lectures, e.g. of a
response: “It has allowed for active participation and has given me the opportunity to share
my views on various topics.” The capability of the system to immediately show on screen
other students answers to the question was a benefit directly identified by 3 students. Text of
one such response is: “The ability to get an insight into other students’ thoughts & opinions
199
and to allow me to critically think about certain questions that I would not have thought about
before.”
Looking at the totality of the responses from students with respect to the benefits they
believed they obtained from the SRS, it should be noted that none answered that they had
received no benefit – even though this was allowed for in the question as it started with the
words: “What benefits (if any).” The overall picture of what emerges from an overview of
the answers is that the students benefited from the system as it enabled them to actively
engage in and contribute to their learning while in the lecture and they could do this in a way
that respected their answers by maintaining their anonymity.
3.2. Benefits to students learning

The second question asked, was more specific – asking students to reflect on how they
believed the system had helped their learning. The answers given were very similar in opinion
and focus to the responses to the first question on overall benefits. The similarity of the
answers to the two questions strengthens the argument that the students saw the SRS as
facilitating and enabling their learning, rather than just something that only had the functional
purpose of recording their attendance and assessing them, or just being entertaining.
The categorised responses are shown in Table 2 below. Looking at the responses collectively,
what emerges is that the students believed the SRS helped their learning by facilitating them
in being active, engaged and reflective learners while being in the lecture hall. The benefit of
obtaining continuous assessment marks because of answering the questions posed was only
identified by 5% of the students as a way the system had helped their learning.
Table 2. How students believed the SRS helped their learning.
How helped Learning? Responses % of Responses
Active Learning in lectures 18 34.62%
Active Reflection in lectures 13 25.00%
Engaged learning 9 17.31%
Critical Thinking 6 11.54%
Class Participation Marks 3 5.77%
Other Answers 3 5.77%
Grand Total 52 100.00%
200
Willie Golden
3.3. Students suggestions for improvement

The extant literature on the use of SRS talks a lot about the importance of the questions asked
in terms of the benefits obtained by students. The third question asked sought to ascertain
what ideas the students might have in terms of questions and/or suggestions that would better
improve their learning. The categorisation of the answers given is shown in Table 3 below.
During the module, the students answered 40 questions, yet none of them suggested a way to
improve things was to ask fewer questions, instead 21% suggested asking more questions. A
lot of the suggestions for improvement was on better question construction – with some
wanting more focused questions – such as multiple choice questions, but others looking for
more open ended questions which might be posed at the end of one lecture to be answered at
the start of the following weeks lecture.
10% of the respondents would have liked to see more discussion in class on the answers
provided. This was done on a few occasions during the module, but the suggestion indicates
that students found benefit in this and allowing for more of this is likely to further increase
the sense of engagement and active learning experienced by the students. One of the features
within Vevox is that the answer to a question can be displayed as a word cloud. To enable
the word cloud to be displayed effectively, Vevox limits each individual answer to 20
characters. This character limit on answers was cited by 3 students as something that they
suggested might be usefully removed to allow a more complete answer.
Table 3. Suggestions for improvement.
Suggestions for improvement Responses % of Responses
No Suggestions 13 25.49%
Improved Question Construction 13 25.49%
More Questions 11 21.57%
More discussion in class on answers provided 5 9.80%
No Character Limit on Answer 3 5.88%
More Time to Answer 3 5.88%
Use for Assessment 2 3.92%
Vevox Software Capability Improvement 1 1.96%
Total 51 100.00%
201
4. Conclusion
The use of a student response system consisting of open-ended questions to enable in-class
interaction with students in a large lecture is something that is highly regarded by the students
as being beneficial to their learning experience. The answers given by students via the SRS
were graded and could account for up to 15% of their final grade, yet this benefit from the
system was only cited as a benefit by less than 10% of the students. Instead, the key self-
selected benefits that students identified were active engagement, active participation and
real-time feedback on their answers relative to their peers. This case study clearly shows that
used in the correct way asking open-ended questions via a SRS enable simultaneous
individual student engagement and learning in a large classroom setting.
References
Bogdanović, Z., Barac̈, D., Jovanic̈, B., Popovic̈, S., & Radenkovic̈, B. (2014). Evaluation of
mobile assessment in a learning management system. British Journal of Educational
Technology, 45(2), 231-244. doi:10.1111/bjet.12015
Chou, P. N., Chang, C. C., & Lin, C. H. (2017). BYOD or not: A comparison of two
assessment strategies for student learning. Computers in Human Behavior, 74, 63-71.
doi:10.1016/j.chb.2017.04.024
Kay, R. H., & LeSage, A. (2009). Examining the benefits and challenges of using audience
response systems: A review of the literature. Computers and Education, 53(3), 819-827.
doi:10.1016/j.compedu.2009.05.001
Nikou, S. A., & Economides, A. A. (2016). The impact of paper-based, computer-based and
mobile-based self-assessment on students' science motivation and achievement.
Computers in Human Behavior, 55, 1241-1248. doi:10.1016/j.chb.2015.09.025
Pimmer, C., Mateescu, M., & Gröhbiel, U. (2016). Mobile and ubiquitous learning in higher
education settings. A systematic review of empirical studies. Computers in Human
Behavior, 63, 490-501. doi:10.1016/j.chb.2016.05.057
Spector, J. M., Ifenthaler, D., Sampson, D., Yang, L. J., Mukama, E., Warusavitarana, A.
Gibson, D. C. (2016). Technology enhanced formative assessment for 21st century
learning. Educational Technology and Society, 19(3), 58-71. Retrieved from
https://www.scopus.com/inward/record.uri?eid=2-s2.0-
85000365267&partnerID=40&md5=cc86917cd1b6eb2271e4bd27ac990dfd
Wood, R. D., & Shirazi, S. D. (2020). A systematic review of audience response systems for
teaching and learning in higher education: The student experience. Computers and
Education, 153. doi:10.1016/j.compedu.2020.103896
202
A mixed reality laboratory for developing competencies in control

engineering
Manuel Navarro-Gutiérrez1, Carlos Renato Vázquez2, Alejandro Guajardo-Cuéllar2,
Noé Marcelo Yungaicela-Naula2
1
Institute of Advanced Materials for Sustainable Manufacturing, Tecnologico de Monterrey,
Mexico, 2Escuela de Ingenieria y Ciencias, Tecnologico de Monterrey, Mexico.
Abstract
This work presents a novel mixed reality laboratory for the development of
competences in control engineering. This laboratory simulates various control
systems that may interact with different control devices. The mixed reality lab
has three components: (i) a virtual environment, (ii) a virtual-electronic
interface, and (iii) a control unit. The virtual environment displays a virtual
representation of the system under control. The electronic interface uses a
microcontroller to solve the differential equations that models the simulated
system. The values solved by this interface are sent to the virtual environment
for driving its animation. Moreover, input and output signals are connected to
the control unit for the implementation of a control law. The functionally of the
proposed mixed reality laboratory is showcased by modeling and controlling
a Planar Vertical Take-off and Landing (PVTOL) system. Furthermore, the
motivation of a group of students, after experimenting with the laboratory, is
reported.
Keywords: Mixed reality labs; virtual systems; educational innovation; higher
education.

A mixed reality laboratory for developing competencies in control engineering
1. Introduction
In the engineering education community, there is a consensus that laboratory experiences are
fundamental to learning processes. Through these experiences, students develop relevant
procedural and disciplinary competences since they offer an interactive learning strategy. In
recent years, universities worldwide seek innovative solutions to reconfigure or succeed
traditional laboratories. The area of control and automation engineering has been an essential
field for the implementation of innovative solutions to overcome the limitations of
conventional laboratories. In general, control and automation theory is abstract and difficult
to understand, since it consists mainly of theoretical models. Although traditional hands-on
laboratories allow students to associate theoretical concepts with physical phenomena, they
have some limitations (Hernández-de-Menéndez et al., 2019). The most common limitations
are: (1) equipment for control engineering experiments is often expensive; (2) equipment
needs to be shared by students in groups; (3) both laboratory opening hours and space are
limited; (4) experimental sessions are always detached from theoretical courses; and (5)
experimentation with real systems could be risky.
The rise of virtual and remote laboratories is removing the limitations of traditional hands-
on laboratories (Li et al., 2020; Macías García et al., 2020; Morales-Menendez et al., 2019;
Tejado et al., 2021). Among these laboratories, those that use mixed reality and augmented
reality laboratories have demonstrated several advantages over others because they combine
the benefits of physical and virtual components (Liang & Liu, 2018; Zata et al., 2017). In
particular, mixed reality laboratories have been successfully applied to train technicians in
different fields, such as medical procedures (Aebersold et al., 2018; Barsom et al., 2016),
industrial maintenance and assembly (Gavish et al., 2015), among others. Regarding control
engineering, mixed reality laboratories have shown significantly improves for learning
specific concepts (Frank & Kapila, 2017).
This work presents a mixed reality laboratory that aims to develop the procedural and
disciplinary competences of engineering students taking control engineering courses.
Engineering competencies are important skills and abilities required to solve modern
challenges (Guajardo-Cuéllar et al., 2020, 2022). Control engineering is not exempt from
these modern challenges. The proposed laboratory has three elements: (i) a virtual
environment that simulates the system to be controlled, (ii) a control unit that uses a physical
electronic device to control the simulated system, and (iii) an electronic interface that
communicates the virtual environment and the control unit. A Planar Vertical Take-Off and
Landing (PVTOL) system is used to showcase the functionality of the proposed laboratory.
Furthermore, seventeen students from the Tecnologico de Monterrey used the laboratory and
reported their experience in a survey. The students found the mixed reality laboratory very
useful, easy to use, and realistic.
204
Manuel Navarro-Gutiérrez et al.
2. Virtual Environment
This section describes the virtual environment. Electro-mechanical systems are designed
using a CAD software, then they are imported to a game engine for creating the virtual
environment.
Figure 1 Two systems designed with a CAD software.
For the development, the systems must be firstly designed using a CAD software. Two
examples are shown in Figure 1: (left) a Planar Vertical Take-Off and Landing (PVTOL)
system and (right) an inverted pendulum. The PVTOL system consists of a beam that rotates
about its central axis, whose movement is induced by the thrust generated by the rotors
located at both ends of the beam. This system is a simplified version of a quad-rotor, in which
engineering students can apply control concepts. The second system is an inverted pendulum
on a cart driven by a direct current (DC) motor. The center of mass of the pendulum is above
the pivot point. Therefore, the pendulum is unstable without a control algorithm. Through
this system, engineering students can understand and apply stability concepts that arise in
applications such as rocket guidance.
The CAD models of the two systems are then exported to Unity, a well-known game engine,
where they are integrated with different assets in a virtual environment. Afterward, Unity
generates an application containing the virtual environment, which can be executed in a
personal computer, tablet, or smartphone (a link to a video is provided in the next section).
3. Electronic Interface
This section shows how the differential equations of a system can be implemented using a
microcontroller. A numerical method for solving the equations of the system is codified on
an Arduino board. It is enabled for sending the data to the application via serial
communication, it also writes these values on the electrical outputs of the microcontroller.
3.1. Mathematical model implementation

The system's dynamics is solved in the electronic interface. In detail, the model of the PVTOL
system is:
205
𝑑! 𝜃(𝑡) 𝑑𝜃
𝐽 = 𝑙)𝑓" (𝑡) − 𝑓! (𝑡), − 𝐶 (1)
𝑑𝑡 ! 𝑑𝑡
where 𝜃 is the angle of the beam, 𝐶 is the viscous friction coefficient, 𝑙 is the distance from
the center of the beam to the rotors, 𝐽 is moment of inertia of the system, and 𝑓" and 𝑓! are
forces on both ends of the beam. 𝑓" and 𝑓! depend on the rotor speeds as follows:
𝑓# = 𝛼𝑢#!
where 𝑢# is the speed of the 𝑖-th rotor, and 𝛼 is a proportional constant that relates the force
and speed of the motor. Equation (1) is solved and implemented in an Arduino Mega board
that is included in the electronic interface.
3.2. Connection with the virtual environment

In order to animate the simulated system, it is necessary to send the system variables
computed by the microcontroller to the virtual environment.
The virtual environment was programmed for receiving the data via serial communication
and display it. Moreover, the values are used to animate the simulated system. In the case of
the PVTOL, the application obtains the values of aforementioned variables: θ, ω, u" , and u! .
Moreover, the virtual beam is rotated according to the angle θ, received by the application.
3.3. Connection with the control unit

In order to interact with electronic controllers, the interface microcontroller must read input
signals from certain analog inputs, and it must provide output signals through analog outputs.
For the inputs, the microcontroller reads ports attached to Analog to Digital Converters
(ADCs), obtaining signals in the range from 0 to 5 Volts. The input signals can be provided
by a control unit (microcontrollers, PLCs, DSPs, FPGAs, etc.) or other electronic devices,
such as potentiometers and operational amplifiers.
On the other hand, the interface provides analog signals representing values of state variables
(e.g., angles, velocities and forces). These values are provided by the microcontroller as pulse
width modulation (PWM) signals. Then, an electronic circuit transforms the PWM signals
into analog ones, by using Resistor-Capacitor (RC) filters. Thus, these electrical signals can
be read by the control unit, emulating electronic sensors.
3.4. Mixed reality lab demonstration

The following link opens a video about a simple simulation of the PVTOL system using the
virtual environment and potentiometers as external signals: https://youtu.be/RNu05flYITs
The video shows the application running in a personal computer. First, the application is
initialized and the “Balancín” option is selected to open the virtual classroom containing the
206
PVTOL system. Then, the user walks around the classroom using the keys ‘a’, ‘s’, ‘d’, ‘w’,
and the mouse. Later, the serial port “COM14” of the computer is connected with the
application, this is the port where the Arduino board is also connected. At this moment, the
sound of two propellers starts because the application is now receiving the data corresponding
to the rotor speeds from the microcontroller. These rotor speeds are defined by the value of
two potentiometers that can be observed in the second screen (displayed at the bottom right
of the video). Then the value of left potentiometer (left rotor) is increased, this produces a
negative torque on the beam, and consequently it rotates until it reaches -45° (this is a limit
value imposed in the code). After that, the value of the right potentiometer (right rotor) is
increased. When it overpasses the left potentiometer value, it produces a positive torque on
the beam, and consequently the beam rotates until it reaches 45°. Finally, the user tries to
balance the beam by regulating the speed of the right rotor; however, this is a very difficult
task for a human being because the system is unstable and the user fails to balance the beam.
The virtual environment and the electronic interface comprise the Mixed Reality Labs
proposed in this work. The former provides two significant advantages: it is possible to
design and simulate several engineering systems, and it provides a visualization of the system
behavior. The latter allows to interact physically with the simulated system, which can be
controlled using a wide variety of control devices.
4. Control Unit
Mixed Reality Lab
External Controller
inputs
outputs
Figure 2: Scheme for connecting the Mixed Reality Lab to a control unit.
Figure 2 shows a scheme with the connections between the mixed reality lab and an external
microcontroller, which in this particular case is an Arduino Uno board. For the PVTOL
system, the outputs are the beam angle θ and its velocity 𝜔, their electrical signals are sent
from the mixed reality lab to the external controller. In the same way, the inputs of the
PVTOL system are the rotor speeds 𝑢" and 𝑢! , which are driven by the electrical signals
received from the external microcontroller.
The output variables of the system are used by a Proportional-Integral-Derivative (PID)
control algorithm for computing an actuator signal effort, necessary for maintaining a
207
required reference value. This signal is provided as an input to the system. The PID algorithm
is running in an Arduino Uno board, for maintaining the angle of the beam at a required
value. The following link is a video showing an implementation of this controller, it maintains
the beam angle in different required positions: https://youtu.be/WDKJmbDNWWM
First, the user opens the application and connects the Arduino Mega that simulates the
PVTOL. Then, the user uploads the PID controller code in an Arduino Uno board and
launches a plotter for visualizing the required angle and the current angle of the PVTOL
system. During the experiment, the video shows the virtual environment and the plotter at
the same time. In the plotter, the red line represents the required angle for the beam, the values
are -20°, -10°, 0°, and 10°. The blue line represents the current angle of the PVTOL system.
This behavior can be appreciated in the virtual environment, where the angle of the PVTOL
system is maintained. The main characteristic of the control unit that commands the behavior
of the virtual system is its versatility.
5. Implementation with students

The mixed reality laboratory was used in a group of undergraduate engineering students from
the Tecnologico de Monterrey, in order to asses their motivation when working with this
platform. The group included seventeen students of the Mechatronics Engineering program.
This project has the approval of the experimental protocol submitted in December 2022 and
approved in January 2023. It is important to note that the sample used for this study
previously signed an informed consent. After the experimentation with the mixed reality
laboratory, a survey was applied to the students to measure their motivation regarding this
laboratory. The formulation of the survey questionnaire was based on similar work on remote
control laboratories (Achuthan et al., 2021; Liang & Liu, 2018; Tejado et al., 2021). The
reliability of the questionnaire used was validated in previous studies (Achuthan et al., 2021),
using the Cronbach's Alpha method and the instrument was found to be reliable (𝛼 > 0.79).
The survey questionnaire consisted of eleven positive claims, regarding the impact of the
laboratory in three aspects: (1) learning motivation; (2) comprehension without supervision;
and (3) realistic experimentation. Seventeen students answered the survey, Figure 3 shows
the scores averages grouped by aspects.
208
5.00
4.50
4.00
3.50
3.00
2.50
2.00 3.93 3.77
1.50 3.42
1.00
0.50
0.00
Learning mo4va4on Comprehension Realis4c
without supervision experimenta4on
Figure 3: Score averages of the survey claims grouped by the aspects.
The results of the survey demonstrated that, through the use of the mixed reality laboratory,
the students were motivated to learn about the implementation of control strategies on
hardware.
6. Conclusions
A mixed reality laboratory for control engineering courses was described in this work, which
combines simulations of control systems in a virtual environment and an electronic device
that functions as interface with external control units. This scheme allows students to practice
the implementation of any kind of controller, in any kind of electronic device. Moreover, the
motivation of students working with the laboratory was assessed. The measurement of
competencies development, regarding control engineering, is left as future work.
Acknowledgments
The authors would like to acknowledge the financial support of NOVUS (Grant number:
N21-219), Institute for the Future of Educations, Tecnologico de Monterrey, Mexico, in the
production of this work. The authors would like to acknowledge the financial support of
Writing Lab, Institute for the Future of Education, Tecnologico de Monterrey, Mexico, in the
production of this work.
References
Achuthan, K., Raghavan, D., Shankar, B., Francis, S. P., & Kolil, V. K. (2021). Impact of
remote experimentation, interactivity and platform effectiveness on laboratory learning
outcomes. International Journal of Educational Technology in Higher Education, 18(1).
https://doi.org/10.1186/s41239-021-00272-z
Aebersold, M., Voepel-Lewis, T., Cherara, L., Weber, M., Khouri, C., Levine, R., & Tait,
A. R. (2018). Interactive Anatomy-Augmented Virtual Simulation Training. Clinical
Simulation in Nursing, 15. https://doi.org/10.1016/j.ecns.2017.09.008
209
Barsom, E. Z., Graafland, M., & Schijven, M. P. (2016). Systematic review on the
effectiveness of augmented reality applications in medical training. Surgical Endoscopy,
30(10). https://doi.org/10.1007/s00464-016-4800-6
Frank, J. A., & Kapila, V. (2017). Mixed-reality learning environments: Integrating mobile
interfaces with laboratory test-beds. Computers and Education, 110.
https://doi.org/10.1016/j.compedu.2017.02.009
Gavish, N., Gutiérrez, T., Webel, S., Rodríguez, J., Peveri, M., Bockholt, U., & Tecchia, F.
(2015). Evaluating virtual reality and augmented reality training for industrial
maintenance and assembly tasks. Interactive Learning Environments, 23(6).
https://doi.org/10.1080/10494820.2013.815221
Guajardo-Cuéllar, A., Cárdenas, D., & Lepe, M. (2020). Fabrication of a Boomerang Using
Computational Tools to Develop Engineering Competences. International Journal of
Mechanical Engineering and Robotics Research, 9(10).
https://doi.org/10.18178/ijmerr.9.10.1406-1410
Guajardo-Cuéllar, A., Vázquez, C. R., & Gutiérrez, M. N. (2022). Developing Competencies
in a Mechanism Course Using a Project-Based Learning Methodology in a
Multidisciplinary Environment. Education Sciences, 12(3).
https://doi.org/10.3390/educsci12030160
Hernández-de-Menéndez, M., Vallejo Guevara, A., & Morales-Menendez, R. (2019).
Virtual reality laboratories: a review of experiences. International Journal on Interactive
Design and Manufacturing, 13(3). https://doi.org/10.1007/s12008-019-00558-7
Li, C., Zhao, R., Wang, Y., Liu, Q., & Dong, D. (2020). Module-based virtual lab design
model with matlab/simulink in automation control theory. ACM International
Conference Proceeding Series. https://doi.org/10.1145/3380625.3380638
Liang, Y., & Liu, G. P. (2018). Design of Large Scale Virtual Equipment for Interactive HIL
Control System Labs. IEEE Transactions on Learning Technologies, 11(3).
https://doi.org/10.1109/TLT.2017.2731772
Macías García, M. E., Cortés Pérez, A. A., & Izaguirre Alegría, A. R. (2020). Cyber-Physical
Labs to enhance engineering training and education. International Journal on Interactive
Design and Manufacturing, 14(4). https://doi.org/10.1007/s12008-020-00704-6
Morales-Menendez, R., Ramírez-Mendoza, R. A., & Guevara, A. J. V. (2019).
Virtual/Remote Labs for Automation Teaching: A Cost Effective Approach. IFAC-
PapersOnLine, 52(9). https://doi.org/10.1016/j.ifacol.2019.08.219
Tejado, I., González, I., Pérez, E., & Merchán, P. (2021). Introducing systems theory with
virtual laboratories at the University of Extremadura: How to improve learning in the lab
in engineering degrees. International Journal of Electrical Engineering and Education,
58(4). https://doi.org/10.1177/0020720919876815
Zata, N. M., van Niekerk, T. I., & Fernandes, J. M. (2017). A process control learning factory
with a plant simulation integrated to industry standard control hardware. 2016 Pattern
Recognition Association of South Africa and Robotics and Mechatronics International
Conference, PRASA-RobMech 2016. https://doi.org/10.1109/RoboMech.2016.7813175
210
How students manage peer feedback through a collaborative

activity in a CS1 course
Géraldine Brieven1, Laurent Leduc2, Benoit Donnet1
1
Institut Montefiore, Université de Liège, Belgium, 2IFRES, Université de Liège, Belgium.
Abstract
In order to boost students’ motivation in practicing their problem-solving skills
and give them opportunities to get feedback, we broke our CS1 course routine
with a disruptive cross-skilling activity. It relies on collaboration between
teams of students where peer feedback (using rubric) stands as the cornerstone
to design and build a solution responding to a given problem.
This paper aims at formally assessing the peer feedback process across three
activity sessions. It also highlights the different success factors supporting peer
feedback in that context through a cause and effect diagram. We show that
peer feedback fosters primary problem-solving foundations. We also discuss
its limitations, namely due to an insufficient granularity in the provided
checklist as well as a lack of transversal skills from students, making them less
comfortable with peer feedback. Although, by repeating the activity, students
could manage it better and better and take more advantage of peer feedback.
Keywords: Peer feedback; team-based learning; checklist; problem-solving;
CS1.

How students manage peer feedback through a collaborative activity
1. Introduction
In a Introduction to Programming Course (CS1) dedicated to First Year students, a particular
importance is given in teaching a problem-solving method as the resulting skills are crucial
in the long run to be able to handle new challenges (Choudhar et al., 2022). Although, in
practice, students often lack motivation during traditional programming exercise sessions,
taking lots of time to solve a given exercise and missing feedback (Sharmin et al., 2020).
To overcome that issue, the Collaborative Design and Build (CDB) activity (Brieven et al.,
2022) was deployed in our CS1 course. That activity implements Assembly Line Learning
(Rosario et al., 2020) and Team-Based Learning (Burgess et al., 2021) where peer feedback
is drawn on a checklist. To motivate that process, CDB was designed such as the reviewers
need to give a feedback as clear and as precise as possible since they will have to rely on the
productions they have reviewed to progress the solution. Previous work (Brieven et al., 2022)
has shown that students feel motivated in taking part to CDB, due to its social dimension and
its authentic aspect. CDB also appeared to boost students’enhancement in solving problems.
This paper addresses the particular focus of peer feedback, aiming to assess it and identify
how it could be optimized. This is tackled through two research questions: (RQ1) How
relevant is peer feedback in the context of CDB in a CS1 course ? (RQ2) Which prerequisites
influence the feedback process in that context ? Answering those questions, this paper shows
that students provide correct feedback based on a given checklist (RQ1) while their capacity
to integrate it is moderate (RQ1). To explain that gap, we namely point out a lack of accuracy
of the checklist criteria and students’ difficulty in communicating. This diagnosis is
corroborated through a cause and effect diagram depicting the feedback process in CDB in
relation with other dimensions emerging from the discussion related to RQ1, such as
students’ skills (not only disciplinary) or the activity parameters (like the checklist) (RQ2).
2. CDB Activity
The CDB activity (Brieven et al., 2022) is made up of two phases: the Design (solution
design) and the Building phase (solution implementation). Fig. 1 shows how CDB is setup.
The right side of Fig. 1 (“Classroom Configuration”) illustrates that, for N participants, the
CDB activity relies on G Groups of students, each Group being split in T Teams (each Team
comprising S students, with S ≥ 2). The goal of each Group is to solve T problems in a limited
amount of time. The left part of Fig. 1 draws how the T problems are getting progressively
solved, in parallel, over time, following the T steps required to frame a problem-solving
process. This conception is inspired by real professional life as, in large-scale development
projects, the different steps in solving a problem are performed by different teams.
212
Design Phase Building Phase

Transition 1 2 Transition T-1 T
Feedback Loop Feedback Loop
Classroom Configuration
criteria OK ? Comments criteria OK ? Comments
c1 c1 Group 1 Group G
Step 1 c2
Step 2 c2
Step T
... ... ...
cC cC
Owner of Team 1 Team 1

production Reviewing productionT production Reviewing production B production
Problem 1 A B ...
Adapting production A T Adapting production C
Owner of Team 2 Team 2

production Reviewing production A production Reviewing production C production
...
Problem
B Adapting production B A Adapting production D C
2
... ...
...
...
...
...
Owner of Team T Team T
Problem production Reviewing production T-1 production
Reviewing production A production
T Adapting production T T-1 Adapting production B A ...
T
00h00 <00h30 <01h <01h20 <01h40 02h00

Resolution over time
Figure 1. CDB generic set-up in the particular instance of two students per Team (S=2).
Going into more detail, at the beginning, each Team receives and gets responsible for one
problem. For a given problem, the T steps are sequentially addressed, each Team being busy,
turn-by-turn, with a specific one. At the end of each step, each Team work moves to the next
Team, clockwise, similarly in each Group, as depicted by the plain arrows (Fig. 1, right part).
Then, a transition period is dedicated to allow each Team (reviewers) to report a feedback
about the production provided by the previous Team (submitters). During that period, every
Team holds two roles in parallel. For instance, Team 2 is reviewer of Team 1 (as expressed
through the dashed arrow (Fig. 1, right part) linking the reviewers to the submitters) while it
is submitter of Team 3. Note that the motivation behind that feedback loop is to limit the
impact of a “poor quality work” on the next productions that are based on the previous ones.
Figure 2. Checklist related to the first step.
213
Table 1. CDB parameters values.
Session N G T S
1 63 10 3 2 or 3
2 49 8 3 2 or 3
3 48 8 3 2
The feedback is based on a rubric checklist (Bharuthram & Patel, 2017), as shown in Fig. 2.
The first column lists the criteria a step output should meet. They are picked from the rubric
supporting the course evaluations. In CDB, they are kept quite general, as the purpose is
making students responsible for putting forward a solution rather than gradually disclosing it
through the criteria. The next two columns allow each Team to specify whether a criterion is
checked or not and attach comments to it. Once filled, the checklist is returned to the
submitters who should adapt their work based on the boxes and comments in the checklist.
3. Method
Over the semester, three CDB sessions were organized, with an increasing complexity of
problems to solve from one session to another. It is worth mentioning that participation was
not mandatory. Table 1 summarizes the different paremeters values for each CDB session.
We collected data during those three sessions. In particular, on the one hand, an anonymous
survey was addressed to students at the end of each session. Every survey included Likert
scale questions, related to different aspects of the activity. On the other hand, all students’
productions and feedbacks were collected and analysed afterwards. More specifically, first,
each step production was qualified as reliable enough or not to support further the solution.
Legend : 0 1
Reviewers Educational Team Feedback Correctness Rate Improvement Rate

Box B1 B2 B3 ipi = (B1ci== 0 && B3ci==1)
fci= (B1ci==B2ci)
Version 1 of criteria OK ? Comments OK ? OK ? = ( ? )
the solution
c1 B1c1= ... B2c1= B3c1= fc1=1 ( == ) ipc1 = 0 ( )
c2 B1c2= ... B2c2= B3c2= fc2=0 ( == ) ipc2 = 0 ( )

... ... ... ... ...
Version 2 of cC B1cC= ... B2cC= B3cC= fcC=1 ( == ) ipcC = 1 ( )
the solution
Submitters
Figure 3: Illustration on how feedbacks were evaluated by the education team.
214
Next, each reviewers’ checklist (reflecting the feedback) (see Fig. 2) was assessed. Fig. 3
depicts that process through a matrix in which the first column lists the C criteria related to a
given step (each one being labeled ci). The next column contains the boxes selected by the
reviewers. Those are compared to the ones checked by the teacher (referred by the column
“B2”) used as benchmark to derive the Feedback Correctness Rate (FCR). Finally, the last
column represents the checklist filled from the refined solution. It allows to compute the
Improvement rate (IR), measuring how much the reviewers’ feedback conducted submitters
in meeting the unchecked criteria. For both rates, closer to 1 the better, closer to 0 the worst.

Fig. 4 exposes a cause and effect diagram where peer feedback is evaluated in CDB (RQ1).
Then, the factors a successful feedback loop relies on are distilled (based on inferences
emerging from RQ1 and the surveys’ outcomes), classified, measured (on the grounds of
students’ views, at each session), and put in relation with each others (RQ2). Those links are
quantified through the Pearson coefficient and its corresponding p-value.
Transversal Skills r=0.3, p=1.62e-04
Problem-solving Communication Analyse someone's production

S1 : S1 : S1 :
Skills S2 :
S3 :
S2 :
S3 :
S2 :
S3 :
r= 6 06
0.2 e0 2e
8, =2 p=
r=0
,p
03
p= 9 7,
5e 3
0.3 0.
4e-
-0
4 r= r=
r=0.35, p=8e-04
.79,
p=
Activity Fitting in the timing Granularity of the criteria

0,
p=3
0.3
Management
S1 :
e-21
S1 :
r=
S2 : S2 :
S3 : S3 :
r=0.18, p=9e-02
r=0.33, p=3e-05
Step S1 to S2:
Better Production Quality
Productions S2 to S3 :
r=0.44,p=1e05
Successful Feedback Loop

Relevance of feedback Feedback Integration
Peer
r=0.3, p=
S1 : 86%± 2% S1 : 26%± 9%
S2 :
S2 : 83%± 3% 21%± 8%
Feedbacks S3 :
80%± 6%
S3 :
46%± 20%
3e03
FCR IR
r=0.22, p=5.75e-03
Students' Opinions
Strongly agree
Reliable foundations for the next steps
Agree
Intermediate 58% 43% 65% Neither agree or disagree
Disagree
Solutions session1 session2 session3 Strongly disagree
Figure 4. Cause and effect diagram depicting the feedback process in CDB.
215
4.1. (RQ1) How relevant is peer feedback in the context of CDB in a CS1 course ?
Considering the main frame of Fig. 4 (“Peer Feedbacks”), first, it shows that students seem
quite mitigated about the relevance of the feedbacks they received during the first sessions.
However, from the second to the last one, 30% of students found it richer, which is consistent
to how comfortable they felt in integrating the feedback, rising from 49% to 72%. To further
corroborate students’ perception, it was put in perspective with results derived from their own
productions and feedbacks. There, in contrast to students’ opinion, a constant high FCR was
computed. That overestimation with respect to students’ opinion may be due to the fact that
students missed tags in their production and comments as more accurate guidance. Those two
feedback components were not involved in the FCR computation while they were taken into
account in students’ view. This explanation gets confirmed by noting that, over all the filled
checklists, only 10% of criteria had a comment on the side. That may have prevented students
from clearly identifying and fixing their gaps, especially those related to more complex
criteria. Finally, the distribution of students behind each FCR mean is provided in Fig. 5a.
1.0 1.0
Cumulative Students Proportion
Cumulative Students Proportion

Session 1
Session 2
0.8 Session 3 0.8
0.6 0.6
0.4 0.4
Session 1
0.2 0.2
Session 2
Session 3
0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Feedback Correctness Rate Improvement Rate
Figure 5. Cumulative distribution of students (a) emitting a correct feedback and (b) improving from a given
feedback.
It namely shows that, for the three sessions, 60% of reviewers checked at least 80% of the
criteria correctly. It substantiates the gap between how submitters perceive the feedback and
the actual feedback correctness in terms of checked boxes. In response to that, the list of
criteria could be refined to tight better the expected solution (although it should not disclose
it) and the reviewers should develop their feedback further too (through more comments
typically). Next to this, Fig. 3 illustrates that the average IR remained limited (less than 30%)
across the two first sessions but got better in the last one (reaching 46% (± 20%)). However,
the large confidence interval related to that last result reflects the heterogeneity of submitters’
perception in adapting to the feedback. That heterogeneity is distilled through Fig. 6b
depicting that 25% of students did not improve at all, 35% improved on 50% to 80% of the
criteria that were not inially met and the rest of the submitters standed in between. More
generally, considering the three sessions, it can be noticed that submitters are likely to
enhance from the feedback, but remaining gaps are persisting. Possibly, some students were
216
missing time in tailoring their solution. Those results get consolidated seing the proportions
of reliable refreshed production being above the average. However, they do not reach top
values, which confirms that further refinements are still needed and suggests that students
may lack problem-solving skills. It also converges to previous findings stating that feedback
intervention shifts the attention away from the task itself and closer to the self (referring to,
e.g., motivation, critical-thinking, communication) (Kluger and DeNisi (1996)), which is also
partially highlighted through RQ2. From an evolution point of view, the proportion of reliable
productions rises from 58% to 65% over the sessions despite the last session proposed harder
statements, which emphases the production quality improvement over the CDB sessions.
4.2. (RQ2) Which prerequisites influence the feedback process in the context of CDB?
Let us now strengthen the inferences stated in Sec. 5.1 by studying the factors assumed as
influencing peer feedback. They are grouped in categories defined upon “Peer Feedbacks” in
Fig. 5, and ordered based on how they impact on each others. Typically, students’ “Skills”
influence the way students handle the activity parameters (included in “Activity
Management”). Then, student’s ability to build and digest a feedback depends on the quality
of the submission as well as how comfortable they are in managing the activity and
mobilizing their skills.
In practice, all the components belonging to the three upper floors follow a similar trend than
the feedback dimensions, getting better and better managed by students across the sessions,
except the problem-solving skills. It is likely due to the nature of the problems that had to be
tackled with a more formal angle students were not used to yet in the last session. Although,
thanks to better transversal skills (Borova et al., 2021), students could counterbalance that
difficulty by dealing better with the activity itself. More precisely, first, 80% of the students
were initially struggling in fitting to the timing and, eventually, 50% were still facing that
issue. That last proportion partially explains why the IR remains under the average : likely,
besides some misunderstanding of the feedback submitters received, they lacked of time to
analyze and integrate it. Next, the criteria of the checklist sounded missing accuracy for about
half of the submitters (although their perception got more moderate), which comforts the idea
that criteria should be refined. Finally, prior to that, students improved in collaborating with
each others by communicating better and feeling more comfortable in analyzing the
submitters’ production, allowing them to deal better with the activity itself and, eventually,
manage peer feedback, as underpinned through the positive correlation between those factors.
5. Perspective and Conclusion

To wrap up, CDB implements peer feedback embedded in a real-life scenario where students
get responsible for sequential tasks on successive problems. To solve them, they need to give
217
their best contributions,by providing valuable feedback and by shaping their outputs using
feedbacks they receive in order to consolidate their ground and the ones of their reviewers.
In practice, in the context of CDB, students are quite good at identifying gaps in submitters’
productions by relying on a checklist. However, digesting a feedback from reviewers appears
more difficult (as also raised and deepened in other studies (Carless & Boud, 2018)). This
paper explains that gap through three main reasons. First, students have difficulties in
managing the limited time allocated to peer feedback. To overcome that, one could split the
feedback loop period in two subparts to make sure students do not spend the whole period in
building the feedback. Next, some criteria of the checklist lack of accuracy. Finally, most of
the reviewers do not spontaneously distill them through further explanation, they only check
boxes. In response to that, on the one hand, some criteria could be detailed and, one the other
hand, students could improve their transversal skills that play a key role in peer feedback at
both sides. Over the three CDB sessions, students felt more comfortable with respect to those
last factors, which got reflected on the feedback process itself and the final productions
quality. It confirms that CDB stands as an essential building block in our teaching methods,
giving to students the opportunity to train transversal skills and learn from peer feedback.
References
Bharuthram, S., and Patel, M. (2017). Co-Constructing a Rubric Checklist with First Year
University Students: A Self-Assessment Tool. Apples - Journal of Applied Language
Studies, 11(4), 35–55. doi: 10.17011/apples/urn.201708073430.
Borova, T., Petrenko, V., Ved, T. and Pyvovarov, V. (2021). Developing Students’
Transversal Competences Through Cultivation of Health Literacy Competence.
Marketing of Scientific and Research Organizations. 40. 35-52. doi: 10.2478/minib-
2021-0008.
Brieven, G., Leduc, L., and Donnet, B. (2022). Collaborative Design and Build Activity in a
CS1 Course: A Practical Experience Report. In 8th International Conference on Higher
Education Advances. Universitat Polytecnica de Valencia.
doi:10.4995/HEAd22.2022.14386.
Burgess, A., Roberts, C., Lane, A.S., Haq, I., Kalman, E., Pappalardo, N., and Bleasel, J.
(2021). Peer Review in Team-Based Learning: Influencing Feedback Literacy. BMC
Medical Education, 21(1), 426. doi: 10.1186/s12909-021-02821-6
Carless, D., and D. Boud. (2018). The Development of Student Feedback Literacy: Enabling
Uptake of Feedback. Assessment & Evaluation in Higher Education 43 (8): 1315–1325.
doi:10.1080/02602938.2018.1463354.
Choudhar, S., Bi, N., Singh P. N., and Talwar P. (2022). Study on Problem Solving Skills
and Its Importance. World Journal of English Language, 12(3), 1-47. doi:
10.5430/wjel.v12n3p47.
218
Harackiewicz, J. M., Smith, J. L., and Priniski, S. J. (2016). Interest Matters: The Importance
of Promoting Interest in Education. Policy Insights from the Behavioral and Brain
Sciences, 3(2), 220–227. doi: 10.1177/2372732216655542.
Kluger, A. and DeNisi, A.. (1996). The Effects of Feedback Interventions on Performance:
A Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory.
Psychological Bulletin. 119. 254-284. doi: 10.1037/0033-2909.119.2.254.
Rosario, J., Lebowitz, D., Webb, A. L., Ganti, L., Vera, A., Macintosh, T., Walker, A., and
Rubeor, J. (2020). Assembly Line Education: A Novel Educational Technique for Today's
Learners. Cureus, 12(2). doi: 10.7759/cureus.7065.
Sharmin, S., Zingaro, D., and Brett, C. (2020). Weekly Open-Ended Exercises and Student
Motivation in CS1. 20th Koli Calling International Conference on Computing Education
Research. ACM. doi: 10.1145/3428029.3428035.
219
Facilitating international transdisciplinary collaboration in a

virtual academic exchange project
Department of Art and Design, Bauhaus-Universität Weimar, Germany.
Abstract
What challenges does virtual collaboration pose to fundamental conditions of
collaborative work such as building relationships and trust and establishing
balanced participation? We approached this question in a virtual seminar in
transdisciplinary Public Art across six international universities, with funding
from the German Academic Exchange Service (DAAD) International Virtual
Academic Collaboration (IVAC) program. Together with our partner
institutions and accompanied by an intercultural facilitator, we established a
collaborative methodology for teaching and learning. As means of
participation and relationship building, we developed a two- step method for
matching student teams and used collaborative tools such as playful exercises
and open agenda meetings. We conducted a qualitative study to monitor the
collaborative process throughout the course. We held semi structured one-on-
one interviews with fifteen students and analyzed their written reports. The
results illustrate how our methodology supported the students’ collaborative
creative practices. Additionally, we describe challenges and consider possible
solutions.
Keywords: Online teaching; collaboration; public art; art education;
relationship building; collaborative methodology; academic exchange.

Facilitating international transdisciplinary collaboration in a virtual academic exchange project
1. Introduction
Since the outset of the COVID-19 pandemic, online teaching has gained substantial
importance in higher education. While technical tools and solutions have quickly developed
and improved (García-Morales, 2021; Carrruana Martín et al., 2022), scholars have worked
to retain the social importance of relationships, trust and care in online teaching
environments. As Burke and Larmar (2021) review, students in online courses are more likely
to experience isolation and loss of a sense of identity and personal engagement with other
students and teachers than they are in face-to-face classes. In online education, opportunities
for casual encounters, personal exchange and spontaneous feedback are reduced. Recent
studies suggest that an environment for collaboration should be purposefully designed and
facilitated (Herrera-Pavo, 2021; Uukkivi et. al. 2022). Herrera-Pavo (2021) reviewed studies
on social and educational factors as a basis of virtual collaboration in higher education and
proposed that teachers assume not only the role of educators but also of mediators.
We designed a virtual teaching collaboration, with a focus on social qualities. In the realm of
interdisciplinary creative practice, our aim was to create an environment for participation and
free experimentation. We followed the extracurricular aim of forming a community of
practice in socially engaged public art. In the spring and summer of 2022, six international
universities co-developed and realized the joint seminar called “Public Arts Garage”,
integrating perspectives from artistic research, performative art, architecture, anthropology
and literature and language. The partnering universities included Bauhaus-Universität
Weimar, Concordia University Montreal, Rennes 2 University, Queen’s University Belfast,
University College Cork and University of Barcelona. The project served as a prototype
within a newly founded international graduate school on creative approaches to public space
(GS-CAPS). Students were matched across different countries and disciplines to develop
artistic projects in virtual exchanges that took place in public spaces in the respective
locations. Due to different academic schedules, the seminar was divided into two cohorts –
one with five and one with three partner universities – that followed the same basic
methodology. We followed an idea of deep collaboration, guided by our intercultural
facilitator Susanne Wille, who accompanied the entire process including course design and
moderation during the sessions. Her approach was informed by a relational understanding of
cultural complexity (Baumann Montecinos and Grünfelder, 2022) and focused on the quality
of attention and intention (Scharmer, 2016). With her support and with participation of all
partner institutions, we established a collaborative methodology, consisting of a two-step
process for matching student groups and collaborative tools like playful exercises and weekly
open-agenda meetings. In this qualitative study, we investigated what drove the collaborative
processes in the student groups, how our methodology supported these processes and what
challenges emerged in virtual group work.
222
2. Methods
The data collection was undertaken as a means of evaluation and as part of the lead author’s
doctoral thesis. The qualitative study focused on results that describe social qualities and
pedagogical methods as a basis for virtual student group work. The data consists of interview
transcripts, fieldnotes and written reports produced by the students. Semi-structured, one-on-
one interviews were held (Brinkmann and Kvale, 2018), lasting between 45 min and one
hour. An interview guide was designed using open-ended questions about the participants’
experience during the course. Follow-up questions were used to delve deeper into relevant
aspects (Flick, 2007). The data were analyzed with an inductive approach, allowing findings
to arise from the data through initial and focused coding (Charmaz, 2014; Saldaña, 2015).
Fifteen students participated in the study with informed consent. Participation comprised of
an interview and analysis of reflective reports produced during the seminar. A request for
participation was sent to all 51 students. Additionally, individual requests were sent to
participants who had the most relevant experience in the subject (Flick, 2007). The final
selection included students with particularly positive or negative experiences in their group,
students who dropped out and students who had experienced both cohorts of the seminar.
The authors were at the same time initiators, coordinators and co-teachers in the seminar.
Being involved in multiple roles provided profound insights, while it could be seen as a
limitation of bias in the study. The researchers were sensitive to potential conflicts of interest.
3. Results
The interviews revealed how students connected to their groups and how they collaboratively
developed projects. We could confirm that our methodology was able to support group
formation in a way that students felt connected and could develop creative work through
virtual exchange. Additionally, we identified challenges that emerged and considered how
these could be met in future collaborative projects.
3.1. Matching method to build positive group relationships

Our matching method supported the formation of teams that were able to develop positive
dynamics and more or less equal commitment to collaboration. Positive experiences were
reported in groups where students found commonalities such as similar personalities,
interests and cultural backgrounds.
Most of the students had not met before even if they were enrolled at the same institution.
For supporting the formation of groups across the international partner universities, we
developed a two-step matching process consisting of a kick-off workshop and a survey. The
223
kick-off workshop was held in a virtual1 meeting space. Students worked on creative tasks in
small groups in different constellations. They brainstormed about connecting features in
public spaces, such as bodies of water or public monuments, between their home countries.
The results were then presented to the whole cohort. The aim was to foster curiosity and
interaction as an alternative to customary introductory rounds, which often promote self-
presentation. In the second step, the participants filled out a survey as a basis for the teachers
to build groups of two to four students for the seminar project work. The participants were
asked to name preferences for potential team members. Furthermore, they were asked open
questions about their expectations and skills they would like to share, followed by ranking
questions about experience in creative collaborations and digital exchange. Finally, questions
about expected time commitments and learning styles were included. In the selection, priority
was given to the choice of potential group members, followed by the other questions in the
order as listed above. All groups were mixed across institutions and disciplines. 60 percent
were mixed across academic levels of MA and Ph.D. based on our survey, at least one student
in each group was experienced in collaborative work and virtual exchange, so that other
group members could benefit from this experience. Our intercultural facilitator helped design
the matching method and moderated the kick-off workshop. As non-subject-specific teacher,
she was in a neutral role and could thus fully focus on the aspect of building relationships.
Through initial coding, we recognized that most participants highlighted positive experiences
regarding their relationships in the groups. With focused coding, we determined what made
the experience positive. Eight participants of both cohorts emphasized a sense of openness
for equal participation. For example, they felt that there was “room for change” to bring in
ideas and that the group members would “listen” which made it “easier to share”. Two of
those students remarked team spirit, describing that they were “tied together as a team” and
experienced “some sort of synergy that you can't really organize or assume from the
beginning”. Others mentioned “trust”. Importantly, in many cases positive experiences of
group constellations in both cohorts were reported as mostly being related to commonalities
that the participants identified in their team-mates. One participant depicted that their group
had similar “personalities”, others told that they shared a similar “worldview” and “that we
think about the same things at the moment.”2. Four of the students from three different groups
emphasized cultural background as a connecting factor. For example, a student from
Germany suggested that their group worked well together “maybe because of culturally being
congruent, being kind of from the same place”. This group consisted of three partners who
lived in Germany, Canada and Brazil, while sharing a Latin American heritage. They were
able to use this commonality to co-develop a performance for different public spaces. They
1
An exemption were the students from Canada in the second cohort, who met in a hybrid setting in a room.
2
Original quote in German: “dass wir uns gerade um die gleichen Dinge Gedanken machen.” (Translation by the lead author)
224
chose the topic of sharing dreams, because the group could relate to the topic from their own
cultural backgrounds. They were “feeling very close culturally to the importance of the
dream space” and were “very aware of what this means to indigenous communities, to
indigenous cultures in Latin America.”
The interdisciplinary seminar across six international universities offered multiple ways for
students to benefit from differences like different disciplinary backgrounds, academic levels
and cultural environments. At the same time, differences were accompanied by challenges,
like working across different time zones and learning cultures. While collaboration entailed
embracing differences and learning from each other’s diverse experiences, ideas and
approaches, our data show that fruitful creative group work was essentially supported by
elements of commonality. Herrera-Pavo (2021), has argued that virtual group work is more
likely to succeed if the group shares common interests. Our results confirm this finding while
our case additionally involved the factors of personality and cultural commonalities. The two-
phase matching process was a helpful instrument. Even limited to virtual meetings,
participants were able to find fellow students with whom they felt connected. The findings
offer a way to refine this method, by more consciously addressing important commonalities.
3.2. Collaborative tools for developing creative group work

One of the most rewarding outcomes of the collaboration was reported when the group work
led to new and surprising results. Next, we consider how the collaborative tools of “playful
exercises” and “Maker’s Lounge” meetings supported this outcome.
Throughout the seminar, playfulness was used to encourage curiosity, leaving known paths
and creating new ideas together with others. In the first cohort of the seminar, Maruška
Svašek, a partnering professor from Queen’s University Belfast, developed a game that the
student groups were asked to play. They explored public spaces based on an experiment of
playful ethnography (Svašek, 2019). In the second cohort, students were encouraged to
design their own playful exercises in reference to “unlearning exercises” used in a
collaborative course (in-person) at the Academy of Fine Arts Vienna (Baldauf, 2016, p. 169).
In the first round of coding the interviews, it appeared that some students emphatically
highlighted positive experiences when they were improvising together in their groups.
Different participants pronounced that they “clicked” together, that they had a “magical time”
discovering “surprises”, and one even described the experience as akin to being “on the
biggest high”. In focused coding, we became aware of reports of seven students from both
cohorts, who experienced rewarding moments in their collaboration when they came upon
surprising new ideas through collective improvisation. Some students pointed out that the
playful exercises helped them to improvise together. In the second cohort, a student from
Northern Ireland declared that using games “was really fun” and “added a different kind of
225
approach”. A teammate from Canada remarked that it “brought up a lot of new avenues”. In
the first cohort, some groups developed their final seminar projects based on their playful
exercises. The written reports of two students from Germany and Northern Ireland illustrate
how this game helped them to collaboratively develop a project. The pair reached a dead end
with their initial approach and used the game to recalibrate. Initially, they worked on the base
of a previous project from the German partner, creating posters that addressed issues of
climate change. The student from Northern Ireland felt unsure working with posters as artistic
medium and wrote: „I believe, for both of us there was an instability in the concept.“ After
they had exchanged some drafts, the German student “found it difficult to work on this
graphic design task at a distance.” Hence, the pair gave up the poster idea and developed a
new collaborative project. The German student recalled that “The playful task (…) gave us
the important impulse.” The task was an explorative walk through the students’ respective
cities letting the dice decide the direction. They set rules for where to start and what to do
during the walk. Their rule was to document plants growing in the cracks and corners of the
cityscape. From this experience, the German student noted that it became “clear that the real
recipients of our public art should be the uninvited flora in the streets.” Instead of posters,
the group created a performance, addressing plants with messages and poetry. The game
encouraged the partners to develop a creative group process even through remote exchange.
The case of the two students from Germany and Northern Ireland also illustrates the role of
another collaborative tool developed for the seminar, a weekly open agenda meeting called
the “Maker’s Lounge”. When the student from Northern Ireland was in doubt about the
group’s first poster idea, she was not immediately able to exchange her concerns with her
team-mate due to timing issues and shaky internet. Thus, the student attended the Maker’s
Lounge, where she was reassured by teachers and peers to reconfigure and to embrace the
playful exercise for new ideas. This case was typical for the role of the Maker’s Lounge in
the seminar. The weekly two-hour time slot was a reliable opportunity to exchange ideas
outside of the regular sessions. It served as an important instrument of feedback and enabled
a facilitation role of teachers as suggested by Herrera-Pavo (2021). Additionally, some
students from both cohorts declared that the meetings gave them orientation in the course. A
student from Canada in the second cohort told that when she “wasn't sure (…) what was most
important for me (…) it was always grounding, when we got together and had conversations
about it.” Burke & Larmar (2021), report that feelings of isolation and disorientation are
more likely to emerge among students in online settings than in-person. The Maker’s Lounge
was a connective feedback instrument that helped strengthen bonds and provide orientation.
3.3. Challenges
While many groups recounted positive experiences, some challenges emerged, and some
students dropped out before the seminar ended. Not all challenges and dropping out were
related to group work. However, some findings were pertinent in the context of this study.
226
Interviews with students who dropped out showed that asymmetries in leaving and staying
within one’s own comfort zone created negative experiences of unequal participation. In this
practice-based seminar, two students from Germany in the second cohort especially felt that
they had to work outside their immediate fields of interest and their artistic media, while their
teammates did not. One of them affirmed that “this is really not my safe ground” The student
used to work with objects and “do things in the streets”, while her group worked fully
digitally. The student “liked that a lot”. However, she added: “it was difficult because I felt
so far away from the things I was used to. So, for me, it was really hard to connect with
them.” The other two members were also on a higher academic level. As the group worked
in an area that was very familiar for those two, the younger student with less practical
experience felt isolated and demotivated. Another student from a different group reported a
similar problematic. She also worked object based, while her group designed a performative
experiment. The researcher’s fieldnotes recount that the student normally felt comfortable
and “would take the lead in such group situations”. In this group “she did not feel qualified
to do so because she was working outside of her strength in fields that her teammates were
more qualified in and she was ‘not a specialist’ in.” The seminar demanded students and
university staff to master multiple challenges including working remotely with public spaces
across different countries. Not being able to work in one’s own fields of interest and artistic
media created additional challenges for the two participants quoted above. The asymmetrical
situation that their teammates were working in their fields and media was experienced as a
barrier to equal participation. Instead of benefiting from the others’ expertise, the students
felt intimidated and did not see a way of meaningfully or equally contributing.
4. Conclusion
In our seminar, similar personalities, interests and cultural backgrounds were essential
preconditions for fruitful creative group work in an international virtual setting. Our tools
helped students from different backgrounds to develop meaningful and rewarding
collaborations and provided orientation. In some cases, negative experiences emerged when
group members asymmetrically worked within and outside their fields of interest and artistic
media. This challenge can be met by monitoring and mediating asymmetries in group
processes. With a focus on social factors of collaboration and with attentive support of a
facilitator, the seminar established relationships among students and faculty, some of which
are still lasting and have led to follow-up projects. This indicates an important first step for a
community of practice to form and to establish virtual exchange in public art.
Acknowledgments
We thank our partners Marion Hohlfeldt, Anne Goarzin, Taylor Still, Patrick Leroux, Silvy
Panet-Raymond, Keely Whitelaw, Eva Urban-Devereux, Maruška Svašek, Yvon Bonefant,
Jools Gilson, Eloi Puig, Jo Milne, Ramón Parramón and Maria Eugenia Agusti Cami.
227
References
Baldauf, A. (Ed.) (2016). Publication series of the Academy of Fine Arts Vienna: volume 18.
Spaces of commoning: Artistic research and the utopia of the everyday. Berlin: Sternberg
Press.
Baumann Montecinos, J., & Grünfelder, T. (2022). What if we focus on developing
commonalities? Results of an international and interdisciplinary Delphi study on
transcultural competence. International Journal of Intercultural Relations, 89, 42–55.
doi: 10.1016/j.ijintrel.2022.05.001
Brinkmann, S., & Kvale, S. (2018). Doing Interviews (2nd edition). Qualitative Research
Kit. London: SAGE Publications Ltd. doi: 10.4135/9781529716665
Burke, K., & Larmar, S. (2021). Acknowledging another face in the virtual crowd:
Reimagining the online experience in higher education through an online pedagogy of
care. Journal of Further and Higher Education, 45(5), 601–615. doi:
10.1080/0309877X.2020.1804536
Carrruana Martín, A., Alario-Hoyos, C., & Delgado Kloos, C. (2022). Smart Groups: A
system to orchestrate collaboration in hybrid learning environments. A simulation study.
Australasian Journal of Educational Technology, 38(6), 150–168. doi:
10.14742/ajet.6776
Charmaz, K. (2014). Constructing grounded theory (2nd edition). Introducing qualitative
methods. London, Thousand Oaks Calif.: SAGE.
Flick, U. (2007). Designing Qualitative Research. Qualitative Research Kit. London: SAGE
Publications, Ltd. doi: 10.4135/9781849208826
García-Morales, V. J., Garrido-Moreno, A., & Martín-Rojas, R. (2021). The Transformation
of Higher Education After the COVID Disruption: Emerging Challenges in an Online
Learning Scenario. Frontiers in Psychology, 12, 616059. doi:
10.3389/fpsyg.2021.616059
Herrera-Pavo, M. Á. (2021). Collaborative learning for virtual higher education. Learning,
Culture and Social Interaction, 28, 100437. doi: 10.1016/j.lcsi.2020.100437
Saldaña, J. (2015). The coding manual for qualitative researchers (3rd edition). Los Angeles:
SAGE.
Scharmer, C. O. (2016). Theory U: Leading from the future as it emerges. the social
technology of presencing (2nd edition). San Francisco, California: Berrett-Koehler
Publishers.
Svašek, M. (2019). The Art of the Game: Visual Methods, Artistic Engagement and Playful
Ethnography. Irish Journal of Anthropology, 22(1), 205–223.
Uukkivi, A., Labanova, O., Petjärv, B., & Nõuakas, K. (2022). How to support cooperation
in hybrid learning? In 8th International Conference on Higher Education Advances
(HEAd'22), 299-306. doi: 10.4995/HEAd22.2022.14484
228
Introducing a collaborative learning strategy in a hybrid and

traditional laboratory for undergraduate computer science
students
Sam Ivan O’Neill, Aidan Mooney
Department of Computer Science, Maynooth University, Ireland.
Abstract
Collaborative learning is an advantageous pedagogical strategy to include in
e-learning. In this study we aim to demonstrate a methodology created to
implement this strategy in an undergraduate student population. We will
present our findings from running this experiment in two scenarios, namely an
in-person environment and a simulated remote environment. We analyse the
impact this methodology has on students understanding of the topic along with
how their assessment compares to their peers in a previous year. We also
analyse the different perception of personal and collaborative work in both
scenarios. We conclude that this methodology is effective in increasing a
student's understanding of the work being assessed and contributes to an
increase in students' assessment grades.
Keywords: E-learning; collaborative learning; hybrid; undergraduate.

Introducing collaborative learning strategy in a hybrid and traditional undergraduate laboratory
1. Introduction
The exponential rise of accessible e-learning recently has brought forward many different
types of strategies and methodologies. Turlaram (2018) discusses that there are several
variables that contribute to the success of an e-learning strategy including technological
awareness of students, general mathematical ability, and teacher competence. Mhouti et al.
(2017) consider how the use of Web 2.0 can be integrated into e-learning practices and how
collaboration, not only between students, but between students and teachers, is crucial. They
do however report that this aspect of collaboration is usually difficult to achieve if the
contents presented are not adapted to the context of collaborative e-learning.
Active learning engages the student and enables the knowledge to be cemented with practice
and understanding. When creating an e-learning methodology, it is essential to consider how
a student's perception of a task changes in an offline, hybrid and fully remote scenario. Jensen
et al. (2017) contribute that e-learning can only reach the same benefits as traditional teaching
if an active learning approach supports it. In comparison to passive learning, where students
absorb information from a lecture or tutorial, active learning places an effective outcome on
the effort put in by the student. Khan et al. (2017) state that integrating active learning into
course materials is crucial to engage students, regardless of the learning environment. They
acknowledge that when it comes to online courses, an appreciation of the unique approaches
to active learning is required.
Motivating the student to the same standard as in-person teaching in a hybrid or fully remote
environment can be challenging but it is essential to the student's development in the course
material. Collaborative e-learning, as a strategy, enables students to have motivational
benefits from social interactions with their peers, even when remote. Collaborative learning
as a strategy enables students to relay and gain information from their peers and put into
practice the theory that they have learned passively. Each student is different, but many who
participate in e-learning positively display similar traits. Punnoose (2012) found three
personality variables positively influence students when it comes to e-learning, namely a high
level of conscientiousness and extraversion along with a low level of neuroticism. This study
aims to foster these traits by means of collaboration and social interaction.
This paper aims to contribute to research around introducing an active e-learning strategy to
higher education students. We have focused in this study on how active learning can be
facilitated through e-learning and looked at what role collaborative learning plays within that.
This study will include a deeper dive into understanding what methodology creates the best
learning environment for an e-learning study group, and how that methodology can be
replicated outside of one subject. For this study, we have focused on computer science
students at the start of their respective degrees, where we have incorporated active learning
230
components within weekly laboratory assignments as a crucial part of their learning

experiences.
One priority of this study was to innovate and build upon current educational practices.
Students' typical laboratory work involves completing a set list of coding problems, where
the majority are released at the beginning of the week and can be completed at home. A
further one or two problems are released within the set laboratory time; these are considered
more challenging than the initial set. While this work is active learning, the addition of
collaborative learning allows students to discuss problems and difficulties they are having
with their peers and encourages students to explain their work clearly and concisely.
Introducing a mobile/web application has enabled students to keep a digital log of their
discussions and notes. Utilising tools such as this is integral to innovating the teaching and
learning experience as the expectation to be technology literate within the workplace is ever-
growing. Roehl et al. (2013) conclude that “one's adaptability to new technologies is crucial
for graduate students to succeed in the workplace.” This underlines the need for the provision
of technology-infused learning environments at educational institutions, where their
pedagogical benefits have been identified. Many workplace tools involve a collaborative
element, be it paired work or online communication. Building these skills within the
classroom helps set students up for success and create a well-rounded workforce.
2. Methodology and Data Gathering
2.1. Methodology steps

This study aimed to understand how students collaborate with coding problems in offline and
emulated online environments. To achieve this goal, we designed a strict set of parameters
for both settings, as follows:
1. Group Size: A group size of 4 students was used.
2. Access to Questions: Students were only given access to the coding problem, for the
collaborative session, at the beginning of the session.
3. Review Time: Students were given 10 minutes to review the question and write
notes on the topic.
4. Group Discussion: Students were placed into groups and given 20 minutes to discuss
and create a robust pseudo-code algorithm to answer the question.
5. Assistance: Students were allowed to ask for help from a lab demonstrator in the
final 10 minutes if they needed help understanding the problem.
6. Survey: Once the session was finished, students completed a survey describing their
experience with the session and how helpful each section was in completing the
231
algorithm. The survey also requested information on the student's background and
their contributions to the discussion.
The experiment was run in two separate scenarios:
(1): In-Person, students were face-to-face and expected to communicate and write down
information on paper. (2): Simulated Online, students were asked to communicate in the
simulated online scenario through a web/mobile app. They were expected to discuss the
problem and ideas they had for a solution through text chat in the app.
After completing the session, the algorithm was looked over by an experienced demonstrator
in both scenarios, and the students were provided feedback on their work.
2.2. Data gathering

In this study, data from 3 distinct student populations were selected over a 12-month period.
1. Group 1 - Undergraduate students in their second semester of university. (n = 476)
2. Group 2 - Higher diploma students in an accelerated introduction to programming
module over three weeks. (n = 47)
3. Group 3 - Undergraduate students in their first semester of university. (n= 570)
Group 1 and Group 3 undertook weekly lab assignments, where a new topic was assessed
each week. Due to the accelerated nature of the content that Group 2 was receiving, they had
multiple lab sessions within a week. For the purposes of this paper, we will report Group 1
and Group 3 in a weekly fashion and Group 2 in a session fashion.
To evaluate the effectiveness of the collaborative sessions, we used an initial survey1
(https://forms.office.com/e/MbS9vWMs0T) with 12 different items representing a student's
background, administered in the first week of the study with each group. Participants
provided information on their education, parental education, gender, and geographic
upbringing, as well as questions on their perception of collaborative learning.
A follow-up survey2 was administered within each weekly session, gathering information on
the session itself. Participants in the weekly follow-up surveys provided information on their
perception of the session, including their understanding of the topic at the beginning of the
session versus the end, how effective the group work was on a scale of 1-10. They were also
asked about how effective the solo work was on a scale of 1-10.
1
https://forms.office.com/e/MbS9vWMs0T
2
https://forms.office.com/r/7xmmNHcbnD
232
In total, we received 2341 completed surveys. They were broken down as follows:
1. Group 1 - 571 surveys over four sessions. (202 initial, 369 weekly session)
2. Group 2 - 122 surveys over nine sessions. (15 initial, 107 weekly session)
3. Group 3 - 1648 surveys over seven sessions. (233 initial, 1415 weekly session)
Group 1 took part in the in-person experiment, while Group 2 took part in a simulated online
experiment. Group 3 completed the experiment in both an in-person and simulated online
scenarios. For Group 3 the students were divided into two groups: a larger group of 540
students who took part in the session in person and 30 students who utilised both in-person
and simulated online sessions throughout the seven sessions.

To determine the effectiveness of this e-learning strategy, we investigated three factors of the
student's learning experience. These were:
1. Did the student find the collaborative session useful in increasing their knowledge
of the topic?
2. Did the personal or group work affect the students' understanding of the problem?
3. Did the students’ grades improve compared to those that did not utilise collaborative
learning sessions in the previous year?
3.1. Group 1’s findings

We first introduced this methodology to a cohort of undergraduate computer science students,
Group 1, after the midpoint of their semester. We have chosen to exclude this data from our
final conclusions as it depicts data from the methodology running in a much less strict
environment and we could not conclude whether these changes had an impact on the
experiment. This group was, however, instrumental in helping to refine the methodology for
Group 2 and Group 3. From the feedback we did gather, we know this group had a positive
perception of the methodology and could see the benefit it would provide in the future.
3.2. Usefulness of collaborative session

We wanted to gauge the students’ perception of the usefulness of collaborative sessions
within their weekly laboratory session. Figure 1 presents the average perception of how
useful the students in Group 2 found the sessions throughout the experiment. The average
response was 7.6 out of 10. The perception of usefulness fell to 6.6 at the midway mark before
recovering to its highest in session 8. In general, we can see that the perception was positive
and is something the students identify as valuable.
233
10 10
5 5
0
0
Se on 1
Se on 2
Se on 3
Se on 4
Se on 5
Se on 6
Se on 7
Se on 8
W 1
W 2
W 3
W 4
W 5
W 6
7
n
io
k
si
i
i
i
i
i
i
i
ee
ee
ee
ee
ee
ee
ee
ss
ss
ss
ss
ss
ss
ss
ss
s
Se
W
Figure 1. Group 2 students' average perception of session Figure 2. Group 3 students' average perception of
usefulness over a 9-session period. session usefulness over a 7-session period.
In relation to Group 3, despite there being a large spread of values week to week on the
students' perception of how useful the collaborative learning sessions were, it can be seen in
Figure 2 that overall, the students found the session helpful and gained both knowledge and
confidence in the topic they were working on. In the initial two weeks, the average perception
was scored at 7.25 and 6.1, respectively. The average then balanced towards 5.7 as the
students grew used to the sessions taking place.
When asked “how helpful did you find the session”, it can be seen in Figure 3, that for each
week a minority of students rated the session as having a low impact on their knowledge of
the topic, peeking in week 5, with 25.1% of respondents giving the session a low rating. The
inverse of this is that at least 74.9% of participants each week found value in completing the
session. During Week 1, 70.1% of participants ranked the sessions highly, which fell to 42%
by week 7. Week to week, the level of difficulty in the sessions' problems rose gradually, and
students' technical skills increased as they built upon past topics.
Variable High rating 10 – 7 Mid rating 6.9 – 3.1 Low rating 3 – 0
Week 1 83 27 7
Week 2 80 61 30
Week 3 77 84 52
Week 4 64 64 36
Week 5 53 57 37
Week 6 62 54 23
Week 7 53 47 26
Figure 3. Group 3 students’ perception of the usefulness of the session over a 7-week period.
3.3. Preference for session work type

In relation to a preference of the type of session work within Group 2, either personal work
or group work, the preference seemingly varied widely from week to week, as shown in
Figure 4. The average response was 46% in favour of personal work and 54% in favour of
234
group work. Due to the lack of eye-to-eye social interaction, seemingly the students felt little
difference between the two.
Figure 4. Group 2 students' preference of session work personal vs group.
For Group 3, students mostly preferred to work in their group's week to week. The largest
variance in responses received was in week 7, with a difference of 47.9% between the two
options, as shown in Figure 5. The smallest difference occurred on week 4, with a difference
of 13.5%. The results, in general, follow a parabolic curve with the precipice at week 4. There
was a consensus that the group work benefited the student the most. The mid-term break
scheduled after week four may have impacted the students' motivation to engage with their
classmates and gain more benefit from their discussions.
Figure 5. Group 3 students’ preference of session Figure 6. Group 3 students' Grades in 2022 in
work personal vs group. comparison to 2021.
3.4. Comparison with previous years’

The data presented in Figure 6 shows the average score of students in the module, where
100% means they got the answer fully correct, and 0% means they did not receive a grade
for the lab. We identified an increase in the average grade of a student who participated in
this study compared to their peers who took the same module in the previous year. Excluding
week 6, the 2021 average grade was 1.4% higher than in the 2022 cohort. When comparing
student grades with students in previous years, we need to do so with care and note a threat
to validity around this. It is worth noting that for this analysis, the lecture content remained
the same year by year, along with the lecturer delivering the content and the content covered
in the labs.
Figures 3 and 7 show that students' confidence in their understanding of the topic can be
mapped to the average grade received each week. A higher perception of understanding
correlates with a higher grade received overall.
235
4. Conclusion
The introduction of a collaborative learning strategy drives engagement and motivation in
students to discuss issues with their peers, and consequently gain a greater understanding of
the solution to the problem. Our work has shown that most students prefer to work within
groups compared to on their own, when it comes to solving technical problems in an in-
person session. In an online session, students are more evenly balanced in their preferences.
The grades of students have marginally increased in previous years when this collaborative
learning strategy was implemented. Again, we present this result with care, cognisant of the
inability to correctly compare distinct student cohorts.
The perception of the usefulness of such a collaborative learning approach fell to its lowest
around midway through the module for both in-person and online students. The use of a
simulated online medium to utilise this strategy has been successful and opens future work
opportunities in re-examining this experiment in a fully online format. In a fully online
scenario, it is important to provide adequate social interaction features to ensure students get
the most benefit out of the collaborative work possible.
References
El Mhouti, A., Nasseh, A., Erradi, M., & Vasquèz, J. M. (2017). Enhancing collaborative
learning in Web 2.0-based e-learning systems: A design framework for building
collaborative e-learning contents. Education and Information Technologies, 22, 2351-2.
Jensen, J. L., Kummer, T. A., & Godoy, P. D. D. M. (2015). Improvements from a flipped
classroom may simply be the fruits of active learning. CBE—Life Sciences Education,
14(1), ar5.
Khan, A., Egbue, O., Palkie, B., & Madden, J. (2017). Active learning: Engaging students to
maximize learning in an online course. Electronic Journal of e-learning, 15(2), pp107-
115.
Roehl, A., Reddy, S. L., & Shannon, G. J. (2013). The flipped classroom: An opportunity to
engage millennial students through active learning strategies. Journal of Family &
Consumer Sciences, 105(2), 44-49.
Punnoose, A. C. (2012). Determinants of intention to use eLearning based on the technology
acceptance model. Journal of Information Technology Education: Research, 11(1), 301-
337.
Tularam, G. A. (2018). Traditional vs Non-traditional Teaching and Learning Strategies-the
case of E-learning!. International Journal for mathematics teaching and learning, 19(1),
129-158.
236
School, university and community collaboration for inclusivity:

the Turin experience with the university courses “expert in the
processes of inclusive education” and “expert in educational and
didactic processes”
Paolo Bianchini, Cecilia Marchisio, Alessandro Monchietto
Department of Philosophy and Educational Sciences, University of Turin, Italy.
Abstract
The COVID-19 pandemic has shed light on the persistent disparities in the
Italian education system and the urgent need for inclusive education. This
study focuses on the crucial role of university training programs in addressing
the issue of school dropout and promoting inclusiveness in the education
system. The University of Turin has taken a pioneering approach by launching
programs for secondary teachers and social educators aimed at providing
them with the necessary skills and competencies to design and implement
effective inclusive educational interventions. The university has also
established partnerships with other organizations to further foster the culture
of inclusiveness in the region. The programs offered by the University of Turin
serve as a model for bridging the gap between inclusive theories and socially
just practices, paving the way for a more equitable and inclusive education
system.
Keywords: Expert; inclusion; multi-professional work; school dropouts;
social educator; teacher.

School, university and community collaboration for Inclusivity
1. Introduction
In Italy, a comprehensive reform of teacher training programs is underway to meet the
demands of a changing society. This is especially crucial given the challenges posed by the
COVID-19 pandemic, which have further exacerbated existing disparities in the education
system. According to the Italian National Institute of Statistics [ISTAT] (2022), students from
socio-economically disadvantaged backgrounds, those who speak a different language, or
those with disabilities still face difficulties accessing and participating in school activities.
This is reflected in the rising rates of school dropouts and failures, as well as declining levels
of learning (AGIA, 2022; Save the Children Italia, 2022).
A 2016 report from the OECD [Education at a Glance] revealed that Italy has one of the
lowest rates of educational mobility in Europe. Only 8% of young Italians between the ages
of 25 and 34 whose parents did not complete upper secondary school obtain a university
degree, compared to the OECD average of 22%. The percentage rises to 32% among young
people with parents who have completed secondary education, and 65% among those with
parents who have a university degree. These figures demonstrate the persistent impact of
socio-economic status on educational outcomes, which the OECD refers to as a "lottery of
nature".
It is also important to note that equalizing action of the school can only be effective if pupils
from different socio-economic and ethnic backgrounds have equal access to the same
schools. If school segregation is high, the school may be formally open to all but not equally
accessible to all students, thereby limiting its ability to provide equal opportunities (Caritas,
2022; Pacchi & Ranci, 2017; Ranci, 2019).
In addressing the issue of exclusion in education, it is crucial to consider various factors
beyond socio-economic conditions. For example, factors such as belonging to ethnic
minorities, having a disability, or being female, can contribute to marginalization and reduced
educational opportunities (Migliarini et al., 2022). To mitigate these challenges, it is
necessary to provide teachers with theoretical and practical tools that are suitable for their
daily work environment (Tino & Fedeli, 2015). Additionally, social educators must also be
equipped with up-to-date and relevant skills, as they play a crucial role in supporting students
with disabilities and collaborating with classroom teachers. By employing workshop-style
didactics that emphasize student participation, both teachers and social educators can
effectively enhance the learning experience. Ultimately, the objective is to not only
strengthen the theoretical and operational abilities of teachers and educators, but also promote
inter-professional collaboration and exchange, resulting in more comprehensive and
impactful educational interventions.
238
2. Enhancing Competencies of Educational Professionals: Addressing Gaps in

Training for Inclusive and Equitable Education
According to several studies on the competencies of educational staff (Sharma, 2012; Caena,
2011; OECD, 2019), the implementation of training programs that specifically target these
competencies is crucial for promoting inclusive educational practices. This shift in
perspective moves away from the notion of "fixing" differences, and instead aims to empower
pupils to become critical individuals who can live and participate as conscious citizens.
Unfortunately, many teachers continue to have a rigid and unchanging approach to their
teaching methods, which can hinder the creation of inclusive classrooms (Aiello et al., 2019;
Mura & Zurru, 2019). Without proper training, teachers can perpetuate inequalities and
struggle to provide educational opportunities for students with special needs (Fiorucci, 2019;
Romano et al., 2021). This is particularly concerning in Italy, where at the moment there is a
lack of university courses specifically designed to prepare secondary school teachers.
The promotion of inclusive practices within educational institutions requires the engagement
of educational staff who are capable of embracing diversity and utilizing it to advance
educational and pedagogical outcomes for all students (Bocci, 2021). However, a tendency
to treat students with special educational needs as a challenge to be managed rather than as a
valuable asset to be cultivated has been documented in the literature (Harry, 2014). This
mentality, rooted in a charitable rather than a rights-based approach, is partially the result of
insufficient and inconsistent training opportunities for school professionals (Gavosto, 2020).
The absence of specialized training programs for secondary school teachers in the Italian
education system has significant consequences for the education system as a whole.
Furthermore, social educators often navigate the complex educational landscape without
sufficient preparation to design and implement effective interventions. As a result, they are
forced to develop their own competencies, relying on their own strengths and dedications to
compensate for gaps in their training.
3. Collaborative Efforts for Inclusive Education: An Analysis of the University

of Turin's Inter-regional Partnership for Teacher Training
The University of Turin has been making efforts to bridge the gap between the theories and
practices of inclusion in the education system by providing support to the schools. To achieve
this, the University has launched a series of initiatives aimed at empowering educational
professionals with the necessary tools to drive positive change (Damiani et al., 2021). In
accordance with a memorandum of understanding signed between the University of Turin,
the Piedmont Region, the Regional School Office for Piedmont, the Metropolitan City of
Turin, the University of Eastern Piedmont, in collaboration with the Trade Unions
239
representing schoolteachers, the Department of Philosophy and Educational Sciences at the

University of Turin launched the University Course of Professional Development (CUAP)
“Expert in the processes of inclusive education” in the academic years 2020/21 and 2021/22.
In the academic year 2022/23, the University further strengthened its offerings by launching
the First Level Master Course "Expert in educational and didactic processes at school". The
aim of this course is to prepare future teachers and social educators to operate in an
increasingly inclusive and conscious manner, by providing them with the necessary
knowledge and skills to engage in inter- and multi-professional work. The course has been
designed in a modular format that takes into account both the demands of high-level
university teaching and the training needs of working students. The goal is to support the
professional growth of teachers and social educators, filling the void left by the abolishment
of qualifying pathways and promoting didactic innovation.
These initiatives align with the national teacher training guidelines set by the italian Ministry
of Education (Pettenati, 2021), which prioritize the development of "facilitator teachers" to
support continuous and widespread training within schools.
4. The Development of the Competences of Educational Professionals through

Interdisciplinary and Inter-Professional approach
The development of non-disciplinary competencies among university students is becoming a
crucial aspect of higher education. These competencies, including personal traits such as
character, judgment, and empathy, are significant for both employability and active
citizenship (Chignoli et al., 2020).
This shift in focus is supported by the literature, which stresses the need to study teachers'
attitudes and personal competence, as well as to consider the interplay between technical and
disciplinary competencies and personal qualities in professionals. Effective teaching involves
constant adaptation, evaluation, and response to student engagement and participation.
Hence, training approaches must be based on comprehensive and integrated perspectives that
account for the multi-dimensionality of competencies. The literature highlights the
importance of "Teacher Thinking" (Perla, 2015; Buitink, 1993; Clark & Lampert, 1986) and
participatory approaches (Cardarello et al., 2009) that showcase the unique professional
aspects of teaching. Training models such as those based on Transformative Learning Theory
(Taylor & Cranton, 2012), mixed method approaches (Suchman, 1987), and educational
principles (Fabbri, 2007) are instrumental in supporting the development of non-disciplinary
competencies. For example, the Embedded approach integrates cross-curricular skills and
teaching of disciplinary content (Yorke & Knight, 2004). Non-disciplinary competencies are
not distinct from other technical knowledge or skills and must be incorporated into the
curriculum (Gopalaswamy & Mahadevan, 2010).
240
4.1. Fostering the Professional Growth of Teachers and Social Educators through
Training
The courses "Expert in Inclusive Education Processes" and "Expert in Educational and
Teaching Processes at School" aim to enhance the professional development of teachers and
social educators in the area of inclusiveness and to prevent school dropout. These courses
aim to foster positive relationships, improve teachers' emotional and relational skills, and
enhance their overall well-being. Additionally, they aim to increase both the technical and
disciplinary knowledge and skills of school professionals, aligning with the core principles
outlined in the European profile of inclusive teachers (EADSNE, 2014). The courses place
equal emphasis on both the non-disciplinary and disciplinary aspects, encompassing cultural,
organizational, cooperative, and strategic awareness, as well as emotional-relational and
body dynamics.
The didactic structure of the courses is founded on a Competency-based education and
training approach. The CUAP pathway consists of a total of 100 hours of blended training,
divided into general and workshop parts, where trainees work together to develop a shared
culture and exchange good practices. On the other hand, the modular master's course consists
of 60 ECTS and 1600 hours, which are tailored to the needs of educators, teachers, and social
professionals. Trainees have the option of taking the entire course or specific modules,
depending on their requirements. Furthermore, the core of the course is a 100-hour internship
coordinated by a university supervisor and a 50-hour discussion reflecting on the experience.
The internship takes place in schools participating in the ‘Play It Again, Sam' project, which
has a primary prevention intervention implemented in the majority of Turin schools, aiming
to prevent unfavorable learning conditions and school attendance. Trainees participating in
this project gain a unique opportunity to gain a multi-professional outlook on education,
working alongside educators, teachers, and social service workers.
4.2. Teacher Inclusiveness in Teacher Training: Achieving Personal and Social

Competencies
The training approach outlined in earlier sections emphasizes the practical application of
acquired competencies and is in line with the Embedded model. The focus of this training is
to enhance the personal and social competencies essential for both learning support teachers
and teachers combating school dropout, through a blend of lessons, workshops, curricular,
and transversal activities. These activities were carefully designed to promote inclusiveness,
both in terms of methodology and content, and were well-received by the participants for
their effectiveness in mitigating stressors such as online learning or a heavy workload. The
participants emphasized the importance and effectiveness of the focus on inclusive teaching,
particularly in terms of professional development. They recognized that both the emotional
and communicative aspects are just as crucial as the technical and educational ones, with the
241
element of sharing and awareness consistently highlighted as a key factor in developing a

professional demeanor that promotes inclusion. It was noted that the creation of inclusive
networks is the most effective means of achieving this goal (Damiani et al., 2021).
5. Conclusions
The training approach described in this article constitutes a valuable contribution to the field
of pedagogy and teacher education. The innovative blend of practical lessons, workshops,
curricular and transversal activities, as well as the focus on personal and social competencies,
make this approach unique in its nature (Di Masi et al., 2023). Furthermore, the approach has
demonstrated positive results in enhancing the professional skills of learning support teachers
and reducing school dropout rates.
In conclusion, the present study provides novel insights into the support of teachers in
creating inclusive educational environments and promoting student well-being. The potential
impact of the Expert in Processes of Inclusive Education and Expert in Educational and
Didactic Processes at School courses on the international scientific community serves as a
stimulus for future research and advancement in the field of teacher training and professional
development. The results of this project reinforce our conviction that universities have a
critical role to play in advancing inclusive processes and treating education as a means to
effect social change, rather than perpetuating existing inequalities.
References
AGIA – Italian Authority for Children and Adolescents. (2022). La dispersione scolastica in
Italia: un’analisi multifattoriale. Documento di studio e di proposta. Roma. Retrieved
from https://www.garanteinfanzia.org/sites/default/files/2022-06/dispersione-scolastica-
2022.pdf
Aiello, P., Sharma, U., Di Gennaro, D., Dimitrov, D., Pace, E.M., Zollo, I., & Sibilio, M.
(2019). A study on Italian teachers’ sentiments, attitudes and concerns towards inclusive
education. Formazione, lavoro, persona, 20, 10-24.
Bocci, F. (2021). Pedagogia Speciale come Pedagogia Inclusiva. Itinerari istituenti di un
modo di essere della scienza dell’educazione. Milano: Guerini Scientifica.
Buitink, J. (1993). Research on teacher thinking and implications for teacher training,
European Journal of Teacher Education, 16(3), 195-203.
Caena, F. (2011). Literature review. Teachers' core competences: Requirements and
development. Brussels: European Commission.
Cardarello, R., Martini M. C., & Antonietti M. (2009). La voce degli insegnanti: competenze
per insegnare nella scuola di base. In G. Domenici & R. Semeraro (Eds.), Le nuove sfide
della ricerca didattica tra sfide, comunità sociali e culture (pp. 587-604). Roma:
Monolite.
242
Caritas Italiana. (2022). L’anello debole. Rapporto 2022 su povertà ed esclusione sociale in
Italia. Teramo: Edizioni Palumbi. Retrieved from https://www.caritas.it/wp-
content/uploads/sites/2/2022/10/rapportopoverta2022b.pdf
Chignoli, V., Leone, E., Carbone, F., Carotenuto, A., & De Lorenzi, A. (2020). Le
competenze trasversali nella formazione universitaria: un percorso online di sviluppo
delle soft skills per l’inserimento nel mondo del lavoro dei giovani neolaureati. Reports
on E-Learning, Media and Education Meetings, 8(1), 76-81.
Clark, C., Lampert, M. (1986). The study of teacher thinking: Implications for teacher
education. Journal of teacher education, 37(5), 27-31.
Damiani, P., Curto, N., Monchietto, A., & Marchisio, C.M. (2021). The training of trainers
of the specialization course in special educational needs. The Turin experience of the
University Course of Professional Development (CUAP), Italian Journal of Health
Education, Sports and InclusiveDidactics, V(4).
Di Masi, D., Seira Ozino, M., Monchietto, A. & Berretta, C. (2023). Ruoli e funzioni del
tutor esperto: il tirocinio nella formazione dell’insegnante inclusive. In C. Coggi & R.
Bellacicco (Eds.), Percorsi inclusivi (pp. 198-212). Milano: FrancoAngeli.
EADSNE - European Agency for Development in Special Needs Education (2012). Teacher
education for inclusion. Profile of inclusive teachers. Denmark: Odense. Retrieved from:
http://www.european-agency.org/sites/default/files/Profile-of-Inclusive-Teachers.pdf
Fiorucci, A. (2019). Inclusione, disabilità e formazione docenti. Uno studio sulla rilevazione
degli atteggiamenti e dei fattori associati in un gruppo di futuri insegnanti. La scala
OFAID. Italian Journal of Special Education for Inclusion, VII(2), 271–293.
Gavosto, A. (2020). Formazione e reclutamento dei docenti in Italia: criticità e proposte. In
M. Baldacci, E. Nigris & M.G. Riva (Eds.), Idee per la formazione degli insegnanti (pp.
85-96). Milano: FrancoAngeli,.
Gopalaswamy, R., & Mahadevan, R. (2010). The ace of soft skills attitude communication
and etiquette for success. India: Pearson Education.
Harry, B. (2014). The disproportionate placement of ethnic minorities in special education.
In L. Florian (Ed.), The Sage handbook of special education. (pp. 73-96). London: Sage.
Italian National Institute of Statistics [ISTAT]. (2022). L’inclusione scolastica degli alunni
con disabilità – A.S. 2020-21. Retrieved from
https://www.istat.it/it/files//2022/01/REPORT-ALUNNI-CON-DISABILITA.pdf
Migliarini, V., Stinson, C., & Hernandez-Saca, D.I. (2022) “It feels like living in a limbo”:
exploring the limits of inclusion for children living at the global affective intersections of
dis/ability, language, and migration in Italy and the United States. In S.A. Annamma, B.
Ferri & D. Connor (Eds.), DisCrit Expanded: Reverberations, Ruptures, and Inquiries.
(pp. 62-80). New York: Teachers College Press.
Mura, A., & Zurru, A.L. (2019). Professionalità docente e processi di inclusione:
dall’indagine sulle pratiche didattiche alla rilevazione dei bisogni formativi.
L’integrazione scolastica e sociale, 18(1), 43-57.
Organisation for Economic Cooperation and Development [OECD]. (2016). Education at a
Glance 2016: OECD Indicators. Paris: OECD Publishing. Retrieved from
https://doi.org/10.1787/eag-2016-en
243
Organisation for Economic Cooperation and Development [OECD]. (2019). What TALIS
2018 implies for policy. In TALIS 2018 results (Volume I): Teachers and school leaders
as lifelong learners. Paris: OECD Publishing. Retrieved from
https://doi.org/10.1787/1d0bc92a-en
Pacchi, C., Ranci, C. (2017). White flight a Milano. La segregazione so-
ciale ed etnica nelle scuole dell’obbligo. Milano: Franco Angeli.
Perla, L. (2015). Learning and teaching in Research & Training. The possible mediation.
Pedagogia oggi, 2, 152-169.
Pettenati, M.C. (2021). Paese formazione. Roma: Carocci editore.
Ranci, C. (2019). Separati a scuola. La segregazione scolastica a Milano. Social Cohesion
Papers, 1.
Romano, A., Rullo, M., & Petruccioli, R. (2021). The assessment of learning outcomes and
core competencies of inclusive education teachers. A pilot study. FORM@RE, 1, 188-
203.
Save the Children Italia. (2022). Promuovere la partecipazione giovanile nel contrasto alla
dispersione scolastica e alla povertà educativa. Roma. Retrieved from
https://s3.savethechildren.it/public/files/uploads/pubblicazioni/promuovere-la-
partecipazione-giovanile-nel-contrasto-alla-dispersione-scolastica-e-alla-poverta.pdf
Sharma, U., Loreman, T., & Forlin, C. (2012). Measuring teacher efficacy to implement
inclusive practices. Journal of research in special educational needs, 12(1), 12–21.
Suchman, L.A. (1987). Plans and Situated Actions: The problem of human-machine
communication. Cambridge: Cambridge University Press.
Tino, C., & Fedeli, M. (2015). I bisogni formativi dei docenti nella complessa realtà
scolastica di oggi. Giornale Italiano della Ricerca Educativa – Italian Journal of
Educational Research, 14 (7), 265-286.
Taylor, E.W. & Cranton, P. (Eds.) (2012). Handbook of transformative learning: Theory,
research and practice. San Francisco: Jossey-Bass.
Yorke, M., & Knight, P.T. (2004). Learning, curriculum and employability in higher
education. London: Routledge Falmer.
244
Experiential learning using short-term global virtual team

projects
Stephanie Swartz
School of Business, Mainz University of Applied Sciences, Germany.
Abstract
Institutions of higher education are facing new challenges when preparing
their graduates for today’s global workplace. Digital communication and
project management skills as well as intercultural competence are imperative
in light of the transitions to remote and virtual teamwork. Global virtual team
(GVT) projects provide students with experiential learning opportunities,
increasing their chances at employability and professional success. This
qualitative study involves a client-based GVT project conducted over the
course of four semesters and engaging approximately 400 students enrolled at
universities in Germany and the United States. Responses regarding students’
perceptions of the project’s value for self-efficacy and employability confirm
the importance of incorporating short-term GVT projects in university
curricula in order to offer students opportunities for professional development.
Keywords: Global virtual team projects; digital communication competence;
employability; experiential learning in higher education; interdisciplinary
project work.

Experiential learning using short-term global virtual team projects
1. Introduction
More now than ever, business schools are facing the pressure of adequately preparing
graduates for the contemporary global workplace. In the post COVID-19 pandemic era, the
focus for higher education has turned to imparting skills and abilities conducive to remote
work. The skills fundamental for post-pandemic employment include digital fluency, virtual
project management skills, as well as intercultural communication competence (Crawford,
2021; Elliot, 2021; Engler, 2020).
An effective method for developing these valuable skills in students involves engaging them
in short-term, global virtual team (GVT) projects (Burleson & Peters, 2021; Swartz &
Shrivastava, 2021). Over a designated period, students work on projects with their peers in
other countries using digital communication channels. Ideally, these projects are embedded
in the syllabi, and instructors share similar learning outcomes and assessment criteria. The
students as well as the focus of the projects may involve similar disciplines or be of cross-
disciplinary nature.
Short-term GVT projects belong to a growing movement of incorporating Virtual Exchange
(VE) at institutions of higher learning, also known as Collaborative Online International
Learning or COIL.1 COIL projects offer students the opportunity to gain valuable
international experience (Barbosa, et al., 2019; Starke-Meyerring, 2010; Swartz &
Shrivastava, 2021). VE has been especially valuable in providing students, who would
otherwise not be able to take advantage of study abroad programs, experiential learning
opportunities for intercultural exchange. During the pandemic, when students were unable to
participate in study or internship abroad opportunities, GVT projects filled the void
(Logemann et al., 2022).
This study investigates the value of short-term GVT projects for developing business
students’ skills and thus increasing graduates’ employability. The GVT projects under
investigation were carried out several times over the course of several semesters between
2020 and 2022. The projects involved approximately 400 students enrolled in an
undergraduate business administration program at a German university of applied sciences.
These students collaborated with peers from a partner university in the United States of
America over the course of six weeks on a project involving the design of an app. Randomly
chosen and heterogeneous teams of five to six students communicated through the
collaborative platform SLACK, ZOOM and other digital communication channels in order
to create a collaborative virtual presentation. After the project, the German students submitted
reflective papers in which they answered questions regarding the relevance of such projects.
1
Collaborative Online International Learning (COIL) is a pedagogical approach, which includes projects involving participants in
different countries utilizing digital communication channels (https://coil.suny.edu/).
246
Stephanie Swartz
The study involves a qualitative investigation of students’ responses to questions about their
project experiences and the project’s significance for preparing them for future employment.
The findings underline the importance of digital and intercultural competence for the post-
pandemic global workplace, as well as illustrate how experiential learning through GVT
projects can contribute to students’ development of this skill set. Finally, the results
encourage higher education instructors across all disciplines to incorporate COIL projects
into their curricula.
2. Background
GVTs refer to geographically and culturally diverse teams that have minimal face-to-face
contact, and communicate primarily through digital channels, utilizing collaborative
platforms such as MS Teams or SLACK (Chudoba & Maznevski, 2000).2 Their team
members represent a wide diversity of backgrounds, national and management cultures,
global variations of English and professional jargons (Neely, 2013).
Although virtual team projects can have a long duration, there has been a growing tendency
towards short-term, temporary projects, requiring team members to change their constellation
often as projects end and new ones begin (Hollema, 2020). As a result, team members must
continually adapt to new teammates with different cultures, personalities and priorities
(Edmonson, 2012; Gupta, 2018; Hollema, 2020).
GVTs face many challenges, which include working across diverse time zones, overcoming
cultural barriers while communicating exclusively through digital communication channels.
The lack of non-verbal cues creates further obstacles to working across cultural differences
and building rapport. Furthermore, the short duration of the projects requires that team
members develop “swift trust” (Crisp & Jarvenpaa, 2013).
In order to overcome these challenges, a specific set of skills, including digital
communication and intercultural competence, is required (Makarius & Larson, 2017).
Underlying these attributes is team members’ self-efficacy in dealing successfully with
digitally and culturally diverse environments (Hollema, 2020). GVT projects embedded in
the curricula of higher education institutions ensure that graduates bring the skillset required
for today’s global workplace (Crawford, 2021; Brewer et al., 2015; Leask & Bridge, 2013;
Starke-Meyerring, 2010).
2
Microsoft Teams and SLACK are software solutions, which allow for the forming of teams according to projects, as well as the
sharing of files, messaging and video conferencing among team members.
247
3. Methodology
3.1 The Project

Over the course of six weeks, teams of German and American students worked on a client-
based project for a leading global tech company. The students collaborated across a six-hour
time difference using digital communication and collaborative platforms (ZOOM, SLACK,
and WhatsApp). The project involved creating an app, which was meant to solve an issue
such as parking on campus, incorporating sustainability efforts at their universities or finding
study buddies. In order to support them in the conception of the app, the students took part
in virtual software workshops carried out by the tech company. The apps were presented in
the form of a virtual pitch.
The learning objectives of the GVT project included developing students’ project
management and digital communication skills, as well as honing their intercultural
competencies. The instructors communicated regularly before, during and after the project;
they synchronized the tasks, the deadlines and the expectations. The projects were embedded
in the respective course modules, and the deliverables constituted at least one third of the
students’ final grades.
The pitches were evaluated with a grading roster using the same criteria. These involved the
approach to solving the issue, user-friendliness, as well as features and creativity. The top
three pitches were presented to a jury of experts from the tech company and external
professionals for the final ranking, who provided in-depth feedback on the finalists’ concept
and presentation. All students received a certificate as well as a digital badge, which could
be posted to social media platforms such as LinkedIn, upon completion of the project.
3.2 Research Method

In order to assess the success of the project in achieving the learning outcomes, students from
the German university were asked to submit a debriefing on their project experience upon
completion of the project. This debriefing, also referred to as reflective paper, involved two
main questions:
1. What impact do you think these types of projects have on your ability to
communicate effectively in an online environment and to lead virtual teams?
2. How do you foresee employers valuing the intercultural communication skills
developed in projects such as this - in your case and in general?
248
Stephanie Swartz
4. Results
4.1 The Impact on Communicating in an Online Environment

The respondents overwhelmingly agreed that GVT projects developed their online
communication skills. In addition to an improvement in their English language, students
noted their experience in overcoming communication barriers. This was especially true
concerning non-verbal communication across digital channels. Although these students
belonged foremost to the digital native and social media generation, the majority pointed out
the benefits of working online with people they did not know and were different from
themselves.
I learned to be more open and direct in my communication, because everything took
place online, and you often couldn't recognize and interpret the direct reactions of
the team members. It also helped me to be more open with people I don't know or
don't know personally.
Furthermore, students acknowledged that participating in GVT projects offered them the
opportunity to recognize challenges they may face working remotely in future and to develop
strategies for dealing with those challenges.
Through this project I had the opportunity, for the first time in my life, to work in a
completely online environment with people outside of my country. Thanks to this
experience I now have a number of difficulties in my head that may occur in such a
digital environment and various tried and tested strategies for overcoming them.
In general, students concurred that engaging in the GVT project developed their self-
confidence and helped them discover abilities that they were previously unaware existed in
themselves.
… in the course of this project, I completely led a team for the first time. This
experience is very valuable for me, as I have realized that I am better at this, than
expected. As a result, I can trust myself more in the future and lead the group work
with more self-confidence.
Almost all students appreciated the opportunity to have taken part in a GVT project during
their studies, and felt that such projects should be an integral part of their studies in order to
develop the skills they would need later on in their careers. Some students recognized also
the gains they had made on a personal level.
An environment in which mistakes are allowed and people meet at eye level without
hierarchies is a valuable platform for experiencing everyday life in international
projects. The basics of communication, the use of different communication media as
well as structured problem solving, are taught in projects of this type using real
249
situations. The ability and competence to communicate effectively in an online

environment and to lead virtual teams can be learned in this way very effectively.
Another student valued the social aspect of the project:
For me personally this experience is priceless! I am really grateful for the given
opportunity to be a part of this international project and to work with people, who I
would have otherwise had no chance of meeting! And last but not least, I am thankful
for the new friendships, that we managed to build!
4.2 Student Perceptions on the Value of GVT Projects for Employers

The majority of student responses agreed that future employers value the skills they have
acquired through having participated in such projects.
Considering that one day I would like to work as a tax consultant or an auditor in a
big company, with offices all around the world, communicating with people from
many different cultures will be an integral part of my everyday work. Participating
in such a project has given me a glimpse at what my future work life might look like.
I learned a lot about how to overcome different barriers, such as different time
zones, how Americans deal with different complications and how they search for
their solutions. Having this project on my resume may, in fact, one day be a decisive
factor if I will get the job that I want or not.
The fact that participation in a GVT project during their studies could provide them with a
competitive advantage when applying for a position at both national as well as multinational
companies was pointed out continually in the responses.
As companies are placing more and more emphasis on these kinds of skills, I
consider these skills to be very well recognized. This is mainly due to the fact that
an increasing number of companies have a lot of contacts abroad, especially due to
globalization. This type of project shows the company, that you can also be deployed
internationally and keep a cool head in special circumstances.
All in all, students’ responses emphasized the value of GVT projects for exposing them to
the challenges of working in a global environment and developing the skills necessary for
success in their future careers, thus increasing their prospective employability and chances
of success. As one student described it:
This group work could have been a pioneer what we will have later in our
professional life. The world is growing together. The world of work has also become
more international. This makes intercultural competence more important in order
to be successful in a global working environment.
250
Stephanie Swartz
5. Discussion
Research conducted on students engaging in GVT projects has found that GVT projects are
conducive to developing students’ skills in digital communication, virtual project
management skills and intercultural competence, thus preparing them for the global work
environment (Crawford, 2021; Swartz, et al., 2021; Hackett, et al., 2023). This study’s
qualitative responses further substantiate these findings. Furthermore, this study shows that
students recognize the potential of GVT projects for acquiring these skills and increasing
their chances of employability and success at the workplace. Thus, this study underlines the
need for incorporating GVT projects into university curricula.
Additional research into the subjects’ career trajectories would provide valuable insight into
determining the impact of GVT projects on developing students’ skill set and improving their
employability as well as job performance. Until now, follow-up studies have not been carried
out on GVT project participants. The extent to which interdisciplinary GVT projects benefit
students is another area for future research. Further studies may guide instructors on the kinds
of GVT projects suitable for various learning outcomes.
6. Conclusion
For instructors and students, GVT projects can be both rewarding and time-consuming.
Instructors must work closely together to coordinate the assignments, monitor progress and
evaluate the learning outcomes. At the same time, students face multiple challenges of
finding meeting times, working together on collaborative digital platforms, and bridging
language and cultural barriers. However, student responses have shown that the rewards are
greater than the challenges. Embedding even short-term GVT projects in university curricula
would offer students the opportunity to acquire the skills necessary to overcome those
challenges, thus better preparing them for the global work environment.
References
Barbosa, B., Swartz, S., Luck, S., Prado-Mezad, C., & Crawford, I. (2019). Learning how to
work in multicultural teams: Students’ insights on Internationalization-at-Home
activities. Interpersona | An International Journal on Personal Relationships, 13, 205-
219. https://doi.org/10.5964/ijpr.v13i2.378
Brewer, P. E., Mitchell, A., Sanders, R., Wallace, P., & Wood, D. D. (2015). Teaching and
learning in cross-disciplinary virtual teams. IEEE Transactions on Professional
Communication, 58(2), 208–229. https://doi.org/10.1109/TPC.2015.2429973
Burleson, D. D., & Peters, U. (2021) Short-term international virtual teams: Preparing
students and employees for a digitized, global workplace. In S. Swartz, B. Barbosa, I.
Crawford, & S. Luck (Eds.), Developments in Virtual Learning Environments and the
251
Global Workplace (pp. 205-227). Hershey, PA: IGI Global. http://doi:10.4018/978-1-

7998-7331-0.ch017
Crawford, I. (2021). Employer perspectives on virtual working: Essential skills for the
globalised, digital workplace. In S. Swartz, B. Barbosa, I. Crawford, & S. Luck (Eds.),
Developments in Virtual Learning Environments and the Global Workplace (pp. 335-
354). Hershey, PA: IGI Global. http://doi:10.4018/978-1-7998-7331-0.ch017
Chudoba, K., Maznevski, M. (2000). Bridging space over time: Global virtual team dynamics
and effectiveness. Organization Science, 11(5), 473-492.
https://doi.org/10.1287/orsc.11.5.473.15200
Crisp, C. B., & Jarvenpaa, S. L. (2013). Swift trust in GVTs: Trusting beliefs and normative
actions. Journal of Personnel Psychology, 12(1) 45-56, http://doi: 10.1027/1866-
5888/a000075.
Edmondson, A. C. (2012). Teaming: How organizations learn, innovate, and compete in the
knowledge economy. San Francisco: Jossey-Bass
Elliot, D. (2021). 6 things to know about the future of skills and workplace learning. World
Economic Forum. https://www.weforum.org/agenda/2021/06/workplace-skills-learning-
linkedin-report/
Engler, S. (2020). Lack of skills threatens digital transformation.
https://www.gartner.com/smarterwithgartner/lack-of-skills-threatens-digital-
transformation?utm_campaign5EVT_GB_2020_GTR_C_NL2_GBS&utm_medium5em
ail&utm_source5Eloqua&cm_mmc5Eloqua-_-Email-_
LM_EVT_GB_2020_GTR_C_NL2_GBS-_-0000.
Gupta, M. (2018). Liquid workforce: The workforce of the future. In P. Duhan, K. Singh, &
R.Verma (Eds.), Radical reorganization of existing work structures through
digitalization (pp. 1-17). IGI Global. DOI: 10.4018/978-1-5225-3191-3.ch001
Hackett, S., Janssen, J., Beach, P., Perreault, M. Beelen, J. & van Tartwijk, J. (2023). The
effectiveness of Collaborative Online International Learning (COIL) on intercultural
competence development in higher education. International Journal of Educational
Technology in Higher Education. 20(5), 1-21. https://doi.org/10.1186/s41239-022-
00373-3
Hollema, T. S. (2020). Virtual teams across cultures. Create successful teams around the
world. Interact Global, Netherlands
Leask, B., & Bridge, C. (2013). Comparing internationalization of the curriculum in action
across disciplines: Theoretical and practical perspectives. A Journal of Comparative and
International Education, 43, 79–101. https://doi.org/10.1080/03057 925.2013.746566
Logemann, M., Aritz, J., Cardon, P., Swartz, S., Terri Elhaddaoui, T., Getchell, K.,
Fleischmann, C. Helens-Hart, R., Li, X., Juan Carlos Palmer-Silveira, J. C., Ruiz-Garrido,
M., Springer, S., & Stapp, J. (2022). Standing strong amid a pandemic: How a global
online team project stands up to the public health crisis. British Journal of Educational
Technology, 00, 1-6. http://doi: 10.1111/bjet.13189
Makarius, E., & Larson, B. (2017). Changing the perspective of virtual work: Building virtual
intelligence at the individual level. Academy of Management Perspectives, 31(2), 159–
178. https://doi.org/10.5465/amp.2014.0120
252
Stephanie Swartz
Neeley, T. B. (2013). Language matters: Status loss and achieved status distinctions in global
organizations. Organization Science, 24(2), 476–497.
https://doi.org/10.1287/orsc.1120.0739
Starke-Meyerring, D. (2010). Globally networked learning environments in professional
communication: challenging normalized ways of learning, teaching and knowing.
Journal of Business and Technical Communication, 24, 259-266. London: Sage
Publications. https//doi.org/10.1177/1050651910363266
Swartz, S., Barbosa, B., & Crawford, I. (2019, October), Building intercultural competence
through virtual team collaboration across global classrooms, Business and Professional
Communication Quarterly, 1-23. https://doi.org/10.1177/2329490619878834
Swartz, S., Barbosa, B., Crawford, I., & Luck, S. (Eds.) (2021, June). Developments in virtual
learning environments and the global workplace. IGI Global.
https://doi.org/10.4018/978-1-7998-7331-0
Swartz, S., & Shrivastava, A. (2021). Stepping up the game– Meeting the needs of global
business through virtual team projects. Higher Education, Skills and Work-Based
Learning, 12(2), 346-368. https://doi.org/10.1108/HESWBL-02-2021-0037
253
Adopting learning circle approaches to equip academic staff for

community engaged research and learning practices
Linde Moriau1, Réka Matolay2, Emma McKenna3, Andrea Toarniczky4, Judit Gáspár5,
Márta Frigyik2, Brecht Van der Schueren1, Sinead McCann6, Caroline McGowan6,
Catherine Bates6
1
Department of Education, Vrije Universiteit Brussel, Belgium, 2Corvinus Science Shop,
Corvinus University of Budapest, Hungary, 3Science Shop, Queen’s University Belfast,
United Kingdom, 4Department of Organizational Behaviour, Corvinus University of
Budapest, Hungary, 5Department of Decision Sciences, Corvinus University of Budapest,
Hungary, 6Programme for Students Learning with Communities, Technological University
Dublin, Ireland.
Abstract
The purpose is to explore the potential of Learning Circle (LC) approaches
with an aim of equipping academic staff for Community Engaged Research
and Learning (CERL) practices. We draw on the experiences from a three-year
Erasmus+ project, CIRCLET. It aimed to meet the demand to better align
higher education with the needs of the rapidly changing 21st century society,
by enhancing the professional development of academic staff and fostering a
culture of engagement. The article presents a case study-based argument that
the LC – as a community of practice approach – is an effective instrument for,
and has the potential to tackle, many of the challenges of professional
development. We draw on a combined method, building case studies from post-
interviews, personal notes and observations. We present four different cases,
and offer suggestions for how LCs created a safe space, thus enabling learning
at different levels which supported academics to build CERL into curricula.
Keywords: Learning circle; community engaged research and learning;
science shop; community of practice; professional development.

Adopting learning circle approaches for community engaged research and learning practices
1. Introduction
The purpose of this article is to explore the potential of Learning Circle (LC) approaches to
equip and support academic staff in their professional development for Community Engaged
Research and Learning (CERL) practices. Research indicates that CERL activities are
capable of benefiting teachers, community partners and students (Moriau et al., 2022).
However, studies also highlight that academic staff do not always feel equipped to properly
design and support such practices, stressing the need for (continued or more appropriate)
professionalisation opportunities (Boland, 2014). In recent years, LC approaches have
emerged as a promising solution for supporting academic professionalisation. LCs provide a
safe and supportive environment for academic staff to reflect on their practices, share their
experiences, and engage in ongoing learning. Through regular meetings and activities,
participants in a LC can build a supportive network of peers and engage in meaningful
dialogue and reflection on their teaching and research practices. In this article, we will
explore both strengths and points of attention for adopting LC methodologies with an aim of
supporting quality implementation of engaged research and learning in academic curricula.
Our research is rooted in CIRCLET (Curriculum Innovation through Research with
Communities: Learning circles of Educators and Technology), a three-year, five partner
Erasmus+ Strategic Partnership Project that ended in 2022. It aimed to enhance the
professional development of lecturers for CERL via facilitated participatory peer learning
processes with the contributions of various stakeholders – community partners, CERL
brokers and support staff, students, educational development staff, university management
and involved 104 academic participants.
This paper is organised in three more parts. In the next chapter we introduce the concepts
discussed, including CERL, LC and professional development. It is followed by our research
methodology and we elaborate on four cases, describing the adopted approach and its impact
on the professional development of the participating lecturers.
2. Theoretical Background
In this chapter we provide conceptual foundation for both CERL and LCs and provide a
picture on contemporary issues and trends regarding professional development of academic
staff.
2.1. Conceptualising Community Engaged Research and Learning

CERL aligns with a socio-constructivist methodology (see e.g. Palincsar, 1998), which
supports learners to co-create knowledge. Our approach draws on constructivism,
experiential learning and critical pedagogies. The relationship between these has been
explored by Tassone et al. (2018). According to Boland (2014) these “approaches to teaching
256
Linde Moriau, et al.
and learning share an explicit civic focus and combine the features of experiential learning
with opportunities for engagement” (p. 180). In CERL students and lecturers are working
with community partners on collaboratively-designed, real-life projects, for mutual benefit,
embedded in the curriculum (McKenna, 2022). Community partners can have a wide variety
of formats from Civil Society Organisations (CSOs), charities, non-profits, via social
enterprises, social cooperatives, statutory bodies, schools, to other communities (Bates et al.,
2022).
CERL intertwines the three missions of higher education institutions: Teaching/Learning,
Research, and Engagement. It is often supported by Science Shops, which are often
university-based units (Fokkink & Mulder, 2004) providing responses to questions of
communities and civil society organisations through research and learning. Although
operations and formats of Science Shops vary, their community “demand driven and bottom-
up approach” (Steinhaus, 2014, p. 72) are common characteristics.
2.2. Contemporary Trends Regarding Professional Development of Academic Staff

The key question underpinning this study is how obstacles to professional development of
academic staff can be tackled by the adoption of LC approaches. Professional development
activities are often seen as an effective means of encouraging academics to alter their teaching
practice. Conventionally, the aim of professional development activities has been to provide
them with new knowledge, building on the assumption that this will act to alter their beliefs,
which in turn will motivate them to rethink their classroom practices. Yet, studies indicate
that teachers’ beliefs are resilient, and as Guskey (2002) argues, most often beliefs only
change after lecturers themselves have experienced the effectiveness of an alternative
practice through improved student outcomes.
Xethakis (2019, p. 46) states that “for any proposed change or reform to be effective and take
hold in the classroom, teachers themselves have to buy into the proposals, change their way
of thinking about teaching and adopt new classroom practices themselves”, calling for
models of professional development that allow teachers to take a more active role in their
professional development. More impactful professionalisation initiatives, according to Drits-
Esser and Stark (2015), demand a shift in agency – a move “away from programs that focus
on creating change in teachers” to activities that focus on “providing opportunities for active
engagement, influencing teachers to take responsibility for their own learning and to reflect
on their practice” (Drits-Esser & Stark, 2015, p. 1). Building on these observations, one might
argue that a different conception of professional development is needed. One in which
lecturers themselves generate the kind of knowledge that can aid them in altering their beliefs
and adopting new practices in order to achieve the successful implementation of reforms and
curriculum changes.
257
2.3. Conceptualising Learning Circles as Community of Practice (CoP)

Our study builds on a LC approach. LCs are small gatherings of people who meet to study a
subject of interest to its members. LC members share their knowledge and experience, learn
to apply and test new information, skills, methods and ideas. The goal of LCs is to support
participants to develop new practices or action plans they can take back to their working
contexts. For the purpose of this study, we will frame LCs as a Community of Practice (CoP).
A CoP promotes collaboration and relationship-building among its members, leading to a
sense of belonging and support (Wenger et al., 2002).
It could be argued that LC approaches offer a flexible, relevant, and sustainable alternative
to traditional training initiatives aiming to inspire academic development of university
lecturers and researchers. By being participant-driven, LCs ensure that the content covered
is relevant and responsive to the needs of academic staff, resulting in a more personalised
and effective learning experience. The peer-to-peer nature of LCs helps to ensure their
sustainability over time, as participants are able to continue learning and sharing with each
other beyond the formal learning event. LCs promote collaboration and relationship-building
among participants, leading to a sense of community and support.
3. Research Method
This qualitative research applies a case study method to show the practical relevance, and the
diverse applicability of the approach applied in CIRCLET. It builds on the works of
Eisenhardt (1989) and Yin (1994) for whom the main goal of the case study methodology
was to build a theory. This aim has been slightly revised as building and showing viable and
inspiring practices with useful implications both to theory and practice. According to Yin
(1994) case studies investigate a contemporary phenomenon – in this case: CERL, supported
by the LCs – within its real-life context, “especially when the border line between the
examined phenomenon and context cannot be drawn clearly, therefore there is an opportunity
for a deeper analysis of contextual factors” (p. 13).
A combined method was used to select cases with a goal of finding confirming, and
theoretically supported ‘typical’ cases (Miles & Huberman, 1994, p. 28.), while also showing
challenges and opportunities. Of the five universities participating in the project, four adopted
similar approaches and these were selected as cases to include for analysis. Due to a very
different local context, the fifth university chose an alternative LC aim, process and structure
which is not discussed in this paper. Following Yin (1994) the research relied on data from
several sources such as LC plans and facilitation guides, interviews with facilitators,
reflective journals of participants in the LC process (including participant lecturers and
facilitators), and participant observation from other contributors, all of these inputs are
coming from different angles for triangulation reasons.
258
4. Findings and Discussion

The LC processes were collaboratively designed across the four cases and consisted of a
series of whole group workshops interfused with small group discussions. Since LCs are
driven by the needs and expectations of the participants to a large extent, the shared basic
design resulted in local variations based on local considerations. Table 1 introduces the key
aspects of LC cases in a summative and comparative manner. Besides enumerating the
considerations in setting up and running an LC, the table also provides insights on the
similarities and differences of local LCs. The Shared/Varied column gives an overview of
the cases from this respect.
Table 1 also describes foci of participants’ diversity, a key element for encouraging peer
discussions and learning on CERL approaches which inherently build on diverse voices. LCs
were designed to make space to incorporate the various experiences of participants and to
integrate the areas of expertise of the lecturers, to give them the opportunity to showcase and
develop their strengths (e.g., their expertise in their field of research and teaching; their
experience of being engaged with community partners outside of work, etc.). Key LC
elements included implementing experiential learning cycles, identifying learning styles (see
Kolb & Kolb, 2005, Manolis et al., 2013), and using reflection tools supporting reflection by
lecturers as well as learning about reflection in a way to support student reflection in CERL
projects (Matolay & Frigyik, 2022).
Our data offers insights into the experiences of participants in the CIRCLET project.
Participants highlighted the feeling of being part of a safe and supportive group and the ways
in which this served as a foundation for awareness raising and learning necessary to build
CERL projects into curricula. They appreciated meeting “people who are like minded”. This
sense of being part of a community created commitment towards the group, its members and
to support each other's learning by sharing their own experiences.
In addition, they highlighted that their experiences in the LC created the condition not only
to reflect on their teaching practices, but to discover their blind spots, those values, beliefs
that may limit their learning new ways of teaching for CERL:“It was quite unexpected to see
how similarly we reacted when ... told us about the communication of one of her projects. It
was uplifting to see that academics all around Europe face the same challenges and struggle
with mental blocks that stop us from doing something in a new way”.
Participants learned teaching techniques to support CERL, that may be put into practice
immediately. This gave a sense of learning about changing their teaching practices but does
not necessarily reflect a readiness for more in-depth changes. The first step towards in-depth
changes were made by increasing awareness on their teaching practice through a better
understanding of CERL: “Community partner speakers ... allow for deeper discussion of the
topics, how to do things better, what to keep, what lose and how to develop further”.
259
Participants learned from each other, so next to relevant content and techniques, they also
learned how to get in meaningful contact with each other, and with other relevant
stakeholders, the community partners and students: “I particularly found useful hearing the
feedback of a student who took part in a CERL course and [of] a community partner.”
Table 1. Learning Circles: Design Considerations and Variations for Embedding CERL.
Aspects Description Shared, Explanation

of LCs of Aspect Varied of Similarities/Differences
Key foundations of Learning Circles for CERL
Goal Professional Shared All LC processes were set up with the shared goal
development of of promoting CERL via engaging and upskilling
lecturers to embed lecturers. Focus was on already existing courses
community with the potential to experience transformation for
engagement into CERL and to prevent obstacles of major changes
existing courses of – e.g. launching new courses in – curricula
Format Facilitated peer Shared LC design for learning with and from each other
learning with the potential to build a CoP, facilitated by
CERL experts/support team
Other Reflection Shared, Reflection by lecturers and equipping lecturers to
Varied support student reflection were central to all LC,
various tools and activities applied though
Learning Circle Participants
Numbers 5-12 lecturers per Varied The general aim was 6-10 persons, this varied due
LC to differences in numbers of local applications
Diversity experience Shared A key aim – reinforced in the call for and selection
- in teaching of participants – was to create diverse groups
- in CERL within the LCs to allow for peer learning
- disciplines
Learning Circle Process and Structure
Length 1 or 2 semesters Varied Building on CERL principles, the 1st was designed
and decided collaboratively, the 2nd was
customised by feedback/needs of local participants
and facilitators
1st semester 3+3 Shared 3 large group sessions of 2-3 hours + each
process followed by 1-hour small group sessions
2nd - Same as in 1st Varied Based on CERL principles, considerable variation
semester - Check-in and – where appropriate to local context, facilitators
processes show & tell worked one-to-one with academics or continued
sessions with full LCs and/or small groups
- Indiv. check-ins
Access Online, in-person, Varied Online due to local characteristics (e.g. distances
both between campuses) and to COVID. In-person
options offered where possible
Source: Compiled based on data collection for cases and Matolay & Frigyik (2022).
The stress and time pressures of academic life impacted the CIRCLET project, resulting in
participants not being present physically or emotionally, not having time to read and prepare,
or to share and fully support peer learning. This generated further negative feelings, an
260
internal conflict: “sadly I had to miss some, but that is not a reflection on my peers but me
as a member”. Lack of time may limit the desired learning and change too: “interesting
conversations, but the benefits are not always that visible or plausible (maybe need more
time to mature)”, “It was an excellent source of learning, exchange of ideas and
experiences!” The LC community could be a source of inspiration and practical learning,
with experiences and impacts on a spectrum from limited reflection to in-depth changes
related to the lecturers’ identity and/or teaching practice.
5. Practical and Theoretical Implications

In conclusion, LCs offer important vehicles for professional development of academic staff
in the context of CERL. CERL is acknowledged as a challenging pedagogy to implement,
and the opportunity for peer-learning, self-reflection and sharing of experience offered by the
LC approach was evaluated as successful by project facilitators and participants,
notwithstanding the time pressures that many experienced. Participants particularly valued
hearing the experiences of others who had undertaken CERL projects, including students,
community partners and other lecturers, and valued the role of the facilitators in supporting
relationship-building with community partners. Whilst participants acknowledged the
challenges for workload in implementing CERL, many have gone on to extend and deepen
their CERL practices. Through CIRCLET, 53 modules were redesigned and over 3,000
students participated in CERL projects.
LCs were seen as safe and inspiring learning spaces, where a sense of belonging was created,
and peer learning was possible. The diversity of participants and the experience sharing
by/with different stakeholders led to a better understanding of CERL, increased awareness
on their own teaching practice and changes needed. The latter requires sustained effort from
the individual even after the LC: learning in LCs serves as foundation, and the relationships
may continue to fuel peer learning, even against the experienced time and workload pressure.
References
Bates, C., McCann, S. & McGowan, C. (2022). Guide for facilitators: Online continuing
professional development module on embedding community engaged research and
learning in higher education curricula. CIRCLET Erasmus+ Project Report. url:
https://circlet.eu/online-module/
Boland, J.A. (2014). Orientations to civic engagement: Insights into the sustainability of a
challenging pedagogy. Studies in Higher Education, 398(1), 180-195. doi:
10.1080/03075079.2011.648177
Drits-Esser, D., & Stark, L.A. (2015). The impact of collaborative curriculum design on
teacher professional learning. Electronic Journal of Science Education, 19(8), 1-27.
261
Eisenhardt, K.M. (1989). Building theories from case study research. Academy of
Management Review, 14(4), 532-550. doi: 10.5465/amr.1989.4308385
Fokkink, A. & Mulder, H.A.J. (2004). Curriculum development through science shops.
Environmental Management and Engineering Journal, 3(3), 549-560. doi:
10.30638/eemj.2004.052
Guskey, T.R. (2002). Professional development and teacher change. Teachers and Teaching,
8(3), 381-391. doi: 10.1080/135406002100000512
Kolb, A.Y. & Kolb, D.A. (2005). Learning styles and learning spaces: Enhancing experiential
learning in higher education. Academy of Management Learning & Education, 4(2), 193-
212. doi: 10.5465/amle.2005.17268566
Manolis, C., Burnett, D.J., Assudani, R. & Chinta, R. (2013). Assessing experiential learning
styles: A methodological reconstruction and validation of the Kolb learning style
inventory. Learning and Individual Differences, 23, 44-52. doi:
10.1016/j.lindif.2012.10.009
Matolay, R. & Frigyik, M. (eds.) (2022). Guide for facilitators – Learning circles for
community engaged research and learning. CIRCLET Erasmus+ Project Report. url:
https://circlet.eu/learning-circles-guide-for-facilitators/
McKenna, E. (2022). Guide for lecturers: Resources to implement community engaged
research and learning in university teaching and pedagogy. CIRCLET Erasmus+ Project
Report. url: https://circlet.eu/implementing-cerl-guide-for-lecturers/
Miles, M.B. & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook.
Thousand Oaks, CA: Sage Publications.
Moriau, L., Tondeur, J., Bertone, J., Huysmans, M., Temmerman, M., & Meurs, P. (2022).
The engagement CUBE: a dialogical tool for designing, facilitating and monitoring
engaged research and teaching strategies. International Journal of Sustainability in
Higher Education, 23(4), 783-798. doi: 10.1108/IJSHE-03-2021-0126
Palincsar, S.A. (1998). Social constructivist perspectives on teaching and learning. Annual
Review of Psychology, 49, 345-375. doi: 10.1146/annurev.psych.49.1.345
Steinhaus, N. (2014). “With or Without You” – The development of science shop and their
relationship to higher education institutions in Europe. In Munck, R., McIlrath, L., Hull,
B. & Tandon, R. (Eds.), Higher Education and Community-Based Research: Creating a
Global Vision (pp. 71-83). New York: Palgrave McMillan. doi:
10.1057/9781137385284_6
Tassone, V., O’Mahony, C., McKenna, E., Eppink, H. & Wals, A. (2018). (Re-)designing
higher education curricula in times of systemic dysfunction: a responsible research and
innovation perspective. Higher Education Journal, 76(2), 337-352. doi: 10.1007/s10734-
017-0211-4
Yin, R.K. (1994). Case study research design and method. Sage.
Wenger, E., McDermott, R. & Snyder, W.M. (2002). Cultivating Communities of Practice.
Boston, MA: Harvard Business School Press. doi:
10.5465/amle.2009.41788855Xethakis, L. (2018). Collaborative curriculum
development: A tool for change. In Bollen, D., Morrow, J.S. & Xethakis, L., (Eds.),
SULTF2018 Selected Papers (pp. 45-54). Kumamoto, Japan: Nankyu JALT.
262
Interdisciplinary week in game design: a learning experience

Bárbara Barroso, Inês Barbedo
Research Centre in Digitalization and Intelligent Robotics (CeDRI), Laboratório para a
Sustentabilidade e Tecnologia em Regiões de Montanha (SusTEC) and Instituto Politécnico
de Bragança, Portugal.
Abstract
Interdisciplinarity promotes competencies like asking meaningful questions
about a complex problem, examine, and synthesize multiple sources of
information, methods, and perspectives, in order to integrate knowledge and
ways of thinking across two or more established disciplines to produce
cognitive advancement. The Interdisciplinary Week of Game Design
challenges the students to demonstrate an interdisciplinary understanding of
a complex problem that students define, organized by teams, having as its
starting point a given theme. Teamwork between members of different
academic years favors the sharing of knowledge among peers with different
aptitudes, technical skills, and degrees of competence.
Keywords: Interdisciplinarity; project-based learning; soft skills; assessment.

1. Introduction
The higher education system has been changing rapidly in recent years, with a greater
emphasis placed on developing not only hard but also soft skills, the inter and intra-personal
skills required to be effective in the workplace, a raised awareness about cognitive diversity
and aligning aspects (cognitive, affective, behavior and context) of self-regulated learning
(SRL). Two of the proposed approaches to developing these skills are through
interdisciplinarity and project-based learning. Interdisciplinary competencies and soft skills,
as the ability to understand issues in a holistic way, to connect both the analysis of problems
and the devising of solutions with relevant disciplinary knowledge and methodologies, and
to reflect on the role of scientific research in solving societal problems (Fortuin et al., 2013)
can be highlighted on a curriculum vitae (CV) or in a job interview as contributions that a
person brings to a future work team, cutting across academic and professional contexts.
In the interdisciplinary paradigm, students must ask meaningful questions about a complex
problem, examine, and synthesize multiple sources of information, methods, and
perspectives, from two or more disciplines, see how they intersect, and use reflective skills
to carry out the solution of a problem or learning outcome (Ashby & Exter, 2019, Repko et
al., 2019;). At its core is the ability to integrate knowledge and ways of thinking across two
or more established disciplines to produce cognitive advancement in ways that would have
been impossible or improbable through mono-disciplinary means (Boix et al., 2000).
However, pedagogical approaches that challenge students to demonstrate interdisciplinary
understanding remain relatively limited (Repko et al., 2019), and there is no single pedagogy
that facilitates interdisciplinary teaching and learning (Ashby & Exter, 2019).
In recent years, interdisciplinary teamwork has become increasingly relevant in higher
education (AWTI – The Dutch Advisory Council for Science, Technology and Innovation,
2022; Ashby & Exter, 2019). Interdisciplinary trends also surface in publication data,
research agendas, and interdisciplinary innovation hubs (Klaassen, 2018). Various authors
point out that at the meeting point of different perspectives there is a space that stimulates
critical thinking, which allows the development of new knowledge, where the co-
construction of learning is triggered (Akkerman & Bakker, 2011; Almasi, 2016). Engaging
students in interdisciplinary experiences helps them develop higher-order metacognitive
skills, guiding them to synthesize disciplinary knowledge to devise innovative solutions
(Ashby & Exter, 2019). Thus, interdisciplinarity often materializes in global thematic areas
such as sustainability, entrepreneurship, or big data, for example, where different disciplines
come together to create solutions, products, or joint explanations of the world (Lam et al.,
2014). This suggests a high level of breadth and complexity in the problems to be addressed.
As the implementation of the European Higher Education Area (EHEA) has evolved towards
shifting the focus of attention from teaching to learning (Moya, 2017) and to the students'
264
own experiences (López-Pastor et al., 2013), the factors that allow students to be aware of
their strengths and weaknesses have been studied. One of them is assessment and the way it
responds to the students' need to judge their own practices and increase their autonomy and
self-management skills (Calkins et al., 2018). The triple assessment approach, which
combines self-assessment, peer assessment and teacher assessment in the same instrument,
before the final grade of a given assessment procedure (Pérez-Pueyo et al., 2019), responds
to this issue. Peer assessment is a very useful learning strategy for improving the feedback
process in students (Kuo et al., 2017), fostering critical thinking (Topping, 2009), and studies
have shown that it is remarkably robust in a wide range of contexts (Double et al., 2020).
Self-assessment contributes to promoting self-regulation and awareness of learning by the
student, helping train future responsible, critical, and reflective professionals (Pérez et al.,
2022). Encouraging student participation throughout the assessment process also favors
extrapolating the learning to various contexts.
Research has also suggested encouraging student innovation by supporting their autonomy
during learning tasks (Martín et al., 2017). Project-based learning (PBL), which is based on
the science of active construction of learning (Krajcik & Shin, 2014), emphasizes the
construction of knowledge. Product creation is one aspect that differentiates PBL from other
forms of learning. Product creation is important because it helps students integrate and
rebuild their knowledge, discover, and improve their professional skills, and increase their
interest in and ability to work with others. Therefore, the final products are the concentrated
expression of various competencies that students can develop during PBL (Guo et al., 2020).
Other authors have stated that the PBL method positively contributes to increasing student
motivation to participate in the learning process (Pan et al., 2019).
Furthermore, understanding how teams effectively deal with continuously and rapidly
changing contexts is an important issue for organizations. Teams capable of effectively
responding to shifting conditions and aligning their member resources into processes that
generate consistency in performance, are more beneficial to organizations than teams that fail
to cope with changes in the environment. A moderate amount of cognitive style diversity has
been shown to enable such team ability since having less doesn’t provide the cognitive
capacity and flexibility to take on tasks that invoke different ways of encoding and processing
information, while too much may negatively impact cooperation and coordination.
Additionally, these team skills or collective intelligence, forecast the rate at which teams
improve in their implicit coordination, processes which are exceedingly important in high-
resistance organizations (Aggarwal et al., 2019). Therefore, interdisciplinary project-based
learning can leverage valuable team skills awareness and perception that will support students
in their future endeavors.
265
2. Interdisciplinary week
The Interdisciplinary Week (IW) is a period, in the middle of the academic semester, in which
all students of the Game Design degree, at the School of Public Management,
Communication and Tourism of the Polytechnic Institute of Bragança, divided into small
teams, reflect on a specific topic, in the form of a response to an unstructured problem,
defined by them, in the three fundamental areas of the degree: game design, computer science
and visual arts.
The IW aims to develop skills that we can divide into four fundamental transversal
competencies (Sá & Serpa, 2018) – problem solving, teamwork, communication, and self-
management; and, furthermore, create positive and productive attitudes, solving problems as
a team through collaboration, debate and the sharing of ideas, research, flexibility, and
interpersonal skills.
It should be noted that the design and development of games implies demonstrating the
thinking, language, and attitudes of different disciplinary fields that are related to each other,
and an integrative approach is key to finding new solutions to problems in this context.
2.1. The process

The week begins with an informative session in which the objectives and skills to be
developed are described, the topic to be explored is revealed, the daily goals to be achieved
are presented, and the week's agenda is defined. and work groups are formed and registered.
The activities, using Discord as a complementing platform, are carried out in face-to-face,
synchronous on-line moments and autonomous work, which can be on-line or face-to-face,
with the aim of meeting the goal of each day:
Goal of day 1 – Problem Definition: through a brainstorming process and based on the
references provided by the teachers and the research carried out by the group, the students
define what they want to delve into within the proposed topic and record what they prove in
the form of a question to examine;
Goal of day 2 – Executive summary: definition of what the student team intends to develop
in response to the question they have defined, which can be a game, an animation, an app, a
website, an interactive narrative, a service, etc.;
Goal of day 3 – Prototype development: through rapid prototyping, the student team begins
to implement the core features and must register what they still intend to develop;
Goal of day 4 – Prototype and poster: refinement of the prototype and articulation of the
result in the form of a poster;
266
Goal of day 5 – Final pitch: Through a 5-minute pitch, each the student team presents the
problem, argument, project features, work process, and demonstration of results.
All results and processes are recorded in templates provided for this purpose and are reviewed
throughout the design and development work cycle.
On the second and fourth day there are moments of interaction between peers, fostering
dynamics between students that contribute to the construction of useful feedback to the
design and development cycle and to the structuring of reflective skills that, in turn, will
inform the formative evaluation.
During the week, teachers supervise the work from their perspective of the topic, for which
they are invited to make a short presentation.
2.2. Assessment
The developed work is evaluated among colleagues from the same team and between teams,
by all those interested in the developed projects, and by the coordinating committee,
contributing up to 10% to the classification in all curricular units in which the student is
enrolled in the semester.
In order to assess students’ progress in the project’s evolvement, quality feedback is sought
every day, on the one hand to facilitate students’ development and task improvement, on the
other as a contributor to the quality of the student experience and as an enabler of team’s
identifying their strengths and weaknesses to further improve (Lizzio & Wilson, 2008).
Assessment is understood as integral to the learning process and, as such, it is the locus of
letting students formulate personal learning goals, activating prior knowledge to support
debate, teachers informing students (collectively or individually) on what to accomplish next,
as well as activating students as instructional resources for one another and as holders of their
learning paths during the week, according to formative assessment strategies (Leenknecht et
al., 2021). Therefore:
Feedback of day 1 – the student teams pitch the ideas discussed in the brainstorming process
in order to define the problem and teachers give oral feedback in order to develop and
improve the task;
Feedback of day 2 – each team registers the feedback resulting from the interaction with other
teams and gives feedback to other teams about the conceptual results already achieved, as
well as developing an Executive Summary for the project taking all collected feedback into
account;
Feedback of day 3 – the students start prototyping, articulating their ideas in a product and
pitching the first implemented features as well as identifying what they want to do; other
267
teams give structured feedback identifying a positive aspect, something to improve and also
giving a suggestion about something that could be added;
Feedback of day 4 – teams test the prototypes from each other and register the feedback
gathered from observation of testing and debate during the interaction;
Feedback of day 5 – on the last day teams publish a poster of the project and pitch the final
results gaining feedback from those in terms of generated debate between teams as well as
voting.
Hence, to complement what has already occurred in terms of assessment throughout IW, at
the end of day 5’s tasks the developed work is evaluated:
- from the student’s perspective, by filling in a form with questions relating to the internal
teamwork, interaction with other teams, the contributions and fulfillment of the tasks
performed, and reflecting on learning throughout the process;
- from the point of view of all those interested in the developed projects, by voting for the
ones they consider the 3 best projects considering the theme/problem identified, the
originality, the degree of development, and the quality of the presentation;
- from the perspective of the coordinating committee, by evaluating all documentation
produced during the process and the active participation of students.
3. Results
The IW has now been running for eight editions and has contributed greatly to the culture of
the students and the degree in Game Design. The exploration of a topic allows unfolding
research methodologies, supporting the double approach necessary for interdisciplinary
thinking – having knowledge and having skills (Chatterjee & Das, 2021). Based on the
iterative cycle of design and development, the learning process is itself iterative, with goals
in which questions are asked.
Teamwork between members of different academic years favors the sharing of knowledge
among peers with different aptitudes, technical skills, and degrees of competence.
Widespread interactions between peers, as well as between students and teachers, combine
to develop a critical stance essential to move away from notions of absolute knowledge and
consider and apply different points of view and reconsider the strategies used within the
project, favoring intellectual maturation of the students. The templates that support the
autonomy of the students during the learning tasks in the IW prove to be sufficient
scaffolding. They also enable a focus on personal characteristics, described in the literature
as sub-competencies of interdisciplinary thinking: curiosity, respect, openness, patience,
diligence, and self-regulation (Chatterjee & Das, 2021).
268
Table 1. Interdisciplinary Weeks that already occurred.
Number of Number of
Date Theme
students the projects
1st 26.11 to 30.11.2018 INTERCONNECTED 109 21
2nd 01.04 to 05.04.2019 (UN)USEFULL 88 15
3rd 11.11 to 15.11.2019 POST-HUMANISM 160 22
VIDEOGAMES 2020
4th 26.11to 03.12.2020 international 112 25
conference
5th 06.04 to 09.04.2021 MEMORY 89 16
06.12 to 10.12.
6th FUTURES 128 23
2021
7th 04.04 to 08.04.2022 MAGIC 95 18
8th 14.11 to 18.11.2022 DOPPELGÄNGER 127 22
The final results presented in the IW are usually divided into three types of artifacts –
documentation, physical objects, and digital prototypes. As a concentrated expression of the
various skills developed by the students, they are an incorporation of the reconstruction of
their knowledge, a reflection of the discovery and improvement of their skills of ideation,
critical analysis, discursive articulation, communication, and collaborative capacity. They
also demonstrate productive moments in the student's academic itinerary.
References
Aggarwal, I.; Woolley, A. W.; Chabris, C. F. & Malone, T. W. (2019). The Impact of
Cognitive Style Diversity on Implicit Learning in Teams. Frontiers in Psychology,
10:112. doi: 10.3389/fpsyg.2019.00112
Akkerman, S. F. & Bakker, A. (2011). Boundary Crossing and Boundary Objects. Review of
Educational Research, 81, 132–169. https://doi.org/10.3102/0034654311404435.
Almasi, J. F. (2016). Crossing Boundaries in Literacy Research: Challenges and
Opportunities. Literacy Research: Theory, Method, and Practice, 65, 24–46.
https://doi.org/10.1177/2381336916661542.
Ashby, I. & Exter, M. (2019). Designing for Interdisciplinarity in Higher Education:
Considerations for Instructional Designers. TechTrends, 63, 202–208.
https://doi.org/10.1007/s11528-018-0352-z
AWTI – The Dutch Advisory Council for Science, Technology and Innovation. (2022).
Unbounded research. Promoting interdisciplinarity through two distinct government
roles. Advisory report. Available: www.awti.nl.
269
Boix Mansilla, V., W. C. Miller & Gardner, H. (2000). On Disciplinary Lenses and
Interdisciplinary Work. In: Wineburg, S. & Grossman, P. (eds.) Interdisciplinary
Curriculum: Challenges of Implementation, 17–38. New York: Teachers College Press.
Calkins, A., Conley, D., Heritage, M., Merino, N., Pecheone, R., Pittenger, L., ... & Wells, J.
(2018). Five Elements for Assessment Design and Use to Support Student Autonomy.
Students at the Center: Deeper Learning Research Series. Jobs for the Future
Chatterjee, S., & Das, S. (2021). Interdisciplinary Education Trends in School and Higher
Education: A Review. Advances in Science Education, 12, 12-18.
Double, K. S., McGrane, J. A. Hopfenbeck, T. N. (2020). The Impact of Peer Assessment on
Academic Performance: A Meta-analysis of Control Group Studies. Educational
Psychology Review, 32, 481–509. doi: 10.1007/s10648-019-09510-3
Fortuin, K., Van Koppen, C., & Kroeze, C. (2013). The contribution of systems analysis to
training students in cognitive interdisciplinary skills in environmental science education.
Journal of Environmental Studies and Sciences, 3(2), 139–152
Guo, P., Saab, N. Post, L. S. & Admiraal, W. (2020). A review of project-based learning in
higher education: Student outcomes and measures. International Journal of Educational
Research. 102, 101586. doi:10.1016/j.ijer.2020.101586.
Klaassen, R. G. (2018). Interdisciplinary education: a case study. European Journal of
Engineering Education, 43(6), 842-859. doi: 10.1080/03043797.2018.1442417
Krajcik, J. S., & Shin, N. (2014). Project-based learning. In R. K. Sawyer (Ed.). The
Cambridge Handbook of the learning sciences (pp. 275–297). (2nd ed.). doi:
10.1017/CBO9781139519526.018
Kuo, F., Chen, J., Chu, H., Yang, K., & Chen, Y. (2017). A Peer-Assessment Mobile Kung
Fu Education Approach to Improving Students' Affective Performances. International
Journal of Distance Education Technologies (IJDET), 15(1), 1-14. doi:
10.4018/IJDET.2017010101
Lam, J. C. K., Walker, R. M. & Hills, P. (2014). Interdisciplinarity in Sustainability Studies:
A Review. Sustainable Development, 22 (3), 158–176. doi:10.1002/sd.533.
Leenknecht, M.; Wijnia, L.; Köhlen, M.; Fryer, L.; Rikers, R. & Loyens, S. (2021). Formative
assessment as practice: the role of students’ motivation. Assessment & Evaluation in
Higher Education, 46(2), 236-255, doi: 10.1080/02602938.2020.1765228.
Lizzio, A. & Wilson, K. (2008). Feedback on assessment: students’ perceptions of quality
and effectiveness. Assessment & Evaluation in Higher Education, 33(3), 263-275, doi:
10.1080/02602930701292548.
López-Pastor, V. M., Pintor, P., Muros, B. & Webb, G. (2013). Formative assessment
strategies and their effect on student performance and on student and tutor workload: the
results of research projects undertaken in preparation for greater convergence of
universities in Spain within the European Higher Education Area (EHEA). Journal of
Further and Higher Education, 37(2), 163-180, Routledge. doi:
10.1080/0309877X.2011.644780
Martín, P., Potočnik, K., & Fras, A. B. (2017). Determinants of students’ innovation in higher
education. Studies in Higher Education, 42(7), 1229–1243. doi:
10.1080/03075079.2015.1087993.
270
Moya, C.E. (2017). Using Active Methodologies: The StudentśView. Procedia - Social and
Behavioral Sciences, 237, 672–677. doi: 10.1016/j.sbspro.2017.02.040
Pan, G., Seow, P.-S. and Koh, G. (2019). Examining learning transformation in project-based
learning process, Journal of International Education in Business, 12 (2), 167-180. doi:
10.1108/JIEB-06-2018-0022
Pérez, M. C. I., Vidal-Puga, J. & Pino Juste, M. R. (2022). The role of self and peer
assessment in Higher Education. Studies in Higher Education, 47(3), 683-692. doi:
10.1080/03075079.2020.1783526
Pérez-Pueyo, Á., Hortigüela Alcalá, D., Gutiérrez-García, C., & Hernando Garijo, A. (2019).
Andamiaje y evaluación formativa: dos caras de la misma moneda. Revista Infancia,
Educación y Aprendizaje, 5(2), 559–565. doi: 10.22370/ieya.2019.5.2.1775
Repko, A. F., Szostak, R., & Buchberger, M.P. (2019). Introduction to Interdisciplinary
Studies. 3rd ed. SAGE Publications. ISBN: 978-1544379401.
Sá, M. J. & Serpa, S. (2018). Transversal Competences: Their Importance and Learning
Processes by Higher Education Students. Education Sciences, 8(3), 126. doi:
10.3390/educsci8030126
Topping, K. J. (2009). Peer Assessment. Theory Into Practice, 48(1), 20-27. doi:
10.1080/00405840802577569
271
“Strategy just isn’t like that”. A case study on the use of a

coopetitive game to experience the strategy roller coaster
Jonathan Fanning
School for Business and Society, University of York, United Kingdom.
Abstract
This is a case study of teaching strategy based around a geo-political
“fantasy” simulation to give students a taste of both theory and practise.
It was designed as the public image of consultants offering little useful help
has a basis in truth. Many do not have a firm grasp of the realities of “doing
strategy”. They fail to understand that problems of implementation are not
individual failings but inherent to the academic teaching of strategy based on
economic models. They either teach a rational approach to decision making
or produce populist texts based on emotions and apparent first-hand
knowledge without evidence. Both approaches suffer from a reality gap either
by abandoning practice or academic rigour.
The intention is to introduce students to the emotional roller coaster that is the
strategy journey. Strategy is about relationships and conflicts, emotional
interactions in a world of bounded rationality, and negotiating within and
between participants.
Keywords: Strategic Planning; simulation; educational games; emotion;
teams; communications; skills; negotiation.

“Strategy just isn’t like that”. Case study on the use of a coopetitive game
1. Introduction
There has long been a public image of a know-nothing consultant who enters public sector
institutions and offers little useful help and lacks understanding of the culture of non-profits.
In doing so they cause more problems than they cure as they try and introduce marketization,
or some other concepts adapted from models and concepts developed for profit-oriented
organisations. Although a crude stereotype this has a basis in truth. Many consultants and
strategy educators do not have a firm grasp of the realities of “doing strategy”, they have
learnt only through applying models to case studies and trying to copy from the greats.
The possibility that the problems of implementation are not individual failings but inherent
to the academic teaching of strategy, including political strategy, is seldom addressed. They
teach a rational approach to decision making based on carefully designed models crafted by
intensive research based around industrial economic models and then express surprise when
people do not respond. Others abandon any pretence at academic rigor and produce populist
texts based on emotions and apparent first-hand knowledge without evidence, often these sell
well at airports but last only as long in the memory as the single serving meal provided on
the airline (Mintzerg, H. Ahlstrand, B. W., Lampel, J, 2008). Both approaches suffer from a
reality gap as there is a need to maintain academic rigour while introducing students to the
emotional roller coaster that is the strategy journey, while these approaches only, at most, do
one or the other.
I struggled for many years to answer the following questions in a manner that provided for a
good student experience:
1. How, in the classroom, do you best capture the strategy journey in complex and
changing environments?
2. How do you teach emergent strategies rather than just prescriptive?
3. How do you replicate the complex, messy nature of the strategy world?
As we looked into these questions it became clear that many of these missing elements of
strategy are also key skills many employers are looking for, and that employers often criticize
universities for not producing these in students, and that this is not a coincidence.
I decided to answer these by making a dramatic change in approach by introducing a dynamic
strategic geo-political simulation with elements of both competition and co-operation, set in
a fictional world where great powers struggle to forge alliances across a steam punk continent
emerging from years of domination and oppression to new statehoods. This is a case study
report on this simulation or game.
274
Jonathan Fanning
Figure 1The Big Continent: The World of The Game.
275
2. Simulations
The use of simulations is not in itself a new idea, though the intensity and the academic
interaction is greater in this than in any others of my experience, which have tended to be
either short or more intensely related to employability skills and not theory, this game
develops both and is thus a learning innovation.
2.1. Assessing Learning

Capturing and assessing the learning was a challenge as the importance of the experiential
learning had to be driven in via assessment or many students would not engage (Hand, L.,
Sanderson, P., O’Neill, M., 1996) and therefore 20% of the marks are allocated by a peer
review and negotiation under strict conditions to stop agreements to merely share marks and
with a backstop of a lower grade should the students fail to reach agreement, 20% by the
ability to act strategically in a group and 60% individual reflective diaries kept during the
game.
2.2. Groupthink
The natural tendency to groupthink in strategy is difficult to overcome, and the anxiety levels
of this module are often high exacerbating this,
The students are a highly diverse group, normally all continents except Australia (and
Antarctica) are represented, and the involvement develops both academic skills. such as
critique and synthesis, and employability skills including difficult to develop ones such as
Resilience, Cultural Awareness and Adaptability.
3. Interest
The module has proved very successful with students, with very positive feedback but also
very good outcomes with a number using the game in interviews and applications and
reporting employer interest.
Example student feedback includes:
• “the simulation game triggered extreme enthusiasm”

• “the simulation game helped us to realize how to Do not only Explain things”
• “the game was run over one whole term, which helped to try new approaches and
acknowledge the messy, emergent character of strategy”
• “The module was very interactive providing opportunities for learning that a normal
lecture and seminar would not do”
276
Jonathan Fanning
• “I definitely enjoyed participating in 'the game' - …….. Also, I believe that by

having practical sessions, it …. allowed me to understand what it is like to be a
stratigikós stochastic.”
A recent further innovation is the development of an advisory group for this module that
consists of former students, 5 of whom returned to the university recently to talk to the present
students as guest lecturers and also to give a careers talk in the evening attended by over 200
students, which is more than three times bigger than the department previous record
attendance. The advisory group has a none binding role in designing the module and the
assessment and has proved useful in improving both and is a major contributor to positive
student outcomes. outcomes.
4. Method
In the department of Politics at the University of York, on the MPA/MA module Strategic
Planning, a simulation based around a scenario where teams representing international great
powers have to deal with dividing up a continent of recently independent nations to make
allies and drive development and/or exploitation was developed and introduced.
Teams are provided with details of their initial resources and over a period of 9 weeks they
make a series of moves and develop and implement a strategy. Unusually for such a “game”
this one allows for extensive cooperation as well as intense competition, and the individual
roles within teams also are a source of cooperation and competition.
Each week of the game students are introduced to new strategic models and they are guided
in their groups to apply them to the game. They learn to recognize the strengths of the models
but also the weaknesses of them in a dynamic environment influenced by the behaviour of
others.
Over the period of the game there are several events that affect the players and the game
board, randomly assigned, and the students need to react to these while building up their
assets. They negotiate both with the game masters and with other players, and they receive a
final group mark after which they must negotiate with their fellow team members to receive
a share (equal awards are forbidden by the rules). This way the students are introduced to a
real impact of their game playing.
5. Covid Adaptions
Covid caused the game to be moved onto the VLE creating unique challenges but the virtual
tools that were developed to allow diverse and dispersed students to work together have
proved so successful they have now been implemented into the game, much of the work is
277
extremely varied and helped to improve student engagement to a huge extent both during
covid and after.
6. Conclusion
Many students struggle at first, the intensity is high and they feel the pressure, time can be
an issue but eventually it settles down and feedback suggests students enjoy the module and
find it useful.
Success measures include:
• High Average grade;

• High “Real Learning”;
• Good understanding of theory and reality;
• Creation of community of learning that continues after graduation;
• Students start to care.
However, a problem has been the development of the whole gamut of emotions, with
examples of:
• Tears;
• Fights;
• Walkouts.
During the game, but in the long run the feeling is the experience has been worthwhile for all
involved.
References
Carter, C., Stewart R., Clegg, Kornberg, M., A Very Short, Fairly Interesting and Reasonably
Cheap Book About Studying Strategy
Chapman, J. (2006), Anxiety and defective decision making: an elaboration of the groupthink
mode Management Decision, Vol: 44, Issue: 10
Hand, L., Sanderson, P., O’Neill, M., Fostering deep and active learning through assessment,
Accounting Education, Vol. 5
Minter, H. Hallstand, B. W., Lempel, J., Strategy bites back: it is a lot more, and less, than
you ever imagined ... /, 2008.
278
Combining computer-based training, virtual, or augmented

reality with peer teaching in medical and bio–technological
education
Interdisciplinary Biomedical Education and Research Center BioMed Laboratories, Munich
University of Applied Sciences, Germany.
Abstract
The Interdisciplinary Biomedical Education and Research Center (BioMed) at
our university offers a modern approach to biomedical education that
addresses the challenge of understanding complex medical fundamentals and
devices and their application in a clinical setting. This is especially relevant
for our students of biomedical engineering, biotechnology, or bioengineering.
The concept combines real medical devices with advanced simulation
technologies and realistic training through computer-based training (CBT),
virtual reality (VR), and augmented reality (AR). Peer teaching, small group
activities, and tangible CBT are also incorporated into the hands-on approach,
along with a focus on training with current and future technologies to prepare
the students for the medical research/-engineering industry. Students
responded overwhelmingly positively to the first peer-taught course that
utilized VR and AR e-learning experiences, resulting also in improved exam
results. This article provides an overview of the concepts used and their
implementation in the biomedical engineering curriculum at our university.
Keywords: Blended learning, virtual reality, augmented reality, education,
peer teaching, computer-based training,

Combining computer-based training, VR or AR with peer teaching in medical education
1. Introduction
In a rapidly changing environment where new technologies emerge frequently, it is crucial
to keep up with progress and integrate state-of-the-art as well as upcoming innovations into
medical and biomedical curricula. Only by doing so can students be prepared for their future
careers. Peer teaching has been shown to benefit academic performance, particularly in more
practical subjects (Brierley et al., 2022), and the potential of technology such as virtual reality
(VR) or augmented reality (AR) to enhance education has already been demonstrated (Wu et
al., 2020). Studies have also shown that using 3D models to teach anatomy is superior to 2D
learning materials (Ye et al., 2020). Peer teaching can foster teamwork, collaboration, and
communication skills as students work together to teach and understand the material, which
are essential skills in both academic and professional settings. When students are actively
involved in teaching each other, they are more likely to retain the information, as they are
forced to think about the material in a different way and explain it to their peers. However,
in many cases, medical education is still taught in isolation from medical devices, neglecting
the circumstances under which medical equipment (e.g. intensive care or emergency gear) is
used or relying on traditional, non-interactive teaching resources like front-line teaching.
At the Interdisciplinary Biomedical Research and Training Center (BioMed), directed by
Prof. Dr. med. Dipl. Ing. Christian Hanshans, a holistic concept has been developed and
successfully implemented in cooperation with the University Hospital of Würzburg. It
combines traditional teaching methods (such as lectures, self-study, and textbooks) with
advanced digital supplements, including VR or AR-based simulations (e.g., 3D anatomy and
physiology training), case-based training, anatomical models, and wall charts. The approach
also includes computer-based training with learning success control, peer-guided tutorials,
and the option for unguided self-study using all media. Furthermore, medical devices are
demonstrated in a hands-on manner within a realistic clinical environment (such as functional
diagnostics, operating rooms, or research settings). This hybrid teaching approach integrates
anatomical and physiological expertise with technical skills and cross-links different classes,
leading to a better understanding of complex subjects and promoting method competency.
The purpose of this paper is to introduce the holistic approach and present preliminary data
on student feedback obtained through evaluations, as well as compare pre- and post-
intervention exam scores.
2. Concept structure
The learning concept is divided into three main subject orientations: (1) anatomical and
physiological basics, (2) pathologies and medical imaging, and (3) operating principles of
medical equipment and fundamentals of biomedical signal measurement. The concept is
designed to provide students with fundamental knowledge in the first semesters, followed by
280
an introduction to how dysfunction of the previously introduced organs leads to diseases and
disorders. In subsequent semesters, the focus shifts to the application of medical devices for
proper diagnosis or therapy. The following paragraphs will provide a detailed description of
each of the aforementioned categories.
2.1. Anatomy and physiology

As students begin their studies, they often struggle with the sheer amount of material they
have to learn in a short amount of time. The courses, enriched with media and supported by
tutors, provide an opportunity for students to deepen their understanding of anatomical
structures and physiological processes using VR and AR. By allowing students to explore
complex medical procedures, anatomy and physiological systems in a 3-dimensional and
interactive way (see Figure 1), VR can provide a highly immersive experience for students.
With VR or AR they get one more opportunity to consolidate the information they have
already learned from lectures or books. Take the human heart, for example. You can read
about its structure, how electrical impulses are generated and conducted, and when the
different valves open and close. But because pictures cannot convey a sense of depth or
movement your next step is to look at a plastic model to combine the 3D perspective with
tactile feedback. But that still doesn't do much for your understanding of bio-mechanics or
physiology. And this is where VR can be a great addition to the teaching portfolio. You can
walk around a 3D animation of the heart contraction while simultaneously comparing it to
the output of an electrocardiogram (ECG). This way, students learn to directly associate the
state of the contraction with the spikes in the ECG. Unlike a video of these mechanisms in
VR, students can scale, rotate, scroll through layers, hide and show structures like arteries,
veins, or muscle fibers at will.
Another benefit of using VR and AR is that it allows for targeted teaching and learning.
Bioengineering students, for example, focus more on histology and cellular processes,
students of clinical optometry need to know more about the anatomy of the human eye, its
diagnostic, related pathologies and clinical therapy, while the curriculum of biomedical
engineering is aligned to the operating principle of medical sensors and devices.
A key factor of this learning concept is the use of different media and senses to approach the
same topic. For example, as shown in Figure 1b , students can take a skeleton apart and do a
practice exercise to find a particular bone. Seeing it on the AR image, they can compare and
reinforce their knowledge by trying to find the same bone on a solid anatomical model and
with that associating the same information with a haptic sensation.
281
Figure 1 a) Student while studying cardiac anatomy and physiology using a virtual reality headset. 0000b)
Students using an AR software on a tablet to virtually dissect a skeleton while plastic 0000000models like the one
in the display case in the background are used for comparison.
2.2. Pathophysiology and medical imaging

Various pathophysiological conditions such as tumor genesis, cardiovascular diseases, or
neurological disorders are implemented in the curriculum through the use of VR, AR, or
animations. This approach helps students to comprehend the underlying mechanisms and
effects of these conditions on the human body. Additionally, it provides insights into the
dynamics of diseases, such as the Covid-19 infection (see Figure 2).
Subsequently, students are taught about various medical imaging techniques such as X-rays,
CT, and MRI, to enable them to comprehend the principles and applications of these methods
and improve their ability to interpret and analyze medical images. This approach also
provides a cross-reference to their prior knowledge of anatomical structures and pathologies.
282
Figure 2 Displaying the progression of a virtual Covid-19 infection on a patient simulated to be in a real-life
training replica of an intensive care unit with real medical equipment, using AR glasses.
2.3. Functionality and use of medical equipment

Finally, students have the opportunity to apply the theoretical knowledge gained from
lectures to a variety of real medical devices presented in a realistic setting. Armed with their
knowledge of anatomy, physiology, and pathology, they have the basic skillset to better
understand the requirements of medical devices and the needs of their users, such as doctors
and nursing staff. The actual use or simulation of medical devices and procedures prepares
students for their future work environment in clinics or medical device development. It is
highly motivating and reinforces the knowledge acquired in previous semesters. VR is a
valuable tool to simulate equipment that cannot be obtained for safety (e.g. infectious
diseases or radiation), logistical, or ethical reasons (section course or visiting a real surgery
or intensive care unit with an entire class), or due to financial constraints. This hands-on
training can boost their confidence and prepare them for real-life situations. For instance, if
they are tasked with improving medical instruments or devices, they will be better able to
assess their clients' needs because they have gained experience using them.
3. Impact on student satisfaction and exam results

By providing a more interactive and engaging learning experience than traditional classroom
and book-based instruction, this multimodal approach can increase student engagement and
motivation. To investigate this hypothesis student evaluations were conducted. A five-point
Likert scale (1 - strongly disagree, 2 - disagree, 3 - neither agree nor disagree, 4 - agree, 5 -
283
strongly agree) and free text fields for positive and negative feedback were used to evaluate
the teaching concept. For this sample, the course "Human Biology" in the 2022 summer
semester was used. This module combines the stated learning objectives of basic anatomy
and physiology, pathology, and medical instrumentation. In addition, it is offered in three
different undergraduate programs (Bioengineering n=26, Medical Technology n=12, and
Clinical optometry n=10) resulting in a diverse total cohort of n=48 students participating
(30 female,17 male, 1 Not specified).
Many students praised the vividness that a VR or AR implementation allows. Statements
like: "When I was learning the eye movements, I had a hard time understanding which muscle
did what just from the script. Being able to turn the virtual eye the way I wanted and then let
one muscle contract helped a lot." or "I found the animation where you could adjust the
severity of the macular degeneration with the slider to see how someone sees the world with
this condition very vivid" where submitted. As shown in Figure 3a, 92% of participants
agreed or strongly agreed that VR helped them understand anatomy more than the lecture
alone would have. The responses to the statement about the benefits of combining VR and
AR with plastic models and the lecture for learning anatomy in Figure 3b were even clearer,
with 94% of students agreeing or strongly agreeing.
Figure 3 a) Results of the student evaluation regarding the benefits of adding VR/AR compared with the sole use
of lectures for learning anatomy.
b) Results of the student evaluation regarding the benefits of combining VR/AR with the use 00of plastic models
and the lectures for learning anatomy.
Some reported that they were more motivated to engage with the content because of the way
it was delivered. They stated "It was easier to motivate me to go to the lab for tutoring than
to go to the library because it feels less like learning and you have more fun in a group." and
“Because looking for bones on the tablet was like playing a game, I can remember them
better now.“ Feedback also emphasized the immersive nature of the experience. “It was kind
of depressing to see the Covid patient having more and more trouble breathing. It really
looked like he was sitting there on our bed struggling to breathe while his machines were
284
beeping.” one student wrote in their text field. The peer-teaching approach also received
positive feedback. Praise like "I liked that I could just ask stupid questions. ","The patellar
tendon reflex was well explained by the tutor." or "The tutor's mnemonic about the exits from
the aortic arch was funny" was given. The survey results demonstrate that the first statement
quoted was not an isolated case. 73% (n=35) fully agreed and 27% (n=27) agreed with the
statement “Being supervised by a student tutor instead of the professor made me feel more
confident to ask questions.”
There has been negative feedback as well. Some students reported discomfort and dizziness
while using the VR goggles. Three students had to stop the exercise early due to motion
sickness. It has also been criticized that the university does not provide funding to make the
software shown in the tutorial available to students as a campus license. Because of this, the
software can currently only be used by one student at a time due to the single license, which
has sometimes led to considerable waiting times. As a result, students who wanted to use the
software on the tablet always had to come to the university to use the lab's device-locked
license instead of being able to use the software at home for private study.
In addition, a comparison of pre- and post-intervention exam scores was made using the
bioengineering students as an example. This group was chosen because they made up the
majority of those attending the tutorial and were therefore the most representative group in
terms of the effectiveness of the tutorial. A peculiarity worth mentioning is that the Munich
University of Applied Sciences offers the possibility for students not to take an exam, even
though they are registered, without having to face any consequences. Especially in the
medical subjects only a fraction (40 - 70%) of the students enrolled in the course actually
take the exam. The subject Human Biology was chosen for the evaluation because this basic
subject is perceived as particularly difficult by students and tends to have lower results than
other subjects. In 2021 n=33 and 2022 n=30 of n=59 in 2021 and n=62 students in 2022
visiting the lectures that took the exam were compared. In both years, the exam was an online
electronic exam (using Moodle) with a mixture of multiple/single choice questions, matching
questions, picture marking questions, and mathematical problems. The grade average (1 to
6, with 1 being the best and 6 the worst) showed a visible improvement in the grade
distribution (2021: x = ̅ 3.50, x =
̃ 3.30; 2022: x ̅=3.07, x ̃=2.85). Although two more students
failed in 2022, the upper range of scores improved considerably. While in 2021 no student
scored between 1.0 and 1.7, the number increased by 5 in 2022 (1.0 n=2;1.3: n=1 1.7 n=2).
The correlation between the introduction of the peer-taught tutorial and the improvement in
grades suggests that this intervention may be particularly helpful for students in the lower to
middle performance spectrum to achieve better exam results.
285
4. Conclusion
Overall, the use of a consistent implementation of blended learning with traditional and peer
teaching, the introduction of interactive technologies such as VR, AR, case based training
and real-world models or devices in a realistic environment can elevate the education of
medicine and biomedical technology to a new didactical level. However, although the student
evaluations and improved exam results provided a valuable first glimpse into the full
potential of this broad didactic concept, further studies are needed to objectively quantify the
success of the intervention on a larger scale and to identify and eliminate potential
confounding factors. Yet despite the limitations this study showed that blended learning can
promote motivation, comprehensive understanding, enhances social skills and collaboration
and allows competence oriented teaching. In addition, students come in contact with
emerging technologies (such as VR, AR) and are provided with an immersive learning
experience, which can improve their understanding and readiness for real-world scenarios
and their future professional career.
References
Brierley, C., Ellis, L., & Reid, E. R. (2022). Peer-assisted learning in medical education: A
systematic review and meta-analysis. Medical Education, 56(4), 365–373.
https://doi.org/10.1111/medu.14672
Wu, B., Yu, X., & Gu, X. (2020). Effectiveness of immersive virtual reality using head-
mounted displays on learning performance: A meta-analysis. British Journal of
Educational Technology, 51(6), 1991–2005. https://doi.org/10.1111/bjet.13023
Ye, Z., Dun, A., Jiang, H., Nie, C., Zhao, S., Wang, T., & Zhai, J. (2020). The role of 3D
printed models in the teaching of human anatomy: A systematic review and meta-
analysis. BMC Medical Education, 20(1), 335. https://doi.org/10.1186/s12909-020-
02242-x
286
They want to fly! – International students attitudes concerning the

climate crisis and their air travel behavior
Dimitri Prandner
Faculty of Social Sciences, Economics & Business, Johannes Kepler University of Linz,
Austria.
Abstract
What do international students think about the impact their travels have on
climate change?
As the relationship between academia and higher education institutions with
regards to protecting the environment and mobility continues to be complex,
universities and other higher education institutions are trying to reduce their
carbon footprint. However, international students often travel globally for
their study programs and other associated trips, raising questions about their
awareness concerning the impact of their travels on climate change.
A qualitative case study conducted in two international study programs in
Austria reveals the presence of five different types of students. While most
students are aware of climate change, only one group is taking active steps to
address it at an individual level. Furthermore, there is also a distinct group of
students who are unaware of basic climate change-related issues.
The findings of the study highlight the importance of incorporating climate
change education into international study programs on both a curricular and
practical level, from the admission process to program-related travels. It is
crucial for universities and higher education institutions to address this issue
and promote sustainability in international student programs.
Keywords: Climate change; international study programs; case study; travel
behavior.

They want to fly! – Inter. Students attitudes conc. the climate crisis and their air travel behavior
1. Introduction – The complicated relationship between climate change,

internationalization in academia and higher education
What do international students think about their contribution to climate change and climate
change-aware behavior overall? The answer should be simple, as the effects of human-
induced climate change have led to a wide range of increasingly catastrophic events. The UN
is urging nations around the world to take substantive action to combat climate change (IPCC,
2022). Despite decades of research supporting this call to action and more than 196 parties
promising to fulfill the Paris Agreement to limit global warming, actual actions are limited
(Nunez et al., 2019). As a result, research and higher education institutions must increase
awareness and strengthen their efforts to combat the climate crisis (Borgermann et al., 2022).
However, the situation in universities and higher education institutions is not straightforward.
From a normative perspective, internationalization and mobility are considered essential for
academic excellence (Altbach et al., 2009). Consequently, many universities and research
institutions aim to attract a diverse range of international scholars and students (Uzhegova &
Baik, 2022), and they actively organize events such as conferences, research stays, staff and
student exchanges, and international study programs (Nursey-Bray et al., 2019). Most of
these activities depend on extensive travel, often via heavily greenhouse gas-emitting planes.
This tension between academic internationalization and climate change has sparked a
widespread discussion in the international higher education community.
At the organizational level, research and educational institutions are reassessing their
ecological impact, taking steps to reduce it, and trying to act as role models when it comes to
sustainability (Eskander & Istiak, 2022; Filho et al., 2022). Many universities have
established guidelines to evaluate and restrict staff travel, substituting on-site events with
virtual or hybrid ones, and fostering the creation of an environment that allows for more
decentralized research (Filho et al., 2022; Nikula et al., 2022).
When it comes to faculty, the scientific community recognizes the impact of their intensive
travels and saw the COVID-19 pandemic as an opportunity to establish a new normal,
permanently cutting back on travel and greenhouse gas emissions (Kreil, 2021; Filho et al.,
2022). However, individual scholars report that they are afraid of competitive disadvantages
if they are not mobile and perceive air travel as intricately tied to doing their work 'well'
(Kreil, 2021, p. 60).
For students, the situation is somewhat similar problematic, but even more pronounced.
While many students are "demanding action, as evident in the agendas and activities of
groups such as Students Organising for Sustainability International in the UK, the Young
Academy in the Netherlands, and the Erasmus Student Network in Europe" (Nikula et al.
2022, p. 2), recent surveys indicate that international students, in particular, do not care about
the environmental impact of their travel and greenhouse gas emission. For example, only
288
Dimitri Prandner
about 5.5% of Erasmus students engage in sustainable behaviors during their international
travels, according to a report by Green Erasmus (2022, p. 26). This supports previous research
suggesting that despite the rapid pace of climate change and the potential catastrophes it may
bring in the future, many international students view global travel and study abroad
experiences as a right rather than a privilege (Reilly and Senders, 2009).
These findings highlight a gap in understanding between organizations and researchers in
higher education and students regarding the impact of international mobility on climate
change. While institutions and scholars recognize the need to reduce their carbon footprint,
students' attitudes towards sustainable behaviors while traveling can be paradoxical.
Therefore, this paper aims to contribute to this field of research by addressing the following
questions:
1. What perceptions and attitudes do students of international study programs show
regarding climate change?
2. How do they evaluate their personal environmental impact?
3. How do they view internationalization as well as their own mobility, when it comes to
dealing with the issue of flying and greenhouse gas emissions in particular?
To gain a deeper understanding of this complex issue, a qualitative case study was conducted
in Austria during the winter semester of 2022/2023, focusing on two groups of international
students – one attending a bachelor’s program at the Johannes Kepler University of Linz and
one a master’s program at the Paris Lodron University of Salzburg. The second section will
provide a short discussion about the state of the research in the field, before section three
outlines the study design used, describes the sample, and gives insight into methodology. The
fourth section will present the empirical findings and provide an in-depth analysis of the data,
highlighting the different types of students that could be identified in the case study. The
paper will conclude by drawing relevant conclusions and offering recommendations on how
international study programs may address climate change education and foster an
environment that encourages students to be more aware and conscious of their actions in an
international setting.
2. State of knowledge
In response to the urgent issue of climate change, the academic community has initiated a
self-reflection process on its greenhouse gas emissions. Numerous universities are
implementing policies to reduce their environmental impact, and research shows that
scientists are increasingly mindful of their international travel (Eskander & Istiak, 2022;
Filho et al., 2022). Similarly, there is a growing body of research that examines students'
attitudes towards climate change. While students generally display a high level of awareness
of the issue, awareness and concern tend to be greater among those studying science and
289
academic subjects rather than practical or applied programs (Eskander & Istiak, 2022).
Moreover, international students and those in international study programs exhibit notable
differences from their peers:
1. International students, particularly those from developing countries or regions severely
affected by climate change, tend to display a relatively high awareness of climate
change (Uzhegova & Baik, 2022; Eskander & Istiak, 2022).
2. Students participating in international study programs, both short-term and full
Bachelor's and Master's degrees, are more likely to engage in environmental campaigns,
use public transportation, and recycle than regular students, but show little willingness
to limit their travel behavior (Green Erasmus, 2022). Case studies have demonstrated
that the level of education on climate change is less important than engagement in
climate change relevant actions and workshops (Akrofi et al., 2019).
3. Researchers have noted a cognitive dissonance among students in recognizing the need
for action to limit the effects of climate change and committing to changes in their
personal behavior, particularly with respect to travel and consumption (Green Erasmus,
2022; Kreil, 2021; Nikula et al., 2022).
Given these observations, it is necessary to explore international students' views on climate
change, their responsibility with respect to the climate, and how they perceive their academic
and non-academic travels in this context (Reilly & Senders, 2009).
3. Study Design and Sample

The presented study is based on qualitative interviews conducted with students from two
different international study programs at two medium-sized universities in Austria - one for
a Bachelor's degree program that focuses on management and the other for a Master's degree
program that focuses on leadership. Both programs target international students, require them
to spend time studying abroad after their initial stay at the host university, are taught entirely
in English, and emphasize in their mission statements that their students are expected to
become leaders and/or internationally active experts in their respective fields. The study
involved interviewing eight students from the Bachelor's program and 21 students from the
Master's program. The interviews were conducted using a peer process, where trained student
interviewers interviewed the selected group of students to minimize the effect of social
desirability. Cluster sampling was employed to select the classes for the interviews, and all
students within each selected class were interviewed. The selected class for the Bachelor's
program started their studies in October 2021, so they were in their second year of studies,
while the selected class for the Master's program started in October 2022, and they were in
their first year of the Master's program. The interviewees were given the flexibility to choose
290
Dimitri Prandner
the time and place for their interviews. Those were competed in CAPI – computer assisted
personal interviews – mode and thus the talks were in person, recorded and transcribed.
The age of the respondents ranged from 20 to 36 years. Three of the bachelor students
identified as male, five as female. When it came to the master students seven were men, and
fourteen were women. While most of the people in the bachelor’s program came from Europe
(two eastern Europe, five central Europe, one from Asia), the masters’ program had a more
diverse class. Around a quarter of them came from Latin America, another quarter from
South-East Asia, a quarter from central to northern Europe. The rest came from the Middle
East. One individual was from the USA.
The average length of an interview was around twenty minutes, with the shortest one taking
ten minutes, and the longest one forty. Before conducting the study, interviewers were
introduced to the interview guide, could prepare themselves for one week and could request
coaching from the head researcher. While only a limited number of interviewers requested
coaching, all of them had some training on the method in corresponding classes.
A semi-structured interview guide was used, including a consent form as well as a metadata
sheet to be completed by the interviewers. The first question dealt with the interviewees travel
behavior, before continuing with questions about the relevance of travel for their personal
and professional lives, their attitudes towards climate change and flight shaming. The
interviews were wrapped by a summary provided by the interviewers, that offered the
interviewees a chance to amend their statements or add further information.
Next to the four main questions that had to be covered in the pre-arranged order, the
interviewers were provided with a set of follow-up questions in case initial responses were
insufficient.
The analysis of the data followed the method of typology building through qualitative content
analysis, as outlined by Kuckartz (2016). The data was systematically organized, taking into
account the relevant aspects previously identified (see section 2). Specific responses were
categorized based on commonalities, coded accordingly, and then compared and contrasted
with each other. Demographic information was also taken into consideration during this
inductive research process. The following section presents the results in the form of a
classification.
4. Results and Classification

Taking the interviews into account, five distinct types of students could be identified:
(A) The first type of student comes mostly from less developed regions in Africa or South-
East Asia and either consciously or unconsciously contributes to actions against climate
change, such as protesting against deforestation or promoting sustainable agriculture.
291
However, this group shows little concern about their (air) travel behavior and sees it as a
means to create professional and social bonds and pursue individual dreams, prestige, and
stature. Anchor quote: “[Air Travel] improves the quality of life where people travel to pursue
dreams, connect, and gain massive achievements... I’m not seeing myself as environmentally
conscious when it comes to traveling... but I want to contribute to a safe world in terms of
climate stability.”
(B) The second type of student mostly comes from Western and Northern European countries
as well as the USA. They are mostly younger, female students, with only one exception being
male. They believe in individuality and, especially, their right to travel because it is highly
important to them on a personal and professional level. They feel that they cannot be
criticized, even if they harm the environment, as their individual enjoyment is more
important. They are oblivious of environmental and societal issues at large and become
highly emotional, defending their positions. Anchor quotes: (1) “The plane would fly to
Australia with or without me... I’m an individual, I want to live my life to the fullest, and my
choices don’t matter in the grand picture.” (2) “The media says flying is really bad... But
everybody flies... It can’t be that bad for the environment as everybody does it.”
(C) The third type is the counterpart to the second one and consists of older students, mostly
in the master's program, who are well-reflected on their environmental impact and individual
behavior. They come from European countries as well as Latin American ones. They are
highly aware of the issues of climate change and see hegemonic structures in place that work
against substantive change. They see it as their civic duty to act against climate change and
bring examples of their personal life, comparing their past actions with their present
understanding of the issue. They highlight that, at least in Europe, it is easily possible to
travel without resorting to air travel. Anchor quotes: “I travel by train. I plan to move from
[Country A] to [Country B] for my next study destination via sharing a pickup with my
classmates”; “I flew a lot when I was younger. I would sometimes flight-shame myself now.”
(D) The fourth group is mostly composed of students from South-East Asia and the Middle
East. Before participating in international study programs, they were not too concerned about
climate change and the impact of their often-frequent air travel behavior. However, they now
reflect that more sustainable forms of travel are available. They now recognize the dangers
of climate change and believe that individuals can make a difference. Anchor quote: “[…] I
think my travel behavior is not environmentally conscious. Especially after this interview,
I'm a bit ashamed of my answers. I've realized that some people consider other modes of
transportation because of the environment and I didn't even know that.”
(E) The final group is not particularly aware of the environmental impact of their travels or
of climate change in general. They also do not place much emphasis on travel, either because
they have traveled in the past or because they have never considered it as important and see
292
Dimitri Prandner
it as more of a necessity. They are open to learning about climate change and there is no clear
pattern in terms of their socio-demographics or background. They are a residual category.
Anchor quote: “I don't think that traveling is that important to me. It's not because of climate
change that I don't fly at the moment. I don't really care either way.”
5. Conclusion and Recommendations

The aim of this paper was to investigate the attitudes of international students towards climate
change, their personal environmental impact, and internationalization and mobility. The
results of the qualitative case study revealed worrying attitudes towards air travel and its
prioritization among most interviewees. While the results cannot be quantified, they align
with previous quantitative studies while providing a nuanced starting point for future studies,
that may aim for generalization.
Firstly, it is concerning that international students, who are being educated to become future
decision-makers, lack an understanding of the severity of the climate crisis and their own
contribution to it (Nikula et al., 2022). Such apathy and lack of awareness can have disastrous
effects on the planet and future generations. Secondly, five distinct groups of students could
be identified, with different attitudes. Of those only one group (B), consisting of young,
Western, mostly female students, who placed high value on personal enjoyment, showed no
reflection on their behavior and were dismissive. Three other groups had varying levels of
awareness and willingness to reflect on their impact, either before or after the interviews.
And the final group (E) did not have strong opinions on climate change or travel but was
willing to learn more.
Those different groups of international students with varying levels of awareness and
willingness to reflect on their impact on climate change also hint at a more hopeful outlook
concerning the issue, suggesting that universities and higher education institutions can play
a crucial role in promoting environmental awareness and sustainability among international
students. To do so international study programs need to provide transparent information on
the environmental impact of individual mobility during the admission process, creating early
awareness. Furthermore, international programs – no matter the discipline – should include
mandatory courses on climate change and embed the topic into regular classes as well. And
finally, universities offering international study programs should incentivize sustainable
mobility among their students and actively promote sustainable transportation for all
program-related travels. Grants need to be tied to climate friendly behavior.
By implementing such practices, students can continue to engage in internationalization and
mobility programs while gaining a deeper understanding and later promoting sustainable
practices, fostering a culture of environmental responsibility.
293
References
Akrofi, M. M., Antwi, S. H., & Gumbo, J. R. (2019). Students in climate action: A study of
some influential factors and implications of knowledge gaps in Africa. Environments,
6(2), 12-27.
Altbach, P. G. (2009). Peripheries and centers: Research universities in developing countries.
Asia Pacific Education Review, 10, 15-27.
Amaral, L. P., Martins, N., & Gouveia, J. B. (2015). Quest for a sustainable university: a
review. International Journal of Sustainability in Higher Education, 16(2), 155-172.
Borgermann, N., Schmidt, A., & Dobbelaere, J. (2022). Preaching water while drinking wine:
Why universities must boost climate action now. One Earth, 5(1), 18-21.
Eskander, S. M., & Istiak, K. (2022). Energy efficiency and CO2 emissions: evidence from
the UK universities. Applied Economics, 0(0), 10.1080/00036846.2022.2130872
Filho, W. L., Caughman, L., Pimenta Dinis, M. A., Frankenberger, F., Azul, A. M., & Salvia,
A. L. (2022). Towards symbiotic approaches between universities, sustainable
development, and cities. Scientific Reports, 12(1), 11433.
Green Erasmus (2022). Research on the habits of Erasmus students. Brussels, BE: Green
Erasmus.
King, L., van den Bergh, J., & Kallis, G. (2022). Transparency crucial to Paris climate
scenarios. Science, 375(6583), 827-828.
Kreil, A. S. (2021). Does flying less harm academic work? Arguments and assumptions about
reducing air travel in academia. Travel Behaviour and Society, 25, 52-61.
Kuckartz, U. (2016). Qualitative Inhaltsanalyse. Methoden, Praxis, Computerunterstützung
(Grundlagentexte Methoden, 3., überarbeitete Auflage). Weinheim: Beltz Juventa.
Nikula, P.-T., Fusek, A., & van Gaalen, A. (2022). Internationalisation of Higher Education
and Climate Change: A Cognitive Dissonance Perspective. Journal of Studies in
International Education, 0(0), 10.1177/10283153221145082
Nunez, S., Arets, E., Alkemade, R., Verwer, C., & Leemans, R. (2019). Assessing the impacts
of climate change on biodiversity: is below 2° C enough? Climatic Change, 154(3), 351-
365.
Nursey-Bray, M., Palmer, R., Meyer-Mclean, B., Wanner, T., & Birzer, C. (2019). The fear
of not flying: Achieving sustainable academic plane travel in higher education based on
insights from South Australia. Sustainability, 11(9), 2694.
Pörtner, H. O., Roberts, D. C., Poloczanska, E. S., Mintenbeck, K., Tignor, M., Alegría, A.,
Craig, M., Langsdorf, S., Löschke, S., Möller, V., & Okem, A. (2022). IPCC, 2022:
Summary for policymakers. In Climate change 2022: Impacts, adaptation, and
vulnerability (pp. 3-33). Cambridge, UK: Cambridge University Press.
Reilly, D., & Senders, S. (2009). Becoming the change we want to see: Critical study abroad
for a tumultuous world. Frontiers: The Interdisciplinary Journal of Study Abroad, 18(1),
241–267.
Uzhegova, D., & Baik, C. (2022). Internationalisation of higher education in an uneven
world: an integrated approach to internationalisation of universities in the academic
periphery. Studies in Higher Education, 47(4), 847-859.
294
A project-based learning approach for engaging undergraduate

students in UN SDGs using GIS
Faculty of Engineering & Materials Science, German University In Cairo (GUC), Egypt.
Abstract
The paper presents project-based learning conducted within a Geographic
Information System (GIS) course offered to senior-level undergraduate
students enrolled in the civil engineering program at the German University
in Cairo (GUC) during the academic semester of Spring 2022. Project-based
learning enables students to achieve competencies that meet modern society
needs in relation to monitoring and assessing world countries attainment of
the United Nations Sustainable Development Goals (UN SDGs). The United
Nations Economic and Social Council (ECOSOC) established the United
Nations Committee of Experts on Global Geospatial Information Management
(UN GGIM) in July 2011, where the UN GGI mandates were identified as a
motivation for conducting project-based learning with the GIS course by the
author. The emphasis in project-based learning was on; SDG 11 on
Sustainable Cities and Communities, and SDG 13 on Climate Action. A Model
of United Nations (MUN) students’ group at the university provided a
simulated UN General Assembly at the end of the project-based learning
activities. During this Assembly, students enrolled in the GIS course, playing
roles of country delegates, delivered to the Assembly their GIS analyses and
findings on the current status of SDG 11 and SDG 13 at the end of the project-
based learning. The success of the project-based learning in achieving its
objectives, along with engagement and interest shown from both GIS enrolled
students and MUN group have resulted in plans to conduct another project-
based learning during Spring 2023.
Keywords: Project-based learning; geographic information systems (GIS);
geospatial information; UN SDG.

A Project-Based Learning Approach For Engaging Undergraduate Students In UN SDGs Using GIS
1. Introduction
Many studies have proposed project-based learning for achieving competency, where
students are active learners; integrating knowledge, skills, and values (De los Ríosa, el al,
2010). Problem-Based Learning (PBL) is a teaching method in which complex real-world
problems are used as the vehicle to promote student learning of concepts and principles as
opposed to mere presentation of facts and concepts. In addition to course content, PBL can
promote the development of critical thinking skills, problem-solving abilities, and
communication skills” (Center for Excellence in Learning and Teaching, 2022).Project-based
learning allows students to acquire newly knowledge based on what they learned earlier.
Power (2019) has indicated that student engagement and collaboration allow for the
consolidation of such new acquired knowledge. Upon the successful implementation of a
project-based learning, examples of competencies that could be achieved are; teamwork,
risk-taking, accountability, problem-solving, innovation, critical thinking, and
communication skills.
The study presented herein is concerned with making the students aware of the benefits of
geospatial information in support of sustainable development. Student awareness of the use
of geospatial information and the development of their technical skills and capabilities in
handling geospatial information is considered within a project-based learning approach.
United Nations Sustainable Development Goals (UN SDGs) indicators are practicable
objectives for an academic course. The objectives of a GIS course taught by the author at the
university to senior undergraduate students were lately augmented. Course objectives
included awareness of UN SDGs. Early in 2015, prior to the project-based learning
implementation, the UN SDGs related topics and themes were discussed through lectures
and case studies relating to GIS and UN SDGs. Low student engagement was detected in this
important topic, namely, UN SDGs. More recently during academic year 2021-2022.
The author considered project-based learning approach to further engage students in UN
SDGs, and make them more aware of the importance of the use of geospatial information and
the implementation of geospatial analyses tools and techniques when assessing UN SDGs
indicators worldwide. Project-based learing may be categorized into three “closely related
teaching techniques: case studies, role-plays, and simulations”, (Center for Excellence in
Learning and Teaching, 2022). The author selected role-plays which is most fitting for a UN
General assembly, where students play the roles of country delegates.
296
2. Project-based Learning in GIS Course
2.1. About the course

The course in the context of this paper is related to Geographic Information System (GIS)
for Civil Engineering Applications. GIS is a system designed to; collect, store, handle,
analyze, manage, and present spatially georeferenced data; i.e. referencing data to geographic
locations on the earth, commonly known as geospatial data. Further, GIS links tabular
attribute textual and numerical data with a GIS geospatial database. Through querying the
GIS geospatial database, users would be able to; spatially analyze, manage large datasets,
and display information in a map-like and other graphical representations for better insight
and in support for decision-making processes. (Dempsey, 2001).
One of the courses that the author teaches at the university is a course on using GIS in Civil
engineering applications, for the senior civil engineering students. The course introduces
senior undergraduate civil engineering students to GIS in terms of; concepts of GIS, and
practice GIS-based processes. First, students become familiar with types of geospatial data,
such as; engineering surveys, digital maps, satellite images, and Global Navigation
Positioning System (GNSS). Next, students use GIS-based geospatial analysis tools and
techniques in relation to civil engineering tasks and applications, such as; terrain mapping
and analysis, location selection, construction management, route planning, watershed
analysis, and environmental impact studies. Students become familiar through hands-on
activities with; capturing geospatial data and integrating it with attribute data as input to
develop a GIS geospatial database, use GIS-based tools and analyses to query the geospatial
database, and produce output depicting spatial relationships, patterns, and trends, using
histograms, charts, photos, satellite images and thematic maps, for better insight and in
support for decision-making processes
2.2. Motivation for the Selected Project-Based Learning

Geographic Information System (GIS) was recognized by the United Nations (UN) as an
enabler in implementing and tracking progress on the UN’s seventeen Sustainable
Development Goals (SDGs), UN SDGs targets, and indicators. Further, it was stated that GIS
and location are key technologies that enable counties to monitor and assess progress at local
levels, and in worldwide setups (Brown, C. and C. Brigham, 2021)
The United Nations Economic and Social Council (ECOSOC) established the United Nations
Committee of Experts on Global Geospatial Information Management (UN-GGIM) in July
2011, to further the cause of using geospatial information in sustainable development
processes worldwide. UN GGIM adopted a Global Statistical Geospatial Framework (GSGF)
as the basis for supporting efforts for combining geospatial and statistical information (Scott
and Rajabifard, 2017).
297
The UN GGIM in 2019 specifically demanded using geospatial information and GIS-based
analysis approach to improve the availability, quality, timeliness, and disaggregation of data
to support the implementation of the new development agenda at all levels (FIG Task Force
on FIG and SDGs (2019). In view of this important global drive that happens to overlap
with the GIS interest in utilizing geospatial information, and as the instructor of a GIS course,
the author decided to introduce a project-based to help civil engineering students become
more aware of GIS potential contributions to UN SDGs, and have the students use their
acquired GIS understanding and analyses skills, through role-playing, in project-based
approach designed specifically to engage GIS enrolled undergraduate students in UN SDGs.
Worth mentioning that over the years of course delivery, the use of GIS and geospatial
addressed UN SDGs among topics covered by lectures and reports. However, the project-
based approach presented herein was first introduced in this format in Spring 2022.
2.3. Description of the Selected Project-Based Learning

In Spring 2022, students enrolled in the GIS course were asked to form teams, and select
countries, where they acted in role-playing mode as country delegates. Using GIS, students
were to present as country delegates their respective GIS assessments of the selected SDG
indicators to a simulated UN Session to be held at the end of the semester. The emphasis in
the Project-based Learning was on; SDG 11 on Sustainable Cities and Communities (Make
Cities inclusive, safe, resilient and sustainable) and SDG 13 on Climate Action (Take urgent
action to combat climate change and its impacts). An appendix is given at the end of the study
describing the tasks required fro students.
As course instructor, along with teaching assistants, we regularly met with students during
project milestone activities. Occasional question and answer sessions were held for task
clarifications and guidance through project-based learning activities during Spring 2022
semester. Each group of students selected a country from a short- list of nine countries
(worldwide, namely; Belgium, Brazil, Chili, Denmark, Ecuador, Greece, Kenya, Mexica,
and Norway). During the course, students were asked to role-play as country delegates, and
be prepared at the end of the semester to present the report in a simulated UN General
Assembly meeting. During the semester, students researched and collected geospatial data
and conducted GIS-based analyses as the course progressed.
A Model of United Nations (MUN) at the university participated in the closing session of the
project-based learning. MUN mirrors the mandates and activities of United Nations
ECOSOC. Which made cooperating with MUN give a realistic environment for the project-
based learning. Most student members of the current MUN were not enrolled in the GIS
course, except for one student. MUN is considered one of the most popular and prestigious
student activities in the university. MUN activities included simulating different UN
functions and activities, to enhance MUN student members’ personal, intellectual and social
298
skills of the Egyptian Youth. MUN provides hands-on experience to undergraduate students,
and future leaders in the international area of diplomacy.
3. Results
Table 1 shows growing student interest and appreciation of the tasks related to the project.
Early in the second week of the semesters just more that 55% of the students were engaged
and had no inquiries or complaints, by the sixth week of the course the percentage of engaged
students were over 70%. By the end of the course in the twelve week, and before the PBL
group peresentations, all students were excited and were looking forward for presenting their
work to other student groups. To date, I still hear from alumni words of appreciation about
this PBL experience and its impact on their views of their roles of them as civil engineers.
Table 1. Growing student interest and appreciation of the tasks related to the project.
For the presentation of PBL group work, a simulated UN General Assembly was held, at the
end of the semester during June 2022. An active working group of students known as Model
of United Nations (MUN), cooperated. The venue for the in conclusion of the GIS project-
based learning activities was held on campus, in a large rounded hall; with layout setup, the
procession of the meeting, and document format exchange, method of addressing the
Assembly, were as much as possible similar to UN General Assembly official sessions.
Figure 1 presents the simulated UN General Assembly session.
Figure 1. The simulated UN General Assembly session
299
Over seventy students enrolled in CIG 1002 GIS for Civil Engineering Applications attended
as country delegates. Each country delegate delivered oral reports, in UN SDG Country
Report format, presentations supported by their GIS analyses findings on the current status
of each of the nine countries. The MUN General Assembly was held for three hours. Each
country delegate presented a short PowerPoint summarizing the SDG country report and
findings. Each country delegate was handed a UN "Resolution Template" to complete and
read to the Assembly. The event was recorded by the Media Production Center (MPC) for
documentation and to be used to introduce the project in the coming semesters to more
students. Figure 2 presents a group photo taken at the closing ceremonies of the event.
Figure 2. A group photo at the closing ceremonies of the event
4. Conclusions
The project-based learning presented herein enabled students to independently collect new
knowledge, and present it in written and oral formats that are novel to them as country
delegates during the simulated UN sessions. Hence, enhancing their technical capabilities
and soft skills , addressing one of the critical mankind challenges; namely, achieving UN
SDGs. Further, the project-based learning enhanced engineering students’ engagement and
awareness of UN SDGs. As well as, encouraged students through critical thinking to relate
UN SDGs to their engineering major, provided a platform for meaningful collaboration
among the teams of students enrolled in the course, and fostered cross-discipline between
engineering students with interest in GIS course and students with MUN student members
with backgrounds from other majors. More interesting was that the project-based learning
stimulated interest of civil engineering students in MUN. Owing to the success of the project-
based learning implementation in the GIS course, and interest of the MUN student members
at the university, plans are currently being made to conduct another project-based learning
during Spring 2023 semester within the next GIS course delivery, with a new batch of
300
students and new country delegates, ultimately working with new student country delegates
adressing simulated UN General Assembly.
References
Brown, C. , and C. Brigham (2021), How GIS and Geography are Playing a Strategic Role
in Implementing the SDGs. United Nations Statistics Division, Development Data and
Outreach Branch, United Nations World Data Forum.
https://unstats.un.org/unsd/undataforum/blog/geography-in-implementing-the-SDGs/ .
Center for Excellence in Learning and Teaching, (2022). University of Illinois Urbana-
Champaign, USA. https://citl.illinois.edu/
De los Ríosa, I. , A. Cazorlaa, J. M. Díaz-Puentea, J. L. Yagüea, (2010). Project–based
learning in engineering higher education: two decades of teaching competencies in real
environments, Procedia Social and Behavioral Sciences 2, 1368–1378.
https://www.academia.edu/11671597
FIG Task Force on FIG and SDGs (2019). Task Force on FIG and the Sustainable
Development Goals Responsibility of FIG towards the SDGs. FIG Working week 2019
Vietnam, April. https://www.fig.net/organisation/tf/sdg/index.asp
Power, R. (2019), Technology And The Curriculum: Summer 2019. Power Learning
Solutions , [eBook]. ISBN ISBN: 978-1-9993825-1-3 Surrey, BC, Canada; Available at
https://pressbooks.pub/techandcurr2019/chapter/pbl-competencies/
Scott, G. and A. Rajabifard (2017) Sustainable development and geospatial information: a
strategic framework for integrating a global policy agenda into national geospatial
capabilities, Geo-spatial Information Science, 20:2,
https://www.researchgate.net/publication/317572574
301
Appendix: Tasks Required from Students

_________________________________________________________________________
CIG 1002 GIS in Civil Engineering Assigned 17 May 2022
Spring Semester 2022
A Brief UN SDG Country Report
On Current Status With Focus On Geospatial Data
(Emphasis on SDG 11 and SDG 13)
_________________________________________________________________________
Introduction:
The scientific literature delineates numerous benefits of geospatial information in
support of sustainable development. Accordingly, the presence of geospatial information
within the United Nation Sustainable Development Goals (SDG) should provide more
quantifiable measures to the SDG indicators. The United Nations Global Geospatial
Information Management (UN GGIM) adopted a Global Statistical Geospatial
Framework (GSGF) as basis for supporting efforts for combining geospatial and
statistical information. (Baraka, 2021).
https://fig.net/resources/proceedings/fig_proceedings/fig2021/papers/ts08.2/TS08.2_ba
raka_10997.pdf
Requirements:
Accordingly, you are asked to prepare brief country report, and make a presentation
during the GUC MUN (Model of United Nation) on Thursday, 2 June 5:00 – 8:00 pm.
(location to be announced). The country SDGs report will be limited to SDG 11 and
SDG 13 goals, with focus on the role of the geospatial data.
Instructions:
1. You will be divided into 10 groups based on your tutorials. (Check GUC CMS)
2. Each group will select ONE “country” to represent, from country pool list assigned to
each group. (Check GUC CMS)
3. The representative and co-representative of each group will notify by email Prof.
Moustafa Baraka, of the country selected by his/her group, no later than Wednesday 18
May, 2022, 5:00 pm
4. As a group you will work together, prepare and upload online your country SDG report
and findings, no later than Tuesday, 31 May, 11:00 pm. (link to be announced)
5. As a group you will attend a GUC MUN General Assembly meeting event at GUC. Some
members from each group will be randomly selected during the event to present a 10
minutes PowerPoint to the GUC MUN, summarizing the SDG country report and
findings on Thursday, 2 June 5:00 – 8:00 pm. (location to be announced).
Required Report Content & Format:
A Cover page with group members names, IDs, and the country represented
Acceptable report page range 4-6 pages including maps, figures, and tables.
A reference section for all material used have to be included
Required Presentation Content & Format:
A Cover page with group members names, IDs, and the country represented
Presentation no to exceed 10 minutes; including maps, figures, and tables.
A reference section for all material used have to be included
Important note:
For resources for the report and presentation check posted: CIG1002 - GIS - UN SDG
Country Report.pdf
302
Take the challenge: compute the CO2e emissions of your

programming course
Joaquim Gabarró, Maria Josep Blesa, Amalia Duch, Maria Serna
Department de Ciències de la Computació, Universitat Politècnica de Catalunya, Spain.
Abstract
Sustainability constraints ask for quick and drastic changes in the ways to
teach at university. Here we study the amount of carbon emissions of lecturing
a first course in programming. In addition to fix costs estimations, our study
relies on a pool on student mobility patterns at our university. We estimate,
per campus and level of presentiality, the amount of CO2e emissions of the
course. Our results show that the main contributed cost of the course is not
always due to transportation. Indeed, the difference in emissions between
virtual and face-to-face is not huge and in some cases face-to-face is even
better. This offers a discussion about the convenience of replacing face-to-face
by virtual teaching through minimizing transportation emissions. We claim
that the former can be made CO2e competitive by disseminating the lectures
across the city avoiding the use of costly traveling means.
Keywords: CO2e footprint; virtual teaching; face-to-face teaching;
programming courses.

Take the challenge: compute the CO2e emissions of your programming course
1. Introduction and Methodology

As it is well known, the recent coronavirus pandemic caused drastic changes in Universities.
The most important one was having to transform teaching from face-to-face to online in
record time. The change had various consequences, some positive and others not so much.
Among the positive ones we find the beneficial impact on the environment thanks to reducing
CO2e emissions due to transport mainly in large cities. Of the refusals, there is talk above all
of the serious impact of confinement and online teaching on the mental health of young
people mainly due to the lack of socialization and movement [Sahu (2020)]. As a
consequence of this facts there is a growing discussion on the necessity of rethinking the
teaching methodology at the university level moving from traditional face-to-face teaching
to complete online teaching through virtual meetings. One of the benefits of a pure online
teaching model is the possibility of implementing the so called Distributed university [Heller
(2022)] in which a distributed effort to design the online courses could be used. Another one
is to reduce traveling time which reverts in less CO2e emissions.
In this paper, we provide a first study analyzing the teaching design among both extremes
(totally online versus totally face-to-face) taking into account the environmental impact of
the implementation of the course. For doing so we take as a base for our study the
PROGRAMMING1 course (PRO1) of the Barcelona School of Informatics (FIB) at the
Universitat Politècnica de Catalunya (UPC).
In order to estimate the total carbon emission, we separate the estimation in two parts on one
side we estimate the carbon emissions due to teaching and on the other the ones due to
transportation. For the first estimation, we take a coarse-view to the main carbon emission
sources involved in the teaching, we identify the following ones:
• Computer Usage: a running computer can generate between 52 and 234 gCO2e/h.
• E-mails: It depends on the size of the message and how large the attachments are. It
oscillates between 4 gCO2e (no attachments), to 19 gCO2e (single simple attachment), up
to 50gCO2e (large multiple attachments, e.g., images) (eCo2greetings).
• Cloud Storage: The data centers’ electricity consumption represents a total emission of
97 MT (million tonnes) of CO2e a year [Masanet et al. (2020), Obringer et al. (2021)].
• Virtual Meetings: As for a quite standard video meeting, we could assume a 1-hour
time, 25 people in it, with HD 1080p video quality. That would emit around 210 gCO2e
according to Zoom emissions’ calculator. Turning off the camera, lowering the
streaming quality and having small groups reduces the environmental footprint of the
meetings drastically (up to by 96% [Obringer et al. (2021)]).
Among the previous sources, we considered only computer usage and virtual meetings. The
main data centers’ suppliers (i.e., Google and Microsoft) declare themselves as carbon neutral
companies and are rapidly moving to buy renewable energy to match all their energy usage.
304
In view of that, we did not consider this source of emissions. The amount of paper in teaching
is very small and thus can be neglected. It is difficult to measure the e-mail traffic, however
in our experience the volume of e-mails due to teaching has been similar in face-to-face and
online teaching; therefore, we did not take it into account.
The second source of carbon emissions is transportation. An average European car emits
around 120.4 gCO2e/km, a local bus around 82 gCO2e/km and the underground 14 gCO2e/km,
approximately (European Environment Agency). These values have to be averaged by the
number of people traveling together. Nevertheless, figuring out the particular transportation
patterns of the students is not easy. In recent years, the UPC has given increasing importance
to the sustainability of its studies and infrastructures. Taking advantage of its scientific and
technical nature, and as a preliminary need to make strategic operational decisions that adapt
to the future, the UPC is carrying out very interesting studies in this regard (UPC Sostenible).
One of them is a study on the carbon footprint of student transportation to its different
campuses [Zamata Romero (2020)]. UPC campuses are distributed throughout the province
of Barcelona, but they are of different sizes and are located in cities of different size,
connectivity and importance. It can be seen from the study that there are big differences in
the transportation patterns (and consequent carbon emissions) of the different campuses. We
analyze the carbon emissions due to student mobility when face-to-face teaching is done in
one of the UPC campuses.
2. Case of Study: PROGRAMMING1 at FIB-UPC

The PRO1 course for the Bachelor Degree in Informatics Engineering at FIB-UPC receives
each semester around 700 students. It is a 7.5 ECTS course, consisting in 5 hours of lessons
per week (2-hours theory with 60 students/group, plus 3-hours lab with 20-22
students/group), plus approximately 7 hours of autonomous study. The course has two
midterm exams and a final exam. Every exam runs for 2 hours in a lab classroom (with a PC
for each student). The UPC dedicates around 20 lecturers for covering all the teaching
necessities of PRO1 and two of them take also care of the coordination. Our estimations
assume that all students attend all lectures. The results can be easily tuned to the adequate
abstention percentages, when it is the case.
In order to calculate the cost in CO2e emissions of this course, we identify the elements that
are involved in its production for each type of lecture and use the power consumption
calculators at Energuide and DisplaySpecification websites. Let us start with the lab sessions,
where every student (and lecturer) uses a PC for working [Adán Navarro (2020)]: Lightning:
600Wh (32 LED tubes); Purifiers: 170 Wh (2 per room); Monitors: 300 Wh (20 of them);
PCs: 2000 Wh (20 PC towers); Projector: 300 Wh; Jutge.org: 300 Wh (3 servers at
maximum computation).
305
We are not considering the costs of heating/cooling classrooms since this data is difficult to
estimate and, moreover, one can consider that it is probably quite equivalent to the costs
incurred by heating/cooling private homes when working 100% virtually.
We consider the differences in cost for the theory sessions with respect to the lab sessions:
• PCs and Monitors: now only one for the lecturer that consumes about 150 Wh.
• Laptops and Tablets: we observed that at most 20% of the students at FIB-UPC use an
electronic device to take notes during the theory lesson; so, assuming that 10 out of 60
do that, that represents a consumption of 800 Wh.
Gathering all together, a 1-hour lab class (for 20 students) at FIB-UPC consumes around
3670 W, while a 1-hour theory class (for 60 students) consumes about 2020 W. Thus, every
3-hours weekly lab session of PRO1 requires about 11 kW (2.85 kgCO2e, considering that a
kWh produces 259 gCO2e) and every 2-hours weekly theory session of PRO1 uses
approximately 4 kW (1 kgCO2e).
The 700 students enrolled in the course are organized in 12 groups for the theory lessons, and
in 35 groups for the lab sessions. Thus, the total approximated carbon footprint of the PRO1
lessons for those 700 students sum up to 112 kgCO2e per week. Multiplied by 15 weeks, that
represents an approximated total emission of 1680 kgCO2e per semester (that we approximate
as 1.7 mt – metric tonnes –). Please note that these costs do not include transportation, which
we take care of in the next section.
3. Different Scenarios at the UPC Campuses

In what follows we calculate the costs in CO2e emissions of two opposite (in terms of
transportation costs) models of lecturing: completely face-to-face and completely virtual.
Face-to-face Model: Many of the big Universities in the world are placed in big cities since
it is an easy way to maximize the number of students that have the university relatively near
from home. Depending on the cities, these Universities are sometimes placed in huge
campuses (where everything is centralized) or they may have its schools and faculties
distributed within the city.
There is also a type of distributed model of big university in which the campuses are around
the big city, placed in smaller cities that have a good transport connection with it. One of the
big problems with this model is the high daily traveling distance of many of the students,
since they use to live in the big city or in other smaller cities close to it. Cities are too close
to each other to make students change their living place, but too far to avoid carbon-producing
transportation. This is the case of the UPC that is distributed across the province of Barcelona
and structured in big and medium-size campuses. The bigger concentration of students is to
be found at the combination of Campus Nord plus Campus Sud, which are both placed at the
SW-border of Barcelona (at one end of the 50-meters wide grid-crossing Diagonal street in
306
Barcelona). At the opposite NE-end of the Diagonal, 12 km away from the Campus Nord and
Campus Sud, there is the medium-sized Diagonal-Besòs campus. UPC has other medium-
sized campuses in smaller cities like Terrassa and Vilanova i la Geltrú.
As currently designed, students are enrolled to a specific campus and do not have to move
through several campuses. However, lecturers might teach in different campuses on the same
day with the corresponding impact in fatigue, time consumption and CO2e emissions due to
their mobility (even if this is not significant in comparison with the travel emissions due to
all the students). As an example of this let us note that a lecturer that teaches both at Campus
Nord and at Campus Diagonal-Besòs on the same day, needs more than one hour by
(probably highly-crowded) bus to travel from one campus to the other (even that both
campuses are inside the same city, Barcelona).
Coming back to our PRO1 course, let us calculate, in terms of CO2e emissions, the cost of
transportation of the 700 students of the course to each of the UPC campuses. Just to attend
our PRO1 course, each student needs to move to campus twice a week (one day for the theory
lessons and one day for the lab sessions); this makes 4 trips a week, which for the 15 weeks
of classes it is a total of 60 trips per student. Multiplied by the 700 students of the course, it
represents a total of 42000 trips.
In order to have the corresponding impact on CO2e emissions of the total amount of trips per
campus, we multiply this amount by the average distance of trip per person by campus and
the cost in CO2e emissions per kilometer per person per campus. The obtained results are
shown in Table 1.
To obtain the total number of emissions due to the PRO1 course as if it were given at a
particular campus it suffices to sum the total emissions due to the transportation of the
students to the emissions calculated in previous section (around 1.7 mt – metric tonnes –)
where only the transportation cost was missing.
Virtual Model: Under this model, students and teachers work completely remotely from
somewhere outside the university, typically at home. A number of traditional Universities
also offer nowadays an online option for students who wish to pursue a degree but cannot
attend courses in a traditional classroom setting. The virtual university model uses the
Internet to deliver classes and seminars. As we already mentioned, this is optimal in terms of
transportation costs but it is not optimal in students’ achievements and mental health [Son et
al. (2020), Sahu (2020)].
307
Table 1. Transportation to the different UPC campuses. The 1st column states for the UPC
campus, the 2nd and 3rd columns contain information concerning students of the different
campuses individually [Zamata Romero (2020)]. The 4th column is the result of multiplying the
first two (and this refers also to a single student), and the last one represents the whole emission
of the semester (4 trips per week multiplied by 15 weeks and applied to the 700 students).
Campus avg. km/trip avg. gCO2e/km gCO2e/trip KgCO2e/course
Vilanova 40 0.8 32 1344

Nautica 13 1.46 19 798
Terrassa 22 0.25 5.5 231
Sant Cugat 19 1.05 20 840
Manresa 50 0.67 33.5 1407
Campus Sud 15 0.12 1.8 75.6
Baix Llobregat 27 0.39 10.53 443
Diagonal-Besòs 19 0.29 5.51 232
Campus Nord 19 0.13 2.47 104
In order to calculate the total cost of the course in a pure virtual model we consider as fixed
cost the around 1.7 mt (metric tonnes) of CO2e calculated previously. This is a simplification,
since we consider that the cost of lighting a classroom of 60 students is almost equivalent to
the up to 60 home-spotlights used virtually, we also consider that the cost of the projector
plus the air purifiers and others might be, roughly, similar to the cost of using personal
computers, laptops, cell phones or tablets to view the zoom transmission during theory
lectures. Considering that an hour of virtual meet for a group of 25 students contributes with
210 gCO2e, the five hours per week along the 15 weeks of the course for the 28 required
groups we obtain a total of 441 kgCO2e. The total is about 2 mtCO2e.
4. Conclusions and Further Work

On-line working, climate change and associated health problems (both mental and physical)
are today trending topics. University has to be aware of that. University studies need to be
redesigned to deal with this new reality. To do that, more data and more models should be
developed. This work is a step in this direction. Our study quantifies in terms of CO2e
emissions the cost of lecturing the PRO1 course in the different UPC campuses. The results
of our study show that, in terms of CO2e emissions, transportation is in some cases
competitive with on-line teaching. The total amount of CO2e emissions of the PRO1 course
at FIB-UPC is of about:
• 1.7 mt (metric tonnes), without taking transportation into account.

• 2mt, when taught 100% virtually.
308
• 0.5 mt are strictly due to virtuality. This is similar of the amount of CO2e due to
transportation of the Campus Baix Llobregat and –surprisingly– some campuses (such
as the Campus Nord and the Campus Diagonal Sud, for instance) are more efficient in
CO2e emissions than the fully virtual version of the course.
We can observe that the campuses with highest CO2e emissions per km, Nautica and Sant
Cugat, are small campuses, below 800 students, offering quite specialized studies. The
campuses with highest total CO2e emissions, Vilanova and Manresa, are the ones with largest
transportation time. They are located in small cities at around 50 Km from Barcelona. The
other campuses are located inside the city of Barcelona and the average CO2e emissions show
that their students reach the university mostly by low emissions public transportation.
We can draw from our studies some ways to reduce the carbon emissions due to lecturing. In
those cases, with costly travel, a balance among face-to-face and on-line teaching together
with improvement on the public transportation will reduce emissions.
Complete on-line teaching has relevant drawbacks on mental health and sociability as it was
observed in [Sahu (2020)] and therefore it should be avoided or tuned. On the other hand, as
we see it, it makes no sense that every day, a student takes private transportation to travel for
about 45 minutes to go to university. With the city design in mind, we envisage a model of
university that tries to avoid the transportation cost by “spreading” the university across the
city. We advocate for a solution having campuses distributed in the different neighborhoods
of the city in a similar model to the one of undergraduate schools agreeing with the general
design principles of the 15 minutes city planning [Moreno et al. (2021)]. With this idea in
mind it should be possible to place university at most at 15 minutes walking distance from
students and (in most cases) lecturers’ homes. The university could re-use and share existing
infrastructures like civil centers for instance which in general are underused and improve on
the assignment of students to campuses. In the case of late afternoon lectures schools could
be reused. It makes no sense to duplicate to infinity expensive technical labs. To allow the
students to work with expensive technical material they can be asked to go where those labs
are placed, in general the big campuses, but only once a week or every two weeks, depending
on the course, the lab, the specific topic, etc.
The data in which the present study is based corresponds to the first pool on mobility
launched by our university. Due to this fact, the level of participation, especially in some
smaller campuses, was low. We expect, that in the forthcoming pools participation will
improve. We plan to perform a comparative study as soon as new data is available.
We are working towards devising a model quantifying other aspects involved in the learning
process with regard to sustainable development goals. In particular, we want to relate the
learning effort and (a quantification of) the well-being with a teaching paradigm that
considers a combination of face-to-face and virtual lessons. As our goal is a medium-sized
309
macroscopic approach because we are not interested in countries but in university campuses.
One line of work is to ground the model in the Cobb-Douglas production function [Cobb et
al. (1928), Jones (2002)], that measures the GDP production of a country in terms of the
capital and the labor.
Acknowledgements
We wish thank Lluís Gabarró for bring our attention to this topic. This work was partially
supported by the Spanish Agencia Estatal de Investigación, under grant PID2020-
112581GB-C21 (MOTION).
References
Adán Navarro, A. (2020). Mòdul de correcció distribuït per a Jutge.org. Bachelor’s thesis,
Universitat Politècnica de Catalunya. (In Catalan).
Cobb, C.W. and Douglas, P.H. (1928). A theory of production. The American Economic
Review, 18(1):139–165.
Heller, R.F. (2022). The Distributed University for Sustainable Higher Education. Springer.
Jones, C.I. (2002). Introduction to Economic Growth. Norton and Company.
Masanet, E., Shehabi, A., Lei, N., Smith, S., and Koomey, J. (2020). Recalibrating global
data center energy-use estimates. Science, 367(6481):984–986.
Moreno, C., Allam, Z., Chabaud, D., Gall, C., and Pratlong, F. (2021). Introducing the “15-
minute city”: Sustainability, resilience and place identity in future post-pandemic cities.
Smart Cities, 4(1):93–111.
Obringer, R., Rachunok, B., Maia-Silva, D., Arbabzadeh, M., Nateghi, R., and Madani, K.
(2021). The overlooked environmental footprint of increasing internet use. Resources,
Conservation and Recycling, 167:105389.
Sahu, P. (2020). Closure of universities due to coronavirus disease 2019 (COVID-19): Impact
on education and mental health of students and academic staff. Cureus, 12(4):e7541.
Son C., Hegde S., Smith A., Wang X., and Sasangohar F. (2020). Effects of COVID-19 on
college students’ mental health in the United States: Interview survey study. Journal of
Medical Internet Research, 22(9):e21279.
Zamata Romero, R. (2020). La Movilidad en los Campus de la UPC. Análisis y propuestas.
Bachelor’s thesis, Universitat Politècnica de Catalunya. (In Spanish).
310
Investigating learners’ perceptions of completion and certification

in MOOCs
Meghan Perdue
School of Humanities, Arts and Social Sciences, Massachusetts Institute of Technology,
United States of America.
Abstract
Understanding learners’ perceptions of their own learning outcomes is critical
for accurately interpreting those outcomes and planning interventions to help
improve them. Past research in massive open online courses (MOOCs) shows
that many learners enroll in courses they do not finish, and much research has
been conducted investigating the patterns and trends driving this. This paper
uses a qualitative methodology to understand how learners perceive course
completion and certification, and why they do or do not meet their learning
goals. Data were analyzed from fifteen interviews with learners who had
enrolled in at least one MOOC. The data suggests that learners have a complex
understanding of completion that varies depending on their own goals and
access to the material. It also shows that they see certification as distinct from
completion, and will only be willing to pursue certification under certain
circumstances.
Keywords: MOOCs, completion, certification, retention, qualitative.

Investigating learners’ perceptions of completion and certification in MOOCs
1. Introduction
Understanding student’s perceptions of their own educational experiences is crucial towards
interpreting their behavior, and designing materials that best meet their needs. Massive Open
Online Courses (MOOCs) pose a particular challenge towards understanding learner’s
experiences, as they are asynchronous, online courses designed to accommodate a large scale
of learners with minimal instructor feedback. Online platforms have the ability to capture
user’s actions in the course, allowing researchers to study behavioral patterns and trends in
and across courses. Although this data is very rich, it does not give insights into the learners
perceptions of their own behavior and reasons motivation this behavior. Research has shown
that MOOCs have a high dropout rate, with many learners enrolling in courses that they don’t
complete (Khalil et al, 2014). Others have sought to characterize completion and dropouts in
MOOCs (Anderson, 2013; Kizilcec et al, 2013; Zhong et al, 2017), and predict what types of
learner engagement will lead to completion (Al-Shabandar et al, 2018; Kloft et al, 2014).
However, defining completion in MOOCs is complicated by the various options facing
learners enrolling in courses. Typically, learners have the option to audit most of the course
for free for a limited time period, or to pay a small fee for the opportunity to access the full
course and the opportunity to earn a certificate of completion. Much of the recent research
investigating completion rates in MOOCs focuses on learners who have paid for the course,
assuming that the act of paying signifies their motivation to complete the course (Reich &
Ruiperez-Valiente, 2019; Roy et al, 2022). However, many studies have shown that learners
have complex motivations for enrolling in courses that may not always necessitate
completing the entire courses as designed (Kzilcec & Shnieder, 2015; Milligan & Littlejohn,
2017; Salmon et al, 2017). This project seeks to shed light on learner’s definitions of
completion and certification, and the underlying reasons surrounding their course outcomes.
1.1. Related Work

Previous research has investigated learners conceptions of course completion and the reasons
behind when they do not complete. Early work from Fini (2009) in a MOOC offered in 2008
showed that learners had varying definitions of course completion, and there was not a single
standard that was uniformly shared by all learners. Researchers have since sought to unpack
the reasons behind these variations. Tanner (2013) found that learners often dropped out of
courses because they did not meet their expectations. Others pointed to a range of factors
such as conflicts arising in learners home and work lives, their perception of the course design
or pedagogy, the perceived usefulness of the material, or the learner’s ability to manage time
effectively (Eriksson et al, 2017; Hood et al, 2015; Wang & Baker, 2018). Several studies
have found that the possession of certain personal characteristics such as goal orientation,
grit, and time management had an impact on completion rates (Doo et al, 2021; Fellman et
al, 2020; Gupta, 2021; Kroll & Reed, 2017). This work has given important insight into the
312
Meghan Perdue
broad trends of learner behavior in MOOCs, though questions still remain about how the
learners describe their own course behavior and outcomes, particularly regarding when they
choose to pay or audit the course for free, earn a certificate or not, and how they define
completion. Drawing on the previous literature, this research sought to understand how
learners interpret their own completion of a course, when and why they choose to purchase
certificates, and what are the factors that they note for not completing a course or earning a
certificate.
2. Methodology
This project utilized qualitative research methods to address the research questions, as the
key interest is in the learner’s own interpretation of their learning experience. The study
population of interest were adult learners (18+ years old) who have enrolled in at least one
MOOC. The study used a structured sample (Shah & Gorbatai, 2015) which identified
learners based on pertinent variables to the study: gender, course completion, and verified
status. A sample of 60 learners was drawn from 5 courses on edX covering the topics: history,
political science, research methodology, mathematics and music. 15 learners consented to
participate in the study, which are described in Table 1. Semi-structured interviews were
conducted for data collection, and grounded theory was used for data analysis (Strauss &
Corbin, 1997). The interviews were approximately ~30 minutes to 1 hour long, and
participants were invited to speak about all of the MOOCs they had taken, not only the one
they had been sampled on. The interviews were recorded, and transcripts were created. These
transcripts were analyzed in nVivo, and coded twice. The first codes were derived from the
text itself, and a codebook was created based on these condensed codes, then the data was
analyzed again using the codebook and synethesized into the findings presented in this study.
Table 1. Demographic Percentages of Sample 1, N=15.
Gender* Course Activity* Status*
M F Sample Explore Complete Audit Verified
% 53 47 20 47 33 40 60
Note. *Indicates variable was included in the structured sample criteria.
3. Findings
Learners described a range of outcomes and reasons for those outcomes, based on their
motivations for taking the course, the quality of the course, and their own circumstances.
While definitions differed slightly, overall participants defined “completion” as having
completed the parts of the course relevant to their interests and motivations. As one learner
313
noted, “completion is to be exposed to the most relevant points of importance to me.” The
outcomes presented use this interpretation, assuming learners were able to complete the
course according to their own determination. The learners’ ability to complete the course is
additionally hampered by their willingness to pay for gated-content, with audit students
having only partial access to the material and verified students having full access. Paying for
verified status also gives learners the opportunity to earn a certificate of completion if they
obtain a high enough grade on the course. Participants described the reasons why they might
or might not pay for verified status, and the circumstances under which they would try to
earn the certificate. Table 2 presents these various outcomes for learners, and learner’s
reasons for these outcomes.
Table 2. Outcomes for learners, and learner’s reasons for these outcomes.
Completed Did not Complete
- “Well-designed, engaging course” - Unexpected

- “Complete as much as I can access for conflicts
Audit free” - “Wrong course”
Verified (No - “Well-designed, engaging course” - Unexpected

Certificate) - “Purchased course for full access only” conflicts
- “Too difficult”
- “Slacked off”
Verified - “Well-designed, engaging course” - Unexpected conflict
(Certificate) - “Need the certificate for credibility” - “Did not need too
- “Sense of accomplishment”
All varieties of learners reported outcomes of completion, regardless of verification or

certification status. They all also reported the quality of course design, and a positive learning
experience in the course as a reason for their completion. Participants noted that “[a faculty
member] who is really engaged, and seems to know their subject matter and loves teaching”
as a factor in their decision to complete. Another felt that their level of learning was an
important factor, noting “there was a lot of new information to learn, and I was enjoying the
course actually, so that’s a huge part of it.” Others noted the course design itself, including
the length of the videos and whether there was an opportunity to apply what they learned.
Many participants described being uninterested in paying for certificates, generally because
they did not have a professional use for it. As one learner explained, “For work purposes,
certificates are still important and in Brazil specially still important. But for personal things,
I don’t think this is an important thing, I mean it’s cool to show friends and family but, in the
end, it doesn’t affect much.” Others noted that the money was a factor, saying “I audit the
courses and I don’t pay, and I am actually a bit furious that they push [the price], and more
314
Meghan Perdue
and more to push for the certificate.” Still, learners reported completing as much of the course
as they had access too, saying “if I don’t pay for it, I do all of [the course] that I can.”
Some learners reported paying for verified status and completing the course, but not earning
a certificate. As one learner described, “I think for me, the problem is not the certificate, I
want to have the full access to all the content, that is why I pay the fee.” These learners may
choose to complete the course on their own schedule, once they have access to all the
materials. For learners who paid for the verified status and did earn the certificate, the reason
given was either out of a personal sense of accomplishment, or to use the certificate as a
marker of credibility for professional or personal reasons. One participant said, “I think if
one has a certificate, it also gives a sense of achievement, although maybe it won’t matter
much at my level, I still felt it like happy, okay I’ve done this.” Another reported “I took
those certificates because I wanted to be proud of my [skills] and I put it down on my
Facebook so that all my friends could see.” Some noted that they earned a certificate because
it may help them professionally in the future, “I don’t really have a use for the certificate, but
in long term I can see that my interest is in [this field] and I definitely feel like in the future
it will probably come in handy.” Many reported needing the certificate for professional
reasons, noting the “credibility”, or the importance of certificates in their work culture, “in
the Indian system they’re always looking for certificates whenever we are promoted, so you
have to get a certificate for everything.”
Learners also reported many reasons for not completing a course. No matter whether the
learner paid for verified status or not, all learners reported unexpected conflicts as a barrier
to completing the course. As one participant said “I want to complete them most of the time,
but I’ll be honest, I might have enrolled in three or four other courses which I have not been
able to finish because of others things, you know I have my work assignments, I have family
to look after, and I have all those requirements and then also do courses.” Other participants
noted that they missed important deadlines due to a conflict that caused them not to be able
to finish the course. One said, “I actually a failed a course because I didn’t make it to the final
exam, because the three days it was available I was sick.”
Many participants described exploring a course before deciding to pay, to ensure that the
course was worth taking. They described many reasons for dropping out of courses due to it
being the “wrong course” for them. This could be because the course is different than they
expected, “you realize that it’s not what is on the label, you know you get in it’s just
something else.” It could also be that the course is inappropriately challenging, “they were
way too easy, some of them are just so easy and silly”, or it is above their level, “sometimes
the content is too much”. Others felt that the faculty teaching the course was “boring”,
causing them to lose interest in completing the course.
315
Some participants who paid for verified status were not able to complete the course or earn
a certificate due to the course being “too hard”. As one participant said, “You have to apply
yourself a bit more but yeah, the ones that are hard you try to read up if you can, and if you
don’t get it you don’t’ get it. Some of them are really hard. I see the comments from the other
students that they are having the same challenges.” Others took personal responsibility for
not finishing, saying “I was kind of slacking, yeah I was not watching not so often”. Another
blamed their own lack of grit, saying “actually some of that has to do with self-discipline, I’ll
be honest not finishing courses is probably my thing”. Participants also reported earning a
certificate, but not actually completing the entire course. As one participant explained, “I
wasn’t able to submit the essay on time, because I was working on the policy paper [for work]
and they had the same deadlines, but I had already received a passing grade. You know,
maybe I will still write that essay though, and try to publish it somewhere else.”
4. Discussion
These findings show that learners have a dynamic understanding of completion in MOOCs,
depending on their interests in the material, and their willingness to pay. Learners reported
feeling a sense of completion when they met their own goals for themselves, regardless of
whether they had earned a certificate or technically completed all sections of the course.
Indeed, learners are aware of the limitations in their ability to technically complete the course
without paying to earn a certificate. As one learner said, “Usually I do everything. If I pay
for it, I will do all the exams of course. And if I don’t pay for it, I do all the exams that I can.”
Or as another learner explained, “If you are not interested in the certificate, you can just skip.
I prefer it if I don’t see the tests, I just don’t need that I would like to keep going with the
classes.” Learner’s benchmark for completion is dependent on fulfilling their own learning
goals, their intention to earn a certificate, and their willingness to pay for full access to the
content. Indeed, some learners paid for verified status for full access to the content, and did
not attempt to earn the certificate but nonetheless felt that they had completed it to their
satisfaction. This shows that learners view completion as a distinct outcome from earning a
certificate. This is further illustrated by learners who reported feeling that they had not
completed the course, despite earning the certificate. Interestingly, learners noted that the act
of earning the certificate disincentivized them from completing the course “because [they]
did not need too”.
This sheds light on previous research on retention in MOOCs which uses certification rates
as a proxy for completion (Reich and Ruiperez-Valiente, 2019), suggesting that the
certification rate in a course might be poorly aligned with the learners’ sense of completion.
Despite this, the findings show alignment with much of the literature investigating retention
in MOOCs. For instance, learners reported a lack of self-discipline and poor time
management as a cause for not completing a MOOC (Doo et al, 2021; Kroll & Reed, 2017;
316
Meghan Perdue
Fellman et al, 2020; Gupta, 2021; Hood et al, 2015; Cohen & Magen-Nager, 2016; Wang &
Baker, 2018). A common reason cited for dropping out of course was that they did not find
it engaging, either due to the course design, the instructors, or the level of the course (Hone
& El Said, 2016; Goopio & Cheung, 2021; Chen et al, 2020). They also described having to
drop out of a course because it was too hard, and they didn’t feel they had access to the
support that they needed (Greene et al, 2015; Aldowah et al, 2020). Finally, many learners
did not complete courses due to external conflicts arising from their professional and personal
lives (Eriksson et al, 2017).
While learners’ sense of completion varied based on the circumstances, their understanding
of certificates was fairly uniform. Most learners reported choosing to earn a certificate
because they had a use for it. As one learner explained that they did not pursue certificates in
MOOCs because, “I don’t need the certificates for anything”. Others noted that they chose to
earn a certificate because they planned to put it on their social media profiles to boost their
credibility and demonstrate their skills. As others have found, pursuing certificates could be
used for short term professional goals, or to support long-term career trajectories (Littlejohnet
al, 2016; Dillahunt et al, 2016). For some, however, a sense of accomplishment was enough
of a reason to earn a certificate. One learner said, “You spend enough time reading, putting
in all the hours, so why don’t you get a certificate? I have to get the certificate.” Another
reported collecting almost 50 certificates in a binder as keepsakes to demonstrate her learning
journey. Learners decisions about when and why to earn certificates depends on whether they
had an internal or external use or need for the certificate.
5. Conclusion
This study sought to characterize the ways in which learners perceive completion and
certification in MOOCs by conducting interviews with MOOC learners. The data suggests
that learner’s definition of completion is complex and relative to their own personal goals
and perception of the material. It also shows that learner’s perceptions of completion are
distinct from the process of earning a certificate, which may or may not be in alignment
depending on the circumstances. Finally, the data shows that learners typically only choose
to earn a certificate if they feel they have a use for it, whether personal or professional. Futher
research could shed light on how pervasive these variations are across the population of
learners, as well as point to interventions that could be useful for some learners to help them
accomplish their goals.
Acknowledgments
Research was sponsored by the United States Air Force Research Laboratory and the United
States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative
317
Agreement Number FA8750-19-2-1000. The views and conclusions contained in this

document are those of the authors and should not be interpreted as representing the official
policies, either expressed or implied, of the United States Air Force or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government
purposes notwithstanding any copyright notation herein
References
Al-Shabandar, R., Hussain, A. J., Liatsis, P., & Keight, R. (2018). Analyzing learners
behavior in MOOCs: An examination of performance and motivation using a data-driven
approach. IEEE Access, 6, 73669-73685. doi: 10.1109/ACCESS.2018.2876755.
Aldowah, H., Al-Samarraie, H., Alzahrani, A. I., & Alalwan, N. (2020). Factors affecting
student dropout in MOOCs: a cause and effect decision‐making model. Journal of
Computing in Higher Education, 32(2), 429-454. Doi: 10.1007/s12528-019-09241-y
Anderson, T. (2013). Promise and/or peril: MOOCs and open and distance
education. Commonwealth of learning, 3, 1-9.
Chen, X., Xia, E., & Jia, W. (2020). Utilization Status and User Satisfaction of Online
Education Platforms. International Journal of Emerging Technologies in Learning
(iJET), 15(19), 154-170. http://www.learntechlib.org/p/217905/
Cohen, L., & Magen-Nagar, N. (2016). Self-regulated learning and a sense of achievement
in MOOCs among high school science and technology students. American Journal of
Distance Education, 30(2), 68-79. Doi: 10.1080/08923647.2016.1155905
Dillahunt, T. R., Ng, S., Fiesta, M., & Wang, Z. (2016). Do massive open online course
platforms support employability? In Proceedings of the 19th ACM conference on
computer-supported cooperative work & social computing (pp. 233-244). Doi:
10.1145/2818048.2819924
Doo, M. Y., Bonk, C. J., & Kim, J. (2021). An investigation of under-represented MOOC
populations: Motivation, self-regulation and grit among 2-year college students in
Korea. Journal of Computing in Higher Education, 33(2), 419-440. Doi:
10.1007/s12528-021-09270-6
Eriksson, T., Adawi, T., & Stöhr, C. (2017). “Time is the bottleneck”: a qualitative study
exploring why learners drop out of MOOCs. Journal of Computing in Higher
Education, 29(1), 133-146. Doi:10.1007/s12528-016-9127-8
Fellman, D., Lincke, A., & Jonsson, B. (2020). Do individual differences in cognition and
personality predict retrieval practice activities on moocs?. Frontiers in psychology, 2076.
Doi: 10.3389/fpsyg.2020.02076
Fini, A. (2009). The technological dimension of a massive open online course: The case of
the CCK08 course tools. International Review of Research in Open and Distributed
Learning, 10(5). Doi: 10.19173/irrodl.v10i5.643
Goopio, J., & Cheung, C. (2021). The MOOC dropout phenomenon and retention
strategies. Journal of Teaching in Travel & Tourism, 21(2), 177-197. Doi:
10.1080/15313220.2020.1809050
318
Meghan Perdue
Greene, J. A., Oswald, C. A., & Pomerantz, J. (2015). Predictors of retention and
achievement in a massive open online course. American Educational Research
Journal, 52(5), 925-955. Doi: 10.3102/0002831215584621
Gupta, K. P. (2021). Understanding learners' completion intention of massive open online
courses (MOOCs): role of personality traits and personal innovativeness. International
Journal of Educational Management, 35 (4), 848-865. doi: 10.1108/IJEM-01-2020-0042
Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A
survey study. Computers & Education, 98, 157-168. Doi:
10.1016/j.compedu.2016.03.016
Hood, N., Littlejohn, A., & Milligan, C. (2015). Context counts: How learners' contexts
influence learning in a MOOC. Computers & Education, 91, 83-91. Doi:
10.1016/j.compedu.2015.10.019
Khalil, H., & Ebner, M. (2014, June). MOOCs completion rates and possible methods to
improve retention-A literature review. In EdMedia+ innovate learning (pp. 1305-1313).
Association for the Advancement of Computing in Education (AACE).
Kizilcec, R. F., Piech, C., & Schneider, E. (2013, April). Deconstructing
disengagement:analyzing learner subpopulations in massive open online courses.
In Proceedings of the third international conference on learning analytics and
knowledge (pp. 170-179). Doi:10.1145/2460296.2460330
Kloft, M., Stiehler, F., Zheng, Z., & Pinkwart, N. (2014, October). Predicting MOOC dropout
over weeks using machine learning methods. In Proceedings of the EMNLP 2014
workshop on analysis of large scale social interaction in MOOCs (pp. 60-65).
Kroll, C., & Reed, A. (2017). Towards optimizing MOOC pedagogy: Learner self-control
study. EdMedia+ Innovate Learning, 650-656.
Littlejohn, A., Hood, N., Milligan, C., & Mustain, P. (2016). Learning in MOOCs:
Motivations and self-regulated learning in MOOCs. The Internet and Higher
Education, 29, 40-48. Doi: 10.1016/j.iheduc.2015.12.003
Milligan, C., & Littlejohn, A. (2017). Why study on a MOOC? The motives of students and
professionals. International Review of Research in Open and Distributed
Learning, 18(2), 92-102. Doi:10.9173/irrodl.v18i2.3033
Reich, Justin, and José A. Ruipérez-Valiente. "The MOOC pivot." Science 363.6423 (2019):
130-131. Doi: 10.1126/science.aav7958
Roy, A., Yee, M., Perdue, M., Stein, J., Bell, A., Carter, R., & Miyagawa, S. (2022, June).
How COVID-19 Affected Computer Science MOOC Learner Behavior and
Achievements: A Demographic Study. In Proceedings of the Ninth ACM Conference on
Learning@ Scale (pp. 345-349). Doi: 10.1145/3491140.3528328
Salmon, G., Pechenkina, E., Chase, A. M., & Ross, B. (2017). Designing Massive Open
Online Courses to take account of participant motivations and expectations. British
Journal of Educational Technology, 48(6), 1284-1294. Doi: 10.1111/bjet.12497
Shah, S. K., & Gorbatai, A. D. (2015). Structural sampling: A technique for illuminating
social systems. In Handbook of qualitative organizational research (pp. 251-261).
Routledge.
Strauss, A., & Corbin, J. M. (1997). Grounded theory in practice. Sage.
319
Tanner, L. (2013). Qualtrics and instructure partner reveal top motivations for MOOC
students. http://www.qualtrics.com/press/press-releases/qualtrics-and-instructure-
partner-reveal-top-motivations-for-mooc-students/
Wang, Y., & Baker, R. (2018). Grit and intention: Why do learners complete MOOCs? The
International Review of Research in Open and Distributed Learning, 19(3). Doi:
10.19173/irrodl.v19i3.3393
Zhong, S. H., Li, Y., Liu, Y., & Wang, Z. (2017). A computational investigation of learning
behaviors in MOOCs. Computer Applications in Engineering Education, 25(5), 693-705.
Doi: 10.1002/cae.21830
320
An augmentation framework for efficiently extracting open

educational resources from slideshows
Balthasar Teuscher1, Zhouyi Xiong1, Martin Werner1
1
Department of Aerospace and Geodesy, TUM School of Engineering and Design,
Professorship of Big Geospatial Data Management, Technical University of Munich,
Germany.
Abstract
Open education is a way of carrying out education, often using digital
technologies. This paper gives a brief background on open education and its
constraints. To make courses more accessible, teachers need to have an
effective way to preserve and share their materials. We derive the requirement
of keeping the choice of tools completely open while we pave the way to a
teaching transport format based on open standards allowing for extending
existing lecture material with interactive content. A simple learning system that
helps to detect and save changes of a slide deck was developed. The linkage of
relevant knowledge and interactive context can also be added through this
implementation which outputs a final transport format. This provides a good
starting point for future work on removing learning barriers and widening
access to education.
Keywords: Open education; online material generation; software tool.

An augmentation framework for efficiently extracting open educational resources from slideshows
1. Introduction
Open education is widely acknowledged as important for successful learning for university
students. Research across disciplines has demonstrated that well-designed online learning
can enhance students’ motivation and improve their learning (Hegeman, 2015; Zheng et al.,
2017). Online learning further played a crucial role during the past years with the COVID-
19 pandemic and lockdowns, when attending in-person lectures for teachers and students was
severely constrained.
From a more general perspective, open education is a modern movement aiming to make
access to education easier hoping to stimulate other fields of political and societal whish
including lifelong learning (United Nations, 2015; Zawacki-Richter et al., 2020). In the
current debate, open education is often reduced to open-access learning over the Internet. We
are convinced that this perspective is only partly valid, as open education is not only about
openly accessible education over the internet but is supposed to transform existing learning
paradigms including frontal teaching and teaching in presence in all educational formats from
primary school to high school and university.
One consequence of this perspective is the fact that traditional teaching formats like frontal
teaching using a blackboard, slideshow presentations, conversations in class, group work and
project work are to be part of open education and we want to make them more visible, as they
are the backbone of education for a long time. Most professors in universities and teachers in
schools are expected to give classes in presence. For traditional methods of teaching, it is
evident that slideshows – despite us knowing that they have limitations – find wide adoption.
Unfortunately, transforming such slideshows into interactive online teaching material turns
out to be difficult and laborious.
Furthermore, playing out such educational material remains challenging. While software like
PowerPoint is routinely used to generate slideshows that often serve their educational purpose
quite well, the shareability is limited. Very often, slideshows are converted to PDF to share
online and prevent editing, in which animated slides cannot be read very nicely.
This paper aims to describe a novel and simple procedure to enable teachers using traditional
formats such as a slideshow to progress in opening and enhancing their education towards
more aspects of good open education while avoiding a radical change such as changing the
underlying toolkit. We comment that a similar framework can be applied to any temporal-
visual system, for example, a blackboard lecture could be held once on a digital blackboard
and augmented very much along the same lines. Furthermore, we note that the procedure is
future-proof in the sense that future updates to the teaching material do not break
augmentation and that it does support advanced visual aids such as animations out of the box.
322
2. Background
In response to COVID-19 pandemic lockdown measures, higher education like universities
closed their premises. Although such institutions were quick to replace face-to-face lectures
with online learning, these closures affected learning and examinations as well as raised
questions about the value offered by higher education which includes networking and social
opportunities. To remain relevant, universities will need to reinvent their learning
environments so that digitalization expands and complements student-teacher and other
relationships (Schleicher, 2020).
One obvious measure to stay relevant is by adopting state-of-the-art good learning practices
throughout the curriculum. Learning resources are mostly part of a curriculum designed to
fulfil certain learning goals and have the purpose to satisfy certain learning objectives, and it
is therefore essential to achieve a successful learning experience for the students. In the
following, we would like to highlight some aspects of good learning, not to provide a
theoretical discussion, but interlink with our work.
One aspect is the shift from teacher-centred learning to student-centred learning. While the
input slide deck might originate from the former, we see the potential of our tool to help
transition the material to the latter in the process by deconstructing, enriching, and
augmenting it. Another aspect is competencies-based learning, which can be enhanced to the
point of an active learning experience by augmenting an existing slide deck with interactive
content. With respect to another aspect, constructive alignment, one could for example
augment a slide deck with a self-test.
The main goal is to facilitate the transition to an (inter-) active open online learning
experience for students, as it has been shown that a student-centred approach, as well as
online learning and active learning, can be beneficial to the learning outcomes of students
(U.S. Department of Education, 2009; Hegeman, 2015).
3. A Simple Learning System Concept

Many learning systems are implemented, designed, and share a large set of features,
unfortunately, combined with high complexity in using them all. For example, the online
teaching system Moodle has found wide adoption and provides a decent set of features most
of which are, however, difficult to implement especially in hybrid courses which shall be
taught synchronously in presence and over the internet at the same time. Excellent teaching
material is currently non-open and frequently available as slide decks while open education
would require a modularized presentation based on open standards supporting interactive
contents and linking of open resources. In simple words, an experienced teacher might ask
you: “How do I get my slide deck into your system?”
323
We propose to maximize the simplicity of turning existing education into a form in which
they can grow towards open education by replacement of problematic sections1 and
augmentation within the teacher’s tools. While teaching material for teaching in presence
requires considering temporal constraints of the teaching situation (e.g., lecture time), open
online education should provide a more flexible temporal layout in which the learner can
progress self-paced. For building a bridge between existing, high-quality teaching material
with a strict temporal organization used in class and flexible open education, we need to
define a system that supports segmentation into smaller pieces (“units that can be skipped or
replaced with an imported other unit”) and augmentation with additional material
(background information, additional pieces to consider, challenges, quizzes, tasks, etc.). With
many teachers relying on slide deck presentations, an online teaching system for a diverse
body of teachers and a diverse audience needs to provide a few key features:
• Feature 1: The framework shall enable teachers to convert their existing material
into open education material with zero or low complexity in terms of work (time)
and learning.
• Feature 2: The framework shall enable the sustainable extension of existing material
with online-only features such as quizzes and background information linkage.
• Feature 3: The augmentation scheme shall not be limited to content, but also include
curricular logic by, for example, linking or proposing other (micro-modules) to learn
or linking with a knowledge base.
We realize a research prototype of such a system and are aiming at using it in large-scale
research studies within the national research data infrastructure NFDI4Earth for our
education. Based on the existing teaching material of the partners involved, we want to pave
the way to accessible and high-quality online teaching, preferably reusing existing material.
Note that this is perfectly in line with the principles of findability, accessibility,
interoperability, and reusability (FAIR) which have recently been applied to open education
as well. Findability is improved by fragmenting temporal teaching material and augmenting
it with metadata, accessibility is provided by open internet access, towards interoperation, we
use only industry-standard datatypes (e.g., MPEG video) and community-driven
standardized representations (e.g., H5P). And reusability is provided by the editable and
extensible nature of our output H5P containers and the explicit integration of all tools that
can map the temporal teaching material to a video. While we demonstrate the system on slide
decks, note that this can be easily extended and applied even to handwritten text on a digital
blackboard.
1
we decide to use the unclear term problematic here as it covers all legal, ethical, didactical, or otherwise occurring limitations of
the value of a shared version as opposed to a version used in the original learning context.
324
• Decision 1: For the prototype, we decided to first concentrate on slide decks and
rely on the video export features ubiquitously available in slideshow software to
bridge existing slideshow technologies.
• Decision 2: The linkage of background information must be implemented within
the teachers´ toolkit (e.g., PowerPoint) as only in this way future structural
modifications like inserting slides do not break the linkage and the improved version
can still be edited with the already established software knowledge of the teacher.
• Decision 3: As a transport format for open education, we rely on the idea of H5P
containers. Such containers are essentially collections of presentation software and
content that are based on open standards and can be readily deployed to existing
teaching systems like Moodle or ILIAS.
The transformation procedure is then structured as follows: Starting from slides, we first use
the teacher’s software to convert the results to video. Opposed to exporting as individual
slides, we can perverse all animations and videos that might be embedded in the presentation.
Furthermore, presentations on a digital blackboard with a pen can as well be recorded as a
video stream while most digital blackboards do not have a suitable concept of a slide that we
could use and especially in handwritten lecture notes, the genesis of the material as a video
is more valuable than the final page. We then implemented a simple software that decodes
the video frame by frame and tracks significant visual changes. For each longer time period
without significant visual change, we emit an image slide. For longer periods of change, we
emit a slide showing a video segment (e.g., for animations). These sequences are then
automatically assembled into an H5P container using image slides and video slides.
3.1. Basic Augmentation through Pictograms

Now, you can easily imagine that the simple procedure works well but will have some
pitfalls. For example, in animations, it should be possible to generate key frames, that is, we
need a mechanism that the teacher selects that a certain visual state within the animation is
going to be used for longer time as a slide. Furthermore, you might realize that some parts
are good for presence teaching but might not be ideal in online scenarios. For example, a
presentation in a school could contain instructions such as “We need your math book today”
or university lectures could contain information, especially directed to online participants
such as when exactly the lecture starts.
These functions are implemented in our framework with a visual language that contains
certain pictograms triggering the required behavior in the processing of the video slideshow.
Table 1 contains the visual language used for our current version of the prototype. This visual
language is sufficiently simple and can be implemented in any slideshow system that is able
to show images. In addition, good usability can be provided by extending the toolbar with
325
buttons that insert exactly these images or with a small tool that efficiently provides these
images through the clipboard of the operating system.
Table 1. Pictograms (icons) with their conditions, semantics, and actions.
Icon Condition, Semantics, Action
When this picogram is visible, emit a snapshot of the middle frame of visibility of
this pictogram as a still image slide.
Note that this can be used in animations by a sequence of showing followed by
removing this pictogram to generate a still image slide within an already existing
animation.
While this picogram is visible, no output frames can be emitted or (based on a

prerequisite) an error message is shown or the output is blurred significantly and
overlaid with a message.
This marks the start of an autoplay section. That is, from the time on, this is visible, we
will emit the plain video segment as a video slide.
This allows for supporting pseudo-animations, for example, when multiple slides are
used for something that should be an animation. This has often been done to facilitate
reasonable PDF export and printing.
This marks the end of an autoplay section.
Note that this does not emit a still frame, hence, the video might be looping or
whatever preset we are having. If you want to have the result as a slide, you can
combine this with a camera icon.
This icon denotes a chapter start.
In this case, optical character recognition (OCR) can be used to extract the text
depicted in the largest font as the title of the chapter and to build a Table-of-Contents
using this information.
Such a structure can also be used for skipping forward and backward in long
presentations.
4. Demonstrative Implementation
The proposed system is implemented as a computer vision system based on a rather simple
heuristic of change detection. As a first step, we decode the video frame by frame. For each
frame, to save computation time and increase stability with respect to video coding artifacts,
we perform a resize operation to a size of 256 x 256 pixels, convert the color space to
grayscale, and extract edges using the Canny detector with thresholds of 50 and 200. With
this representation, we scan through the video and compare each frame with the next frame
326
using a normalized correlation coefficient 𝑅(𝑥, 𝑦) as given in the following formula for two
grayscale images 𝑇’ and 𝐼’.
For an example slide deck, this leads to a correlation profile between slides as depicted in
Figure 1. In this figure, one sees strong peaks away from almost perfect correlation (e.g., R
= 1) for each slide change. Hence, for detecting a slide change, we propose to use a threshold
of 𝜏 = 0.5 . As a second observation, you see small regular peaks which are related to
keyframes of the MPEG codec and providing artifacts. However, you also see that the third
slide has some non-trivial correlation. This slide has a small animation (e.g., a small circle
moves around). This leads to an overall high correlation as most of each pair of frames is the
same, but we need to detect this movement. Therefore, we employ a trick: We incrementally
build a sum image of all edge-detections while we do not detect a frame change. And as soon
as we get a frame change, the correlation of this sum image with the last image of the previous
frame is computed to detect, if there is an animation. Figure 2 depicts this situation for our
animation: while almost all frames are very similar to their successor, the sum of all frames
clearly depicts the movement of the ball and does not well correlate with any of the frames
of the animation. Hence, if this correlation is low enough, we mark the previous range of
frames as an animated region that needs to be kept in a video format as opposed to creating
a still image representation in H5P.
Figure 1. Video frames correlation profile.
327
Figure 2. Sum of correlations.
During the scan, we also employ template matching with the library of defined symbols. For
this template matching, we assume that the aspect ratio of the symbols is the same and that
the scale is kept similar with the template. Under these assumptions, we can use the same
correlation technique (edge detection followed by computing the normalized correlation
coefficient), but this time sliding the template over the frame. Varying the frame resolution
introduces the required partial scale invariance and symbols can be detected.
This processing stage then provides the following information: the decomposition of the
video in slices related to each slide, the information on whether a video slice contains
significant animation, and for each frame the presence of any of the symbols from the symbol
library. From this information, an H5P container can be generated which contains all the
required features.
5. Conclusion
With the proposed simple method of content augmentation without additional or external
toolsets, just where the teacher is used to be, we hope to enable most university teachers to
share high-quality interactive content and to improve step by step towards open education.
The framework is implemented and demonstrated for PowerPoint, but the augmentation
scheme is general and completely independent of any presentation software or framework,
as the functionality is driven by visible elements.
For future work, the complete set of features including PowerPoint animation, voice
explanation and interactive materials (reference links and quizzes) should all be embedded
328
into one file. Teachers can then easily share it online within and out of their own universities
for open education.
Acknowledgements
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) –
460036893 and the Bundesministerium für Bildung und Forschung (BMBF, Federal Ministry
of Education and Research) – 16DHBKI021.
References
Hegeman, J. S. (2015). Using instructor-generated video lectures in online mathematics
courses improves student learning. Online Learning, 19(3), 70-87.
Schleicher, A. (2020). The impact of COVID-19 on education: insights from Education at a
Glance, OECD. viewed 23 Jan 2023, https://www.oecd.org/education/the-impact-of-
covid-19-on-education-insights-education-at-a-glance-2020.pdf
United Nations. (2015). Transforming our world: the 2030 Agenda for Sustainable
Development. A/RES/70/1.
U.S. Department of Education, Office of Planning, Evaluation, and Policy Development
(2009). Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis
and Review of Online Learning Studies.
https://repository.alt.ac.uk/629/1/US_DepEdu_Final_report_2009.pdf
Yazbek, M. & Bresser, P. (2017). Designing interprofessional modules for undergraduate
healthcare learners. African Journal of Health Professions Education. 9. 111-114.
https://doi.org/10.7196/AJHPE.2017.v9i4.853
Zawacki-Richter, O., Conrad, D., Bozkurt, A., Aydin, C. H., Bedenlier, S., Jung, I., Stöter,
J., Veletsianos, G., Blaschke, L. M., Bond, M., Broens, A., Bruhn, E., … Xiao, J. (2020).
Elements of Open Education: An Invitation to Future Research. The International Review
of Research in Open and Distributed Learning, 21(3).
https://doi.org/10.19173/irrodl.v21i3.4659
Zheng, M., Bender, D., Reid, L., & Milani, J. (2017). An interactive online approach to
teaching evidence‐based dentistry with Web 2.0 technology. Journal of dental education,
81(8), 995-1003.
329
ChatGPT in higher education: the good, the bad, and the

University
Marius Schönberger
University of Applied Sciences Kaiserslautern, Campus Zweibrücken, Department of
Computer Science and Microsystems Technology, Germany.
Abstract
This research examines the opportunities and risks of artificial intelligence
(AI) in the context of higher education, using ChatGPT as an example. The
hype around AI tools in education has led to a skeptical attitude among many
educators towards this technology. Based on a comprehensive literature
analysis, the opportunities and risks of ChatGPT in higher education are
identified and analyzed. The research concludes with a recommendation for a
sensible use of ChatGPT in higher education. The results of this study can
support educators in their decision-making process on whether and how to use
ChatGPT as a tool in their teaching contexts.
Keywords: Artificial intelligence; higher education; ChatGPT.

ChatGPT in higher education: the good, the bad, and the University
1. Introduction
The potential that artificial intelligence (AI) offers for the teaching and learning process is
diverse, providing new opportunities for personalized and effective education (Zawacki-
Richter et al., 2019). AI technologies are gaining increasing importance in universities, as
confirmed by a survey of university leaders in Germany in 2019: AI technologies offer a
range of opportunities for personalized and effective education, with 26.4% of universities
already using AI, machine learning, text mining, and data mining in their research and
teaching activities (Gilch et al., 2019). In addition, universities are increasingly offering
courses in AI, integrating AI modules into the curricula of courses, and developing
interdisciplinary study content to provide basic digital and AI-specific ‘literacy’ (Laupichler
et al., 2022; Mah and Büching, 2019). Thus, AI plays a key role, especially in the context of
digital transformation of higher education, and AI applications, such as intelligent tutoring
systems, teaching robots, learning analytics dashboards, adaptive learning systems,
personalized learning systems, assessment and feedback systems, human-computer
interaction, or intelligent virtual reality, have become established over the years (e.g.,
Bearman et al., 2022; Ouyang et al., 2022; Hinojo-Lucena et al., 2019; Zawacki-Richter et
al., 2019; Ma and Siua, 2018).
However, there are also concerns about AI’s potential risks in higher education, including
reinforcing existing inequalities, increasing plagiarism, and replacing teachers (e.g., Cotton
et al., 2023). In this context, ChatGPT (Generative Pre-trained Transformer), a language
model for providing conversational responses, is attracting a lot of attention in higher
education, with potential applications for text analysis and automation of writing tasks (Zhai,
2022). ChatGPT is developed by OpenAI and has been trained on a massive amount of
human-generated text data using the “Transformer Network” machine learning technique for
natural language processing (OpenAI, 2022). As a result, ChatGPT has a comprehensive
understanding of human language and is able to have human-like conversations. ChatGPT
can therefore be used for a variety of applications in higher education, such as text analysis
or the automation of writing tasks, making it a tool with far-reaching implications for
teaching. Therefore, ChatGPT has the potential to revolutionize applications and services
related to universities. This, in turn, calls for critics as well as advocates.
This research seeks to address this disconnect by exploring the opportunities and challenges
associated with the use of ChatGPT in higher education. Specifically, this paper addresses
the following research questions (RQ):
• RQ1: What are the opportunities and risks of using ChatGPT-based chatbots in
higher education?
• RQ2: How can ChatGPT-based chatbots effectively shape teaching, learning and
administrative processes in higher education?
332
Marius Schönberger
To answer the RQs, this paper identifies and analyses the advantages and disadvantages of
ChatGPT based on a literature review and provides recommendations for the effective use of
ChatGPT in higher education. The aims of the paper are (1) to contribute to the understanding
of the role of ChatGPT in higher education, (2) to provide insights into the opportunities and
challenges associated with its use, and (3) to help educators make informed decisions about
the use of AI in higher education.
To provide an overview of the current use of AI technologies, specifically ChatGPT, in higher
education, a systematic literature review was conducted. However, the heterogeneity of
European universities (Lepori, 2022) makes it difficult to collect a complete and up-to-date
literature review. Therefore, the aim of this thesis is to identify key literature on the
implementation of AI technologies in higher education. As a result, the review results
presented in Chapter 3 do not claim to be exhaustive. Rather, it is intended to provide a
general orientation of the literature. The following section describes the literature review
procedure, which follows the methodology of Mikelsone and Liela (2015).
Prior to conducting a literature review, the following criteria were used to narrow the search
area: (1) Content limitation: The literature reviewed is intended to be as comprehensive as
possible in highlighting different approaches to implementing AI in higher education
contexts. For better comparability of approaches, the term “university” is used generically,
without specific reference to individual disciplinary orientations within higher education. (2)
Linguistic limitation: The focus of the literature review is on European higher education
institutions. Therefore, for better comparability, only English language literature is used for
this study. (3) Type of publication limitation: To achieve the best possible and appropriate
results through the literature review, research papers, conference papers, proceedings,
monographs, books, and dissertations are examined and evaluated. (4) Time limitation: Since
the focus on AI in higher education, especially on the recently published ChatGPT, implies
the identification of only a few sources, there is no limitation on the time of publication.
Nevertheless, an attempt is made to consider current literature.
The literature review is based on an extensive keyword search of the following literature
databases: Google Scholar, Scopus, Ebsco Academic Search, ScienceDirect, Emerald
Insight, and Sage Journals. The search was conducted in several steps: First, a complete
overall search was performed. The keywords were combined into a search string: Each main
keyword was AND-linked with the corresponding thesaurus terms, while the latter were OR-
linked, e.g., “ChatGPT” AND “higher education” OR “university”. The search was then
restricted to article titles, abstracts, and keywords. In a second step, the first results were
limited to fully accessible publications only, and in a third step, duplicates within the results
333
were eliminated. Finally, the results were analyzed by reading the abstracts and references of
the remaining research papers. Again, publications that did not address the research problem
were excluded.
3. Research findings
As predicted before, only a few scientific papers were identified in an initial search. One
reason for this is that ChatGPT was first published in November 2022 (OpenAI, 2022).
Therefore, the keywords “artificial intelligence” and “AI” were added to the initial keywords
and the search was performed again. Finally, a total of 120 literature sources were identified,
of which 88 sources were considered for this study after the elimination of duplicates. The
content of the collected publications can be summarized in the following topics:
• The implementation of ChatGPT in the context of higher education (e.g., Döbeli

Honegger, 2023; Cotton et al., 2023; Zhai, 2022).
• The use of AI technologies in teaching, research, and development (e.g., Ouyang et
al., 2022; Mah and Büching, 2019; Zawacki-Richter et al., 2019).
• The teaching of AI-specific digital literacy and curriculum development (e.g.,
Laupichler et al., 2022; Wannemacher and Bodmann, 2021; de Witt et al., 2020).
• The development and use of AI-based learning systems, including chatbots or
feedback tools (e.g., de Witt et al., 2020; Seufert et al., 2019).
• Ethical aspects of the use of AI applications in higher education (e.g., Bearman et
al., 2022; Zhai, 2022; de Witt et al., 2020).
• Future scenarios and potentials of AI technologies in higher education (e.g.,
Wannemacher and Bodmann, 2021; de Witt et al., 2020).
The following sections discuss possible trends, opportunities and challenges, and
recommendations for implementing ChatGPT and AI in higher education.
3.1 Opportunities and risks

First, the opportunities for ChatGPT in higher education are summarized before the identified
risks are listed. This provides an answer to the first research question.
Developing digital literacy: ChatGPT can be used in general to learn how AI works, what
the consequences of its use are, and how AI specifically changes working with and on texts.
In addition, students can learn to critically question the origin, composition, and quality of
AI-generated data (e.g., Laupichler et al., 2022; de Witt et al., 2020).
Supporting scholarly practices: ChatGPT can support researchers in carrying out scientific
work, such as conducting literature searches, analyzing and evaluating data, recording
experiments, or producing scientific texts (e.g., Cotton et al., 2023; Zhai, 2022)
334
Marius Schönberger
Automated student support: ChatGPT can be used to quickly and efficiently assist students
with questions about their courses, assignments, exams, or other academic matters (e.g.,
Cotton et al., 2023; Zawacki-Richter et al., 2019).
Personalized learning support: ChatGPT can be used as a tutor to help students with concepts
and skills they are struggling with. For example, ChatGPT can optimize texts based on given
criteria and adapt them to different needs (e.g., Bearman et al., 2022; Ma and Siua, 2018).
Encourage creativity: ChatGPT can function as a creativity technique, using unexpected or
incorrect answers constructively, e.g., to deviate from well-trodden paths of thought or to
stimulate one's own thought processes (e.g., Cotton et al., 2023; Zhai, 2022).
Generation of text passages, summaries, and formats: ChatGPT can quickly generate
suggestions for text passages, analyze and summarize longer texts, and generate suggestions
for special formats (e.g., press releases, blog posts) from existing texts (e.g., Döbeli
Honegger, 2023; Zhai, 2022).
Encourage interaction and collaboration: ChatGPT can be used to motivate and help
students interact and collaborate with each other by serving as a moderated platform for
discussions and questions (e.g., Cotton et al., 2023; Ouyang et al., 2022).
However, ChatGPT is not perfect and may sometimes generate inaccurate or inappropriate
responses. The following additional risks exist when using ChatGPT:
Biases and training through inputs: Using ChatGPT can introduce various types of bias
because the data used to train the model may contain certain distortions and imperfections
that become embedded in the model and may be reflected in the responses it generates.
Examples of possible biases include (1) gender bias, (2) race and ethnicity bias, (3) political
bias, or (4) incomplete data bias (e.g., Brennan, 2023; Zhai, 2022).
Misinformation: ChatGPT not only generates text, but also offers explanations for scientific
contexts. In addition to incorrect, randomly generated citations or sources, misinformation is
also possible (e.g., Döbeli Honegger, 2023; Cotton et al., 2023).
Difficulty in evaluating the results: It is difficult to distinguish texts generated by ChatGPT
from those written by humans, as the source of the result remains opaque. This also makes it
difficult to detect misinformation (e.g., Döbeli Honegger, 2023; Cotton et al., 2023).
Lack of consideration of current and scientific sources: Because ChatGPT’s database is not
currently up to date, relevant information may be missing. Also, results from scientific studies
that are not freely available are not considered (e.g., Döbeli Honegger, 2023; Cotton et al.,
2023).
335
Unclear authorship: It is currently unclear how text generated by ChatGPT should be cited
in publications. It is also unclear to what extent ChatGPT’s use of existing documents may
violate copyright, even if the content is not copied literally (e.g., Döbeli Honegger, 2023).
3.2 Recommendations for implementation in higher education

Based on the results of the literature review, the following section provides recommendations
for the implementation of ChatGPT in higher education. This will answer the second research
question.
ChatGPT as a tool for teachers: ChatGPT is particularly suitable as a working tool for
teachers. The focus here is less on automated text generation. Instead, experimenting with
ChatGPT should sensitize teachers and give them ideas for their lessons, such as ideas for
quiz questions, arguments for pro-contra discussions, or impulses for role plays. ChatGPT
can also help create individualized materials, such as assignments for students. It can also
transfer existing content into new formats, such as scripts for podcasts or instructional videos.
It can also help streamline instructions, overviews, and the like, and create standardized text
types such as event descriptions.
ChatGPT as a didactic element of courses: Teachers should use the chatbot as part of their
teaching approach, which limits the privacy issue to the teacher's data. In addition, the use of
ChatGPT should be transparent to address the potential and risks of AI systems and to
promote the development of digital literacy among students. Didactic scenarios could include
identifying fake news, managing discussions, comparing summaries, comparing text formats
and writing styles, and developing criteria for a successful scientific text.
Use of ChatGPT in exams: The use of ChatGPT in the context of examinations (e.g., written
exams, term papers, presentations) naturally poses an increased risk of cheating, especially
since current plagiarism detection software does not yet recognize ChatGPT-generated texts
as plagiarism. Even though tools are being developed to detect ChatGPT texts, concerned
and uncertain lecturers should refrain from traditional term papers or take-home exams, or
only use them in combination with an oral defense. If ChatGPT is to be used as a tool to assist
students in the future, there will inevitably be new rules requiring students to indicate which
tools they have used. ChatGPT could also be a reason to change the culture of examinations
at universities to one where students refrain from cheating and recognize the value of
academic integrity.
4. Conclusions and outlook

In conclusion, the use of AI, particularly ChatGPT, in higher education is attracting attention
due to its opportunities and implications for teaching and learning. However, there are also
concerns about its potential risks. This paper aims to address this disconnect by exploring the
336
Marius Schönberger
opportunities and challenges associated with ChatGPT in higher education and providing
recommendations for its effective use. A literature review of European universities and their
use of AI technologies in higher education was conducted to provide a general orientation of
the field. Based on the results of the literature review, this research also aims to contribute to
the understanding of the role of ChatGPT in higher education and to help educators make
informed decisions about using AI in higher education.
Currently, any specific recommendations for the implementation of ChatGPT should be
understood as impulses for reflective experimentation and as an invitation for discourse on
the design of AI-based teaching in higher education. It should be emphasized that an
uncritical and automated use of the results of ChatGPT is not recommended. Furthermore,
the opportunities and risks summarized in this paper should always be understood in relative
terms, as they are based solely on the results of the literature review. An exact representation
of the real opportunities and risks is hardly presentable, since a feasible evaluation always
depends on the concrete context and the objective.
Like other new digital tools, ChatGPT presents both opportunities and risks. However, by
making AI accessible at a low threshold, ChatGPT can be expected to make qualitative leaps
compared to previous digital developments, the consequences of which cannot yet be
assessed with certainty. Regarding the implementation of AI-based technologies in higher
education, Hinojo-Lucena et al. (2019: 1) state that “this technology is already being
introduced in the field of higher education, although many teachers are unaware of its scope
and, above all, what it consists of.” In this context, this paper contributes to filling this gap,
and the authors accordingly recommend further research, both qualitative and quantitative,
on the implementation of AI in the different activities of higher education, not only in the
context of teaching and research, but also for administrative, recruitment or accreditation
tasks.
References
Bearman, M., Ryan, J., & Ajjawi, R. (2022). Discourses of artificial intelligence in higher
education: a critical literature review. Higher Education, 1-17.
Brennan, K. (2023). ChatGPT and the Hidden Bias of Language Models.
https://thestoryexchange.org/chatgpt-and-the-hidden-bias-of-language-models/,
accessed 06 February 2023.
Cotton, D. R., Cotton, P. A., & Shipway, J. R. (2023). Chatting and Cheating: Ensuring
academic integrity in the era of ChatGPT. Preprint. https://doi.org/10.35542/osf.
io/mrz8h, accessed 06 February 2023.
de Witt, C., Rampelt, F., & Pinkwart, N. (2020): Künstliche Intelligenz in der
Hochschulbildung. Whitepaper, October 2020. Berlin: KI-Campus.
337
Döbeli Honegger, B. (2023). ChatGPT & Schule. Einschätzungen der Professur

„Digitalisierung und Bildung“ der Pädagogischen Hochschule Schwyz.
https://mia.phsz.ch/pub/MIA/ChatGPT/2023-chat-gpt-und-schule-v128.pdf, accessed 06
Februar 2023.
Gilch, H., Beise, A. S., Krempkow, R., Müller, M., Stratmann, F., & Wannemacher, K.
(2019). Digitalisierung der Hochschulen: Ergebnisse einer Schwerpunktstudie für die
Expertenkommission Forschung und Innovation (No. 14-2019). Studien zum deutschen
Innovationssystem. Berlin: Expertenkommission Forschung und Innovation (EFI).
Hinojo-Lucena, F. J., Aznar-Díaz, I., Cáceres-Reche, M. P., & Romero-Rodríguez, J. M.
(2019). Artificial intelligence in higher education: A bibliometric study on its impact in
the scientific literature. Education Sciences, 9(1), 51.
Laupichler, M. C., Aster, A., Schirch, J., & Raupach, T. (2022). Artificial intelligence literacy
in higher and adult education: A scoping literature review. Computers and Education:
Artificial Intelligence, 3(2022), 100101.
Lepori, B. (2022). The heterogeneity of European Higher Education Institutions: a
configurational approach. Studies in Higher Education, 47(9), 1827-1843.
Ma, Y., & Siau, K. L. (2018). Artificial intelligence impacts on higher education.
Proceedings of the 13th Midwest Association for Information Systems Conference
(MWAIS), 42(5), 1–5.
Mah, D., & Büching, C. (2019). Künstliche Intelligenz in Studium und
Lehre. Überblickstudie zu Professuren und Studiengängen der Künstlichen Intelligenz in
Deutschland. Berlin: VDI/VDE Innovation+Technik GmbH.
Mikelsone, E., & Liela, E. (2015). Literature review of idea management: Focuses and gabs.
Journal of Business Management, 2015, No. 9, 107-121.
OpenAI (2022). ChatGPT: Optimizing Language Models for Dialogue. 30. November 2022.
https://openai.com/blog/chatgpt/, accessed 06 February 2023.
Ouyang, F., Zheng, L., & Jiao, P. (2022). Artificial intelligence in online higher education:
A systematic review of empirical research from 2011 to 2020. Education and Information
Technologies, 27(6), 7893-7925.
Seufert, S., Guggemos, J. & Moser, L. (2019). Digitale Transformation in Hochschulen: auf
dem Weg zu offenen Ökosystemen. Zeitschrift für Hochschulentwicklung, 14(2), 85-107.
Wannemacher, K., & Bodmann, L. (2021). Künstliche Intelligenz an den
Hochschulen. Potenziale und Herausforderungen in Forschung, Studium und Lehre
sowie Curriculumentwicklung; Hochschulforum Digitalisierung: Berlin, Germany.
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of
research on artificial intelligence applications in higher education–where are the
educators? International Journal of Educational Technology in Higher Education, 16(1),
1-27.
Zhai, X. (2022). ChatGPT user experience: Implications for education. Available at SSRN
4312418.
338
ChatGPT in the classroom: friend or foe?

Josep Domenech
Department of Economics and Social Sciences, Universitat Politècnica de València, Spain.
Abstract
The use of Artificial Intelligence (AI) and specifically, OpenAI's ChatGPT, has
been increasing in the educational field as a tool to enhance learning and
teaching processes. Understanding the potential of ChatGPT in the classroom
is crucial for instructors and educators because it can provide significant
benefits in the preparation of classes, exams, and instructional materials. But
it is even more crucial to understand how students may use it because many
will use it regardless of its adoption by the instructor.
This paper analyzes the potential of ChatGPT in the classroom, exploring its
applications as a writing assistant, study tool, personal tutor, and more. Our
aim is to shed light on the present capabilities of ChatGPT in higher education
and to provide a comprehensive view of its advantages and challenges.
Keywords: Artificial intelligence; ChatGPT; pedagogy, technology.

1. Introduction
The integration of Artificial Intelligence (AI) in education has been a growing trend in recent
years. Early applications of AI in education focused on providing more efficient and effective
ways to support the learning process, such as automated grading and personalized learning
(Zawacki-Richter et al. 2019). With the advent of language models such as ChatGPT, the use
of AI in education has become even more accessible, as these tools offer a more human-like
interface.
ChatGPT is a language model developed by OpenAI (OpenAI Team 2022) that utilizes AI
techniques to generate human-like text. It is based on a Generative Pretrained Transformer
(GPT) model, which predicts the probability of each word to occur next conditioned on the
previous words in a text, and based on the training dataset (Floridi and Chiriatti 2020). This
way, it is capable of completing sentences, paragraphs, or entire articles based on the context
provided as input. The model has been trained on a vast amount of text data and in multiple
languages. Its ability to perform language-based tasks offers a great opportunity for
instructors in a wide variety of fields. The natural language interface is its disruptive
characteristic, as it enables instructors and students to easily incorporate AI without requiring
dedicated software or coding skills.
In a higher education context, ChatGPT can be used to enhance the teaching and learning
process in a number of ways. For example, it can assist students in generating written
responses to prompts or questions, helping them to improve their writing skills. Additionally,
ChatGPT can provide students with personalized feedback on their work and answer their
questions in real time. It can also be used as a study tool by generating summaries of key
concepts and providing students with practice questions to help them prepare for exams.
Another way to use ChatGPT is as an assessment tool, where it can automatically grade
student responses to some open-ended questions, providing teachers with data on student
understanding of a topic. It can also be used as a content creator by generating educational
materials such as quizzes, worksheets, and lesson plans.
Understanding the potential of ChatGPT in the classroom environment is crucial for
instructors and educators because it can provide significant benefits in the preparation of
classes, exams, and instructional materials. But it is even more crucial to understand how
students may use it because many will use it regardless of its adoption by the instructor, so
assignments and exams must be prepared accordingly. The objective of this paper is to
examine the current state of AI in education and its potential as a tool to support the teaching
and learning process. The paper aims to provide an overview of the benefits of using
ChatGPT in the classroom and to discuss the main challenges and pedagogical concerns that
must be considered. It is worth noting that the use of AI in education is still in its early stages,
and further research is needed to fully understand its effects and potential limitations.
340
Josep Domenech
The remainder of the paper is structured as follows. In Section 2, a literature review is

presented, which covers the current state of AI in education and its potential applications.
Section 3 explores how ChatGPT can be used for good, to create content and assist students.
Section 4 reviews some of the challenges regarding the integration of AI tools in classroom.
Finally, Section 5 draws some concluding remarks.
2. Earlier uses of AI in education

There have been a number of previous applications of AI in education that have aimed to
enhance the learning and teaching process, from planning to assessment (Pender et al. 2022).
One of the most notable examples is the use of AI-powered educational games and
simulations. These games and simulations use AI algorithms to adapt the level of difficulty
and provide personalized feedback, making the learning experience more engaging and
effective (Bennani et al. 2022).
Another area where AI has been applied in education is in the development of intelligent
tutoring systems (ITS). These systems use AI algorithms to provide students with
personalized feedback, guidance, and support based on their unique learning needs. ITS have
been shown to be effective in improving student performance, particularly in subjects such
as mathematics and science (Mousavinasab et al. 2018).
The utilization of automated writing evaluation and natural language processing (NLP) in the
analysis of student writing is also an application of AI in education. NLP algorithms can be
used to identify and correct grammatical errors, provide feedback on the coherence and
organization of writing, and identify plagiarism. This can help to improve student writing
skills and reduce the workload of teachers who have to grade student work (Wang et al. 2013;
Bohra and Barwar 2022).
AI has been used in automated assessment and scoring, where AI-based algorithms are used
to grade student responses to open-ended questions (Zhang et al. 2022; Smith et al. 2020).
This can help to provide teachers with data on student understanding of a topic and can also
reduce the time required to grade assessments. This is particularly relevant in large class
environments or in Massive Open Online Courses (MOOC) (Vittorini et al. 2021).
Chatbots are one of the closest antecedents to ChatGPT in education. They have been used
in e-learning platforms to simulate one-to-one interaction with instructors and provide
personalized feedback (Clarizia et al. 2018), such as suggesting to follow some tutorials or
watch some videos according to some rules. However, the versatility of such chatbots is
limited because they are like scripted conversations with linear flows (Kuhail et al. 2022).
ChatGPT’s advanced natural language processing capabilities enable it to generate more
sophisticated and contextually appropriate responses than traditional rule-based chatbots.
341
3. ChatGPT for good
3.1. Content creation

ChatGPT can be utilized to create custom quizzes on a variety of topics and modalities, such
as multiple choice, true/false, fill-in-the-blank, or matching questions and answers. By asking
it to "Generate a multiple choice quiz on the Consumer Theory," for example, it can provide
a set of well-crafted questions and answers that can be used in classes or exams. The ability
to easily generate quizzes on any subject matter makes ChatGPT a valuable resource for
educators. Figure 1 offers an example of how these questions can be generated.
User: Prepare a multiple-choice question on shifts in the demand in competitive markets.
IA: Which of the following factors can cause a shift in the demand curve in a competitive market?
A) A change in consumer tastes and preferences B) A change in the price of a substitute good
C) A change in the population size D) All of the above.
Figure 1. Example of a multiple-choice question created by ChatGPT.
Open-ended questions are another type of content that can be created using ChatGPT. In
response to the user's request, it can generate questions that encourage reflection, discussion,
explanation, creativity, or opinion-based responses. By asking ChatGPT to "generate open-
ended questions about the Consumer Theory," instructors receive a list of thought-provoking
questions that can be the basis for class discussions, written assignments, or other interactive
activities that promote deeper learning and understanding, as illustrated in Figure 2.
ChatGPT can also provide relevant information on a wide range of subjects. When asked for
an explanation of a topic, such as "How does nuclear power work?", ChatGPT can respond
with a detailed response that can be used to supplement lessons or as a starting point for
further study. Therefore, it can become an excellent tool for creating class materials.
Finally, ChatGPT can be used to create simple games, such as trivia, and role-playing games,
to engage students with educational material. For instance, it can generate a trivia game about
a certain topic by simply being asked. Role-playing games, such as the dialogue presented in
Figure 3, allow students to understand complex situations by experiencing different
perspectives.
3.2. Assisting students

There is a number of ways in which ChatGPT can be used as a study tool to assist students
in their academic pursuits. Some of them are reviewed below.
Firstly, it can be used as a personal tutor and offer custom guidance. Students can ask
questions to gain relevant information and explanations on a topic in real-time. ChatGPT can
also offer personalized feedback by evaluating students' answers to questions and providing
342
Josep Domenech
User: Prepare a comparative question that considers AI, GPT, and BERT.
IA: How does the language generation capability of GPT compare to that of BERT, and how do
both models differ from traditional AI approaches in terms of their language processing
capabilities?
User: Generate an open-ended question to encourage critical thinking about the Consumer Theory
IA: What are some of the real-life situations in which the principles of Consumer Theory can be
applied, and how would you go about applying them in each situation?
Figure 2. Examples of open-ended questions created by ChatGPT.
some insights into their understanding of the material. Furthermore, ChatGPT can create
personalized study plans following some requirements by asking, for example, “Create a 10-
hour study plan for the consumer theory for a visual learner”.
Secondly, ChatGPT can be employed as a writing assistant in multiple ways, such as helping
students generate written responses to questions, summarizing key concepts, and providing
feedback and corrections. For instance, students can use ChatGPT to create a first draft of an
essay, summarize a text, or receive feedback on their writing regarding grammar, vocabulary,
and coherence. This way, it can assist students in improving their writing skills,
understanding the material, retaining information, and identifying areas for improvement.
Thirdly, ChatGPT can provide students with opportunities for conversational practice and
language development. For example, students can ask “Let's engage in a conversation about
the weather in Valencia” to practice writing in a foreign language or to develop their
communication skills in their native language. In a conversation with ChatGPT, students can
improve their fluency and vocabulary and can also receive feedback on their writing abilities.
This can help students to become more confident and proficient communicators.
4. The Dark Side

Despite the potential benefits of integrating ChatGPT in the classroom, it is not exempt from
some potential challenges that educators should consider.
Firstly, ChatGPT's ability to generate answers to any question can be tempting for students
looking to cheat on exams or assignments. Besides encouraging honesty and ethical behavior,
instructors should be very aware of the capabilities of technologies such as ChatGPT when
designing exams or assignments. Similarly, when used as an assistant, students can use
ChatGPT to complete homework assignments without fully understanding the material.
Including self-reflection and critical thinking exercises are essential to prevent this.
Secondly, ChatGPT may provide students with false or misleading information or simply
wrong answers, as illustrated in Figure 4. In this vein, ChatGPT is particularly inadequate in
343
User: Let's start a role-playing game. You are a bookstore manager that is experiencing a decrease in
sales. I am advising you on how to improve your business. Please, keep this role until I say
otherwise.
IA: Understood, I am now the bookstore manager facing declining sales. Please proceed with your
advice.
User: When did the decline start? What do you think the reason is?
IA: I would say that the decline in sales started about 6 months ago. (…) Some of the possible
reasons could be increased competition from online bookstores, changes in consumer spending
habits, a shift towards digital reading, or a decline in foot traffic to the area where the store is
located.
User: You mentioned a decline in foot traffic near the store? Are there construction works around?
IA Yes, I did mention a possible decline in foot traffic due to changes in the area around the store.
To my knowledge, there are currently no construction works happening in the immediate
vicinity of the store, but there could be other changes, such as road closures or new
developments, that could be affecting the flow of foot traffic.
User: And what about your costs? Are they mainly fixed or variable?
IA: I can say that our costs are a mix of both fixed and variable expenses. Fixed costs, such as rent,
insurance, and salaries, remain relatively constant regardless of our sales volume. Variable
costs, such as the cost of books, shipping expenses, and advertising, can fluctuate depending on
our sales and other factors. (…)
User: Which percentage of costs is variable?
IA: I don't have exact figures on hand, but I would estimate that around 30-40% of our costs are
variable and the rest are fixed. This can fluctuate from month to month, depending on factors
such as the cost of books, shipping expenses (…).
User: Since the foot traffic has declined, you might try to negotiate a reduction in the rent. How is
your contract?
IA: I can say that rent is a significant fixed cost for our business and negotiating a reduction in rent
could definitely help improve our bottom line. Our current lease agreement is set for a fixed
term, and any changes to the agreement would require the consent of both parties. (…)
Figure 3. Example of a role-player game with ChatGPT.
providing references and Digital Object Identifiers (DOIs) that can be used to track down the
original source of the information. This lack of information makes it even more challenging
to determine the accuracy and reliability of the information provided. Although future
developments in the technologies could improve the accuracy of the information provided,
fostering critical thinking is again key for avoiding wrong learnings.
344
Josep Domenech
User: Demonstrate that x+7 is even.
IA: If x is an even number, then it can be represented as 2k for some integer k. Therefore, x + 7 =
2k + 7, which can also be represented as 2(k + 3). Since the result is in the form of 2n for some
integer n, it is an even number. Therefore, if x is an even number, x + 7 is an even number.
(…)
IA: Suppose the demand for apples is given by the equation P = 120 - 2Qd and the supply is given
by the equation P = 2Qs. What is the market equilibrium price and quantity?
User: Q=40
IA: That's correct! Now let's move to scenario 2. Suppose there is a technological breakthrough in
apple production, which reduces the cost of production. How does this affect the market
equilibrium price and quantity?
User: Both price and quantity will increase.
IA: Exactly! An increase in supply, as a result of a technological breakthrough, will shift the supply
curve to the right. This will result in a higher quantity and a lower price at the new market
equilibrium. Great work!
(…)
IA: To find the market equilibrium, we need to find the intersection of the supply and demand
curves, where the quantity supplied is equal to the quantity demanded. We can start by setting
the demand and supply equations equal to each other:
120 - 2Qd = 2Qs
Solving for Q, we get: Q = 60. Then, substituting Q = 60 into either the demand or supply
equation, we can find the market equilibrium price:
P = 120 - 2(60) = 120 - 120 = 0
So the market equilibrium quantity is 60 and the price is 0.
Figure 4. Three examples of wrong answers by ChatGPT.
Furthermore, there is no guarantee that ChatGPT is available during class time. At this
moment, it is offered as a free research preview with limited availability, and its access might
be canceled at any time. This makes it unreliable for educational purposes and raises concerns
about the dependability of the technology.
5. Conclusions
Artificial intelligence has the potential to revolutionize the way we teach and learn. One
promising AI application is the use of large language models, such as ChatGPT, in various
educational contexts.
345
ChatGPT can be used as a writing assistant, study tool, personal tutor, assessment tool,
content creator, game-based learning tool, and language learning tool. It can generate
instructions, procedures, explanations, dialogues and responses to students’ inquiries. In all
these applications, ChatGPT can help to make the learning process more interactive and
engaging, and to provide students with exposure to the target content.
As a natural language model, ChatGPT is well-suited for educational settings where plain
language is used to convey knowledge, for example, in the humanities or social sciences. It
can also be useful in scientific and technical studies since it can understand specific jargon
or terminology. Being based on a natural language model is a primary advantage, but it is
also a significant drawback because its limited understanding of mathematical and specific
notations may restrict its effectiveness in fields that rely heavily on them.
In order to reach learning objectives, future educators should learn which prompts spark
productive conversations with ChatGPT. This process is analogous to coding but utilizes
natural language and is therefore accessible to instructors without programming skills. Future
learners should include ChatGPT into their repertoire of tools for both academic and
professional endeavors while developing critical thinking skills to evaluate the quality of the
responses provided by the AI.
Despite the potential benefits of integrating ChatGPT in the classroom, educators should be
aware of the potential challenges it poses, such as those related to cheating and availability,
since they may affect the class even if the instructor does not adopt it. The limitations of AI,
evidenced by the inaccuracy of the information provided, emphasize the need for human
intelligence to promote critical thinking.
References
Bennani, S., Maalel, A., & Ben Ghezala, H. (2022). Adaptive gamification in E‐learning: A
literature review and future challenges. Computer Applications in Engineering
Education, 30(2), 628-642.
Bohra, A., Barwar, N.C. (2022). A Deep Learning Approach for Plagiarism Detection System
Using BERT. Lecture Notes on Data Engineering and Communications Technologies,
111, 163-174.
Mousavinasab, E., Zarifsanaiey, N., R. Niakan Kalhori, S., Rakhshan, M., Keikha, L., &
Ghazi Saeedi, M. (2021). Intelligent tutoring systems: a systematic review of
characteristics, applications, and evaluation methods. Interactive Learning Environments,
29(1), 142-163.
OpenAI Team. (2022). ChatGPT. OpenAI. Retrieved February 3, 2023, from
https://openai.com/
346
Josep Domenech
Pender, H. L., Bohl, L., Schönberger, M., & Knopf, J. An AI-based lesson planning software
to support competency-based learning. 8th International Conference on Higher
Education Advances (HEAd'22), 1033-1041.
Smith, G. G., Haworth, R., & Žitnik, S. (2020). Computer science meets education: Natural
language processing for automatic grading of open-ended questions in ebooks. Journal
of educational computing research, 58(7), 1227-1255.
Vittorini, P., Menini, S., & Tonelli, S. (2021). An AI-based system for formative and
summative assessment in data science courses. International Journal of Artificial
Intelligence in Education, 31(2), 159-185.
Wang, Y. J., Shang, H. F., & Briody, P. (2013). Exploring the impact of using automated
writing evaluation in English as a foreign language university students' writing. Computer
Assisted Language Learning, 26(3), 234-257.
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of
research on artificial intelligence applications in higher education–where are the
educators? International Journal of Educational Technology in Higher Education, 16(1),
1-27.
Zhang, L., Huang, Y., Yang, X., Yu, S., & Zhuang, F. (2022). An automatic short-answer
grading model for semi-open-ended questions. Interactive learning environments, 30(1),
177-190.
347
“Building” knowledge by creating manipulatives with the 3D

printer: a course for mathematics student teachers
Institute for Didactics of Mathematics and Computer Science, Goethe-University, Germany.
Abstract
3D modelling and printing (3DMP) is a suitable context for education. With it,
own manipulatives for mathematics education can be created, which was not
possible easily before. We develop a seminar at Goethe University, in which
use 3DMP to teach mathematics student teachers about mathematics,
pedagogy, and technology. In this paper, we present the general 3DMP
process, and how we integrate it in the seminar. The structure and contents of
the seminar, as well as results from the students are laid out. The hardware
and software used are shown and explained, as well as our approach to 3D
modelling with the help of programming. The individual iterations of the
seminar and the conclusions show that students mainly benefit from being able
to realize their ideas very early and having the freedom to choose their own
topic.
Keywords: 3D modelling and printing; mathematics education; project-based
learning.

“Building” knowledge by creating manipulatives with the 3D printer
1. Introduction: 3D Printing
With 3D printing it has become possible to create anything that fits on or in a printer. There
are some variations on how 3D printers work: some work by using powder and glue, while
others use lasers to harden special resins. The method mostly used in affordable printers is
additive manufacturing (FDM, “Fused Deposition Modeling”), which will be referred to in
the following as 3D printing (3DP). In 3DP, the printer heats a special form of plastic
(“filament”) until it reaches the right temperature (and therefore viscosity), and then “prints”
it on top of a print bed—or the previous layers (see figure 1).
Figure 1: The layers of an additive 3D-print. The upper layers are printed on top of the lower layers.
In general, 3D modelling and printing (3DMP, see Anđić et al., 2022, Läufer & Ludwig, in
press) follows the following steps (also shown in figure 2), with an example of a
mathematical object (Menger sponge with recursion depth of 2) in figure 3:
1. The process starts with a concept or aim that is to be achieved and notes and calculations
are written down.
2. Inside a modelling program or application, a 3D model is created with the help of said
notes (Figure 3, top left).
3. This 3D model is then imported into a so-called “slicing”-Software, which “slices” the
model into layers (figure 3, top right) and creates a file with a series of commands
specific for each printer model.
4. The printer then prints the model in layers by following the commands inside the file
(Figure 3, bottom left).
Figure 2: The 3D Modelling and Printing (3DMP) process, divided into phases, tools, and outputs. From every
step, we can go towards an earlier step in case a change is needed. After the process has been completed, the
manipulative can be validated if it fulfills the aim by using it in class.
350
This process is iterative. After a first prototype has been printed, multiple changes might be
done in following iterations. Either expanding or modifying the original concept (by shifting
focus), model, or commands for the printer in the file. This does not only happen after a
process has been finished. It may also occur in previous phases. For example, in the slicing
program, we can see whether a 3D model may be printable or not, sometimes requiring a
remodelling in the modelling program before printing.
Figure 3: The Menger sponge in the modelling program (top left), the slicing program (PrusaSlicer, top right),
during the print (Prusa I3 Mk3S+, bottom left) and printed (bottom right).
In their meta-study, Ford and Minshall (2019) have compiled multiple educational uses of
3D printers. Teachers using 3D printers in their lessons is popular, which has also shown in
the studies by Pearson and Dubé (2022) and Kit Ng, Tsui, and Yuen (2022). Ford and
Minshall (2019) also report that 3DP in school brings a range of benefits, namely increased
motivation and realism inside the classroom, raised student engagement, interest in subject
material, and student understanding in the respective fields. Pearson and Dubé (2022)
identified multiple studies showing promising effects for (mathematics-) education, such as
increased mental rotations skills and understanding through visualization. (Huang & Lin,
2017; Katsioloudis & Jones, 2015) However, these effects are only possible with significant
effort of the teacher. (Ford & Minshall, 2019)
While this opens possibilities for students, teachers, and educators, the negatives of 3D(M)P
are considerable: long print times for even small objects, a large technical overhead, and a
high need for instructor support and training indicate that the complexity of 3D(M)P may not
be fully advantageous and integration of 3DMP into lessons faces difficulties. Educators find
it difficult to change from their traditional methodology towards a new one (Medina Herrera,
Castro Pérez, & Juárez Ordóñez, 2019), and the professional development opportunities
needed to use 3DMP in school as a teacher are missing in current teacher education. (Ford
& Minshall, 2019; Simpson, Williams, & Hripko, 2017)
351
2. The Seminar
As described above, 3DP is a rich possibility to produce even complex models according to
one’s own ideas. Therefore, we offer a 3DP seminar at Goethe-University, in which student
teachers can develop and print suitable manipulatives for their teaching ideas in the context
of mathematics education. The seminar (see figure 4) has been developed for the education
of 10-12 mathematics student teachers (at least in the 3rd semester). It is the aim to teach them
the entire process of 3DMP in one semester. In particular, the seminar aims to increase the
mathematical, technical, and pedagogical knowledge and to foster the creativity of the
students. The first iteration of the seminar (in the following: “first iteration”) was held from
April-July 2022 (13 weeks). The second iteration of the seminar (in the following: “second
iteration”) was held from October 2022 through February 2023 with a span of 15 weeks.
Between the sessions (2x 45 min/week) the students could make appointments to print their
models with the university's printers under the condition of them being present for the entire
printing time.
Figure 4: The structure, basic contents, and duration of each phase of the seminar per iteration. On the left side
are the elements of the first iteration. The right side holds the second iteration. The “first print” of the first
iteration (left) was the first time the students completed the 3DMP process themselves. In the second iteration, the
“first print” was in the second session.
The student teachers must create and present manipulatives (individually) that are unique,
not easily manufactured by other means, available in shops for a reasonable price, embedded
in a lesson plan, complex to a certain degree (i.e., a single simple irregular shape is not
sufficient), and do not exist or do only exist virtually, like a sketch or drawing in a book.
2.1. Software use, hardware used and script-based modelling

3DMP is a technologically advanced topic. At least 3 of the 4 phases of the 3DPM process
(3D modelling, Print Preparation, and Print, see figure 2) use specific technology and
applications. Common 3D modelling applications used for 3DMP may be TinkerCad,
352
Fusion360, SketchUp, and many more. All of them are complex programs, which try to find
the balance between usability and powerfulness of the modelling possibilities. We set the
criterion that the students must use their mathematical skills to think about all aspects,
geometric objects used for, and dimensions of their manipulative before starting the
modelling.
Therefore, we chose the free, open-source modelling program Blender. Blender meets the set
criterion since it supplies an integrated interface to use scripts – small programming snippets
– for 3D modelling. We then developed a library of programmatic functions that reduce the
complexity of Blenders’ internal interface. This allows script-based modelling of 3D models
with a low-threshold programming language, with us being able to adjust the complexity of
the programmatic functions and therefore scaffold the students receive. For example, as
shown in figure 5, the function “box” only needs two points to create a cuboid. It also supplies
the benefit of being precise and the students needing to be prepared before they start
modelling, requiring them to think before they do. Depending on the object, demanding
calculations are required by the students during 3DMP (Kostakis, Niaros, & Giotitsas, 2015;
Nemorin & Selwyn, 2017, cited by Pearson & Dubé, 2022). Depending on the complexity of
the manipulative, this may be different from other modelling applications where objects can
be dragged, dropped, and edited. A precise measurement may only be needed for complex
objects. Additionally, algorithmic structures like loops can be used in the scripts as well,
allowing manipulatives like the hundreds chart in figure 5 to be created efficiently and
precisely, without needing to place 100 individual cuboids manually.
Figure 5: The hundreds field (left, zoomed out to fit) and the code that created it (right) in the Blender scripting
area. There are two nested loops in lines 1260 and 1261, each iterating the respective variables – representing the
row j and the column i – from 1 to 10. The programmatic function “box()” in lines 1270-1273 creates a cuboid
where one side is parallel to the x-y plane, and where the given points (line 1271 is one (x,y,z) point, line 1272 is
the other) are in the corners diagonally away from each other.
Slicing software takes the 3D model and creates a set of instructions for a specific type of
printer, due to different types and brands having different components and calibrations. The
slicing software, PrusaSlicer, is not only free, but tailored to our printers by the creators
themselves, reducing the need to fine-adjust the settings in the slicer to get good prints.
353
The printers we use (Prusa I3 Mk3S/S+) are well suited for shared use, like in the university
or school, due to the features the printer has, like a detachable print bed, auto-calibration of
the print head, and a filament sensor, which makes switching filaments easy. We chose these
printers because they are reliable and there is a slicing software tailored to them.
2.2. First Iteration

The first iteration started with a presentation of the basics of manipulatives and the basics of
the 3DMP process. By showing an example of how to create a 3D puzzle, we presented each
step. Afterward, we introduced the students to the scripting library, and they started
conceptualizing their first ideas, which were then presented shortly to the group in the 5th
week. Feedback on the progress was given. This phase was called input-phase.
The first conceptualizations showed that the students did not understand the printers’ limits
or misunderstood the justification for 3DP to create manipulatives. Some designed long,
floating pieces that cannot be printed without significant changes in their design, while others
used the 3D printer just for the reproduction of pre-existing manipulatives that can be bought
cheaply or easily created with alternative materials. We traced this back to us not enabling
the students to print their ideas in the beginning. The first time that students printed something
of their design was after 7 weeks (see Figure 4). After the feedback, we continued giving
input on the phases and then began the work-phase, where attendance was not needed. These
could be used for working on the concepts, printing, and asking questions. This phase is
called hands-on-phase. The objects that the students then created and printed were (among
others):
• A puzzle for the cubic formulae with different levels of complexity and distractors,
• A puzzle on approaching the surface of a sphere with triangles,
• A board and pieces for visualizing a path in statistics as a game.
2.3. Changes in the second iteration

Learning from the first iteration, we began the second iteration similarly, but let the students
create and print their own mathematically themed cookie stamps in the beginning. Due to the
small size and limited complexity, these could be printed in about 20–30 minutes. With our
3 available printers, we managed to print them for almost all students. The only part in the
3DMP we took over was the slicing of the model due to the amount of experience required.
After the students experienced the complete 3DMP process themselves and the input sessions
on how to create their own, the students could focus more on the mathematical and didactical
side of the process. In the following sessions, more complex models were created by the
student teachers with scripts. There were more (creative) and realistic ideas than in the
previous iteration, and the students hesitated significantly less to start printing their ideas.
354
We found that allowing the students to print their models (cookie stamps) early on, was truly
motivating. This motivation was upheld by the students, with work-sessions (no attendance
required) being filled with almost all students. The evaluation of the seminar was also very
positive. Some examples of what students are developing are:
• Platonic Solids as a solid object and their dual solid as an edge model.
• A hundreds-field for early and special education, where the value of the numbers can
be felt. (See figure 5 for the 3D model)
• A puzzle of the partner numbers to ten.
We also used one session, right before the hands-on-phase, to present a different way to
create 3D models with TinkerCad, which has been used by some students to create parts of
other models. This was done to allow the students to create simple prototypes more quickly
and test the printer settings (like different layer heights) more easily.
3. Conclusion and what is to come

In this paper, we have presented how 3DMP works, as well as how we used it in the context
of a seminar at Goethe University. The comparison of iteration one and two makes clear that
showing the whole process early on is crucial for student motivation and creativity. The
students used a lot of their own ideas and can handle the 3D modelling, preparing the print,
and printing on their own. Some support, especially on the technical side, needs to be given
nonetheless. The students in the second iteration were motivated and produced innovative
manipulatives for mathematics education. Further integration and studies thereof in other
subjects such as computer science are of high interest.
The following iterations will embrace the work more with project diaries that will be used as
a basis for a PhD-research. Additional interviews with the students on how the design and
print processes work for them individually will be held and analyzed. We believe that letting
students print their own manipulatives increases their mathematical, didactic and technical
knowledge (see TPACK Model, Mishra & Koehler, 2006). The individual 3DMP phases
which the students experience and their relation to the TPACK model are to be investigated
in this research project.
References
Anđić, B., Ulbrich, E., Dana-Picard, T., Cvjetićanin, S., Petrović, F., Lavicza, Z., &
Maričić, M. (2022). A Phenomenography Study of STEM Teachers’ Conceptions of
Using Three-Dimensional Modeling and Printing (3DMP) in Teaching. Journal of
Science Education and Technology. Advance online publication.
https://doi.org/10.1007/s10956-022-10005-0
355
Ford, S., & Minshall, T. (2019). Where and how 3D printing is used in teaching and
education. Additive Manufacturing, 25, 131–150.
https://doi.org/10.1016/j.addma.2018.10.028
Huang, T.‑C., & Lin, C.‑Y. (2017). From 3D modeling to 3D printing: Development of a
differentiated spatial ability teaching model. Telematics and Informatics, 34(2), 604–613.
https://doi.org/10.1016/j.tele.2016.10.005
Katsioloudis, P., & Jones, M. (2015). Technology and Engineering Teacher: Using
computer-aided design software and 3D printers to improve spatial visualization.
07463537, 14+.
Kit Ng, D. T., Tsui, M. F., & Yuen, M. (2022). Exploring the use of 3D printing in
mathematics education: A scoping review. Asian Journal for Mathematics Education,
1(3), 338–358. https://doi.org/10.1177/27527263221129357
Kostakis, V., Niaros, V., & Giotitsas, C. (2015). Open source 3D printing as a means of
learning: An educational experiment in two high schools in Greece. Telematics and
Informatics, 32(1), 118–128. https://doi.org/10.1016/j.tele.2014.05.001
Läufer, T., & Ludwig, M. (in press). Bringing 3D Printing Into Student Teachers’
Mathematics Education. In Research On STEM Education in the Digital Age (ROSEDA),
Porto.
Medina Herrera, L., Castro Pérez, J., & Juárez Ordóñez, S. (2019). Developing spatial
mathematical skills through 3D tools: augmented reality, virtual environments and 3D
printing. International Journal on Interactive Design and Manufacturing (IJIDeM),
13(4), 1385–1399. https://doi.org/10.1007/s12008-019-00595-2
Mishra, P., & Koehler, M. J. (2006). Technological Pedagogical Content Knowledge: A
Framework for Teacher Knowledge. Teachers College Record, 108(6), 1017–1054.
https://doi.org/10.1111/j.1467-9620.2006.00684.x
Nemorin, S., & Selwyn, N. (2017). Making the best of it? Exploring the realities of 3D
printing in school. Research Papers in Education, 32(5), 578–595.
https://doi.org/10.1080/02671522.2016.1225802
Pearson, H. A., & Dubé, A. K. (2022). 3D printing as an educational technology: theoretical
perspectives, learning outcomes, and recommendations for practice. Education and
Information Technologies, 27(3), 3037–3064. https://doi.org/10.1007/s10639-021-
10733-7
Simpson, T. W., Williams, C. B., & Hripko, M. (2017). Preparing industry for additive
manufacturing and its applications: Summary & recommendations from a National
Science Foundation workshop. Additive Manufacturing, 13, 166–178.
https://doi.org/10.1016/j.addma.2016.08.002
356
The complexity of grading student work and the reconstruction of

the meaning of criterion-referenced assessment
Liao Liang
The Office of University General Education, The Chinese University of Hong Kong, Hong
Kong.
Abstract
This study explores how assessment criteria are applied in grading student
work. It is found that explicit assessment criteria do not work as authoritative
guidance as expected and that tacit criteria are more decisive in awarding a
certain grade. Various sources that form idiosyncratic tacit criteria are
identified. These sources, including values on curriculum and assessment,
teaching experiences, and the way of supervising grades, work as different
social contextual elements that influence judgements on grading students’
work. The study suggests that tacit criteria and the different sources that form
the tacit criteria need to be identified, perceived, and communicated in the
community of practice to reduce grade variability and achieve a shared
understanding of grading.
Keywords: Criterion-referenced assessment; grading process; assessment
criteria; tacit criteria; shared understanding.

The complexity of grading student work
1. Introduction
Criterion-referenced assessment (CRA) has been recognised as a common method for
assessing what students have achieved in a specific course (Sadler, 2017; Svennberg,
Meckbach, & Redelius, 2018). Compared with norm-referenced assessment, which evaluates
a student’s work by ranking it within a group (Lok, McNaught & Yong, 2016), CRA is based
on scoring according to a series of explicit criteria (Popham, 1978, 2014). CRA has many
positive impacts on student assessment. As grading in a bell curve is no longer required in
criterion-referencing, pursuing higher grades becomes a matter of an individual’s efforts,
rather than competing with others. Additionally, by providing explicit assessment criteria,
students can form a clearer understanding of what the assessment task entails, how it will be
judged, and what the level of achievement is (O’Donovan, Price, & Rust, 2001).
Although CRA has many advantages, it is still need to note that the grading process is
complex and intermingled with assessors’ internal judgement (Bloxham, Boyd & Orr, 2011;
Orr, 2007). Previous researches indicate that besides explicit criteria, tacit criteria commonly
exist in grading judgement (Sadler, 2005, 2009). Tacit criteria are usually personally
determined, which could easily lead to a substantial difference between different assessors
when judging the same student’s work (Bloxham, 2016a). Consequently, how an assessor
applies the explicit and tacit criteria to award a grade greatly affects the reliability and quality
of an assessment. To review and to reflect upon the quality of student assessment, a thorough
investigation of the process of grading judgement is necessary.
2.1. The complexity of grading process

Several studies reveal similar findings, stating that making judgements in grading is more
likely to happen implicitly rather than being based on external sources. In other words, the
real standards are locked inside the teacher’s head (Bloxham et al., 2016a; Grainger, Purnell,
& Zipf, 2008; O’Hagan and Wiggleswoth, 2015). Furthermore, the interpretation of each of
the standards for the criteria is highly diverse among assessors, thus leading to a large gap in
assessment results for the same student’s work. For example, Bloxham et al. (2016a) invited
24 teachers from four isciplines to assign five students’ writing works respectively; they
found substantial differences in the awarded grades. Only one of the 20 pieces was graded in
the same rank by six teachers in any of the disciplines. The assignment gap for the same
student work was at least three positions. Bloxham points out that assessors’ internal
standards framework plays an important role in awarding grades. This internal framework,
also called tacit criteria, commonly exists among assessors, even when explicit criteria are
provided (Sadler, 2005, 2009; Gonzalez & Burwood, 2003). The nature of tacit criteria is
recognised as idiosyncratic (Matshedisho, 2020; Adie, Lloyd, & Beutel, 2013), to be
358
Liao Liang
perceived unconsciously and expressed uneasily (Sadler, 2009, 2013). Tacit criteria increase
the opacity of the grading process and make grading complicated to understand.
Beside the effect of tacit criteria, grading judgement is also contextually influenced. Shay
(2004) used conceptions of ‘field’ and ‘habitus’ to discuss the basis of decision making in
assessment. It concludes that assessing complex tasks is a socially situated interpretive act.
Assessors’ interpretations are shaped by their disciplinary orientations, years of experience,
and level of involvement with students. These interpretations are constituted not only to
sustain systems of belief but also to maintain identities and interpersonal relations. Given
social practice, grading is not merely decided individually. More precisely, it is closely
related to the backgrounds and experiences of assessors and the institutional culture they
encounter (Watty et al. 2014; Zahra et al. 2017).
3. Research questions
This study aims to explore how teachers internalise, interact with, and interpret explicit
criteria to illuminate the mechanism of using explicit criteria to frame internal decisions. The
role of tacit criteria and its influence in grading judgement will also be investigated. The
following research questions are considered:
1. What are the roles of explicit and tacit criteria in determining the final grade?
2. How are tacit criteria formed and how do they influence the assessor’s grading
considerations?
3. What insights are proposed to improve CRA practice by revealing the formation of tacit
criteria and the nature of the grading process?
4. The case
4.1. Background
A general education programme of University A in Hong Kong was selected as the case
study. There are currently 28 full-time teachers involved in this programme to teach two GE
courses. In 2018, CRA was required to be fully implemented in all faculties and teaching
units at University A. A task force was initiated in the GE programme to respond to the shift,
which involved five teachers from two GE courses and one researcher. Grading rubrics were
developed based on the intended learning outcomes to echo the outcome-based approach
advocated by the Quality Assurance Council. In addition to developing grading rubrics,
several meetings were also held to discuss CRA and how to use the grading rubrics.
359

Interviews were the main sources of data collection. Seven GE Programme teachers were
invited to attend one-on-one interviews, and each interview lasted 1–1.5 hours. Interview
questions addressed teachers’ considerations of using grading rubrics, understanding the
assessment criteria, and their views on CRA, GE learning goals, and assessment. Five of the
teachers had more than six years of teaching and assessment experiences in this programme.
Two were novice teachers with less than one year of work experience. In addition to
conducting interviews, the author joined the task force, developed grading rubrics with other
task force members, and attended all meetings on CRA. Other sources of data included
reflective notes of CRA meetings, emails of the discussions among teachers, and a brief
survey of teachers. As a complement to teachers’ thoughts on CRA and grading judgement,
some informal talks were also collected and used to support teachers’ viewpoints. In CRA
meetings, the author was an observer and took notes on what happened in the field, such as
questions raised, interactions, and responses among teachers.
5. Results
5.1. Role of explicit and tacit criteria in grading judgment

All teachers reviewed the descriptions of explicit criteria and expressed their evaluations on
them. Grading is not like enacting orders under the guidance of external requirements; it is
rather an application of internalised criteria. A typical problem of internalising explicit
criteria was about abstract expressions such as ‘focused topic’, ‘relevant evidence’, and
‘logical and specific conclusion’. Teachers noted that these wordings were very general and
needed more explanations.
Because elaborating and communicating explicit criteria were not recognised as a regular
practice, most teachers took individual interpretations on these explicit criteria as common;
nevertheless, they did not realise the necessity of illuminating their own interpretations.
Among the seven teachers, only one showed the notes, which explained in detail what the
abstract descriptions in each criterion mean, whereas the other teachers did not specify the
meaning of those abstract wordings. It seemed that teachers had examined the meaning of
criteria implicitly, but they did not explicate and reflect their interpretations or seek other
ways of interpretations.
Sometimes, personal interpretations were in conflict with the explicit criteria. Three teachers
noted that they did not completely agree with the standards for each criterion. Even so, they
had their way of reconciling the external standards with their internal standards. For example,
a teacher noted that they would grade leniently on some criteria which were set higher in
their views:
360
Liao Liang
‘I use different standards in grading reflective journal and term paper. For
reflective journals, standards will be set lower since they’re completed in the
middle of the course and where in this stage, understanding is most important
other than higher-order thinking skills. I would give the score leniently in the
reflective journal. If students show a good understanding of texts, I will give
them a good grade even if they do not exert complex thinking skills’.
Not every teacher would refer to all criteria listed in the grading rubric. Among the seven
teachers, three indicated that they would make a holistic judgement, rather than make one by
referring to every criterion. One teacher stated disagreement in dividing criterion into several
parts, for it will spoil the integrity of student achievement and make each part of performance
separate and irrelevant to each other:
‘All criteria should be related, and they actually reflect the whole. I will focus
on the integrity of student’s performance’.
5.2. The formation of tacit criteria and its influence on grading considerations
What qualities should be cultivated for students affect the interpretation and usage of explicit
criteria. Teachers held different expectations regarding the GE curriculum. These individual
notifications were not fully covered either by learning outcomes or explicit criteria. Both
learning outcomes and assessment criteria were worked out by the representatives of
teachers, instead of all of them. Consequently, these uncovered expectations became tacit
criteria. This may explain why personal interpretations and various grading strategies were
common. For example, a teacher who regarded curiosity as an essential characteristic noted
that ‘he would adjust the descriptions of explicit criteria and inject the element of curiosity
into them’. Another teacher who did not hold the value of curiosity would not interpret the
criteria in this way.
Compared to novice teachers, experienced teachers were more flexible in using explicit
criteria. Experienced teachers were able to interpret the meaning of criterion from a broad
perspective, instead of translating the keywords of a criterion literally. In addition, they were
more confident in determining the ‘right’ grade for students. Experienced teachers seemed to
know how to adapt explicit criteria to make grading decisions more consistent with their tacit
criteria. Such an adaptive strategy was scarce in novice teachers. For novice teachers,
interpreting explicit criteria involved many uncertainties. It was difficult for them to connect
teaching experiences with the criteria to give an appropriate grading judgement. Unlike
‘technically adjusting’ the meaning of explicit criteria, as done by experienced teachers,
novice teachers more often adopted direct methods. They added or deleted criteria to make
explicit criteria more consistent with their internal judgement, although these methods would
lead to various patterns of criterion-referencing. Confidence in grading was also inadequate
361
among novice teachers. Assessment training was more frequently mentioned by these
participants.
‘I hope I could receive some trainings on how to award a grade. Differences
in each grade level, especially between A- and B+ or some similar cases, were
not very clear for me to identify. Maybe we could invite some grading experts
and share with us how to grade accurately’.
The other difference between grading considerations of experienced and novice teachers was
their adoption of holistic judgement. Two novice teachers indicated they would adopt
analytic judgement, whereas, of five experienced teachers, three noted that they preferred
holistic judgement. Although task force members communicated with experienced teachers
that analytical judgement was indispensable in criterion-referencing, it was hard to change
their views. Adopting a holistic judgement was related to the issue of trust and efficiency.
For some teachers, holistic judgement would be more reliable. Teachers worried that the final
grade might be unmatched with their original judgement by adding sub-scores to each
criterion. The other consideration was time. Teachers noted that analytic judgement was too
time-consuming:
‘I have tried grading according to the criterion one by one and find it wasted
too much time. After doing this, I still need to review the grade and examine if
the judgement is appropriate with a holistic approach’.
Besides the influence of teaching experiences that form the different basis of grading
judgement, many teachers treated grade distribution as a boundary to keep assessment results
‘safe’. In other words, although CRA called for ‘giving students a real grade’ and
‘abandoning grading in a bell curve’, in the grading practice, norm-referenced grading was
still being tacitly used.
There were two patterns of tacit practices. One examined the entire grade distribution after
grading and compared it with the previous policy. If any ‘abnormal’ grades were discovered,
they would be adjusted. Some teachers chose to examine grade distribution in the mid-term
to adjust the coming grading strategy:
‘I would first examine the results of the reflective journal, if the grade
distribution is not good, I would reconsider the grading of the term paper. If
there are too much A’s in term papers, I probably would make some
adjustments’.
Another way of applying the principle of grade distribution is to set a line to guide grading.
In this way, grades were regulated during the grading process to satisfy the hidden criteria of
the specific distribution. This approach was more undisclosed and even discerned that
teachers were using norm-referencing. In sum up, Grading was still reckoned on norm-
362
Liao Liang
referencing because of the obscure policy of monitoring grades. In the brief survey results,
many teachers indicated that grade distribution was still applied in their grading practice.
6. Discussion
Although CRA has been practised over the years, this study revealed that the educational
ideas of CRA are not thoroughly manifested in grading practice. The core idea of CRA is to
consider the explicit criteria reflectively, so as to minimize the subjectivity in grading
affected by student image (Shay, 2004), peer relationship (Zahra et al., 2017), department
culture (Deneen & Boud, 2014) and so on. The marginalisation of explicit criteria and
amplification of the power of tacit criteria may be due to a superficial and inappropriate
understanding of CRA. Elaborating and reconstructing the meaning of CRA is, therefore,
necessary to clarify some common misconceptions of criterion-referenced grading and to
ensure a fairer grading process.
First, explicit criteria should not be regarded as the absolute standard, nor should personal
interpretations be taken for granted without examining them in the community of practice.
As judgement making based on teachers’ professional experiences is in terms of whether
academic freedom is respected and protected (Sadler, 2011; Lee, 2006), while explicit criteria
provide a basis for calibrating various individual judgements (Sadler, 2013; Dracuo, 1997),
balancing the function of tacit criteria and explicit criteria is critical in deciding a consistent
and fair grading. Criterion-referencing does not favour the approach of maximising unified
standards, nor does it favour the opacity of the grading process. Letting explicit criteria and
tacit criteria dance together would make them the true criterion-referencing. To balance the
power of tacit criteria, it is critical to have empathy regarding understanding the rationality
of grading judgement by discussing it in the community of practice (Bloxham, Hughes, &
Adie, 2016b). It requires putting aside individual liberty in making grading judgements and
embracing the liberty of the whole (Sadler, 2011; Berlin, 1969).
Second, assessment criteria are in the central position of CRA. Therefore, the appropriate
usage of the assessment criteria needs to be clarified. This study argues that CRA is a social
constructive practice (Rust, O’Donovan, & Price, 2005), which contains two levels of
meaning. The first level represents how assessment criteria should be used among assessors.
It suggests that assessment criteria should be identified, selected, discussed, and
communicated in the assessment team. Furthermore, teachers’ value in the curriculum needs
to be elaborated and absorbed into the learning outcomes and assessment criteria pool (Watty
et al., 2014). The second level represents how assessment criteria should be used among
teachers and students. Assessment criteria and how they are interpreted by teachers should
be communicated with students (Bearman& Ajjawi, 2021). Social constructive practice
means to initiate a dialogue with students to address the meaning of criteria and build an
363
understanding on grading judgement. In this way, CRA plays a role of assessment for
learning which emphasises the concept of student-centred approach and empowers students’
experiences of assessment by involving them into the grading process (Sadler, 1987, 1989).
Third, CRA is a different grading approach other than the regularly-adopted norm-
referencing approach. How grades are supervised and what should be accountable for a given
grade substantially affects whether assessors will adopt real criterion-referencing. If
departments or universities still take grade distribution as the accountability objective, it is
no surprise that teachers will finally depend on grade distribution to award a grade. In this
case, explicit criteria would be marginalised and, more notably, students may not get the real
grade. Therefore, grade distribution as the only standard for reviewing students’ grades
should be abandoned. The focus can be on how the assessment criteria were set. Do they
match course content? Does the assessment task reflect the assessment criteria well? How do
students regard the assessment criteria? To answer these questions, the paradigm of grade
review needs to abandon the result-oriented approach and focus more on the grading process,
the decision making of the assessment team, and students’ feedback on grades and grading.
7. Conclusion
For a long time, many educators believe that the variability of grading is normal and
inevitable due to the existence of tacit criteria. However, should this passive attitude toward
grading continue, it would increase students’ negative perceptions of assessments. The
current study argues that criterion-referencing can reduce subjectivity in grading. This is
realized by reflectively examining the grading judgement, better understanding our own tacit
criteria, and discussing the explicit criteria in the community of practice. Revealing the
formation of tacit criteria and illuminating the sources can help assessors become aware of
the unperceived hidden standards influenced by complex social elements, consider grading
more carefully, pay attention to grade variability issues, conduct grade moderation
effectively, thus making grading and grades more fair. By recognising the characteristics of
explicit and tacit criteria and build a balance between them, the spirit of CRA, which
emphasises the assessment for learning, can be achieved.
References
Adie, L., Lloyd., Beutel, D. 2013. Identifying discourses of moderation in higher education.
Assessment and Evaluation in Higher Education 38(8): 968-977
Bearman, M., and Aiiawi, R. 2021. Can a rubric do more than be transparent? Invitation as a
new metaphor for assessment criteria. Studies in Higher Education 46(2): 359-368.
Berlin, I. 1969. Two concepts of liberty, In: I. Berlin, Four Essays on Liberty. Oxford
University Press: London, 118-172.
364
Liao Liang
Bloxham, S., P. Boyd, and S. Orr. 2011. Mark my words: the role of assessment criteria in
UK higher education grading practices. Studies in Higher Education 36(6): 655-670.
Bloxham, S., B. den-Outer, J. Hudson, and M. Price. 2016a. Let’s stop the pretence of
consistent marking: exploring the multiple limitations of assessment criteria. Assessment
and Evaluation in Higher Education 41(3): 466-481.
Bloxham, S., C. Hughes., & L. Adie. 2016b. What’s the point of moderation? A discussion
of the purposes achieved through contemporary moderation practices. Assessment and
Evaluation in Higher Education 41(4): 638-653.
Deneen C, Boud D. 2014. Patterns of resistance in managing assessment change. Assessment
and Evaluation in Higher Education,39(5):577-591.
Dracup, C. 1997. The reliability of marking on a psychology degree. British Journal of
Psychology 88(4): 691-708.
Gonzalez Arnal, S., and S. Burwood. 2003. Tacit knowledge and public accounts. Journal of
Philosophy of Education 37(3): 377-391.
Grainger, P., K. Purnell, and R. Zipf. 2008. Judging quality through substantive
conversations between markers. Assessment and Evaluation in Higher Education 33(2):
133-142.
Lok,B., McNaught,C.,&Yong,K. 2016. Criterion-referenced and norm-referenced
assessments: compatibility and complementarity Assessment and Evaluation in Higher
Education,41 (3): 450-465.
Lee,D.E. 2006. Academic freedom, critical thinking and teaching ethics. Arts and Humanities
in Higher Education: an International Journal of Theory, Research and
Practice,5(2):199-208.
Matshedisho, R. 2020. Straddling rows and columns: students’ (mis)conceptions of an
assessment rubric. Assessment and Evaluation in Higher Education 45(2): 169-179.
O’Donovan, B., Price, M., & Rust, C. The students experience of criterion-referenced
assessment(Through the introduction of a common criteria assessment grid)[J].
Innovations in Education and Teaching International,2001,38(1):74-85.
O’Hagan,S.,& Wigglesworth,G. 2015. Who’s marking my essay? The assessment of non-
native speaker and native-speaker undergraduate essays in an Australian Higher
Education Context. Studies in Higher Education,40(9):1729-1747.
Orr, S. 2007. Assessment moderation: Constructing the marks and constructing the students.
Assessment and Evaluation in Higher Education 32(6): 645-656.
Popham, J. 1978. Criterion-referenced measurement. Prentice-Hall: Englewood Cliff.
Popham, W.J. 2014. Criterion-referenced measurement: half a century wasted? Educational
Leadership 71(6): 62-66.
Rust, C., B. O’Donovan, and M. Price. 2005. A social constructivist assessment process
model: how the research literature shows us this could be best practice. Assessment and
Evaluation in Higher Education 30(3): 231-240.
Sadler, D.R. 1989. Formative assessment and the design of instructional systems.
Instructional Science 18(2): 119-144.
365
Sadler, D.R. 1987. Specifying and promulgating achievement standards. Oxford Review of
Education 13(2): 191-209.
Sadler, D.R. 2005. Interpretations of criteria-based assessment and grading in higher
education. Assessment & Evaluation in Higher Education 30(2): 175-194.
Sadler, D.R. 2009. Indeterminacy in the use of preset criteria for assessment and grading.
Assessment & Evaluation in Higher Education 34(2): 159-179.
Sadler, D.R. 2011. Academic freedom, achievement standards and professional identity.
Quality in Higher Education 17(1): 85-100.
Sadler, D.R. 2013. Assuring academic achievement standards: from moderation to
calibration. Assessment in Education: Principle, Policy & Practice 20(1): 5-19.
Sadler, D.R. 2017. Academic achievement standards and quality assurance. Quality in
Higher Education 23(2): 81-99.
Shay, S. 2004. The Assessment of Complex Performance: A Socially Situated Interpretive
Act. Harvard Educational Review 74(3): 307-329.
Svennberg, L., Meckbach, J., Redelius, K. Swedish PE teachers struggle with assessment in
a criterion-referenced grading system. 2018. Sport, Education and Society, 23(4): 381-
393.
Watty. K., Freeman. M., Howieson. B., Hancock. P., O’connell. B., Lange. P., and Abraham.
A. 2014. Social moderation, assessment and assuring standards for accounting graduates.
Assessment and Evaluation in Higher Education 39(4): 461-478.
Zahra, D., I. Robinson, M. Roberts, L. Coombes, J. Cockerill, S. Burr. 2017. Rigor in
moderation processed is more important than the choice of method. Assessment and
evaluation in Higher Education 42(7): 1159-1167.
366
The Reggio Childhood Studies PhD as a learning community

Marcella Colacino1, Roberta Mineo2
1
Department of Education and Humanities, Università di Modena e Reggio Emilia, Italy,
2
Department of Education and Humanities, Università di Modena e Reggio Emilia, Italy.
Abstract
Exploring visions and methods to pursue high level education means to study
how each person is allowed to develop new intellectual, aesthetic, and
operational perspective, as well as grasp new insights for growth and
enrichment of one’s understanding of the world. The authors present the pilot
phase of an analysis of the first industrial doctorate in Reggio Chilhood
Studies, organized by the University of Modena and Reggio Emilia in
collaboration with the Reggio Children Foundation: its socio-constructivist
matrix is highlighted. In fact, constructing a community of learners allows
cross-fertilization among diverse disciplinary knowledge and many cultures.
The study hereby presented explores the variables of such learning community,
through the analysis of in-depth interviews of the PhD fellows. Results show
factors and variables that can affect and challenge the existence of the
community. Recommendations are made to continue exploring the doctoral
community, bringing in the picture all the stakeholders involved.
Keywords: Industrial PhD; learning community; multidisciplinary; socio-
constructivism.

1. Introduction
The municipality of Reggio Emilia has long been internationally recognized as an excellence,
for its innovative pedagogical approach. The Reggio Emilia approach has been characterized
by the organization and management of the city's nurseries and preschools since 1963, when
it began to organize educational services, promoting the opening of the first kindergartens
for children aged 3 to 6. Loris Malaguzzi (Edwards, Gandini & Forman, 2010) is the
reference pedagogue who inspired a new way of perceiving the kid, his capacities, and the
collective responsibility that stems from them.
One of the fundamental pillars of Reggio Emilia’s pedagogy is the research component.
Anyone who wishes to participate in the educational process must regard research as an
ontological condition to which one must continually listen to be a part of a continuous route
of knowledge building. In this perspective, learning is viewed as a continuous, individual,
and collective process, whose stimuli and reflection on them are research topics in and of
themselves and apply to any sort of research. Carla Rinaldi (2009) defines the child as the
first researcher because: he is innately capable of listening to the various stimuli coming from
reality and capable of selecting them to build his knowledge based on his skills, thus
becoming a creator subject rather than a passive recipient.
1.1. The specifics of the doctoral program in Reggio Childohhod Studies

The doctoral program in Reggio Childhood Studies was launched in 2019 and now it is on
its fourth cycle. It is organized in collaboration between the University of Modena and
Reggio Emilia and the Reggio Children Foundation. As such, the doctoral program has been
envisioned as a learning community in support of specific research projects. Processes and
procedures that encourage participation, relationships among participants, skill borrowing
between fields of knowledge, and cross-cultural contaminations provide as many stimuli as
possible to interrogate reality and train for a continuous vital exchange with it.
The pursued research projects span from inclusion to biomaterials, semiotics to translation,
the study of prison environments to contemporary parenting models, the history of music to
the fight against the mafias: the underlying belief is that contact between seemingly disparate
fields increases the chances of innovative break-through studies and creative cross-
fertilization.
Training methods range from seminars, lessons, conferences, and laboratory experiences, all
based on innovative methodologies of peer-to-peer education, tutoring, mentoring, and
flipped classrooms. During the years of the COVID-19 pandemic, multimedia was enhanced,
moving formal and informal meetings online, through dedicated platforms and ICT tools.
The theoretical and methodological paradigm underlying this organization is one of
complexity (Morin, 2008), in which the various components, each with its cognitive style
368
and training, as well as the levels of organization of the doctorate itself, are to be understood
as related variables that participate in the construction of learning. Unpredictability or errors
are also included in this system, making them peculiar and vital components for the research
itself.
The theoretical frame of reference of the PhD upholds the concept of education as the creation
of intellectual, aesthetic, and social learning opportunities in which everyone is free to learn
autonomously as well as through the learning modality of others, in a horizon of meaning
that is never definitively given once and for all but must be regained each time.
Reggio Approach inspiring theories are addressed during the mandatory-readings seminars:
such meetings offer the opportunity to also experiment with different forms of
teaching/learning. Becoming co-constructive and intrinsically relational, the PhD allows
students to experience in practice what has been studied on a theoretical level, to monitor if
and how the program has produced transformations.
1.2. The research questions

The present study aims to identify the specifics of a learning community within highly
educated people, what themes emerge from a socio-constructivist approach when people talk
about a community of learners. Specifically, it aims to investigate all those variables that
have a positive impact on the emergence, functioning and sustainability of a learning
community; and those variables that can hinder or negatively affect the existence of a learning
community.
2. Methods and materials
2.1. Research design

The study comprises two phases: in the first pilot stage, the research aims to explore what the
most significant features of a learning community are in the the experience of the student
learners of the Reggio Childhood Studies PhD program. In this pilot phase, only a few
learners are extensively interviewed to investigate their perception of which protective
factors and which risks to the existence of the learning community they experienced in their
years of doctorate.
The second phase is both qualitative and quantitative. At this second stage, not only all the
PhD fellows are involved (a total population of 28 students) but also the faculty and the
managing staff (directors, program manager and technical coordinator). The present article
focuses on the results of the pilot study.
369
2.2. Sample and Tools

In this pilot study, the research team interviewed 2 student fellows per each cohort of the
PhD program, excluding the current year that is only in its first months of activities. Therefore
6 students were interviewed, each couple corresponding to one different cohort of the
program, that we named the “pioneers”, the “pandemics” and the “re-start”.
The criteria used to recruit the interviewees in this first phase were: gender, as males represent
only the 18% of the total population of 28 students; and country of origin, as international
students represent 29 % of the total number of fellows in the first three years of existence of
the PhD.
The semi-structured interview had 5 areas of investigation. The first was the academic and
working background of the interviewees together with their expectations. The second was to
explore the construct of a learning community: what specifics, what limitations, what
innovations etcetera in a constructionist approach (Gergen, 1994). The third area was that of
the protective factors of the learning community, in the experience of the interviewee: they
were asked to give specific examples of what elements of their doctoral life had a positive
effect on the building and maintaining of the learning community. The fourth area of the
interview was on the risk factors: that is what specific elements of their experience played a
negative or challenging role in the making and surviving of the doctoral learning community.
Finally, the last area was on the ideal prerequisites that they would consider when building a
learning community: what were the unquestionable conditions that an educator should be
aware of when fostering a learning community.
The more than three levels of headings, which will be numbered with Arabic numerals
starting at 1. The interviews were transcribed and translated, when needed. The authors
extracted all the themes that were mentioned in each interview; they then built a table with
the lists of themes/variables and pasted the parts of each interview that fitted under the titles
of such themes (see paragraph 3).
3. Results
The results are reported following the distinction in the five categories that the interviews
explored: background, construct, pillars, challenges, ideal prerequisites
370
Table 1. Thematic variables emerged from the analysis of the interviews (only one of examples
are shown, for brevity).
AREAS VARIABLES AND EXTRACTED EXAMPLES
BACKGROUND Plurality and heterogeneity

“Most of us come from regular university structure, where you have for
example to do certain things at a given time, you have like a clearly defined
structure ”.
The extended Community
“It can be extended to all the professors and all the people you meet while
doing the project”.
CONSTRUCT Physical presence
of a LEARNING "The physical presence much requested by the teaching staff has favored the
COMMUNITY construction of this community".
Multidisciplinary
“A learning community should share ways of comparing possibly between
different backgrounds to have different points of view”.
Team work
"It's a group of people who have a purpose, who have common goals."
Leadership
“Nobody forced us, on the one hand it is good and on the other hand it is
bad”.
PILLARS The Reggio Emilia Approach
“We tried to understand the concept of Reggio childhood from different
perspectives”.
Informal networking
"Over time, closeness has become friendship”.
CHALLENGES The pandemic
“Covid has hindered and blocked the informal relationship that was
fundamental humus at the beginning”.
Working online
“The fact that our group was formed online was somehow flawed by it”.
Spaces and times
“The question of spaces:there are not yet the conditions to offer a research
laboratory but only an office”.
“A learning community exists when there are people who meet on an ongoing
basis, with a fixed frequency, to create that type of exchange community”.
Legislation
“Our doctorate follows the national legislation according to which research
is individual”.
Competition/Individualism
“If a group is made up of many individuals, the competitive logic prevails.
Co-planning/co-leading can make a group supportive and non-competitive".
Language
“There's the issue of language. An issue especially for international students
because sometimes you want to bond and come closer to the community; but
because of the language it was not always possible”.
IDEAL Shared co-design
PREREQUISITES “Involving PhD students in activities and in co-planning together with
colleagues from the Foundation”.
Listening
371
“Listening as a form of openness, of recognition of the other, of his diversity,

of his individuality, as a form of curiosity".
Freedom of expression
“The facilitator must not be the one who organizes the meetings but the one
who favors the creation of this spirit of trust”.
Solidarity
“This start, so all together, unanimously, has certainly helped us a lot”.
4. Discussion
4.1. The backgrounds

Phd fellows range in age and have a diverse professional backgrounds, including art history,
biology, history, pedagogy, philosophy, languages, music, and psychology. The PhD students
come from a variety of national and international geographical contexts.
The teaching staff and teachers involved at different levels in the training have heterogeneous
backgrounds and curricula and come from universities all over the world, making a
significant contribution not only to the contents transmitted but also because they embody
the culture and teaching traditions of other universities.
All fellows saw such diversity as an asset of the emerging learning community. Plurality of
backgrounds and expectations grounded a sense of diversity as the true richness of the human
capital of the learning community, being considered both an ehancing factor of collaboration,
but also a challenge in the flow of the process: because working in group is a real job (Clark
& Wilkes-Gibbs, 1986).
4.2. The co-construction of a learning community

To create a culture of continuous learning and critical review of one's practices and beliefs,
the doctoral program is envisioned as a learning community in which the processes of
sharing, collaborative and non-competitive creation of knowledge and skills, and sharing
each person's cultural heritage were constantly encouraged (Gardner, 1987).
Because of such a vision, the PHD program is organized as a residential community, with
significant opportunities for formal, non-formal, and informal exchange, from the sharing of
spaces and times to the actual study and research part, to enhance all the skills acquired,
including soft skills.
The interviews show a general overlap between the theoretical idea of a learning community
and the experience of it in the doctoral program. Concepts like teamwork, leadership,
multidisciplinary are presented through the understanding of how the interviewee
experienced student life in the program. The construct is the result of a process of sense-
making of the experience itself.
372
4.3. The pillars that support and enhance

By experiencing the evolutionary processes of groups, students gradually assume roles of
shared responsibility. In fact, the path provides both a path of reading, analysis, and
comparison of the literature that constitutes the framework of the Reggio Emilia educational
approach, as well as a parallel path, of a methodological nature, which allows for active
participation in work and reflection groups, which are sub-units of the larger community and
its same building blocks. The primary objective of this dual register methodology (cognitive-
speculative and experiential-relational) is to facilitate an individual's affective, cognitive,
emotional, personal, and interpersonal development, which is the foundation for the
acquisition of the relational and professional skills required of PhD candidates in the RCS.
Each working group organizes itself autonomously at first, but with time, the groups begin
to collaborate and increase the intensity of their synergy and mirroring activities. A team that
works on the process level outperforms a task-oriented team. These small working groups
regard the student as an active resource for the gained knowledge. Different personalities,
sensitivities, and attitudes come into play to bring a common project to life in a situation
where the group processes underpinning task delivery are examined. The outcome of the task
groups and their methodological choices are debated and evaluated by the students present at
the end of the classroom presentation: the classroom as a whole is measured with certain
hypotheses on the type of work that brought that particular task group to that particular
exhibition result, to find links between the dynamics of the small group and the dynamics of
the class group.
4.4. The challenges at stake

With the pandemic, the PhD community and workshop model had to be reconsidered, with
the shift to a virtual setting. The loss of bodies, of the physicality of the group, was undeniably
a disadvantage that caused 'distance' and 'randomness'. On the other hand, online settings
enabled more frequent interactions, the creation of intimate spaces in which participants'
homes opened to the community, less formal and more equitable education in which
technology 'levels out' certain inherent asymmetries (Weinberg & Rolnick, 2020). The group
sessions that cannot be held in the physical classroom triggered a reflection on the distance-
proximity polarities (working online breaks down transnational barriers of lockdown),
formal-informal (the private and the public are sometimes contaminated with effort, others
with amazement), learner-teacher (the sense of the role of co-construction of the setting, of
the training methodology, of the constellation of objects of reflection that constitute the
content of the community becomes more intuitive.
The analysis of the interviews also showed other existing challenges, that can threat the
sustainability of the community of learners. The same diversity of the group, which was
previously referred to as a richness of the community, can trigger individualism and rivalries:
373
two dynamics that move the members away from the community, enabling self-centered
modalities of learning and narcissistic drives.
Another theme is the conflict with existing resources: both a legislation that awards
traditional academic research and the lack of sufficient spaces, that can welcome at any time
all the cohorts of students and faculty, are mentioned as risks to the survival of the learning
community.
4.5. The prerequisites of an ideal learning community

Every method of investigation, qualitative or quantitative, is allowed in the view that all
contributions, if justified, have equal dignity and can contribute to the creation of shared
knowledge. In such a diverse learning environment, everyone maintains a well-defined yet
complimentary function to the others to achieve the established objectives, which in this case
concern both specific scientific work and reflective metacognition on the personal style of
learning and uncovering. Sharing and listening are seen as mandatory in the vision of a shared
design of the learning process and its contents.
5. Conclusions and future implications

The goal of the Reggio Childhood Studies PhD is to train researchers through research, to
invest in the creation of personal, original, and new lines of research, attempting to bring
together quite varied domains of knowledge, skills, approaches, and concerns. It is crucial
for the scientific community to be able to communicate with people from different
perspectives and languages, which the PhD translates into a set of methodological and
operational practices. In the complete research design, we intend to analyze all learning
experiences, of all stakeholders involved, to evaluate and self-evaluate the quality (intended
as the attribution of meaning) of the social and intellectual interactions established among
peers and among generations, within contents, and through methodologies. The research can
significantly highlight how such relational and dialogical experiences can become material
to enhance metacognitive and critical skills, as well as to strengthen forms of individual and
cooperative learning, through research-action projects (Bruner, 1996).
References
Bruner, J. (1996). The culture of education. Milan: Feltrinelli.
Edwards, C., Gandini, L., & Forman G. (2010). The hundred languages of children.
Bergamo: Edizioni Junior.
Gardner, H. (1987). Formae Mentis. Essay on the plurality of intelligence. Milan: Feltrinelli.
Gergen, K. J. (1994). Realities and relationships, soundings in social construction.
Cambridge: Harvard University Press.
374
Morin, E. (2008). On complexity. Cresskill, NJ: Hampton Press.

Rinaldi, C. (2009). In dialogue with Reggio Emilia. Listening, researching, learning. Reggio
Emilia: Reggio Children.
Weinberg, H. & Rolnick A. (eds.) (2020). Theory and Practice of Online Therapy: Internet-
delivered Interventions for Individuals, Families, Groups, and Organizations. New York:
Routledge.
375
Financial literacy decision tree game: A system development

exposé
Melany Lotter1, Chioma Okoro2
1
Department of Finance and Investment Management, University of Johannesburg, South
Africa, 2Department of Finance and Investment Management, University of Johannesburg,
South Africa.
Abstract
Poor financial literacy has significant economic implications, with limited
savings, insufficient retirement provisions and higher dependency on debt.
Concerns about the level of financial literacy among the youth or young
graduates motivated this study. The study describes the design of a game to
educate young graduates while imbibing life-long personal financial
management skills. A system development approach was used to present the
information. Scenarios and decisions were developed to guide the student
(player) in planning and making decisions about their budget with real-life
events or cases. The study will be beneficial to university students to transition
smoothly into the workforce and take control of their finances. University
authorities and stakeholders can also implement the game design to assist
students in gaining financial education before venturing into the working
world.
Keywords: Education; financial literacy; game design; system development;
universities; young graduates.

Financial literacy decision tree game: A system development exposé
1. Introduction
There is a strong focus on financial literacy, specifically on younger individuals and
university students who graduate and need to be financially independent. The decisions made
in the early working years may predict the financial well-being for the future, where one
wrong financial decision can lead to another. Offering financial education at a young age is
necessary as this impacts many areas - the economy, organizations, families, communities
and the individual. With adequate financial literacy, working employees can positively attend
to, make decisions and control their personal finances in the short and long term (Pearson et
al., 2017).
Financial illiteracy during the first working years is identified through the lack of the most
basic economic and financial concepts, which can have an unfavorable impact (Lusardi &
Michigan, 2007). Therefore, students will benefit from a sound understanding of financial
concepts including inflation, compound interest, savings, debt, and retirement planning,
among others and avoid economic implications, insufficient savings and retirement
provisions, and higher dependency on debt. Subsequently, there is more reliance on financial
education’s impact on financial literacy levels. Therefore, it is equally important to introduce
financial literacy education and measure its effectiveness, as success is sometimes limited
(Frijns et al., 2014). Improving financial literacy levels is a priority for the economy at large,
and tertiary institutions have an opportunity to offer financial education to improve students’
financial literacy.
Therefore, the study’s objective was to describe the design of a game-based financial
education program that positively impacts students’ financial literacy. A detailed and
systematic approach through a system development methodology (systems thinking) to
create a solution to a problem (Nunamaker et al., 1990). Knowledge is created about the
development process itself, and deeper understanding emerges about the problem that the
system is designed to solve (Hasan, 2003). The paper focuses on the conceptual framework
and game content, to produce knowledge and illustrates how students can acquire financial
education to prepare them for the world of work. The technical development of the game by
the developers falls outside the scope of this study.
2. Conceptual framework
2.1. Game-based learning and theories

Gamification research in the field of finance has been limited although it has become
common in the sector (Bitrián et al., 2021). The game-based learning approach has grown
exponentially in recent years in different disciplines. Academics have explored the use of
gamification to enhance the transfer of knowledge and learning experiences (Nieto-Escamez
378
& Roldán-Tapia, 2021). The learning approach has grown significantly into a popular
instructional approach due to its power to motivate and engage students in complex learning,
problem solving, critical thinking and decision making (Aprea & Ifenthaler, 2021).
Integrating game design elements into non-game activities enhance user engagement and has
a great potential to motivate players and impart knowledge to them (Hoffmann & Matysiak,
2019). Incorporating gamification has gained momentum since the impact of COVID-19,
resulting in learning outcomes (Nieto-Escamez & Roldán-Tapia, 2021).
Games are designed in different formats, types and strategies. Several theories have been
identified in gamification, self-determination, goal setting and the flow theory (ibid.). Self-
determination theory includes basic needs in terms of autonomy, relatedness, and
competence, where at least one should be addressed (ibid.). Taking ownership of your
behavior can be equated with autonomy, which is associated with financial behavior and
decision making (Richter et al., 2015). The goal-setting theory includes student commitment,
feedback, game complexity, and situational constraints. In comparison, the flow theory
focuses on optimal enjoyment and engagement with the game (Nieto-Escamez & Roldán-
Tapia, 2021). Sufficient planning is required in the game-based learning to maximize
participation by students and increase student knowledge. For this study, the focus was on
improving financial literacy.
2.2. Motivation to engage the content

Students think and learn differently, especially Generation Y (millennials) and Z (the students
of today). For millennials, the largest generation, motivating and engaging to learn is very
hard since they consider the traditional teaching methods as boring compared to their
experience with digital technology (Woods et al., 2011). Gamification techniques in financial
education can effectively enhance student engagement and learning for Generation Y, as
demonstrated by Poole et al. (2014), using traditional lectures and game-like techniques.
Nicholas (2020) found that 51% of Generation Z students prefer learning by doing (working
through examples), 38% by seeing and 12% by listening. Generation Y students exposed to
the gamification method of instruction perform better in assessments and experience higher
levels of involvement, while Generation Z is more exposed to technology from a young age.
Hence, motivation and the mode of offering are vital to engaging students in financial
content.
Platz and Jüttler (2022) found that motivated students in finance have a positive link to the
game experience. The success of financial education depends on “learning from doing”. Such
programs should be implemented in a manner with the most significant impact (Massey et
al., 2016). Therefore, tertiary education has a unique opportunity to introduce financial
education that will promote and improve financial literacy by learning from doing in line
with the preferred learning style of Generation Z.
379
Further, studies have shown a positive link between gamification and improved financial
knowledge. Hoffmann and Matysiak (2019) found that gamification of financial education
increased the motivation to engage in an investment topic. Kalmi and Rahko (2022) and Sari
et al. (2022) applied augmented reality (AR) to school children who improved their financial
knowledge more than was observed in the control group (traditional program). Johnson et al.
(2021) evaluated the effectiveness of a free online financial educational program where
financial knowledge improvement was significant. Nordin et al. (2022) investigated AR
gamification technology use; 93% of the students reported that it was a better learning media.
Therefore, gamification can be considered a valuable tool for Generation Y and Z, motivating
students for improved interactivity, engagement, and learning. However, necessary planning
is required to ensure the effectiveness of the game. The right strategy and media will offer
the best potential for financial education (Sari et al., 2022).
2.3. Game design and financial literacy influencers

In developing the conceptual framework to improve financial literacy, gamification was
selected where learning from doing is the preferred learning style for students. Factors that
influence financial literacy were incorporated into the game. Studies established the impact
of life experiences on financial literacy. Liu and Lin (2021) demonstrated that including
students’ real-life experiences and choices in the teaching content promotes learning interest
while teaching financial knowledge and improving literacy. Hoffmann and Plotkina (2021)
found that intervention during an experiment where consumers recall and reflect on
experiences regarding their personal finances promotes higher financial efficacy. Therefore,
game-based learning that simulates real-life experience was developed in this study to
improve knowledge and prevent decisions that have potentially negative impacts on personal
finances.
3. Game methodology
System development was implemented to design the game prototype. The game methodology
was applied to develop game-based learning that would simulate a life experience through
the consequences of events. The basis of the game concept is the ability to process a life
experience, adapt to new conditions with budget adjustments, make impactful decisions and
readjust the budget, while successfully responding to existing game rules, which implies
modification, here and now in a limited time. A similar methodology was used by Kubina et
al. (2021), who designed a budget game model with illustrative elements in real-life
scenarios; players had to adjust and make changes according to the changing conditions in a
limited time. The advantage of gaming is experiencing the impact of a decision or reaction,
copying a life event that adds to life experience.
380
A system development methodology systematically organizes efficient financial education

in a game. The fundamental role of system development is the process of the developed
system and then as an object (the game) that becomes the focus of further and continued
research (Nunamaker et al., 1990). Therefore, system development was suitable in this study
as it efficiently presented an optimal mode to include various elements, descriptions, and
rationale. In developing the game, autonomy in self-determination theory was applied where
the students must manage their finances and budgeting decisions, as suggested by Kubina et
al. (2021).
4. Game description – Financial literacy decision tree game

The game starts with the student graduating from university and being employed for the first
time. Four real-life scenarios were selected that may severely impact a young graduate if not
prepared; thus, relevant situations are often experienced in the local and real context. These
include allocating the budget between wants and needs, black tax (financial dependency of
family members), wealth creation and theft, discretionary 13th cheque and retrenchment.
The game was designed with the following scenarios presented in Table 1 and Figure 1. You
are a young graduate who is the first family member to graduate from a university, with the
support of the entire family that relies on you for financial support in times of need. You get
your first payslip from a big corporate entity. You have accommodation, transport, study
loan, and credit card expenses. You must make provisions for needs and wants (groceries,
toiletries, clothes, airtime, entertainment, eating out and credit card debt). Further specific
information is provided regarding lunch at the office canteen, your current office outfits, data
bundle expenses, streaming channels costs, your current savings and credit card debt, lack of
insurance and medical aid and the monthly costs to make provision for it.
The students are then expected to consider all the information provided and draft a budget of
their expected spending, how often they eat out or buy takeaways, their savings behavior and
any other relevant information considering the information provided. They can enter R0 for
entries that they are not utilizing or that do not apply to them. Once the student finalizes their
budget based on the information provided, they are faced with four scenarios presented in
table 1. Each scenario represents a real-life event that could impact a young graduate’s
finances if they do not plan accordingly. The four scenarios will have financial implications
for the student. Each scenario will test their preparation for the unknown and how well they
planned and managed their personal finances. The student makes another decision based on
the specific scenario, which requires adjustments to the existing budget. The student also has
an opportunity to reassess each line entry and consider the importance of their wants, needs,
and financial well-being. Before they move to the next scenario, the student reviews and
makes changes to their budget.
381
Table 1. Game decisions and scenarios.
Scenario 1 (budget, savings, Scenario 2 (budget, wealth Scenarios 3 (surplus) & 4

emergency fund) creation, insurance) (retrenchment)
Three months later, you have a Four months later, you are At the end of the year, you
family emergency; your mom robbed and need to replace received a 13th cheque. Options:
needs to replace a burst geyser your stolen laptop. Options: 1) save the funds in a savings
during winter and relies on you 1) you have insurance and are vehicle for an emergency fund; 2)
for assistance. Options: 1) you covered entirely; 2) you did pay credit card; 3) pay student
have allocated savings from your not take out insurance and loan; 4) spend it on a holiday.
budget (savings per month x 3), have no funds. You can buy a Readjust your budget. Two
which can be used. If not enough, second-hand laptop or take months later, you are retrenched.
use the credit card (with out a personal loan with a You find another employment
increased payments) and reduce higher interest rate. Consider two months later, resulting in one
another expense; 2) you did not wealth creation, debt, interest month without any income.
save and need to use your credit rates and insurance to Consider your current savings,
card. readjust your budget. fixed expenses and necessities.
Content for the specific scenario is then presented; this gives the student additional
information to assess their decisions and how they could better manage their finances.
Subsequently, the student has control over their personal finances and has to take
responsibility for decisions. When a situation occurs, they can take ownership of it and
consider available options (Kubina et al. 2021). That way, they gain financial experience to
help them make decisions in real life. These views were expressed by Frijns et al. (2014),
who found that financial experience positively impacts financial literacy and promotes
improved financial knowledge. The game dynamics also help students to recall and learn by
doing, as supported in previous studies (Massey et al., 2016; Hoffmann and Plotkina, 2021).
bonus - 13th
laptop stolen
cheque
dependency
retrench
family
take credit
for new
holiday
take out
credit
No
No
Allocate budget:
Adjust budget
wants & needs
Life event
credit card
Life event
insurance?
insurance?
Readjust
insurance?
Life event
savings?
Budget
Readjust
Life event
savings?
Readjust
budget
savings?
budget
budget
Yes
savings
Figure 1. Scenario and decision tree.
382
5. Conclusion
Life experience positively impacts financial literacy, and where a gamified experience can
be simulated, the individual may gain knowledge to make better decisions. Subsequently, a
game was developed to allow students to improve their financial literacy before they start
working. Game-based learning is generally effective, and it can promote financial literacy. If
presented to university students, it will enhance the importance of the content as they
transition from students to employees. However, this study did not explore the impact of the
gaming experience on students’ financial literacy. It only includes the description of the
design of the game prototype to be used for further investigation. Future research will include
testing the effectiveness and measuring if it adds to the students’ experience of gaining
financial knowledge.
References
Aprea, C., & Ifenthaler, D. (2021). Game-based Learning Across the Disciplines. Springer.
http://www.springer.com/series/13094
Bitrián, P., Buil, I., & Catalán, S. (2021). Making finance fun: The gamification of personal
financial management apps. International Journal of Bank Marketing, 39(7), 1310–1332.
https://doi.org/10.1108/IJBM-02-2021-0074
Frijns, B., Gilbert, A., & Tourani-Rad, A. (2014). Learning by doing: The role of financial
experience in financial literacy. Journal of Public Policy, 34(1), 123–154.
https://doi.org/10.1017/S0143814X13000275
Hasan, H. (2003). Information Systems Development as a Research Method. Australasian
Journal of Information Systems 11(1), 4-11. https://doi.org/10.3127/ajis.v11i1.142
Hoffmann, A. O., & Plotkina, D. (2021). Positive framing when assessing the personal
resources to manage one’s finances increases consumers’ retirement self‐efficacy and
improves retirement goal clarity. Psychology & Marketing, 38(12), 2286-2304.
Hoffmann, G. & Matysiak, L. (2019). Exploring Game Design for the Financial Education
of Millenials. 11th International Conference on Virtual Worlds and Games for Serious
Applications (VS-Games), Vienna, Austria, pp. 1-2, doi: 10.1109/VS-
Games.2019.8864517
Johnson, J., Spraggon, D., Stevenson, G., Levine, E., & Mancari, G. (2021). Impact of the
FutureSmart Online Financial Education Course on Financial Knowledge of Middle
School Students. Journal of Financial Counseling and Planning, 32(3), 368–386.
https://doi.org/10.1891/JFCP-19-00061
Kalmi, P., & Rahko, J. (2022). The effects of game-based financial education: New survey
evidence from lower-secondary school students in Finland. Journal of Economic
Education, 53(2), 109–125. https://doi.org/10.1080/00220485.2022.2038320
Kubina, E., Bareicheva, M., & Stepanova, N. (2021). Universal sprint game that teaches the
basics of financial literacy. Proceedings of the European Conference on Games-Based
Learning, 2021-September, 437–444. https://doi.org/10.34190/GBL.21.090
383
Liu, H. C., & Lin, J. S. (2021). Impact of Internet Integrated Financial Education on Students’
Financial Awareness and Financial Behavior. Frontiers in Psychology, 12.
https://doi.org/10.3389/fpsyg.2021.751709
Lusardi, A., & Michigan, O. M. (2007). Financial Literacy and Retirement Planning: New
Evidence from the Rand American Life Panel. http://ssrn.com/abstract=1095869
Massey, E., Wyatt, A., & Smit, C. (2016). Evaluating financial education initiatives in South
Africa: The importance of multiple evaluation approaches. African Evaluation Journal,
4(1). https://doi.org/10.4102/aej.v4i1.125
Nicholas, A. J. (2020). Preferred Learning Methods of Generation Z. Faculty and Staff –
Articles & Papers. 74. https://digitalcommons.salve.edu/fac_staff_pub/74
Nieto-Escamez, F. A., & Roldán-Tapia, M. D. (2021). Gamification as Online Teaching
Strategy During COVID-19: A Mini-Review. Frontiers in Psychology, 12.
https://doi.org/10.3389/fpsyg.2021.648552
Nordin, N., Nordin, N. R. M., & Omar, W. (2022). REV-OPOLY: A Study on Educational
Board Game with Webbased Augmented Reality. Asian Journal of University Education,
18(1), 81–90. https://doi.org/10.24191/ajue.v18i1.17172
Nunamaker, J. F., Chen, M., & Purdin, T. D. M. (1990). Systems development in information
systems research. Journal of Management Information Systems, 7(3), 89–106.
https://doi.org/10.1080/07421222.1990.11517898
Pearson, G., Stoop, P. N., & Kelly-Louw, M. (2017). Balancing responsibilities–financial
literacy. Potchefstroom Electronic Law Journal, 20(1). https://doi.org/10.17159/1727
Philippas, N. D., & Avdoulas, C. (2020). Financial literacy and financial well-being among
generation-Z university students: Evidence from Greece. European Journal of Finance,
26(4–5), 360–381. https://doi.org/10.1080/1351847X.2019.1701512
Platz, L., & Jüttler, M. (2022). Game-based learning as a gateway for promoting financial
literacy. Citizenship, Social and Economics Education, 21(3), 185–208.
https://doi.org/10.1177/14788047221135343
Poole, S. M., Kemp, E., Patterson, L., & Williams, K. (2014). Get Your Head in the Game:
Using Gamification in Business Education to Connect with Generation Y. Journal for
Excellence in Business Education, 3(2). http://repository.usfca.edu/ml
Richter, G., Raban, D. R., & Rafaeli, S. (2015). Studying gamification: The effect of rewards
and incentives on motivation. Gamification in Education and Business, 21–46.
https://doi.org/10.1007/978-3-319-10208-5_2
Sari, C. R., Fatimah, R. P., Ilyana, S., & Dwi Hermawan, H. (2022). Augmented reality (AR)-
based sharia financial literacy system (AR-SFLS): A new approach to virtual sharia
financial socialization for young learners. International Journal of Islamic and Middle
Eastern Finance and Management, 15(1), 48–65. https://doi.org/10.1108/IMEFM-11-
2019-0484
384
Fairness matters in higher education: student classroom justice

perceptions and behavioral responses
Rebecca M. Chory
Department of Management, Frostburg State University, United States of America.
Abstract
Fairness in the higher education instructional context, i.e., classroom justice,
is fast becoming a salient and widespread concern among scholars,
instructors, and students alike. Drawing on research conducted in North
America, Europe, Asia, and other world regions, the present article describes
university students’ beliefs about what constitutes unfair instructor behavior,
and it explores the relationships between classroom justice and student
behavioral responses. Results of this work indicate that university students
identify grading procedures, instructor feedback, and instructor affect,
attention, and caring as key factors in determining classroom justice. In
addition, university students’ perceptions of (un)fairness are associated with
student learning, engagement, and antisocial classroom behavior. In short,
classroom justice in higher education is integral to student learning and
instruction. Classroom justice matters.
Keywords: Classroom justice; fairness; student-instructor relationships.

Fairness matters in higher education: student classroom justice perceptions and behavioral responses
1. Introduction
Whether driven by the neoliberalization of higher education, rapid changes in information and
communication technology, or the prevailing student-as-consumer orientation, fairness has
become an increasingly salient topic for instructors, administrators, and students (Chory &
Horan, 2018, 2022; Chory, Horan, & Houser, 2017; Chory & Offstein, 2017; Chory, Zhaleh,
& Estaji, 2022). The study of fairness in the instructional context is known as classroom justice
(Chory-Assad, 2002). In the present manuscript, the instructional practices and behaviors
students perceive to be (un)fair and student responses to (un)fairness are examined, with
particular focus on higher education. Findings from the United States of America (USA), where
the study of classroom justice began, are discussed, as is burgeoning classroom justice research
conducted in Europe, Asia, the Middle East, and other regions across the world.
2. What is Classroom Justice?

Classroom justice theory and research were originally grounded in industrial-organizational
psychology, organizational behavior, and management theory and research in the USA (see
Chory-Assad, 2002; Chory-Assad & Paulsel, 2004). As a theoretical construct, classroom
justice consists of three dimensions enacted or evaluated according to multiple principles,
which are applied in various domains (Chory et al., 2022; Rasooli, Zandi, & DeLuca, 2019;
Estaji & Zhaleh, 2021a, 2021b).
Early classroom justice research focused primarily on the three dimensions. Distributive justice
refers to perceptions of fairness concerning the outcomes (e.g., grades, instructor attention)
allocated in the instructional context, whereas procedural justice refers to the fairness of the
processes (e.g., grading procedures) used to distribute said outcomes (Chory-Assad, 2002;
Chory-Assad & Paulsel, 2004). Interactional justice is defined as perceptions of the fairness
of the instructor’s interpersonal treatment of students when processes are executed or
outcomes allocated (Chory, 2007).
A recent theoretical development in the realm of classroom justice centers on justice principles,
i.e., the standards or rules used to judge fairness, and domains, i.e., the course elements or
contexts in which the principles are applied (Rasooli et al., 2019). Classroom justice may be
implemented or violated according to 17 justice principles (Chory et al., 2022; Estaji & Zhaleh,
2021a, 2021b; Rasooli et al., 2019). The domains include assessment, teaching, learning, and
interaction, each of which has subdomains (Chory et al., 2017, 2022; Estaji & Zhaleh, 2021a,
2021b, 2022; Horan, Chory, & Goodboy, 2010; Rasooli et al., 2019).
Research on justice principles and domains revealed that university students reported their
instructors violated the distributive justice principles of equity (in grading) and equality (in
communicating affect) in equal measure (Chory et al., 2022), consistent with prior research
386
Rebecca M. Chory
(Chory et al., 2017; Chory-Assad & Paulsel, 2004; Horan et al., 2010; Rasooli, Zandi, &
DeLuca, 2018). University students most frequently reported instructor procedural justice
violations of the bias suppression principle in the relational communication and student
performance domains, followed by violations of accuracy in performance and reasonableness
in course workload. Finally, students perceived instructors violated the interactional justice
principles of justification in student performance domains and caring and propriety in relational
domains (Chory et al., 2022).
3. What Classroom Justice Violations (Unfair Behaviors) Do Students Identify?

College students identify and label a variety of instructor behaviors, events, and practices as
unfair. Quantitative research reveals that American university students perceive instructors who
are incompetent, unethical, uncaring (Chory, 2007; Chory & Offstein, 2017) and verbally
aggressive (Claus, Chory, & Malachowski, 2012) as unfair. They also find instructors who use
coercive power (Paulsel, Chory-Assad, & Dunleavy, 2005) and do not effectively answer
questions or provide enjoyable interactions (Young, Horan, & Frisby, 2013) to be unfair.
Research across the world confirms the aforementioned findings and provides additional
insight into the types of instructional behaviors and occurrences that university students believe
to be unfair. See Table 1.
As shown in Table 1, university students identify unfair behaviors representing all three justice
dimensions and occurring in various domains. Across samples, the most commonly reported
justice violations pertain to the distribution of grades and instructor affect, grading procedures,
feedback, and the extent to which instructors care about students. Singling students out for
criticism, attendance/make-up policies, and not following through on one’s word are classroom
justice violations that are less commonly identified across groups. Regardless of the particular
instance or form of unfairness, however, the perception of classroom justice has been shown to
lead to observable and meaningful consequences.
387
Table 1. University students’ reports of classroom injustice (i.e., unfair instructor behaviors).
Distributive Justice Violations (the instructor’s distribution of…)
• grades 1, 3, 5
• opportunities 1, 3
• affect or attention 1, 3, 5, 6
• punishment 1, 3
Procedural Justice Violations (the instructor or instructor’s…)
• class procedures 1, 3
• grading procedures 1, 2, 3, 5, 7
• course make-up/late/attendance policies 1, 3
• course schedule/workload 1, 3, 5
• not following through 1, 3
• not providing students desired information for exams 1, 3
• making an error that affects students 1, 3, 5
• not providing students adequate or timely feedback 1, 3, 4, 5, 6, 7
• not being available or accessible to students 1, 3, 5
Interactional Justice Violations (the instructor…)
• was rude/insensitive 1, 2, 3, 4, 7
• implied/stated student was stupid 1, 3
• made sexist/racist/prejudiced comments 1, 3
• singled student out for criticism 1, 3
• did not care about students/nonresponsive 1, 3, 4, 5, 8
• accused student of wrongdoing 1, 2, 3, 7
• did not explain decisions (i.e., informational injustice) 1, 3, 6
Sources: 1Horan et al. (2010), 2Čiuladienė & Račelytė (2016), 3Chory et al. (2017), 4Rasooli et al. (2019), 5Estaji &
Zhaleh (2022), 6Chory et al. (2022), 7Bazvand & Rasooli (2022)
4. Why Does Classroom Justice Matter?

Classroom justice is important because it impacts student learning, classroom climate, and
student and instructor well-being. On the positive side, student perceptions of classroom justice
tend to enhance learning and student-instructor relational outcomes. College students’
perceptions of distributive, procedural, and interactional justice were positively related to
motivation, affective learning, and cognitive learning (Chory-Assad, 2002; Holmgren &
Bolkan, 2014; Horan, Martin, & Weber, 2012; Vallade, Martin, & Weber, 2014). Fairness
perceptions were also associated with students having higher quality leader-member exchange
relationships with instructors (Horan, Chory, Carton, Miller, & Raposo, 2013).
388
Rebecca M. Chory
In contrast to justice’s positive effects, university students’ perceptions of injustice are

associated with outcomes that threaten the learning environment. For instance, perceived
injustice predicted a stronger likelihood that students would engage in indirect interpersonal
aggression and hostility toward their instructors and seek revenge on, deceive, and resist
instructors (Chory-Assad, 2002; Chory-Assad & Paulsel, 2004; Horan et al., 2013; Paulsel &
Chory-Assad, 2005). Perceived instructor injustice was also associated with a stronger
likelihood of college students communicating antisocial compliance-gaining messages to
instructors (Claus, Chory, & Malachowski, 2012), coming to class under the influence of drugs
and alcohol, and communicating aggressively and offensively in the instructor’s course.
Distributive injustice and instructor unethical out-of-class behavior also interacted to predict
university student incivility and offensive communication (Chory & Offstein, 2017).
Departing from quantitative-based research methodology, Horan et al. (2010) and Chory et al.
(2017) used open-ended items to investigate college students’ behavioral responses to
perceived injustice. They observed that the most common behavioral response was dissent,
which included complaining about the unfair incident to the instructor, another school authority
figure, on teacher evaluations, and to family, friends, and other students. The second most
common university student behavioral responses were inaction or acceptance of the unfairness
(Horan et al., 2010) and changing one’s approach to the course (Chory et al., 2017). These
responses were followed by student verbal aggression, hostility, confrontation, and
disengagement.
5. Do Classroom Justice Perceptions and Responses Vary by Country?

Since the introduction of classroom justice theory and research in the USA, scholars across the
world have contributed to the body of knowledge on fairness in the classroom. Classroom
justice has been investigated in Australia (Lizzio, Wilson, & Hadaway, 2007), Italy (DiBattista,
Pivetti, & Berti, 2014), Poland (Lankiewicz, 2014), Serbia (Kovačević, Zunić, & Mihailović,
2013), Lithuania (Čiuladienė & Račelytė, 2016), Russia (Bempechat et al., 2013), Turkey
(Argon & Kepekcioglu, 2016), Cyprus (Uludag, 2014), Iran (Estaji & Zhaleh, 2021a, 2021b,
2022), China (Tata, 2005; Yan, 2021), Nigeria (Kura, Shamsudina, & Chauhan, 2014), and
other countries. In some cases, results are consistent with work in the USA. For example,
perceptions of fairness were related to Italian (DiBattista et al., 2014) and Australian (Lizzio et
al., 2007) university students’ engagement, Serbian college students’ achievement (Kovačević
et al., 2013), Turkish college students’ perceptions of instructor trustworthiness (Argon &
Kepekcioglu, 2016), and Chinese university students’ perceptions of instructor credibility and
immediacy (Yan, 2021). In contrast with US-based work, Nigerian undergraduates’ perceptions
of procedural justice were not related to deviant behavior or honor codes (Kura et al., 2014).
389
6. Conclusion
Fairness in the higher education classroom is associated with a number of significant
educational experiences and outcomes. University instructors enacting fair procedures tends to
enrich college student learning and student-instructor relationships, whereas instructors
violating justice expectations tends to harm student achievement and relationships. These are
issues that deserve continued attention from those seeking to enhance student learning, as well
as student and instructor well-being, in higher education.
In addition, there is a need for classroom justice research in more diverse settings. Future
research should examine and compare justice perceptions and responses across countries with
different cultural values (e.g., individualistic vs. collectivistic), sociopolitical structures (e.g.,
democratic, communist, post-socialist) and histories (e.g., colonization). As more and more
university faculty and students engage in teaching and learning across borders, this work will
grow in significance and utility. Classroom justice matters in higher education and it will
continue to do so.
References
Argon, T., & Kepekcioglu, E. S. (2016). The relationship between university students’
instructors’ credibility and perceptions of justice in the classroom. Anthropologist, 24(1),
347-353. doi:10.1080/09720073.2016.11892024.
Bazvand, A. D., & Rasooli, A. (2022). Students’ experiences of fairness in summative
assessment: A study in a higher education context. Studies in Educational Evaluation, 72,
101118. doi: 10.1016/j.stueduc.2021.101118
Bempechat, J., Ronfard, S., Mirny, A., Li, J., & Holloway, S. D. (2013). “She always gives
grades lower than one deserves:” A qualitative study of Russian adolescents’ perceptions
of fairness in the classroom. Journal of Ethnographic & Qualitative Research, 7(4), 169-
187.
Chory, R. M. (2007). Enhancing student perceptions of fairness: The relationship between
instructor credibility and classroom justice. Communication Education, 56(1), 89-105.
doi:10.1080/03634520600994300
Chory, R. M., & Horan, S. M. (2018). [Re]negotiating power and influence in the classroom.
In M. L. Houser & A. M. Hosek (Eds.), Handbook of instructional communication:
Rhetorical and relational perspectives (pp. 112-125), 2nd Ed. New York, NY: Taylor and
Francis.
Chory, R. M., & Horan, S. M. (2022). Looking back and looking ahead at negotiating power in
the classroom. In M. L. Houser & A. M. Hosek (Eds.), Handbook of instructional
communication: Rhetorical and relational perspectives (pp. 151-170), 3rd Ed. Dubuque, IA:
Kendall Hunt.
Chory, R. M., Horan, S. M., & Houser, M. L. (2017). Justice in the higher education classroom:
Students’ perceptions of unfairness and responses to instructors. Innovative Higher
Education, 42(4), 321-336. doi:10.1007/s10755-017-9388-9
390
Rebecca M. Chory
Chory, R. M., & Offstein, E. H. (2017). Outside the classroom walls: Perceptions of professor
inappropriate out-of-class conduct and student classroom incivility among American
business students. Journal of Academic Ethics, 15(3), 197-214. doi: 10.1007/s10805-017-
9290-1
Chory-Assad, R. M. (2002). Classroom justice: Perceptions of fairness as a predictor of student
motivation, learning, and aggression. Communication Quarterly, 50(1), 58-77.
doi:10.1080/01463370209385646
Chory-Assad, R. M., & Paulsel, M. L. (2004). Classroom justice: Student aggression and
resistance as reactions to perceived unfairness. Communication Education, 53(3), 255-275.
doi:10.1080/0363452042000265189
Čiuladienė, G., & Račelytė, D. (2016). Perceived unfairness in teacher-student conflict
situations: Students’ point of view. Polish Journal of Applied Psychology, 14(1), 49-66.
doi:10.1515/pjap-2015-0049
Claus, C. J., Chory, R. M., & Malachowski, C. C. (2012). Student antisocial compliance-
gaining as a function of instructor aggressive communication and classroom justice.
Communication Education, 61(1), 17-43. doi:10.1080/03634523.2011.619270
DiBattista, S., Pivetti, M., & Berti, C. (2014). Engagement in the university context: Exploring
the role of a sense of justice and social identification. Social Psychology of Education,
17(3), 471-490. doi:10.1007/s11218-014-9255-9
Estaji, M., & Zhaleh, K. (2021a). Exploring Iranian teachers’ perceptions of classroom justice
and its dimensions in EFL instructional contexts. Language Related Research, 12(3), 277-
314. doi: 10.29252/LRR.12.3
Estaji, M., & Zhaleh, K. (2021b). Teachers’ perceptions, experiences, and challenges of
incorporating justice in English as a foreign language classrooms. Polish Psychological
Bulletin, 52(3), 251-269. doi: 10.24425/ppb.2021.137889
Estaji, M., & Zhaleh, K. (2022). Experiences, causes, and solutions for injustice in Covid-19
online classes: Evidence from university EFL students in Iran. LEARN Journal: Language
Education and Acquisition Research Network, 15(2), 436-467.
Holmgren, J. L., & Bolkan, S. (2014). Instructor responses to rhetorical dissent: Student
perceptions of justice and classroom outcomes. Communication Education, 63(1), 17-40.
doi:10.1080/03634523.2013.833644
Horan, S. M., Chory, R. M., Carton, S. T., Miller, E., & Raposo, P. C. J. (2013). Testing leader-
member exchange theory as a lens to understand students’ classroom justice perceptions
and antisocial communication. Communication Quarterly, 61(5), 497-518.
doi:10.1080/01463373.2013.799511
Horan, S. M., Chory, R. M., & Goodboy, A. K. (2010). Understanding students' classroom
justice experiences and responses. Communication Education, 59(4), 453-474.
doi:10.1080/03634523.2010.487282
Horan, S. M., Martin, M. M., & Weber, K. (2012). Understanding emotional response theory:
The role of instructor power and justice messages. Communication Quarterly, 60(2), 210-
233. doi:10.1080/01463373.2012.669323
391
Kepekcioglu, E. S. (2015). The relationship between university students’ instructors’

credibility perceptions and justice in the classroom perceptions. Unpublished doctoral
dissertation. Bolu, Turkey: Abant Izzet Baysal University.
Kovačević1, I., Zunić, P., & Mihailović, D. (2013). Concept of organizational justice in the
context of academic achievement. Management Journal for Theory and Practice
Management, 69, 37-46. doi: 10.7595/management.fon.2013.0024
Kura, K. M., Shamsudin, F. M., Chauhan, A. (2014). Effects of honor codes and classroom
justice on students’ deviant behavior. Procedia - Social and Behavioral Sciences, 112, 77-
86. doi: 10.1016/j.sbspro.2014.01.1141
Lankiewicz, H. (2014). Teacher interpersonal communication abilities in the classroom with
regard to perceived classroom justice and teacher credibility. In M. Pawlak, J. Bielak, & A.
Mystkowska-Wiertelak (Eds.), Classroom-oriented research (pp. 101–120). Heidelberg:
Springer International Publishing.
Lizzio, A., Wilson, K., & Hadaway, V. (2007). University students’ perceptions of a fair
learning environment: A social justice perspective. Assessment & Evaluation in Higher
Education, 32(2), 195-213. doi: 10.1080/02602930600801969
Paulsel, M. L., & Chory-Assad, R. M. (2005). Perceptions of instructor interactional justice as
a predictor of student resistance. Communication Research Reports, 22(4), 283-291.
doi:10.1080/000368105000317565
Paulsel, M. L., Chory-Assad, R. M., & Dunleavy, K. N. (2005). The relationship between
student perceptions of instructor power and classroom justice. Communication Research
Reports, 22(3), 207-215. doi: 10.1080/00036810500207030
Rasooli, A., Zandi, H., & DeLuca, C. (2018). Re-conceptualizing classroom assessment
fairness: A systematic meta-ethnography of assessment literature and beyond. Studies in
Educational Evaluation, 56, 164-181. doi: 10.1016/j.stueduc.2017.12.008
Rasooli, A., Zandi, H., & DeLuca, C. (2019). Conceptualizing fairness in classroom
assessment: Exploring the value of organizational justice theory. Assessment in Education:
Principles, Policy & Practice, 26(5), 584-611. doi: 10.1080/0969594X.2019.1593105
Tata, J. (2005). The Influence of national culture on the perceived fairness of grading
procedures: A comparison of the United States and China. The Journal of Psychology,
139(5), 401-412. doi:10.3200/JRLP.139.5.401-412.
Uludag, O. (2014). Fair and square: How does perceptions of fairness is associated to
aggression? Procedia-Social and Behavioral Sciences, 143, 504-508.
doi:10.1016/j.sbspro.2014.07.424
Vallade, J. I., Martin, M. M., & Weber, K. (2014). Academic entitlement, grade orientation, and
classroom justice as predictors of instructional beliefs and learning outcomes.
Communication Quarterly, 62(5), 497-517. doi:10.1080/01463373.2014949386
Yan, P. (2021). Chinese EFL students' perceptions of classroom justice: The impact of teachers'
caring and immediacy. Frontiers in Psychology, 12, 767008. doi:
10.3389/fpsyg.2021.767008
Young, L. E., Horan, S. M., & Frisby, B. N. (2013). Fair and square? An examination of
classroom justice and relational teaching messages. Communication Education, 62(4), 333-
351. doi: 10.1080/03634523.2013.800216
392
University graduates enrolled in higher VET in Spain: an

upskilling or reskilling choice?
Iván Diego Rodríguez1, Juan P. Gamboa2, 3
1
Valnalon, Spain, 2 Orkestra-Deusto Foundation, Spain 3 Deusto Business School, University
of Deusto, Spain.
Abstract
Post-Baccalaureate reverse transfer students (PRTSs) are students that enroll
at a VET institution after completion of at least a bachelor’s degree. This paper
is an exploratory, descriptive and correlational study of Higher VET graduates
with a previous university qualification in Spain. The purpose of this study is
to contribute to the knowledge of PRTSs in Spain by identifying if it is an
upskilling or reskilling choice. The study uses data from the ETEFIL-19 survey
to examine the demographics, previous degrees, program of enrollment, and
prevalence of university graduates who get VET degrees. The results suggest
that the transition of university graduates in Spain into Higher VET is best
interpreted as an upskilling strategy due to a significant correlation between
the original university field and the VET professional field of destination. A
summary discussion of the results along with implications and
recommendations for the practice, policy and further research are included.
Keywords: Higher VET; postbaccalaureate reverse transfer; university to VET
transitions; upskilling; reskilling; university dropout.

University graduates enrolled in higher VET in Spain: an upskilling or reskilling choice?
1. Introduction
Vocational Education & Training (henceforth, VET) attracts very varied profiles of people
in adulthood who decide to resume their studies at some point in their life (Cournoyer et al.,
2017). In Europe, this trend has been accentuated in recent years and more and more VET
providers are admitting adults to initial vocational training courses (Markowitsch & Hefler,
2019). In Spain, Higher VET programmes are seen to represent an incipient way of re-entry
into the educational system for university graduates who seek to specialize in certain areas
or complement their training (García-Brosa, 2019) but to date, this phenomenon remained
largely unexplored.
This research has a two-fold objective. Firstly, it aims to quantify the number of Higher VET
graduates in Spain with a previous university degree. Secondly, it seeks to establish whether
this educational transition obeys a strategy of specialization (upskilling) or retraining
(reskilling) by analysing the relationship between previous university degree and a Higher
VET programme followed. To achieve our research goals, the paper taps on the Educational-
Training Transition and Labour Insertion Survey (hereinafter, ETEFIL-19) carried out by
the Spanish National Institute of Statistics in 2019 (INE, 2020).
The following section offers a review of the academic literature on this phenomenon. Later,
the method used to carry out the study will be explained, as well as the data that make up the
sample. In the third section the main results obtained will be presented. Conclusions and
limitations of this research will be derived in the final section.
2. The transitions of university graduates towards Vocational Education &

Training
The presence of university graduates in VET is not a new or exclusive phenomenon in Spain.
The research work in this field has been developed mainly in countries such as Australia and
the United States with lines of work aimed mainly at determining the volume of this flow,
the context in which it occurs, the reasons behind this decision and the experience of people
with a university degree in the vocational training system. The “counter-intuitive” nature of
this movement may be one of the reasons for the scant attention that this phenomenon has
received as an object of study in Europe.
In the United States, Townsend & Dever (1999) coined the concept "Post-Baccalaureate
Reverse Transfer" to characterize the movements of people with university degrees towards
Community Colleges (2-year vocational training). Two years earlier, a study for the National
Association of Community Colleges estimated that between 10% and 20% of the students
who accessed this type of training had a previous university degree (Gose, 1997 cited by
Leigh, 2009). Since then there have been several authors (e.g. Friedel & Friesleben, 2017;
394
Leigh, 2009) who have continued this line of research with studies limited to the scope of
certain educational institutions or states, for which reason there is no a clear vision at the
national level.
In Australia, the movement of graduates from university to VET is known as 'reverse
articulation' (Haas, 1999). In a study carried out in the state of Victoria between 1991-1997,
Golding (1999) estimated that around 40,000 students a year entered VET with a previous
university degree and that a similar number of VET students would have previously gone
through university without having completed these studies. These figures were questioned
years later by Moodie (2004) who specified that taking into account the quality of the data
available at that time, the only thing that could be stated was that the movements from the
university to vocational training were between 50% less and 50% higher than movements in
the opposite direction depending on the definition and the transfer measurement strategy
used.
3. Method
3.1. Data source

The key source for the aims of the present study is a survey about the Educational-Training
Transition and to the Labour Market (ETEFIL) (INE, 2020). The National Institute of
Statistics makes the microdata available to third parties completely free of charge. The latest
edition of the survey (ETEFIL-19) surveyed a representative sample of 7,802 people who
had completed a Higher VET Programme in 2013-2014 all around Spain from the whole set
of professional fields (26 fields). The general aim of the survey is to collect data on their
academic and employment situation five years after graduation (in 2019).
3.2. Sample
The study focuses on a subsample of 767 Higher VET graduates who got their degree in 2014
holding a previous university degree or diploma.
The number of people with a previous university degree who had obtained a Higher VET
degree in 2013-2014 amounts to 767, which represents 9.8% of the sample for the entire
country (n=7802). In addition to a university degree, 17.1% had at least a Master's degree.
The average age of this subset of the sample at the time of obtaining the Higher Technician
degree in 2014 was 32 years and 50% were 30 years or older (min=23, max=45, SD=6.45).
Both data indicate that the decision to enroll in a Higher VET programme does not occur
immediately after completing their university studies. 67.5% are women, although an
increase in the percentage of men is observed as the age group increases.
395
Almost half of Higher VET graduates with a previous university qualification come from
the field of “Social and Legal Sciences” (49.9%). They are followed in order of importance
by the field of "Engineering and Architecture" (16.8%) and "Sciences" (13.3%) which,
considered as a whole, reveal that 30% of this group are STEM graduates. Health Sciences
and Arts and Humanities provide a similar number of graduates to the sample, standing in
both cases slightly above 9%.
Table 1. Higher VET graduates with a previous university degree, according to Academic
Discipline of their university studies.
Academic discipline Frequency Percentage

Social Sciences 383 49,9%
Engineering and architecture 129 16,8%
Sciences 102 13,3%
Health Sciences 72 9,4%
Humanities and Arts 70 9,1%
Indeterminable 11 1,4%
TOTAL 767 100,0%
Source: Own elaboration based on data from ETEFIL-19.
Individuals have been classified according to two dichotomous qualitative variables: STEM
professional family (Yes/No) and STEM University academic disciplines (Yes/No). The two
branches of studies assigned to the STEM category are Engineering and Architecture and
Science. In the case of Higher VET, 9 professional fields have been included following the
classification proposed by the National Institute for Educational Evaluation (INEE, 2017).
Table 2. STEM-related Academic Disciplines and Higher VET professional fields.
STEM University Academic Disciplines STEM VET professional fields

1. Science Degrees 1. Building and Civil Works
2. Engineering and architecture Degrees 2. Electricity and electronics
3. Energy and Water
4. Mechanical manufacturing
5. Food industries
6. Information Technologies and
Communications
7. Installation and maintenance
8. Chemical industry
9. Transportation and vehicle
maintenance
Source: Own elaboration based on INEE (2017).
4. Results
4.1. Upskilling or reskilling

Interpreting the university graduates transition to VET as a complement to training or
specialization (upskilling) or as a change of course in their professional career (reskilling)
396
requires determining to what extent the previous university education is related to the higher
technical degree obtained in 2014.
Table 3. Contingency table of STEM VET qualification of destination vs STEM University
Degree of origin.
Previous University Degree

(origin)
VET Qualification
STEM NO STEM Total
(destination)
Observed 134.00 83.00 217.00
Expected 72.10 144.90 217.00
STEM
% line 61.75 % 38.25 % 100.00 %
% column 65.37 % 20.15 % 35.17 %
Observed 71.00 329.00 400.00
Expected 132.90 267.10 400.00
NO STEM
% line 17.75 % 82.25 % 100.00 %
% column 34.63 % 79.85 % 64.83 %
Observed 205.00 412.00 617.00
Expected 205.00 412.00 617.00
Total
% line 33.23 % 66.77 % 100.00 %
% column 100.00 % 100.00 % 100.00 %
The analysis shows 65.4% of university graduates from the branches of Science and
Engineering and Architecture tend to opt for STEM-related Higher VET programs. This
pattern is even more pronounced in the case of HASS graduates, where 79.8% of the
graduates are concentrated in non-STEM Higher VET programs. The re-skilling strategy or
career change is more frequent in STEM graduates (34.6%) than in HASS graduates (20.2%).
The χ2 statistic (χ2 (1) = 122.7, p <.001) shown in Table 4 indicates that there is a significant
relationship between the original university field and the VET field of destination.. The
magnitude of the relationship between the variables shows a moderate to large effect size
(Phi 0.45).
Table 4. Chi-square test results for the correlation between the original university academic
discipline and the VET professional field of destination
Value Degrees of freedom p

X² 122.77 1 <.001 **
N 617
Phi 0.45
397
5. Conclusions
The presence of university graduates in Higher VET programs reflects the growing
individualization, fragmentation and complexity of academic-professional trajectories
(Montes Ruiz, 2019). The study provides quantitative evidence on the ambiguous, prolonged
and indirect nature of educational itineraries and routes to employment (e.g. Golding, 1999;
Moodie, 2004; Taylor & Jain, 2017; Townsend & Dever, 1999). The increasingly blurred
borders between Vocational Education and Training and the University System in Spain point
to the need to leave behind the excessively rigid conceptualisation of VET-University
pathways. This study demonstrates that the flow in the opposite direction, from University to
VET, is also taking place.
Up to now, Post-Baccalaureate Reverse Transfer has been occurring spontaneously. The
magnitude and nature of this flow points to the need to adopt measures that improve and
reinforce the permeability, the exchange of information and credit recognition schemes
between Higher Education Institutions and Higher VET providers.
The results contradict the common assumption that associate the presence of University
Graduates in VET programmes with the need for a career change. We demonstrate the
transition of university graduates into Higher VET is best interpreted as an upskilling
decision both for STEM and HASS graduates and that the re-skilling strategy is more
frequent in STEM graduates than in HASS graduates.
6. Limitations and future research

Although the ETEFIL-19 microdata allows a higher level of disaggregation by VET
professional field, it would still be difficult to estimate the relationship between the two
qualifications (university and VET). In the same way, the complementarity of studies that at
first are not related to each other may end up becoming evident in the medium or long term
if the combination of both qualifications are revealed as a plus in a recruitment process.
Further qualitative studies (e.g. interviews, discussion groups) are needed for the in-depth
exploration of the motivations and reasons informing this decision.
Another line of future research is to determine the medium-term impact of this decision on
employment-related indicators such as the employment rate, the relationship between VET
program completed and the occupation, wages and other employment quality indicators.
University graduates make up only a part of the total volume of people who transit from
University to VET. As a reference, Moodie (2004) estimated that in Australia almost two
thirds of the protagonists of University to VET transitions correspond to students who had
abandoned their university studies. In Spain, a recently released study commissioned by the
Spanish Ministry of Universities found 13.5% of undergraduate students leave University
398
before completing their degree (Fernández Mellizo, 2022). Unfortunately, the ETEFIL-19
data does not include any variables identifying University drop-outs in Higher VET
programmes so we consider that it would be interesting to further explore the destinations of
this group in order to have the most complete picture of the University to VET transitions in
Spain.
References
Cournoyer, L., Fournier, G., & Masdonati, J. (2017). Going Back-to-School in Vocational
Education and Training: Introduction. International Journal for Research in Vocational
Education and Training, 4(3), 196-205. https://doi.org/10.13152/IJRVET.4.3.1
Fernández Mellizo, M. (2022) Análisis del abandono de los estudiantes de grado en las
universidades presenciales en España. Ministerio de Universidades.
https://www.universidades.gob.es/wp-
content/uploads/2022/11/EAU_Informe_abandono.pdf
Friedel, J. N., & Friesleben, K. L. (2017). Postbaccalaureate Reverse Transfer Students in
Iowa: An Expanded Look at Community College Students with Previous Degrees Earned.
Community College Journal of Research and Practice, 41(4-5), 273-287.
https://doi.org/10.1080/10668926.2016.1251360
Garcia-Brosa, G. (2019). La inserción laboral de los graduados universitarios en España.
¿Qué hacen las empresas, las universidades y la Administración para mejorar la
empleabilidad?. En Fundación CYD (Ed.) Monografía: La inserción laboral de los
graduados universitarios en España. Experiencias recientes (pp. 135-139).
https://www.fundacioncyd.org/wp-content/uploads/2019/09/ICYD2018-E-
Monografia.pdf
Golding, B. (1999). When the backwash dwarfs the wave: A case study of the relationship
between research, policy and practice concerning two-way inter-sectoral movement in
Australia. In C. Selby Smith (Ed.), The Impact of R&D on VET Decision Making: A
Range of Case Studies (pp. 5-16). National Centre for Vocational Education Research
(Australia).https://www.ncver.edu.au/research-and-statistics/publications/all-
publications/the-impact-of-r-and-d-on-vet-decision-making-a-range-of-case-studies
Haas, A. R. (1999). Trends in Articulation Arrangements for Technical and Vocational
Education in the South East Asian Region. RMIT University.
https://unevoc.unesco.org/fileadmin/user_upload/pubs/rmit99.pdf
INEE. (2017). Indicadores y estadísticas educativas: las ciencias ¿una opción al alza?
Ministerio de Educación, Cultura y Deporte.
http://educalab.es/documents/10180/640047/JuevesJoseGallego.pdf/14e76a10-7eec-
46d9-8f4b-ebe4588721c0
INE. (2020, November 27). Encuesta de transición educativa-formativa e inserción laboral.
Microdatos.
https://www.ine.es/dyngs/INEbase/es/operacion.htm?c=Estadistica_C&cid=125473605
6996&menu=resultados&idp=1254735573113#!tabs-1254736058554
399
Leigh, R. A. (2009). Post-Baccalaureate Reverse Transfer Students in Iowa: 2006 to 2008.

https://files.eric.ed.gov/fulltext/ED515621.pdf
Markowitsch, J., & Hefler, G. (2019). Future developments in Vocational Education and
Training in Europe: Report on reskilling and upskilling through formal and vocational
education training (JRC Working Papers Series on Labour, Education and Technology
2019/07). European Commission. https://joint-research-
centre.ec.europa.eu/publications/future-developments-vocational-education-and-
training-europe_en
Montes Ruiz, A. (2019). Trayectorias no tradicionales de acceso a la educación superior:
Nuevas y viejas desigualdades en tiempos de expansión educativa [Ph.D. Thesis,
Universitat Autònoma de Barcelona]. TDX (Tesis Doctorals en Xarxa).
http://hdl.handle.net/10803/666871
Moodie, G. (2004). Reverse transfer in Australia. International Journal of Training
Research, 2(2), 24-48. https://doi.org/10.5172/ijtr.2.2.24
Taylor, J. L., & Jain, D. (2017). The Multiple Dimensions of Transfer: Examining the
Transfer Function in American Higher Education. Community College Review, 45(4),
273-293. https://doi.org/10.1177/0091552117725177
Townsend, B. K., & Dever, J. T. (1999). What Do We Know About Reverse Transfer
Students? New Directions for Community Colleges, (106), 5-14.
https://doi.org/10.1002/cc.10601
400
Quality management in Italian Universities: a case study in the

University of Cassino and Southern Lazio
Dipartimento di Economia e Giurisprudenza, Università degli Studi di Cassino e del Lazio
Meridionale, Italy.
Abstract
This paper examines on a wider scale, quality management in Italian higher
education institutions, and then progresses to focus specifically on the case of
the Master Course in Global Economy and Business (GLEB), held in the
University of Cassino and Southern Lazio. An analysis of the way in which the
University of Cassino manages quality is undertaken to identify the
responsibilities of those accountable for quality assurance. A preliminary
study was carried out through the completion of a digital survey distributed to
GLEB graduates, to monitor the success of the course. The results obtained
highlight that this course is quite appealing to students, and what also emerges
is how innovation and improvement are the foundation of quality education.
Being only the first step towards a more exhaustive evaluation, this analysis
must be extended by collecting more data, to compare the results with similar
courses in other universities through a benchmarking process.
Keywords: Higher education; quality management; institutions.

Quality management in Italian Universities
1. Introduction
Education and the formation of qualified professionals are the predominant factors, from an
international level, which significantly impact both the economy and society. Their rate of
development is correlated to the quality of peoples’ lives and the entire development of the
country itself. This produces a domino effect, leading to the search for a more competitive,
dynamic, and capable education system to ensure the country and corporations have a
sustainable growth and achieve their goals of internationalization. From this viewpoint,
higher education (HE) represents a critical factor and thus has assumed a fundamental role in
determining the level of success of a country’s economy, in terms of the development of
capital and innovation. For this reason, this process is in continuous adaptation to our society
(Texeira-Quiros, J. et al., 2022). Being humans the main consumers and beneficiaries of
educational services, defining the level of quality education becomes quite a complex task.
The pressure and need for quality education are increasing. The ceaseless demand for wealth
and security of societies and their populations is what manipulates the degree of quality that
is expected of higher education. This has been the prevailing scheme for the enhancement of
higher education institutions (HEIs) across countries worldwide, because of the proven
effectiveness that these educational systems have had in yielding outstanding professionals
to govern nations. Because the improvement of quality is a continuous process, the
perceptions of its execution and index rate are incredibly decisive in terms of achieving a
successful outcome. However, there is never only one insight available since people perceive
the concept and degree of quality in diverse ways. Both Crosby (1992) and Juran et al. (1988),
had their own perception of quality with Juran et al stating that “Quality is fitness for use or
purpose”, while Crosby believed the idea of quality is more in line with the “conformance to
standards” (Crosby, 1992; Juran et al., 1988). Therefore, to be able to assess the level of
quality, it must be defined by recognizable elements which convey its essence, and in turn
these characteristics must be identified and understood by HEIs (Justino et al., 2022).
Globalization, along with the various developments that are occurring worldwide, are forcing
HEIs to go in search of new instruments to improve the quality of higher education. The
supply and demand model can be referred to when analysing HEI management nowadays, as
universities and other higher education institutions are much more diversified than what they
once were in the past.
Thus, a higher education system that is classified as efficient and effective can only be
obtained through the achievement of a certain degree of quality in terms of teaching and
learning, which are the most important strategic issues in this kind of system (Seyfried, M.
et al., 2018). In recent years, universities have been forced to face several challenges
including internationalization, the increasing need for effective competitive strategies, the
rapid development of indispensable technological changes and the never-ending concern of
needing to reduce and control financial costs.
402
For these reasons, institutions have no other choice but to implement dynamic strategies and
technological innovations, to be capable of achieving and actually exceeding what their
stakeholders expect of them (Texeira-Quiros, J. et al., 2022; Budiharso, T. et al., 2020). Over
the past twenty years or so, high quality management systems have been gradually
established in HEIs, on both governmental and institutional levels. The issuing of enormous
governmental funds has made it fundamental for those involved in the education sector to
guarantee the provision of a productive service in universities and learning institutions, with
the aim to deliver an ever more proficient and outstanding quality of learning (Krymets et al.,
2022). The most frequent models that educational institutions refer to for quality calibration,
adjusting them of course to the institutional environment, include the Malcolm Baldridge
National Quality Award, the European Foundation for Quality Management (EFQM)
excellence model, and SERVQUAL. Many HEIs also make reference to the Common
Assessment Framework (CAF) which is a quality management method based on self-
assessment that enables public organizations to achieve their objectives, facilitating a mode
of comparison with other public entities. The significance of these quality models has been
widely recognized among educational institutions (Sciarelli et al., 2020).
Having introduced these global concepts in relation to HEI quality management, the study
can now progress forward to concentrate particularly on the quality management system
instituted in Italian universities, taking as a case study the University of Cassino and Southern
Lazio, which is the central focus of this paper. The primary objective is to initiate a
preliminary evaluation on the success of the Global Economy and Business master’s degree
course. This case study analysis is simply the first step taken towards a thorough qualitative
evaluation process that will be conducted for the GLEB course. Thus, it cannot be taken as a
general reflection of the overall degree of quality of the master course.
2. Quality Management in Italian Universities

The management of HE and research institutes in Italy is dealt with by the MUR - Ministero
dell’Università e della Ricerca, which is in charge of regulating funding, scrutinizing and
assessing universities and establishing incentives based on their performance; applying
student support policies and endorsing study curricula; making sure that the Italian education
system has a significant international presence and is up to the standards of other institutions
in the European Union. HEIs have been granted autonomy by the Ministry in 1989, provided
they abide by the legal regulatory framework (Capano, 2014). They are free to establish their
organisational governance, create their strategy and mission, design curricula and elaborate
their own research projects (MUR, 2020). The Ministry also has strong ties with the ANVUR-
Agenzia Nazionale di Valutazione del Sistema Universitario e della Ricerca (National
Agency for the Evaluation of Universities and Research Institutes) which was founded in
403
2010 to monitor and assess HEIs as well as to improve the quality of Italian research
(ANVUR, 2022).The ANVUR conducts assessment programs, in accordance with EU
standards, to evaluate the quality of activities undertaken in HEIs with the use of the
following practices:
• AVA - Autovalutazione-Valutazione periodica-Accreditamento (Self-assessment,

Periodic Evaluation, Accreditation): aims to ensure that Italian HEIs are offering
services of adequate quality; to sustain HEIs in utilizing public resources
responsibly and autonomously and in operating appropriate actions in terms of
education, technological transfer, and research activities; to ameliorate the quality
level of higher education and research.
• VQR – Valutazione della Qualità della Ricerca (Evaluation of Quality and
Research): aims to monitor the quality of research project outcomes of national
HEIs, and to stimulate an overall improvement of the quality of research projects,
to accredit doctoral programs, and to assign funding on a merit-basis to national
institutions (OECD & European Union, 2019).
Italian HEIs are aiming to increment the level of involvement of students in assessment
processes by conducting detailed investigations, in order to augment the degree of quality
attributed to the institutions (Fondazione Crui, 2018).
2.1. University of Cassino and Southern Lazio (UNICAS)

UNICAS, founded in 1979, currently has approximately eight thousand students enrolled and
offers thirty degree-courses covering multiple disciplinary sectors (Skuola Network Srl.,
2022). As in all Italian HEIs, the main body responsible for planning and monitoring quality
assurance activities at the level of the course of study is the Quality Assessment Team which
is responsible for:
• verifying the correct performance of the planned activities, the pursuit of the
objectives set by the study program, the resolution of any critical issues.
• managing interrelations between the Joint Teachers/Students Commission and the
Quality Presidium.
• dialogues with the coordinator of the study program, for the identification of
improvement actions, as well as with the external members of the Review Group,
to monitor the effectiveness of the training offer for the employment opportunities
of graduates.
• contributing to the drafting of the annual and cyclical Review Report.
(Quality of Services Office, 2022)
404
The University Quality Presidium is the main actor called upon to supervise and support the
effective implementation of the university's quality policies. Every year the Presidium issues
the Quality Policy of the University, describing the objectives and actions regarding the
aspects of quality and quality assurance (QA), in line with the priorities of the university and
its strategic plans.
In recent years, UNICAS has been particularly working on enhancing its offering of
international courses held entirely in English, one of which is the Global Economy and
Business Master course (GLEB) (Consorzio Interuniversitario AlmaLaurea, 2022).
3. Case Study
The study consists of monitoring the relevance of what the GLEB graduates studied during
the course to their job positions and the overall success rate of the study course. This analysis
was conducted in collaboration with the Quality Assessment Team of the economics
department whose objective is to monitor the performance of the GLEB course and the
satisfaction rate among students enrolled, in compliance with the national standards
established by the ANVUR. As mentioned previously, this study is simply a preliminary
analysis undertaken to initiate a complete evaluation process that will determine successively
the overall quality of the course, highlighting its strong points and weaknesses, as well as
identifying where improvement can be implemented. Therefore, this analysis is not entirely
reflective of the overall results that will be obtained upon completion of the entire evaluation
process.
A questionnaire was conducted between June and December in 2022, for a population
composed of students that graduated from the course, all participating voluntarily. The
sample size was a total of eighty-seven participants who responded out of approximately two
hundred and fifty individuals who had been contacted, in order to understand what kind of
experience these individuals had with the course through the completion of a digital
questionnaire, comprising a selection of multiple-choice questions and only a limited number
of open-ended questions for statistical study purposes.
3.1. The Survey

The survey is composed of questions regarding the demographic profile of the participants,
such as gender, age, nationality, and residence. The questions aimed to find out the graduation
age range, the time taken to complete the course, and the time taken to find a job after
graduation. It is then divided into two sub-sections based on whether the participant is
currently working or not. For those who are currently employed, the survey proceeds to
identify which country they work in, the type of occupation, the sector of work, the job
position, the approximate monthly salary range, the degree of satisfaction with their job,
405
whether the work is related to GLEB studies or not and finally an open-ended question asking
for their personal input on how the GLEB course could be improved based on their work
experience. In the second part, for currently unemployed graduates, participants are asked to
explain why they are currently not working, and for those who have chosen to continue their
education, the questions focus on where they are studying. The survey ends for all
participants with the recommendation section asking them if they would recommend the
GLEB course to others and the reasons why.
3.2. Data Analysis

According to the data obtained approximately 66% of the participants are males and the
remainder females. Regarding the birth country, more than 50% of the participants are
foreign students, while the remainder are Italian. Also, just over 75% of the participants
reside in the European continent, which implies that most of the foreign students who
enrolled in the GLEB course have remained in the continent even after their studies.
Approximately 81% of the participants are currently working and 65% of these were either
already working when they graduated, or they managed to find a job within six months after
graduation. 59% of the working graduates earn an average monthly salary of €1500 or more
(See Fig.1). Another important set of data to consider are the results of the job relevance to
course study rating, in which almost 55% of the working graduates stated that their current
job is related to their GLEB studies (See Table 1). Regarding the question of “Based on your
experience, are there any contents/skills you would suggest providing to the incoming GLEB
students?”, the main suggestions included to introduce more courses and training to acquire
people skills, soft skills, analytical skills for data reading and interpretation, digital skills,
project management abilities, project finance skills, SQL skills, statistical software
utilization, debating skills, and how to use the Microsoft Office package. Other suggestions
regarded the introduction of courses on new topics such as cyber security, cloud computing,
financial modelling, regional and national policy, as well as supply chain management.
Some suggested to increase the knowledge of the students regarding economics by
introducing more courses based on this topic, and to increase the opportunities for students
to build connections with large corporations. Out of the total number of graduates, sixteen
are currently not working, and among these, only two are continuing their studies, while the
remainder are still searching for employment. Finally, over 85% of the total participants
stated that they would recommend the GLEB course to other potential students.
406
Monthly Salary of Working Graduates

1.10% 6.80%
26.37% 16.23%
22.31%
less than 500 500-999 1000-1499 1500-1999 2000 or more

Figure 1. The average monthly salary ranges of working GLEB graduates. Source: Personal Data Source
(2022/2023).
Table 1. The relevancy of the graduates’ job to what they studied in the GLEB course.
Working Graduates
Is job related to GLEB
degree?
Definitely yes 23.94%

More yes than no 30.98%
More no than yes 26.76%
Definitely not 18.30%
Source: Personal Data Source (2022/2023).
4. Conclusion
HEIs should be continuously assuring and enhancing the quality of their educational and
research-based activities. Being well aware that for an advanced country scientific and
technological research is the driving force of society, the University of Cassino and Southern
Lazio makes an important contribution to the advancement of knowledge in the fields of
humanities, social sciences, engineering, and economic-legal studies. From the results
obtained, a critical issue has emerged, in that almost 50% of the working graduates stated
that their GLEB studies are not relevant to their current jobs, thus a further investigation must
be undertaken to verify the reasoning behind this issue. Despite this criticality, most of the
graduates have expressed that they would most likely recommend the course to other
students. This demonstrates that the methodological approach utilized has proven useful for
this study, as criticalities can be identified and consequently rectified. However, it is not
without limitations, since a limited number of participants responded to the questionnaire,
407
and thus, it is not completely reflective of the overall opinion of past students. For this reason,
the results obtained may not be exhaustive and therefore a further study and evaluation must
be conducted to obtain a more complete set of data.
From the suggestions received from the participants in the questionnaire, one of the future
objectives of the University is to implement the program and the variety of courses offered.
To do so as a future prospect, the benchmarking process could be applied to the GLEB course,
firstly, to help maintain a certain degree of quality and stay in line with the institutional
mission and reputation. Secondly, by undertaking a benchmarking analysis, the strong points
and problematic issues currently connected with the course can be unveiled and monitored.
Lastly, by comparing the performance of the course in UNICAS to other similar courses in
other institutions, similarities and differences can be noted and subsequently evaluated to
understand how this program can be improved. The concept of innovation and change is
fundamental, not only because the GLEB course is open also to international students, but
also because by implementing innovative methods, ideas, or processes it becomes more
appealing to both national and international students, thus adding to the university’s overall
competitiveness on an international level.
References
Budiharso, T., & Tarman, B. (2020). Improving Quality Education through Better Working
Conditions of Academic InstitutesJournal of Ethnic and Cultural Studies,7(1), pp.99-115.
Capano, G. (2014). The Re-regulation of the Italian University System through Quality
Assurance. A Mechanistic Perspective, Policy, and Society. 33(3), pp. 199-213.
Consorzio Interuniversitario AlmaLaurea. (2022). AlmaLaurea. Accessed on 30/08/2022,
10:20 AM. https://www.almalaurea.it/gli-atenei/universita-degli-studi-di-cassino-e-
dellazio-meridionale.
Crosby, P. B. (1992). Completeness: Quality for the 21st century. Penguin.
Fondazione Crui. (2018). Osservatorio della Fondazione CRUI per il dialogo e la
cooperazione tra università e imprese. Osservatorio Università-Imprese, Report 2018.
Juran, J., M. & Gryna, F., M. (1988). Quality Control Handbook. 4th ed. McGrawHill, NY.
Justino, M., Antunes, M. G., Mucharreira, P. R., & Nunes, A., (2022). Effects of Innovation,
Total Quality Management, and Internationalization on Organizational Performance of
Higher Education Institutions, https://doi.org/10.3389/fpsyg.2022.869638
Krymets, L. V., Saienko, O. H., Bilyakovska, O. O., Zakharov, O. Y., & Ivanova, D. H.
(2022). Quality Management in Higher Education: Developing the Methodology on the
Basis of Total Quality Management. Review of Education, 10, e3322.
https://berajournals.onlinelibrary.wiley.com/doi/epdf/10.1002/rev3.3322
Ministry of Universities and Research (MUR). (2020) Ministero dell’Università e della
Ricerca website. Italian Government. Accessed on 30/07/2022, 15:26 PM.
https://www.mur.gov.it/it
408
National Agency for the Evaluation of Universities and Research Institutes (ANVUR).
(2022). Agenzia nazionale di valutazione del sistema universitario e della ricerca website.
ANVUR. Accessed on 25/08/2022, 10:04 AM. https://www.anvur.it/en/homepage/
OECD, & European Union (2019). Supporting Entrepreneurship and Innovation in Higher
Education in Italy. OECD Skills Studies. OECD Publishing, Paris.
https://doi.org/10.1787/43e88f48-en
Quality of Services Office (2022). Piano Integrato 2022-2024. University of Cassino and
Southern Lazio.
Sciarelli, M., Gheith, M., H., and Tani, M. (2020). The Relationship between Soft and Hard
Quality Management Practices, Innovation and Organizational Performance in Higher
Education. TQM Journal, 32, pp. 1349–1372.
Seyfried, M., and Pohlenz, P. (2018). Assessing Quality Assurance in Higher Education:
Quality Managers’ Perceptions of Effectiveness. Eur. J. High. Educ. 8, pp. 258–271.
Skuola Network Srl. (2022). Skuola.net. Accessed on 30/08/2022, 16:50 PM.
https://www.skuola.net/universita/ateneo/unicas.
Texeira-Quiros, J., do Rosário Justino, M., Antunes, M. G., Mucharreira, P. R., & de
Trindade Nunes, A. (2022). Effects of innovation, total quality management, and
internationalization on organizational performance of higher education institutions.
Frontiers in Psychology, 13.
409
Robust estimation method for the economic subsidies to

educational institutions in Chile
Valentina Ugalde, Sebastián Dávila-Gálvez, Óscar C. Vásquez
University of Santiago of Chile (USACH), Faculty of Engineering, Program for the
Development of Sustainable Production Systems (PDSPS), Chile, University of Santiago of
Chile (USACH), Faculty of Engineering, Industrial Engineering Department, Chile.
Abstract
In Chile, the Ministry of Education of Chile (MINEDUC) provides resources
to the various educational institutions in the country based on a budget
estimation model, where the "attendance of students" is the main component to
be considered by annual economic subsidies. In the process, this estimation
must be computed in the middle of the year, and therefore, the reduction of the
impact of error in the estimation face to uncertainty is a desirable goal. In this
work, we propose an easy-to-implemented robust estimation method, which is
based on the formulation, implementation, and integration of a set of
mathematical models that target the minimization of the maximum estimation
error. The obtained results show that the proposal obtains a better estimation
than the current MINEDUC estimation model, providing an efficient
alternative of robust estimation method for the decision makers in the process,
exposing insights to be considered in the future.
Keywords: School Subsidies; attendance, mathematical models; robust
method; estimation.

Robust estimation method for the economic subsidies to educational institutions in Chile
1. Introduction
Education is the fundamental basis for people's opportunities throughout their lives and
constitutes the essential underpinning of a country's productivity and development (Marcel
& Tokman, 2005). Therefore, States must finance school education considering its
compulsory nature and guarantee access and equal participation of all members of the
community in national life. In this context, the Ministry of Education of Chile (MINEDUC)
is responsible for ensuring an inclusive and quality education system that contributes to the
comprehensive and lifelong learning of people and the development of the country, through
the formulation and implementation of policies, standards, and regulations, from
kindergarten to higher education (Gobierno de Chile, 2022). For this purpose, MINEDUC
grants municipal and subsidized private education and local education services the concept
of school subsidy.
The schooling subsidy granted by MINEDUC finances an average of 10.595 educational

institutions in Chile, which translates into an average of 3.269.831 students in the country.
Moreover, concerning the distribution of financing resources for the educational system, in
2021, the Chilean government allocated 6.396.670.457 MM CLP, of which 3.670.242.048
MM CLP are destined for the schooling subsidy (MINEDUC, 2021), representing more than
50% of the annual budget ministry.
In this context, MINEDUC has a budget estimation model that is conducted in collaboration
with the Ministry of Finance and is composed of different variables and parameters. The main
assumptions are enrollment and attendance percentage, which are determined by analyzing
the behavior of recent years. For example, the methodology used in the Ministry of
Education's 2023 subsidy estimation considered the most recent enrollment at the time of the
estimation (May enrollment) and a 2,48% increase in the number of PIE students (students
with permanent or temporary special educational needs) according to the growth reported by
the General Education Division (School Integration Program Coordination). To make the
estimation, the percentage of occupied attendance corresponding to the average of the last
four years was used.
However, the attendance estimation for determining the budget to be requested for the school
subsidy delivery needs to be more explicit due to various situations, including social
mobilizations and Covid-19, among others. On the other hand, MINEDUC must annually
make the budget estimation for the following year in May. That is, with the information from
January to May 2023, they must estimate the attendance for all the months of 2024. That
generates uncertainty about the value of estimated attendance, causing a margin of error in
the estimation, either underestimating or overestimating the actual budget. If the calculated
412
budget is lower than that needed in the subsidized educational institutions, MINEDUC must
again request an amount to subsidize the educational institutions that should have been
covered.
1.1. Our contribution
This paper deals specifically with the problem of uncertainty in the number of students
attending educational institutions to determine the amount of school subsidy to be received.
The contribution of this research is based on a robust estimation method, which involves the
formulation, development, and implementation of a set of mathematical models that target
the estimation error. This method allows determining a more efficient estimation of the
number of students attending educational institutions per month, and per teaching modality,
complying with the MINEDUC's request, that is, that the estimation values have at most a
2,5% error. This result will, in turn, be used as a parameter in the financing formula of the
Chilean educational system and will reduce the margin of error in estimating of the total
budget for educational financing. The data corresponding to Chile's municipal and private
educational institutions and local education services were considered for the above.
2. Material and methods

2.1. Statement of the problem
In Chile, schools are characterized according to their administration, for which we consider
a set of educational institutions 𝐷. Each educational institution can offer a set of days for
holding classes, so we consider a set of days 𝐽 for educational institutions. In turn, each
educational institution has different teaching modalities that we denote by a set 𝑃 that
depends on the type of day 𝐽 offered by that institution. Then, we define 𝑃! ⊂ 𝑃 as the subset
of teaching modalities for educational institutions with a full school day and a subset 𝑃" ⊆ 𝑃
as the subset without a full school day. Student attendance is recorded from January to
December for each year. We define a set of months 𝑀 and a set of years 𝐴 that depends on
the collected data that considers information from educational institutions from 2017 to 2021.
As MINEDUC annually must make the budget estimation in May for the attendance
estimation, we consequently consider a subset 𝑀# ⊂ 𝑀 representing the training months to
be used for the estimation.
To estimate attendance, we rely on estimation errors. On the one hand, the mean absolute
percentage error (MAPE), which relates the estimation error to the level of demand, is helpful
to place the estimation performance in its correct perspective, where the objective is to
minimize this error to obtain the lowest possible margin of error in the estimation (Krajewski,
Ritzman, Malhotra, & Krajewski, 2008). On the other hand, the maximization of the
413
minimum of the errors (MINMAX) aims to obtain the minimum of the errors, that is, the
most negative, to represent an overestimation of the real value of the attendance to make this
overestimation the minimum possible subsequently. This last error is considered because, in
conversations with MINEDUC, the latter stated that it wanted to obtain an overestimation in
the estimations.
2.2. Mathematical model
To address the problem above, we propose a robust estimation method based on formulating
two mathematical models that target the estimation error: the MAPE and the MINMAX.
Specifically, we define the decision variables 𝑥)$,&,',( and 𝑒$,&,',( that express the estimation
of the average attendance and the estimation error of the educational institutions of
dependency 𝑑 ∈ 𝐷 with a type of day 𝑗 ∈ 𝐽 with teaching modality 𝑝 ∈ 𝑃 in the year 𝑎 ∈ 𝐴.
A parameter 𝛼 that expresses an estimation error of 2,5%. A non-negative decision variable
𝜆),$,&,' ∈ [0,1] that expresses the relevance of the training month 𝑚 ∈ 𝑀# of the educational
institutions of dependency 𝑑 ∈ 𝐷 with the type of school day 𝑗 ∈ 𝐽 with teaching modality
𝑝 ∈ 𝑃. The parameters 𝑥$,&,',),( and 𝑥̅$,&,',( that express the real attendance and the real
average attendance, respectively, of the educational institutions of dependency 𝑑 ∈ 𝐷 with
the type of day 𝑗 ∈ 𝐽 with teaching modality 𝑝 ∈ 𝑃 in the month 𝑚 ∈ 𝑀 in the year 𝑎 ∈
𝐴. For both models, let us consider the following set of restrictions ∀𝑑 ∈ 𝐷, ∀𝑗 ∈ 𝐽, ∀𝑝 ∈ 𝑃:
𝑥)$,&,',( = < 𝜆),$,&,' 𝑥$,&,',),( ∀𝑎 ∈ 𝐴 (2) < 𝜆),$,&,' = 1 (3)

)∈+! )∈+!
𝑒$,&,',( = 𝑥)$,&,',( − 𝑥̅$,&,',( ∀𝑎 ∈ 𝐴 (4) 𝑒$,&,',( ∀𝑎 ∈ 𝐴 (5)

−𝛼 ≤ ≤𝛼
𝑥̅$,&,',(
where, the set of constraints (2) and (3) define the convex combination for the attendance
estimation that corresponds to the weighted sum of the actual attendance value of the
educational institutions in the training month 𝑚 𝜖 𝑀# . Constraint (4) defines the estimation
error corresponding to the difference between the estimation and the actual average
attendance. Constraint (5) defines the estimation error tolerance as 2,5%.
Then, the objective of the MAPE model is to minimize the mean absolute percentage error
for which we consider an auxiliary variable 𝑌$,&,',( that expresses the linearization of the
absolute value function of the estimation error defined by Expression (6) ∀𝑑 ∈ 𝐷, ∀𝑗 ∈
𝐽, ∀𝑝 ∈ 𝑃:
[𝑀𝐼𝑁] < 𝑌$,&,',( (6)

(∈-
Subject to the following restrictions (2) to (5):
414
𝑌$,&,',( ≥ 𝑒$,&,',( /𝑥̅ $,&,',( , 𝑌$,&,',( ≥ −𝑒$,&,',( /𝑥̅$,&,',( ∀𝑎 ∈ 𝐴 (7)
where, the set of constraints in (7) defines the linearization of the absolute value function of
the estimation error corresponding to the mean absolute percentage error. On the other hand,
the objective of the MINMAX model is to obtain the most positive error to represent an
overestimation of the real value of attendance, to subsequently make that overestimation the
minimum possible for which we consider an auxiliary variable 𝑍$,&,' , defined by Expression
(8) ∀𝑑 ∈ 𝐷, ∀𝑗 ∈ 𝐽, ∀𝑝 ∈ 𝑃:
[𝑀𝐼𝑁] 𝑍$,&,' (8)
Subject to the following restrictions (2) to (5):
𝑍$,&,' ≥ 𝑒$,&,',( , 𝑒$,&,',( ≥ 0 ∀𝑎 ∈ 𝐴 (9)
where, the set of restrictions in (9) defines the auxiliary variable that rescues the most
considerable overestimation of the estimation, and the estimation errors must be positive to
represent an overestimation in the attendance estimation.
Then, to determine the attendance estimations of the educational institutions with their
respective attributes, we formulated a compromise mathematical model that integrates the
two models formulated above. Specifically, we defined two weights 𝑊 ∈ [0,1] to give
weight to each mathematical model of estimation error. A 𝐹𝑂+-./ parameter that expresses
the value of the objective function of the MAPE model. And a 𝐹𝑂+01+-2 parameter that
expresses the value of the objective function of the MINMAX model. The above, as shown
in Expression (10):
𝑊 ∗ ∑$∈4 ∑&∈3 ∑'∈. ∑(∈- 𝑌$,&,',( (1 − 𝑊) ∗ ∑$∈4 ∑&∈3 ∑'∈. 𝑍$,&,' (10)
[𝑀𝐼𝑁] +
𝐹𝑂+-./ 𝐹𝑂+01+-2
subject to the following expressions (2) to (9) and ∀𝑑 ∈ 𝐷, ∀𝑗 ∈ 𝐽, ∀𝑝 ∈ 𝑃:
𝑍$,&,' ≥ 0 (11)
where, constraint (11) indicates which auxiliary variable 𝑍$,&,' represents the minimum
overestimation, such that if there is no overestimation ∀𝑎 ∈ 𝐴, its value is zero.
3. Results
3.1. Statement of the problem
To solve the problem above, we use the actual attendance data provided by MINEDUC of
municipal, private subsidized, and local education services educational institutions from
2017 to 2021, corresponding to January to December. For educational institutions whose
dependence is "Municipal" (Mun) or "Private Subsidized" (PSub), the corresponding
415
teaching modalities must have at least three years of historical data. Moreover, the teaching
modalities must have at least two years of historical data for those with "Servicios Locales
de Educación" (SLE) dependency. Attendance estimations will be made by teaching modality
for the year 2021 to be able to compare our results with the results obtained by the MINEDUC
model for the same year. Finally, the scope of this work involves the attendance estimations
for the year 2021 of the educational modalities whose annual average attendance covers at
least 50% of the global average.
3.2. Mathematical model
To solve the instance above, we implemented the mathematical models previously
formulated using the Python programming language, considering the following
considerations: Pyomo to formulate optimization models, Gurobi as a solver on a PC with an
AMD Ryzen 9 5900HS with Radeon Graphics processor at 3.30 GHz and with 32 GB of
RAM. Table 1 shows the attendance estimations for the year 2021 for educational institutions
of dependency 𝑑 𝜖 𝐷 with a type of day 𝑗 𝜖 𝐽 with teaching modality 𝑝 𝜖 𝑃 obtained by the
MAPE model, MINMAX, Compromise Solution with two weightings W equal to 0,5 to
integrate the MAPE and MINMAX models and the MINEDUC budget estimation model
equally. And a comparison of the actual attendance data for the same year, where the
acronyms "C.J" and "S.J" refer to educational institutions with and without a full school day,
respectively, while "B" and "M.H.C" correspond to the modalities of primary and secondary
scientific and humanistic education, respectively.
4. Discussions and Conclusion

In this paper, we address the problem of uncertainty in the number of students attending
educational institutions to allocate the school subsidy in Chile. A robust estimation method
is provided by developing a mathematical model that targets the estimation error, the MAPE,
and the MINMAX and integrating these through a compromise solution mathematical model.
The results obtained are illustrated considering the actual attendance data of the subsidized
educational institutions in Chile from 2017 to 2021 delivered by MINEDUC. For this
research, the selection of the MINMAX estimation error is because MINEDUC requested
that the attendance estimations represent an overestimation of the real data in order not to
have a budget deficit in the school subsidy that affects the administrative management of
MINEDUC and the quality of the student's education. Now, the attendance estimations of the
educational institutions were made at a disaggregated level according to the dependency
𝑑 𝜖 𝐷, type of school day, 𝑗 𝜖 𝐽 and type of education 𝑝 𝜖 𝑃 because MINEDUC's budget
estimation model requires it for the school subsidy financing formula. This implies having a
carry-over error in each estimation, which could mean a significant error at the aggregate
level, i.e., in the final amount of the school subsidy. Therefore, the overestimation could be
416
massive if we only use the MINMAX error for the estimations. As a result, we will obtain a
surplus in the school subsidy budget, which generates inefficiency in the distribution of
resources, on the one hand, in the reallocation of resources to other items within the subsidy
program or, on the other hand, causing another MINEDUC program not to be executed.
Therefore, the MAPE estimation error selection has a regulatory impact at the aggregate level
of the school subsidy budget, i.e., the final amount is associated with a smaller confidence
level or estimation threshold since some estimations will be overestimated. Others
underestimated, where a negative value (underestimated) can save a lot in the overestimation
of the estimation.
This behavior of the estimation errors is supported by the results obtained by each
mathematical model separately. When estimating attendance only with the MAPE model,
most of these estimations at the disaggregated level are below the actual attendance value.
For the MINMAX model, most estimations are above the actual attendance value. Moreover,
comparing the total amount of attendance, it is observed that when using only the MINMAX
model, the total of the estimations is higher and closer to the actual value of attendance than
when using only the MAPE model. Then, by integrating these two models in the compromise
solution model, it can be verified, according to the results obtained, how the MAPE model
attenuates the overestimates of the MINMAX model, finding a balance between
underestimates and overestimates. In fact, at the global level of attendance, it can be seen
how the total value of attendance decreases by only using the MINMAX model, but at the
same time, how the total value of attendance increases by only using the MAPE model.
Finally, the results obtained by our robust estimation method were more accurate than those
obtained by the MINEDUC estimation model, which ratifies the effectiveness of our proposal
and the improvement that our research could contribute.
Something interesting to note about the results obtained by the MINMAX model is that
although these results represent overestimates, they needed to be sufficiently loose. This is
because in the database provided by MINEDUC, the behavior of the data was stable; in fact,
they were similar within and outside the estimation period. It follows that the dispersion of
the missing data, i.e., the data to be estimated, varies a little, so because of applying
MINMAX, the overestimates were not so loose. Therefore, the carry-over error needed to be
bigger, which meant that the disaggregated estimations were relatively accurate in the final
amount of attendance.
417
Table 1. Results of the attendance estimations for 2021 of the set of mathematical models versus
the MINEDUC estimation model.
MAPE MINMAX S.C MINEDU

C
D J P Real attendance Estimation Estimation Estimation Estima

tion
M C.J B. 1°-4° 230.576 230.121 230.313 230.121 230.462
u
n
M C.J B. 5°-6° 141.102 141.033 141.330 141.330 141.070
u
n
M C.J B. 7°-8° 143.919 143.905 143.905 143.905 144.616
u
n
M C.J M.H.C 145.965 144.205 144.241 144.205 148.472
u
n
P C.J B. 1°-4° 304.839 306.319 304.974 306.319 299.127
S
u
b
P S.J B. 1°-4° 189.784 189.607 189.789 189.607 187.603
S
u
b
P C.J B. 5°-6° 213.924 214.254 214.077 214.254 210.310
S
u
b
P C.J B. 7°-8° 205.330 204.433 205.364 204.433 202.180
S
u
b
P C.J M.H.C 308.533 305.253 308.821 305.253 304.686
S
u
b
Total 1.883.972 1.879.130 1.882.814 1.879.427 1.868.526
Source: Own elaboration.
In conclusion, based on the results obtained, for future work, we propose the use of other
mathematical models that target the estimation error, such as the mean absolute deviation
418
(MAD), the absolute mean square error (MSE), among others, to see how the integration of
these in the compromise solution model alters the results of the attendance estimations, either
overestimating or underestimating each of the estimations. Now, although in the compromise
solution model, the weight assigned to the W weight was 0,5, it would be interesting to
analyze how the values of the attendance estimations change as W changes in the objective
function of the compromise solution model and also to study the behavior of the weights to
see the importance of each month in the attendance estimations, that is, to analyze if there is
any trend in the weights of the weights, for example, that March is the most relevant in the
attendance behavior.
Acknowledgements
The authors are grateful for the support of the National Agency for Research and
Development (ANID), FONDECYT Project N°1211640, and the Economic Subsidies Area
of the Ministry of Education (MINEDUC).
References
Gobierno de Chile. (2022). Gob.cl. Retrieved February 8, 2023, from
https://www.gob.cl/ministerios/ministerio-de-educacion/
Krajewski, L. J., Ritzman, L. P., Malhotra, M. K. (Manoj K., & Krajewski, L. J. (2008).
Administración de operaciones: procesos y cadenas de valor. Pearson Educación.
Marcel, M., & Tokman, C. (2005). ¿Cómo se Financia la Educación en Chile? Retrieved from
https://www.dipres.gob.cl/598/articles-21669_doc_pdf.pdf
MINEDUC. (2021). Informe mesa financiamiento del Sistema Escolar. Santiago. Retrieved from
https://www.mineduc.cl/informe-mesa-financiamiento-del-sistema-escolar/
419
Evolution of academic dishonesty in computer science courses

Katerina Zdravkova
Faculty of Computer Science and Engineering, Ss. Cyril and Methodius University in Skopje,
N. Macedonia.
Abstract
Online exams and assignments during the COVID-19 pandemic have
introduced new forms of student cheating. In order to maintain evaluation
criteria and preserve established ethical standards, professors have
introduced new methods to minimize cheating. When returning onsite, the
newly created cheating techniques evolved once again. They were supported
by special groups on social networks dedicated to easier liquidation of exams
and getting better grades. Crowdsourcing became frequent, particularly for
homework assignment preparation. Recently, ChatGPT has become a new ally
of students. This paper presents the evolution of student cheating in several
computer science courses taught by the author of this paper. All examples of
cheating are supplemented by the detecting methods and own applications
used to prevent them from occurring again. The paper ends by predicting who
will win in the eternal war between students and professors, at least in the
short run.
Keywords: Academic dishonesty; contract cheating; crowdsourcing; social
media; versatile chatbots.

1. Introduction
Academic dishonesty is a common phenomenon that has been going on for centuries (Lang,
2013). It is sometimes supplemented by bribing the examiner (Liu & Peng, 2015). Cheating
usually happens during exams, but a large amount of cheating also comes from homework.
The methods are specific to the field and type of study and are usually applied either during
exams or during the preparation of assignments. In the pre-digital era, cheating during exams
was done using more than 20 different techniques (Twomey, White & Sagendorf, 2009). At
the Faculty of Computer Science and Engineering in Skopje (FCSE), the most common were:
• Cheat sheets, which our students hid in their pockets, inside their pants or taped
inside thighs, while the girls hid them under the skirts or in their bras (Erbe, 2007).
Sometimes they were placed in logarithmic tables, hidden inside pencils or stuck on
plastic water bottles.
• Passing notes, prepared by better students after finishing their own solutions and
shared with the students in their vicinity (Yee & MacKown, 2009). For shorter
solutions, they could reach more than five students during one exam.
• Peeking into the test of a student who is ahead or behind (Newstead, Franklyn-
Stokes & Armstead, 1996).
• Whispering or showing the number of the task and the correct answer with fingers
(Twomey, White & Sagendorf, 2009).
• Distracting the proctor, often with banal questions, and enabling other colleagues to
copy from the cheat sheets, passing notes or even from the test of a better student
unhindered (Yee & MacKown, 2009).
The digital era introduced many new techniques, which complemented the previously
mentioned ones (Twomey, White & Sagendorf, 2009). At FCSE, they included:
• Searching the Internet from the computer from which the student takes the exam or
from the mobile phone (Harkins $ Kubik, 2010).
• Entering the solution into a calculator (Kelley & Dooley. 2014), disk of USB flash
drives that is shared with colleagues (Dawson, 2016).
• Sharing the solution using chat communication within the faculty local area network
(Khan & Balasubramanian, 2012).
• Sharing the photo of the solution via mobile phone (Tindell & Bohlander, 2012).
• Turning off the screen to hide the solution that is intentionally left for a colleague
who will take the exam from the same computer (Moten et al., 2013).
• Taking the exam on behalf of another student by logging into his/her account
(Bretag et al., 2019).
422
Katerina Zdravkova
Academic dishonesty during homework preparation did not change significantly with the
transition to the digital era (Ercegovac & Richardson, 2004). It predominantly includes:
• Various forms of plagiarism, which embrace verbatim or Google translated text

(Ducar & Schocket, 2018), fabricated bibliography (Lin & Wen, 2007),
paraphrasing without acknowledging the original author (Newstead, Franklyn-
Stokes & Armstead, 1996), potlucking sentences from several sources to harmonize
them while retaining most of the original text (Myers, 2018) or self-plagiarism from
assignments prepared for other courses (Bretag & Mahmud, 2009).
• Various forms of contract cheating that encompass: delivering or copying from
other assignments with the approval of the original authors (Zdravkova, 2011).
• Hiring a ghostwriter (Zheng & Cheng, 2015), purchasing from paper mills
(Medway, Roper & Gillooly, 2018) or using computer generated solutions (Vasylets
& Marín, 2022).
• Appropriating a joint solution or homework assignment as its own (Newstead,
Franklyn-Stokes & Armstead, 1996).
New learning management systems are Web 2.0-oriented, offering students the opportunity
to work together using discussion forums, wikis, blogs and chats, enabling new forms of
cheating (Aljawarneh, 2020). They introduce several new cheating activities:
• Idea theft in the discussion-based assignments inspired by what colleagues have

written (Ellis, 2022; Zdravkova, 2014). Sometimes, the quality of stolen idea
overcomes the quality of the original one (Zdravkova, 2014).
• Intrinsic plagiarism by combining parts of previously submitted discussions (Stein,
B., Lipka, N., & Prettenhofer).
• Identity fraud when a student gives up his password to a colleague who does
assignments for him/her (Bailie & Jortberg, 2009).
• Deliberate destruction of existing wikis or their fake editing (Zdravkova, 2014).
Last but not least is the use of social media in education (Greenhow & Lewin, 2016). They
contain massive repositories of all information shared among the students, starting with exam
questions and their solutions, through homework solutions, ending with strategies for
outsmarting professors.
The paper continues with a review of observed cheating in two types of courses at FCSE
during online education caused by the COVID-19 virus. All examples of cheating are
supplemented with detection methods and display of proprietary applications used to prevent
them from occurring. The paper ends with the post-COVID student cheating methods and
the personal impression of the outcome of the eternal war between students prone to cheating
and teachers who strive to prevent this academic dishonesty.
423
2. Cheating and prevention methods during online learning

Online education required extraordinary efforts to prevent students from cheating on exams
and assignments. FCSE prepared a completely new environment, which contained many
mechanisms to detect and minimize the effects of students' eternal desire to complete their
assignments with minimum effort and get the highest grade in the process.
All exams were organized using Safe Exam Browser (https://safeexambarowser.org), which
disabled browsing outside the exam website. Before the exam, students were identified
through their smartphones, which were turned into web cameras using the Manycam
(https://manycam.com/) or Droidcam (https://droidcam.en.softonic.com/) applications. The
identification included careful observation of the room from which the exam was taken.
During the exam, the proctors monitored each student, his/her computer screen, and listened
to all the sounds he/she made. In spite of all these efforts, the success rate during exams
increased, probably as a result of communication via alternative communication media.
Exams were recorded and kept for further inspection whenever the proctor suspected that
some students were cheating. Based on the re-checking of the recorded exam, several
fraudsters were discovered and their studies were suspended for one year.
2.1. Detecting and preventing academic cheating in technical courses

Plagiarism of exam solutions and homework assignments in technical courses was detected
with JPlag (https://github.com/jplag/JPlag), a powerful tool that discovers software
plagiarism and collusion in software development by mutually comparing the code. On each
course and during each exam, at least one couple of completely identical and several almost
identical solutions was detected, resulting in a disqualification of all involved students from
the course. The same approach was implemented to prevent contract cheating of assignments.
The exam questions included the available commands of the programing tool presented as
icons that must be correctly selected and assembled to generate the solution (Fig. 1). This
time-consuming activity for the exam creators has proven to be extremely effective to reduce
the use of cheat sheets and avoid the temptation to search for solutions from an alternative
laptop or smartphone. Moreover, it enabled the automatic grading of exams.
Figure 1. Typical exam questions of the Introduction to computer science course.
424
Katerina Zdravkova
2.2. Detecting and preventing academic cheating in soft skill courses

Soft skill courses at FCSE are mainly realized using Web 2.0 features within Moodle
(Zdravkova, 2014; Zdravkova, 2022a). Plagiarism by stealing from online sources was
detected by careful checking of all posts, particularly those with a writing style not typical
for a young computer science student. The professor translated them back into English and
then searched the phrases on the web. This manual strategy resulted in the discovery of
dozens of plagiarized posts out of hundreds, proving the suspicion that students are easily
misappropriating other people's copyrighted material.
Detecting the theft of ideas required careful reading of posts and excellent memorization of
previously published content. To facilitate visual discovery, the forums were divided into
aspects to reduce the amount of published material (Zdravkova, 2022a). The same technique
contributed to the detection of intrinsic plagiarism. In several situations, students whose ideas
were stolen reacted within the same forum, making the professor's detective work easier.
An application for ghostwriting detection was developed to mutually compare homework
assignments according to document metadata, used references, text ngrams and text
similarity (Fig. 2.). The main requirement was that the documents should be delivered as
source files. The application revealed several key facts:
• Students use their own crowdsourcing repositories where they have collected
assignments from many generations. To prepare a new essay or journal, which is a
collection of hot topics related to the curriculum, cheaters compose their own
assignments from several similar assignments, reducing the text similarity.
• Contract cheating is more than obvious. Students from earlier generations and
ghostwriters recruited from the current one create essays that are in accordance with
the defined rules. The common feature of these essays is that they have 0 editing
time. If the student who was suspected of handing in such an assignment knows well
what is written in it, then he / she gets a grade depending on the quality of the work.
• Crowdsourcing was also noticed mainly because the writing style in the homework
was too professional. Few students who were suspected admitted that they bought
the homework from Amazon Mechanical Turk (https://www.mturk.com/).
Figure 2. Ghostwriting detecting application at work.
425
3. Returning to new normal and final remarks

The return to on-site education marked a new shift in academic dishonesty. The long-term
reliance of students on the web during exams was the first challenge FCSE had to deal with.
This was achieved by activating a firewall that prevents access to all websites except the
exam and by disabling access to the exam website from IP addresses ouside the faculty labs.
That decision drew the ire of some students who insisted on online exams for medical
reasons.. When entering the exam, students switch their cell phones to flight mode and leave
them in a special place. It does not necessarily mean that they are prevented from cheating,
because they can always have an alternative device hidden in their clothes.
FSCE has decided not to block the spy earbuds and mobile phones during exams. Typically,
one proctor is responsible for a maximum of 20 students, which is more than enough to notice
attempts to communicate with colleagues taking the same exam or with the cheating partner
via the spy earbud.
The discovery of academic dishonesty in soft skills courses has not changed, as they have
been mostly online for over 15 years (Zdravkova, 2022b). The only on-site activities, which
are part of the student assessment, are presentations and discussions of collaborative projects.
They have always been events that students liked and they never cheated on them. During
the last discussions in January 2023, students revealed that there is “a new kid in town”:
ChatGPT (https://openai.com/blog/chatgpt). Some have already experienced it and were
impressed by its functionalities. It is probable that ChatGPT has already been used for the
preparation of essays in the computer science course. According to the author of this paper
own experience, the writing style in the essays compiled by this versatile chatbot resemble
the writing of young students. If these essays are post-edited after being translated using
Google Translate, there will no longer be a need for contract cheating and crowdsourcing
support. This chatbot will do a great job in replacing days of homework preparation with less
than a quarter of an hour of polishing. Ghostwriting detectors will become obsolete.
The paper presented a plethora of various techniques and activities of cheating on exams and
during homework preparation. Many new techniques and methods of cheating have appeared,
but the old ones still exist and are all widely used by dishonest students. Therefore, it is more
than evident who will win in the eternal war between students and professors.
Cheaters are always at least one step ahead of their opponents. New technologies and
crowdsourcing initiatives, driven by social media and versatile chatbots are their great allies.
No matter how hard the professors try to prevent academic dishonesty, the students will
always have an advantage and will manage to outsmart them.
426
Katerina Zdravkova
References
Aljawarneh, S. A. (2020). Reviewing and exploring innovative ubiquitous learning tools in
higher education. Journal of computing in higher education, 32, 57-73.
https://doi.org/10.1007/s12528-019-09207-0
Bailie, J. L., & Jortberg, M. A. (2009). Online learner authentication: Verifying the identity
of online users. Journal of Online Learning and Teaching, 5(2), 197-207.
Bretag, T., & Mahmud, S. (2009). Self-plagiarism or appropriate textual re-use?. Journal of
Academic Ethics, 7, 193-205.
Bretag, T., Harper, R., Burton, M., Ellis, C., Newton, P., Rozenberg, P., ... & van Haeringen,
K. (2019). Contract cheating: A survey of Australian university students. Studies in
higher education, 44(11), 1837-1856.
Dawson, P. (2016). Five ways to hack and cheat with bring‐your‐own‐device electronic
examinations. British Journal of Educational Technology, 47(4), 592-600.
doi.org/10.1111/bjet.12246
Ducar, C., & Schocket, D. H. (2018). Machine translation and the L2 classroom: Pedagogical
solutions for making peace with Google translate. Foreign Language Annals, 51(4), 779-
795.
Ellis, L. M. (2022). The interpersonal consequences of stealing ideas: Worse character
judgments and less co-worker support for an idea (vs. money) thief. Organizational
Behavior and Human Decision Processes, 171, 104165.
doi.org/10.1016/j.obhdp.2022.104165
Erbe, B. (2007). Reducing test anxiety while increasing learning: The cheat sheet. College
teaching, 55(3), 96-98.
Ercegovac, Z., & Richardson, J. V. (2004). Academic dishonesty, plagiarism included, in the
digital age: A literature review. College & Research Libraries, 65(4), 301-318.
Greenhow, C., & Lewin, C. (2016). Social media and education: Reconceptualizing the
boundaries of formal and informal learning. Learning, media and technology, 41(1), 6-
30.
Harkins, A. M., & Kubik, G. H. (2010). “Ethical” cheating in formal education. On the
Horizon. doi.org/10.1108/10748121011050487
Kelley, R., & Dooley, B. (2014). The technology of cheating. 2014 IEEE International
Symposium on Ethics in Science, Technology and Engineering (1-4). IEEE.
Khan, Z. R., & Balasubramanian, S. (2012). Students go click, flick and cheat... e-cheating,
technologies and more. Journal of Academic and Business Ethics, 6, 1.
Lang, J. M. (2013). Cheating lessons. Harvard University Press.
Lin, C. H. S., & Wen, L. Y. M. (2007). Academic dishonesty in higher education—a
nationwide study in Taiwan. Higher Education, 54, 85-97.
Liu, Q., & Peng, Y. (2015). Determinants of willingness to bribe: Micro evidence from the
educational sector in China. Jahrbücher für Nationalökonomie und Statistik, 235(2), 168-
183. doi.org/10.1515/jbnst-2015-0205
427
Medway, D., Roper, S., & Gillooly, L. (2018). Contract cheating in UK higher education: A
covert investigation of essay mills. British Educational Research Journal, 44(3), 393-
418.
Moten Jr, J., Fitterer, A., Brazier, E., Leonard, J., & Brown, A. (2013). Examining online
college cyber cheating methods and prevention measures. Electronic Journal of E-
learning, 11(2), 139-146.
Myers, C. S. (2018). Plagiarism and copyright: best practices for classroom education.
College & Undergraduate Libraries, 25(1), 91-99.
Newstead, S. E., Franklyn-Stokes, A., & Armstead, P. (1996). Individual differences in
student cheating. Journal of Educational Psychology, 88(2), 229.
Stein, B., Lipka, N., & Prettenhofer, P. (2011). Intrinsic plagiarism analysis. Language
Resources and Evaluation, 45, 63-82. doi.org/10.1007/s10579-010-9115-y
Tindell, D. R., & Bohlander, R. W. (2012). The use and abuse of cell phones and text
messaging in the classroom: A survey of college students. College teaching, 60(1), 1-9.
Twomey, T., White, H., & Sagendorf, K. (2009). Pedagogy, not policing.
Vasylets, O., & Marín, J. (2022). Pen-and-paper versus computer-mediated writing modality
as a new dimension of task complexity. Languages, 7(3), 195.
Yee, K., & MacKown, P. (2009). Detecting and preventing cheating during exams.
Pedagogy, Not Policing, 141.
Zdravkova, K. (2011). Fifty ways to detect a ghostwriter.
Zdravkova, K. (2014). Transformation of student cheating in web 2.0.
Zdravkova, K. (2022a). Managing a successful educational role-playing game. Information
Systems and Technologies: WorldCIST 2022, Volume 2. 433-443.. doi.org/10.1007/978-
3-031-04819-7_41
Zdravkova, K. (2022b). Adapting a Web 2.0-Based Course to a Fully Online Course and
Readapting it Back for Face-to-Face Use, Communications in Computer and Information
Science, vol 1740, 135-146. doi.org/10.1007/978-3-031-22792-9_11
Zheng, S., & Cheng, J. (2015). Academic ghostwriting and international students.
428
Place-based teaching amidst a global pandemic?

Barbara Cabezal Bruno1, Daniela Bottjer-Wilson2, Jennifer Engels1
1
Hawaii Institute of Geophysics and Planetology, School of Ocean and Earth
Science and Technology, University of Hawaii at Mānoa, USA, 2Center for
Teaching Excellence, University of Hawai‘i at Mānoa, USA.
Abstract
How did the COVID-induced switch to online learning impact attitudes and
practices toward place-based teaching? To explore this question, a pair of
surveys was administered to students and faculty in the University of Hawaii’s
School of Ocean and Earth Science and Technology in Fall 2018 (142
respondents) and Fall 2021 (83 respondents). Survey results indicate that PBT
practices are highly valued by students and faculty, even (or perhaps
especially) when courses are in online formats. Faculty report wanting to use
more place-based teaching practices in online courses, but there are obvious
challenges. The paper ends with concrete examples of how place-based
teaching can be effectively implemented in online courses.
Keywords: Place-based teaching, geoscience, STEM education, online
learning.

1. Introduction
The COVID-19 pandemic has had a dramatic and enduring effect on higher education. In
early 2020, when the global pandemic first hit, nearly all colleges and universities shut their
physical classrooms. Online learning rapidly became the norm, forcing students and teachers
to grapple with learning new technologies and modes of interaction at breakneck speed.
Despite a rocky start, students and teachers rapidly adapted. A year or two later, when the
pandemic subsided, many students were no longer interested in returning to in-person
instruction: they wanted to continue learning remotely (Inside Higher Ed, 2021, Clary et al.
2022). Students voted with their feet: universities that insisted on in-person-only instruction
were punished with plummeting enrollments. Many institutions are now offering courses in
various online, hybrid and in-person formats, in an attempt to meet student demand while
maintaining academic rigor.
This paper examines the question: How did the COVID-induced switch to online learning
impact attitudes and practices toward place-based teaching at the University of Hawaii’s
School of Ocean and Earth Science and Technology (SOEST)? Place-based teaching (PBT)
has been shown to increase student interest and motivation (Smith and Sobel 2010),
strengthen relationships to place and local communities (Sobel 2004); and raise
environmental consciousness (Stapp et al. 1969). Whereas PBT can be a successful approach
for engaging all students, it has been shown to be exceptionally effective with indigenous
students (Cajete 1994). [See Böttjer-Wilson and Bruno 2019 for more on PBT].
2. Methods
To assess attitudes and experience regarding PBT at SOEST, we distributed an online survey
to SOEST faculty and students in Fall 2018 (Böttjer-Wilson and Bruno 2019) and again in
Fall 2021. In 2018, there were 142 survey respondents, including 59 faculty and 83 students.
In 2021, the response was smaller (83 total), including 31 faculty and 52 students. Not every
respondent answered every survey item, so the number of responses to each survey item
varies. Comparing the two data sets enables us to gauge how PBT attitudes and practices
have changed between 2018 and 2021. The timing of the 2018 pre-survey was fortuitous, as
we obviously could not have anticipated the COVID-19 pandemic hitting in 2020, and allows
us to consider the effect of the global pandemic on PBT. Throughout this paper, the graphs
present the 2018 survey data in blue and the 2021 survey data in red.
430
3. Data & Results
3.1 Faculty Survey Results

In Fall 2020 and Spring 2021, all SOEST instructors used online course formats, with 92%
teaching online synchronous classes. The majority of instructors (60%) taught only online
synchronous classes, but some also used asynchronous (20%), hybrid (12%) and/or in-person
(16%) formats. When asked which PBT practices they would have normally used during Fall
2020 and/or Spring 2021 but did not due to the COVID-19 pandemic, most faculty listed one
or more practices (Figure 1). Topping the list of PBT practices reported were field trips (used
by 48% of faculty), service learning (20%), and local/regional experts as guest speakers
(16%) – all of which would have been much more difficult to do in online courses. About
one-third (36%) of faculty responded “None of the Above”, which could either indicate that
these faculty used PBT practices pre-pandemic and continued to use them post-pandemic, or
that they never used PBT practices pre-pandemic.
Figure 1. These data on PBT practices were compiled from faculty responses to the 2021 (n=25) survey question:
Which of the following PBT practices would you have normally used during Fall 2020 and/or Spring 2021, but did
NOT because of the COVID-19 pandemic? (Check all that apply). The listed practices (from left to right) are:
Field trips; Service learning; Local/regional experts as guest speakers; Indigenous knowledge, or ways of
knowing; Community workdays; Cultural practices; Local/regional data sets; Hawaiian language terms; and
None of the above.
431
3.2 Student Survey Results

In student surveys administered in both 2018 and 2021, SOEST students were asked to state
their agreement or disagreement to the survey item: “SOEST lab and lecture classes with
strong ties to place (Mānoa, O‘ahu, Hawai‘i or the Pacific region) have greatly improved
my learning experience” by selecting an answer choice on a five-point scale ranging from
Strongly Agree to Strongly Disagree. Results are presented in Figure 2.
Figure 2. Student responses to the 2018 (n=67) and 2021 (n=52) survey item: SOEST lab and lecture classes with
strong ties to place (Mānoa, O‘ahu, Hawai‘i or the Pacific region) have greatly improved my learning experience.
“Neither” means Neither agree nor disagree.
From 2018 to 2021, the combined positive responses (Agree and Strongly Agree) increased
from 81% to 88%, the neutral response (Neither Agree nor Disagree) decreased from 16% to
4% and the combined negative responses (Disagree and Strongly Disagree) decreased from
3% to 2%. [Note: Although Figure 2 suggests the combined negative responses remained at
2%, the combined 2018 value is 3%, due to rounding.] Together, these results indicate that,
compared with the 2018 student respondents, the 2021 student respondents reported higher
levels of agreement. In order to test whether the higher agreement in 2021 (vs. 2018) is
statistically significant, we quantified the Likert Scale on a scale of 1 (Strongly Disagree) to
5 (Strongly Agree). This enables us to calculate mean values and run an unpaired, one-tailed
t-test, using α=0.05. The results were: 2018 mean = 4.18; 2021 mean = 4.42, t-statistic = 1.68
and p = 0.048, indicating statistical significance (as p<α) – that is, the students reported
significantly stronger agreement to PBT learning benefits in 2021, compared with 2018. This
432
result underscores the importance for faculty to use PBT practices, even (or especially?) when
teaching online courses, despite the obvious challenges.
Figure 3 summarizes the types of PBT practices that students would like to see included in
future classes. Each PBT practice listed on the survey was selected by 47-84% of 2018
respondents and 58-88% of 2021 respondents, with field trips (84% and 88%) topping the
list in both years. All (100%) respondents who answered this question selected at least one
practice; no one selected “None of the Above”. The percentage of respondents who selected
each practice increased from 2018 to 2021. One possible explanation could be the COVID-
19 pandemic, which caused many students to feel socially isolated and perhaps strengthened
their desire to be connected to people and places (Smith et al. 2022).
Figure 3. Student responses to the 2018 (n=73) and 2021 (n=52) survey question: What aspects of PBT would you
like to see included in your SOEST lab and lecture classes in future semesters? (Check all that apply). The listed
practices (from left to right) are: Field trips; Local/regional experts as guest speakers; Hawaiian language terms;
Indigenous knowledge, or ways of knowing; Local/regional data sets; Cultural practices; Service learning;
Community workdays; and None of the above.

SOEST survey results indicate that PBT practices are highly valued by students and faculty,
even (or perhaps especially) when courses are in online formats, and faculty report wanting
to use PBT. When they do, students report their learning is greatly improved. Thus, an online
433
course format should not be considered a reason to discontinue PBT. Looking at the strategies
listed in Figure 3, the most in-demand PBT practices were field trips. Through field trips,
student can apply the knowledge learned in the classroom to real-life settings – for example,
in the field, they can identify rocks and minerals in situ, conduct geological mapping
exercises, detect evidence of glaciation, examine volcanic deposits and infer eruption styles,
observe weathering and erosion, and use all of this information to unravel the geologic history
of an area. Field trips also help students create connections with place and with other
participants, strengthening student motivation and their sense of belonging to a learning
community. But can these benefits be realized through virtual field trips?
Prior to the COVID pandemic, virtual field trips existed, but were relatively uncommon in
the geosciences (see Bond et al. 2022 for a literature review of pre-pandemic virtual field
trips). Their popularity exploded during the pandemic, driven by necessity. Virtual field trips
pose unique challenges, and do have limitations, but there are many excellent virtual
geoscience field trips that serve as exemplars in achieving well-defined learning outcomes.
The Science Education Resource Center at Carleton College (SERC 2022) has perhaps the
most extensive and varied collection of virtual geoscience field trips in the world, and
includes a top-tier “Exemplary Collection”. Other useful collections of virtual geoscience
field trips have been compiled by the American Geophysical Union (AGU 2022) and the
European Geosciences Union. The latter highlighted 19 examples of virtual field trips, on
topics ranging from simulations of traditional geologic mapping to gamified experiences, in
a recent joint special issue (Toy et al. 2022).
Although the virtual field trips described above may not contain PBT content relevant to all
areas, they can serve as useful models. In SOEST, instructors were able to quickly adapt their
in-person field trips to virtual formats that retained PBT content, using a variety of strategies.
One adaptation converted a campus-based geology field trip (Bruno et al. 2022) to a self-
guided walking tour, using interactive prompts that the students could access via their cell
phones. Another faculty member took photos and video of sites around O‘ahu, then voiced
over content that allowed students to answer geology questions online without having to
travel to the sites. The Hawai‘i Division of Forestry and Wildlife (2023) hosts virtual field
trips that include geographic, historical and cultural information, which serves as an example
of how locally relevant, pre-made virtual content could be used to fulfill PBT goals with a
minimal time investment on the part of the instructor.
Apart from field trips (88%), several other PBT practices were in high demand by the student
respondents, including Local/regional experts as guest speakers (87%); Hawaiian language
terms (81%); Indigenous knowledge, or ways of knowing (81%); and local/regional datasets
(75%) (Figure 3). While using local and regional datasets is a relatively accessible
pedagogical technique for most faculty, the vast majority of SOEST faculty are neither Native
Hawaiian nor from Hawai‘i, and few are experts in Hawaiian language, culture or indigenous
434
knowledge. Thus, it is essential for faculty to construct their own knowledge as well as
partner with experts. Here are a few suggested starting points for incorporating PBT content
into online classes, beyond virtual field trips and local and regional datasets. Though
conceived for Hawaii-based SOEST faculty, these strategies can be extrapolated for a global
audience:
• Regularly use Hawaiian terms (including place names) in geoscience classes, and
learn to write and pronounce them correctly.
• Learn about history and culture, by visiting local museums (for example, Bishop
Museum in Hawaii) and cultural sites, attending lectures, and reading.
• Volunteer at local community events, to learn, give back, and make connections
with local experts. Investments in the community may result in opportunities to
invite local experts into virtual classrooms.
• Incorporate local and indigenous knowledge into geoscience course content in
consultation with cultural practitioners. Hawaiian-language newspaper archives are
a repository of traditional practices (IHLRT 2018) – see Swanson (2008) for an
example.
• Invest in reciprocal community-research partnerships, following protocols outlined
in Kūlana Noi‘i (2021)
In summary, an online course format should not be considered a reason to discontinue PBT
practices, but an opportunity to explore new and innovative ways of incorporating this
important content.
Acknowledgments
The U.S. National Science Foundation (Award # 2022937) provided support for this study.
We would like to thank Mari MacNeill for assistance with administering the 2021 survey,
and SOEST faculty and students for completing the survey. We are also grateful to two
anonymous peer reviewers, whose comments improved this manuscript.
References
American Geophysical Union (AGU) (2022). Virtual Field Trips.
https://www.facebook.com/watch/AmericanGeophysicalUnion/377422986576360/
Accessed April 6, 2023.
Bond, C.E., Pugsley, J.H., Kedar, L.Ledingham, S.R. Skupinska, M.Z., Gluzinski, T.K. and
Boath, M.L. (2022). Learning outcomes, learning support, and cohort cohesion on a
virtual field trip: an analysis of student and staff perceptions. Geoscience
Communication, 5, 307–323, https://doi.org/10.5194/gc-5-307-2022
435
Böttjer-Wilson, D. and Bruno, B.C. (2019). Place Matters! Fostering place-based geoscience
teaching at the University of Hawai‘i at Mānoa, Proceedings of the 5th International
Conference on Higher Education Advances (HEAd’19), 243-251,
http://dx.doi.org/10.4995/HEAd19.2019.9237.
Bruno, B.C., D. Tachera, J. Engels and S.K. Rowland (2022). A Campus Huaka‘i: Weaving
Geological, Historical & Cultural Perspectives in Hawai‘i, Proceedings of the 8th
International Conference on Higher Education Advances (HEAd’22), 705-713,
https://dx.doi.org/10.4995/HEAD22.2022.15652
Cajete, G. (1994). Look to the mountain: An ecology of indigenous education. Skyland, NC:
Kivaki Press.
Clary, G., Dick, G. et al. (2022) The After Times: College Students’ Desire to Continue with
Distance Learning Post Pandemic. Communications of the Association for Information
Systems, 50(1), 122-142, https://doi.org/10.17705/1CAIS.05003.
Hawai‘i Division of Forestry and Wildlife (2023)
https://dlnr.hawaii.gov/dofaw/education/virtual/ Accessed Feb 1, 2023.
Inside Higher Ed (2021) Students Want Online Learning Options Post-Pandemic.
https://www.insidehighered.com/news/2021/04/27/ Accessed Feb 4, 2023.
Institute for Hawaiian Language Research and Translation (IHLRT) (2018).
https://seagrant.soest.hawaii.edu/institute-of-hawaiian-language-research-and-
translation/ihlrt-about/ Accessed Jan 27, 2023.
Kūlana Noiʻi Working Group (2021). Kūlana Noiʻi v. 2. University of Hawaiʻi Sea Grant
College Program, Honolulu, Hawaiʻi. https://seagrant.soest.hawaii.edu/kulana-noii/
Science Education Resource Center at Carleton College (SERC) (2022).
https://serc.carleton.edu/index.html Accessed Jan 31, 2023.
Smith, G.A. & Sobel, D. (2010). Place- and Community-Based Education in Schools. New
York, NY: Routledge.
Smith, R., Seaborn, K., Goldsmith, A., Curtis, N., Davies, A., Haynes, W., McEnroe, R.,
Murphy, N., O’Neill, L., Pacey, C., Walker, E., Wordley, E. (2022). Student loneliness
through the pandemic: How, why and where? The Geographical Journal, 188(2), 277-
293. https://doi.org/10.1111/geoj.12438
Sobel, D. (2004). Place-based education: Connecting classrooms and communities. Great
Barrington, MA: The Orion Society.
Stapp, W.B. et al. (1969). The concept of environmental education. The Journal of
Environmental Education, 1, 30-31.
Swanson, D.A. (2008). Hawaiian oral tradition describes 400 years of volcanic activity at
Kīlauea. Journal of Volcanology and Geothermal Research, 176(3), 427-431.
Toy, V., Villeneuve, M., Buckley, S.J., Whitmeyer, S. and S. Illingworth, editors (2022).
Virtual geoscience education resources (Solid Earth/Geoscience Communication inter-
journal Special Issue). https://gc.copernicus.org/articles/special_issue431_1145.html
Accessed Jan 31, 2023.
436
Surviving and thriving in COVID

Mary Jo Parker
Scholars Academy, University of Houston-Downtown, USA.
Abstract
Many summer bridge or other programs focus on initiating high impact
activities (Kuh, 2008) as mechanisms of relationship-building among
attendants and the university sponsoring these activities. Additionally, a broad
spectrum of these bridge programs are grant-funded by either state or federal
funding agencies. Mentored research within a university laboratory connects
the undergraduate to career insights and cultural awareness of research.
COVID- has altered but not cancelled this activity over the last two years.
Keywords: Summer bridge; STEM research; undergraduates; DOED MSEIP
grant.

1. Introduction
Summer undergraduate research and summer bridge programs for entering undergraduates
are staples to initial training of college-goers to the possibilities that mentored research offers
their understanding of what the profession involves, what basic or applied research involves,
and provides mentoring by PhD scientists such that undergraduates develop a consideration
of what a career in research could/would involve especially within tier one laboratories.
Undergraduates become aware of theory, practice, success and failure at the bench.
Moreover, they see modeled for them what scientists actually do in the day-to-day routine
demands of their laboratories. Conversations between mentor and student can lead to greater
insights and address limiting ideas the student may have of what it means to discover new
concepts or information.
The summer bridge program involving mentored research for first generation students
becomes a critical component to the student’s consideration of entrance into a doctoral
program as a career. Among institutions which are primarily undergraduate institutions with
no doctoral programs, the summer bridge programs are crucial to introducing first
generations diverse students to the possibilities that research holds as career stepping stones
by enhancing the entire undergraduate experience through increased relevancy, substantive
meaning of a career in science (STEM) (Mogk, 1993; Lopatto, 2004, 2007).
High impact practices often suggest strong connections to student retention and student
engagement as well as character development as it connects to leadership development (Kuh,
2008; AACU, 2008; Kuh & Umbach, 2004). Many four-year colleges and universities
encourage a common practice of mentored undergraduate research as a student support
mechanism and as a career development opportunity, especially within the sciences (Lopatto,
2007). Several studies offer well-documented findings associated with undergraduates
involved in research experiences in under the guidance of laboratory principal investigators
or in the classroom setting using a student-centered, problem-based approach. Likewise,
studies on the effectiveness of mentored research experiences indicate several important
findings, such as: 1) enhancing the entire undergraduate experience and 2) on increasing the
substantive interest in entering a career in science through advanced graduate work (Mogk,
1993). Gains in self-confidence and pathways to science careers, especially among first
generation, minority, and female undergraduates were documented in several studies by
Lopatto (2004, 2007) and Hathaway, et.al, 2002).
In the UHD Scholars Academy program a Department of Education MSEIP grant award
(award #P120A190069) enabled the establishment of a summer bridge mentored research
program supporting both incoming freshmen (FTIC) and current undergraduates in their
initial mentored research experience. Year one occurred during May and June 2020 during
the first lockdown of the COVID-19 pandemic spanning almost all institutions of higher
438
Mary Jo Parker
education. This paper provides explanations, data-gathering, outcomes associated with

implementation of a virtual mentoring bridge program. The award was received in October
2019 and involved multiple targets including: 1) freshman UpSTART programs, 2)
Academic Skill Monitoring, 3) Mentoring, Leadership/Teamwork Development, 4)
Mentoring, and 5) Career and Research Skill Development, which will be the focus of this
paper.
Goals of the awarded grant entitled “UHD Enhancing STEM Success Through Complex
Problem Solving, Preparation and Access” address STEM success program components as
well as some innovative mechanisms. A primary focused outcome is to encourage, prepare
and support minority STEM students, especially females. Another is to target access for first
time in college entering undergraduates through summer bridge program and transition
activities into enrollment. Success is targeted in all years where transitions and rigor form
barriers for undergraduates with some of the success activities to include freshman
UpSTART, academic skill monitoring, mentoring in small learning communities,
career/Research skill development and leadership/teamwork enhancement, then finally
intentional financial literacy skill development. Other targets include support of STEM
females through a Women in STEM series, Women Mentoring Women workshop approach,
and career development through mentored research under the guidance of PhDs.
2. Beginning Under COVID-19 Virtual Environments

The grant administration team was comprised of PI and four PhD Co-PIs (Dr. Gabriella
Bowden, Dr. Mian Jiang, Dr. Ting Zhang, and Dr. Weining Feng) spanning all College of
Science and Technology STEM departments at UHD. Each a subject matter expert in their
field provided the depth of knowledge and experience in mentored laboratory work to excel.
Each had provided mentored research experiences during academic and summer sessions to
undergraduates prior to the quarantining events associated with COVID. However, none had
provided virtual mentoring and as such each became experts through experiential methods.
A little bit about each Co-PIs’ research is useful to best understand the magnitude of impact
going virtual would present in order to reach the targets of the grant program.
Dr. Gabriela Bowden’s research involved measuring the human response to bacterial
pathogens, or the study of spore-forming soil microorganisms. Studies that address the
impacts of environmental contaminants on microbial populations which in turn could impact
humans. Usually this type of research must be completed under strict safety protocols and
within laboratory onsite.
Dr. Mian Jiang’s research involved the study of molecular nanowires. Specifically,
participants will prepare conducting polymer-based nanowires, and nanowires from
439
conducting polymer/carbon nanotube composites. studies will extend the known facts of
nanowires being used for mechanical devices and microcircuits into high capacity
nanobattery and chemical sensing. Considering the sponging and biocompatible nature of
conducting polymers, the proposed polymer nanowire sensors may represent nearly ideal
detector elements since their size match that of biological macromolecules. Again, this type
of research is usually always performed within a laboratory setting onsite.
Dr. Ting Zhang’s research involved the study of robotics, neuroimaging, and artificial
intelligence (AI) on web databases. Specifically, the cutting-edge techniques in computer
science, electrical engineering, and biomedical engineering are investigated to advance
interdisciplinary research approaches. This arena of research was very approachable through
virtual environments.
Dr. Weining Feng’s research involved electronics, power systems, measurement and
instrumentation, control engineering, data communications, and networking. She has worked
extensively in the area of robot manipulator dynamics and control and has brought her
experiences in the areas of system reliability, failure mode, and effects of criticality analysis
to both the research laboratory and her teaching classroom.
With the large numbers of highly interested undergraduate bridge and current students
additional research PhDs were solicited for participation. Each received a small stipend per
semester for their participation. Other PhDs brought a broad spectrum of research across
other STEM disciplines including: mathematics, datascience, statistics, computer science,
web development, robotics, organic chemistry, genetics, cancer research, bioinformatics,
material sciences, microbiology, plant biology, and environmental biology. Each PhD
mentored from 2 to 4 undergraduates per the five week summer bridge program. In keeping
with the targeted support of STEM females, 12 PhDs were female and 5 PhDs were male.
Of these PhDs with minority status comprised only 2 individuals. One minority male was
part of the mentoring PhDs as well.
3. How Mentored Research Occurred in a Virtual Environment

This question was key to bringing all participating PhDs into an understanding of the
outcomes expected for their research group/students. The question also provided some very
interesting opportunities for all undergraduates as they entered into the mentored research
program. The keys to successful outcomes were generated by outstanding, flexible, and
creative PhDs providing the mentoring. Together in an initial meeting discussed the
possibilities of what approach/es and accomplishments could be expected from the
participating students. This conversation occurred via a Zoom video-conference meeting
with the PI/Co-PIs and all other PhDs.
440
Mary Jo Parker
Begin with theory by offering readings of published literature in their SME areas. Generally,
this is not where mentored research begins when normal “in the laboratory” conditions occur.
So this was a revamped approach which provided foundational understanding of the research
to be undertaken. Next, each PhD agreed to move to processes to be undertaken once back
in the laboratory. In the case of Dr. Bowden’s microbiology research students investigated
protocols for creating agar cultures, staining, etc. In the case of Dr. Jiang’s chemical
laboratory, challenges to these students were to transform laboratory safe procedures into
“kitchen chemistry” safe procedure that could be done in a home environment. The goal in
one instance was to create a titration lab using kitchen chemistry. Both students approached
the challenge from different perspectives as one student was a junior chemistry major and
one a sophomore chemistry major, but both developed successful techniques. Further, Dr.
Feng’s students decided upon their projects and she subsequently ordered materials to outfit
kits per each student. These kits were put together and sent to each students’ home where
they would meet with Dr. Feng virtually via Zoom and demonstrate where the project was
moving. All activities took place over as five-week period.
As PI of the overall project, the role involved setting the milestones, meeting dates, mid-
session presentation dates, and submission of final abstracts, posters/slideshows, and video
with all members involved in the creation of the groups’ achieved outcomes. These
deliverables were then sent to the DOED MSEIP program director for inclusion in the
summer research conference (also held virtually).
4. Holding Student Researchers Accountable
4.1. Visual Accountability

Students were asked to document both in paper format and visual formats weekly discussions
occurring with each of the PhD research mentors. These were uploaded into an MS Teams
area within PhD folders. Other artifacts, such as Youtube videos, other videos, websites,
e.g., BLAST were also uploaded as evidence of research and skill development. Review of
safety practices associated with each researcher were documented within the TEAMS data
system.
4.2. Meetings Accountability

Students met with their individual PhD research mentors weekly via Zoom video-conference
meeting. Zoom meetings were to be recorded and uploaded into the TEAMS environment.
All student researchers were asked to work from 8 to 10 hours per week on their research
projects as well as complete a Weekly Research Report form documenting current status in
the project, planned activities per discussion with the PhD, and overall progress. The report
forms would be upload into TEAMS for review and signatures by PhD mentors.
441
4.3. Orientation Meeting/s

The PI/Co-PIs and other participating researchers were asked to attend an initial orientation
meeting of one hour held via Zoom video-conferencing. An overview of the general
objectives of the program was provided. Likewise, weblinks to the Zoom meetings, times,
and attachments were sent through email but also uploaded into the MS TEAMS environment
for transparency and access by all. All meetings were recorded to ensure all could have
additional access to the meeting contents for refresher purposes or in the event they could not
actually attend. As part of the orientation, the PI provided a general discussion related to
goals, hours needed for weekly progress, expected outcomes for student deliverables. This
was necessary as each undergraduate participant would be provided a research stipend of
$700 for the five-week period.
A mid-session Zoom meeting was scheduled for all participants to provide a one-slide
overview of the progress within their research group. All members would have to provide
speaking portions indicating they were all participating.
4.4. Deliverables
Student success was assessed via multiple measures. Deliverables were submitted to
document project progress during the 5-week period of mentored research. Deliverables
included 1) weekly research report forms, 2) attendance of all meetings including weekly
PhD meetings, 3) DOED MSEIP abstract template produced and approved by PhD research
mentor, 4) mid-session slideshow outlining progress, and 5) production of zoom video
describing the cumulative accomplishments of research project/s for the 5-week program.
Participants were asked to complete a SURE Survey for Mentored Research taken from
Lopatto, 2007, 2010. The survey was digitized into a Qualtric survey for ease of analyses.
All participants in years 1 and 2 completed the SURE survey. Additionally, all participants
were asked to complete a Leadership Survey derived from Northouse, 2019. This survey
was placed in a Qualtrics framework and distributed. Approximately 50% completed this
survey. An additional survey was collected investigating confidence, competence, likelihood
of pursuing STEM careers, increased likelihood of finding STEM employment upon
graduation, increased academic capacity due to participation in mentored research, perceived
improvement in critical thinking, technical writing, and presentations, and feeling more
connected to college coursework.
In late July 2020 all student produced abstracts and powerpoint posters were provided to the
DOED MSEIP Program Director for review and selection of presentation as part of the
Competitive Capacity Enhancement Model Student Research Conference (CCEMSRC).
Four abstracts from our university program were selected to present their virtual mentored
research findings. The students included were mentored under the guidance of Dr. Ting
442
Mary Jo Parker
Zhang, computer science and robotics, Dr. Connie Kang, genetics, Dr. Ling Xu, computer
science and web development, and Dr. Sanghamitra Saha, bioinformatics/biology. The
students performed well and learned of other models of virtual research projects undertaken
by other universities with similar awards.
5. Programmatic success measurements

The DOED MSEIP targets support of STEM, STEM minorities, STEM females in particular
as outcomes to be reported. Figure 1 below describes the increasing number of participants
involved in the mentored research virtual programs over the first two years of the grant award.
As can be easily noticed, all participating categories are increasing over the two-year period.
Also, more minorities (who tend to be first generation students) are engaging in mentored
research thereby connecting them to the university, to their course of study, and to the
potential to move into STEM career pathways as they graduate.
A review of participating bridge and undergraduate research students indicates continuance
among a majority of those beginning initially in the virtual environment. Figure 2 describes
the fluctuation in participants but also shows increases in minority participation and females.
Figure 2. Participation by category in academic and summer virtual research.
6. Finding common interests for matching

Undergraduates are matched with on-campus UHD PhD research scientists within a variety
of STEM lab settings. The SME PhD comes to understand the qualities/competencies and
commitment the undergraduates bring to gaining deep learning of not only discipline content
and skills, but also of what research is and what it can become for each of the mentees in the
443
pursuit of individual careers. Early mentored research students are guided by and learn from
an upper division undergraduate who has worked with the PhD for some time (within the
normal settings and possibly within the virtual COVID settings). As the early career research
students develop expertise and competency in the lab and trust on the part of the PhD
researcher, they move up within the lab setting and become perceived as a leader. Thus,
student researchers are trained in skillsets and laboratory leadership is developed through this
mentorship in research. Matching by common interests between PhD and undergraduate
truly has long-term impact on post-baccalaureate graduate/professional program outcomes,
individual ambitions, and confidence to enter STEM careers (Lopatto, 2004; 2007; Mogk,
1993). Funding support provides those engaged in mentored research a workforce
connection encouraging chosen STEM research career pathways as well as transferability of
the skills into the post-baccauleate workplace.
7. UHD, SA and minority research student development

Based on current demographics that Hispanics and other minorities are likely to become the
major ethnic group in Texas by 2030 (Associated Press, 2004) there is a need for Texas to
close the science education gap and to recruit more Hispanic students and other minorities
into university STEM degree programs. A clear need exists for more minority undergraduate
students across a broad natural sciences, computer and mathematical sciences, and
engineering technology degree emphasis to enroll in and complete STEM degrees, thus
lessening an educational attainment gap evidenced among minorities (Dugan & Komives,
2006). At UHD this includes all degree plans within the College of Sciences and Technology.
The ultimate goal of increasing the pipeline of minority students, entering and completing
the baccalaureate degree, is strongly connected to the need for 1) familiarity of the college
experience [most are first generation], 2) pre-college academic support, 3) academic
monitoring, 4) mentoring by STEM faculty and undergraduates already successful in the
STEM arena, 5) broadened exposure to graduate and industry experts and opportunities, 6)
research and career experiences/exposure, and 7) training in leadership development across
various aspects of STEM (DOED MSEIP 2013 award #P120A130040).
Unfortunately, minority students continue to dismiss the four-year university out of fear of
the unknown prior to entrance or out of a lack of understanding of expectations. Minorities,
particularly minority women, continue to fail-out of first-year barrier STEM courses as a
result of no substantive support services within the designated/selected content discipline,
lack of adequate pre-college preparatory coursework, and/or little to no acculturation into the
study hours and techniques needed to grasp rigorous, often difficult STEM content
topics/knowledge as presented in first year courses. First generation minority students, in
particular, have no historical, familial connections to what the collegiate expectations are for
STEM students, thus have less than supportive network available to them once entrance into
444
Mary Jo Parker
the STEM degree curriculum begins. Working with the mentored research program provides
foundational knowledge and experiences to minorities to begin to see or envision their
potential within a STEM career.
Because UHD is an urban MSI/HSI university located in the 4th largest metropolitan city in
the U.S., the mentored research development program, virtual or otherwise, provides an
intentional, purposeful vehicle aimed to 1) support minority students seeking university
degrees through this large university, 2) examine how and what leadership through
mentorship in the research arena influences STEM undergraduates college to career readiness
enhancements, impact, and degree completion (baccalaureate and post-baccalaureate), and
3) creates processes and product knowledge for other universities, both minority-serving and
other, to utilize in the support of undergraduate research development positively influencing
the changing demographic constituencies and leadership capacity of Houston, Texas, and the
U.S.
References
AACU. (2008), High Impact Educational Practices. A brief overview. Retrieved at
http://www.aacu.org/leap/hips.
Clark, I, Romero-Calderon, R, Olson, JM, Jaworski, L, & Lopatto, D, et al. (2009).
‘‘Deconstructing’’ Scientific Research: A Practical and Scalable Pedagogical Tool to
Provide Evidence-Based Science Instruction. PLoS Biol 7(12): e1000264.
doi:10.1371/journal.pbio.1000264.
Dalakoura, A. (2010). Differentiating leader and leadership development: A collective
framework for leadership development. Journal of Management Development, 29, 5, 432-
441. Retrieved at https://doi.org.10.1108/02621711011039204.
Day, D. & Harrison, M. (2007). A multilevel, identity-based approach to leadership
development. Human Resource Management Review, 17, 4, 360-373. Retrieved at
www.sciencedirect.com.
Dugan, J. & Komives, S. (2006). Developing Leadership Capacity in College Students:
Findings from a National Study, The Multi-Insitutional Study for Leadership –A Project
of National Clearinghouse for Leadership Programs. Retrieved from www.nclp.umd.edu.
Ferman, C. & Van Linden, J. (1999). Charcter education for developing youth leadership.
Education Digest, 65, 4, 11-16.
Hathaway, R.S., Nagda, B.A., & Gregerman, S.R. (2002). The relationship of undergraduate
research participation to graduate and professional education pursuit: an empirical study.
Journal of College Student Development , 43 , 614-631.
Komives, S., Owen, J., Longerbeam, S., Mainella, F., &Osteen, L. (2005). Developing a
Leadership Identity: A grounded theory. Journal of College Student Development, 6,
593-611.
Kuh, G. (2008). High-Impact Educational Practices: A Brief Overview. High-Impact
Educational Practices: What They Are, Who Has Access to Them, and Why They Matter.
AAC&U.
445
A Post-Covid comparison of students’ usage of an online learning

platform
Robert A. Phillips
Alliance Manchester Business School, University of Manchester, Manchester, M15 6PB,
United Kingdom.
Abstract
This paper explores the differences in the usage by students of an online
platform for supplementary resources between a first year and final year
entrepreneurship course which are traditionally face-to-face. We found a weak
correlation between use of the online resources and higher marks. The average
amount of time spent and other habits were identified. We can use this data to
improve the online offering element for students at university where
supplementary online resources are offered alongside face-to-face teaching.
Keywords: Enterprise; entrepreneurship education; online resources; online
learning.

A Post-Covid comparison of students’ usage of an online learning platform
1. Introduction
Covid brought a shock to the education system with a sudden pivot online of university
courses that were normally almost exclusively face-to-face. This was a particular challenge
for those teaching entrepreneurship where the best learning environment is much more
experiential. However, despite the far from ideal rushed nature for most of this challenge,
having had time to assess the impact of online Covid teaching (Phillips, 2022), there are
benefits of offering elements of online teaching that might help to augment any course and
allow students to learn in the most appropriate way for their preferred learning style.
Entrepreneurship is a key factor in economic growth and entrepreneurship education is
suggested as a key factor for reducing the fear of failure in potential entrepreneurs (Chapman
and Phillips, 2022) and consequently raising the nascent entrepreneurship rate. The Quality
Assurance Agency document on Enterprise and Entrepreneurship Education (2018) suggests
students should be able to “identify and respond to opportunities using their ideas,
knowledge, skills and confidence to create interventions that will address the challenges they
meet.” They further suggest there are three types of enterprise education: Learning “about,”
learning “for” and learning “through.” Of these, learning “through” is deemed to be the most
effective at stimulating students to start a business. There has been considerable research into
entrepreneurship education, although metrics for success are not clear cut (Duval‐Couetil,
2013; Kuratko, 2005). With Covid reducing teaching interactions to online only, this was
clearly a major issue for Entrepreneurship Education.
At the University of Manchester there is a varied approach to entrepreneurship with both on
curricular (Papadopoulou and Phillips, 2019; Sanchez-Romaguera and Phillips, 2018) and
extracurricular (Phillips, 2010; Phillips, 2017) activities widely used. A range of assessments
are used rather than exams - such as consultancy reports, reflective journals (Phillips, 2008),
posters, pitches, and feasibility studies. However, online learning as with many other
institutions was not a high priority pre-covid. Covid has awakened us to the opportunities of
online learning and the benefits it can bring. Ligiori and Winkler (2020) suggest that
entrepreneurship education was behind even other subjects with its use of online technology,
possibly due to the more active nature of entrepreneurship education with more of a need for
face-to-face interaction, groupwork and networking. Covid has allowed us to re-evaluate
online learning and to understand the reasons for student’s level of engagement with it. For
example, the increase in travel expenses means some students prioritise when to come to
university attendance and poor scheduling might mean a student deciding not to attend in
person if they have just a single session in a day. Some students who are disabled also may
choose to prioritise attending only certain days for maximum efficiency, such as interactive
seminars over a content-based lecture, and those increasing numbers of students with caring
responsibilities or term time employment also might have extensive use of the online
material. There are also some students who have always preferred using the podcasts over
448
Robert A. Phillips
lectures. Most units at the university now are face to face again, but have an online site with
extra reading, lecture slides, quizzes, and links to podcasts of the lectures. Measuring the
interaction with the online sites post-covid by students would be useful to determine how
much they are valued and used by students and what their role should be post-covid in a
mainly face-to-face course. There has been considerable research into online learning, both
where online material such as podcasts are offered as an additional resource to a course that
is principally face to face, and also where online learning comprises the entire course. For
this paper, of particular interest is research into the former, to help understand how online
platforms can augment teaching post-covid, with research suggesting the switch to pure
online out of necessity during Covid is quite different to a more considered approach in
applying lessons learned to post covid education (Adedoyin & Soykan, 2020, Ferri et al.
2020), although there is a sense that some universities are trying to return to pre-covid without
incorporating any of the benefits learned from the sudden switch to online (Watermeyer et
al., 2022). Several researchers have attempted to document the shift to online and analysed
the strengths and weaknesses of different approaches (e.g., Dhawan, 2020; Adnan & Anwar,
2020). There are conflicting views on whether the availability of online material such as
podcasts affects attendance, with purely online courses there have historically been issues
with dropout rates (DiRamio & Wolverton, 2006), and indeed whether it affects the marks
and satisfaction a student gains from a principally face to face course. Bolliger et al., (2010)
suggest that offering podcasts to students is useful as they can repeat sections they do not
understand and that the pace of face-to-face lecture might not be optimal for learning or for
taking notes. They found 3 separate groups of learners – 1 group focussed on podcasts whilst
attending few lectures, group 2 used very little of either, and group 3 made occasional usage
of podcasts. Group 1 was found to be the highest in both achievement and satisfaction with
the course. Traphagan et al. (2010) found that students who used the lecture podcasts as well
as to face-to-face lectures had a similar level of achievement from students who only attended
face-to-face lectures. In other studies, students reported lower anxiety when they could take
notes from lectures they viewed later on their own without having to worry about missing
important information (Owston et al. 2011), and that anxiety was reduced by viewing lectures
before exams and tests (Traphagan et al. 2010). In terms of online learning, researchers have
referred to the TAM (Technology Acceptance Model) (Davis, 1989) which builds on the
Theory of Planned Behaviour (Ajzen, 1980) to predict usage based on the key factors of
awareness, perceived usefulness, attitude to using the system, behavioural intention, and
actual usage of the system. Since the online environment is an additional resource for the
students used asynchronously, perceived usefulness is likely to play an important role.
This paper investigates the usage of the course Blackboard site post-covid, to determine how
they are now viewed by students from two courses in order to maximise the effectiveness of
the supporting services/activities, and to determine whether if there is any linkage between
usage of the site and improved academic performance for a subject which is generally
449
accepted needs to be as hands on as possible. A year 1 and year 3 entrepreneurship course

were examined. Both are open electives for students of a range of different subject areas,
both sciences and arts (Phillips, 2020), and both are designed with the EntreComp
Framework in mind (Bacigalupo, 2016). They also have one piece of course work as the
assessment – an individual project – to be completed by the end of the final week of the
course (See Table 1).
Table 1: Similarities and differences between first year and third year course analysed.
Details First Year Third Year
Format Once session per week of two Once session per week of two hours,
hours, one hour lecture with one lecture with class discussion intermixed
hour class activity (11 weeks) (11 weeks)
Assessment 2000 Word Report due at the end of 4000 Word Report, due at end of course
course submitted via the submitted via the Blackboard Platform
Blackboard Platform
Class size 120 34
Material on Lecture Slides, Submission Portal, Lecture Slides, Submission Portal,

Blackboard Site lecture supplementary material, lecture supplementary material, access
access to podcast site to podcast site
2. Methodology
Quantitative data was collected from the Blackboard software with the first-year course and
third year (both semester one 2022) used as case studies. The class of first years was 120
students whilst class 30011 was 34. Data was collected from activation of the site on 15th
September and ended on 25th January when the final student had submitted coursework. Both
courses were chosen for their similarity – both were 100% coursework with a written report
required at the end of the course. Both had a mixture of students from both science and arts
subjects. It was expected that third years would have a higher average time spent using the
site as they have experienced more online learning during lockdown. It was also hypothesized
that third years would spend more time using the site as third year counts for a
disproportionate amount of the marks for their degree classification and would engage with
the material whilst writing their assignments. It was also hypothesized that the highest marks
would be obtained by those who spent more time on the platform.
450
Robert A. Phillips

Table 2 shows the number of hours of engagement with the online platform by students of
each course and a breakdown by day of the week and average numbers of hours of
engagement per student.
Table 2: The hours of engagement on the Blackboard Platform for each of the courses.
Day of the Week First Year Course Third Year Course
Monday 133.14 183.76***
Tuesday 137.27 50.25
Wednesday 471.86*** 56.95
Thursday 148.55 62.64
Friday 126.28 68.77
Saturday 46.71 18.98
Sunday 78.30 46.39

Total Engagement hours 1142.65 487.75
Ave engagement hours per student 8.46 9.20

*** = the course was timetabled on this day.
It was expected that third years would spend considerably longer on the platform on average
as third years the marks more important for final degree classification. Interestingly, for both
courses, engagement was three-fold higher on the day of the lecture which could be explained
by the observation that many students take a laptop to class and follow the lecture using the
online slides. Despite the assignment deadline being on a Friday, there was no particular
spike in usage for either course at that time, this could be explained by students being more
prepared as draft versions of their assignment were given formative feedback during the
course, this hopefully meant fewer students leaving work until the last minute. There was a
fairly similar profile for both courses with Saturday unsurprisingly the day of least usage.
Table 3: The percentage of time spent on each area of the Blackboard Platform.
Area of the Platform First Year Course Third Year Course
Course Content 64.50 80.18
Course Information 7.88 2.95
Podcasts 3.15 0.76
Coursework Submission Area 20.47 3.35
451
The data in Table 3 shows that the first-year students spend considerably more time on the
submission area for the coursework than the third years, this could be because the online
submission system is new to them and they are more careful to read instructions and get the
submission correct (as failure to submit has big penalties for students’ marks). Figures 1 and
2 plot marks obtained by the students against time spent on the online platform. The trendline
for each is strikingly similar, with an approximate 15% increase in mark between students
with the lowest engagement and the highest. There is however clearly a wide spread of
datapoints with some very high marks gained by some students who had little engagement –
this could perhaps be explained by those students being keen attenders of the face-to-face
sessions having little need to catch up online.
100
90
80
70
60
Marks %
50
40
30
20
10
0
0 5 10 15 20 25 30 35
Hours Spent using the Blackboard Site
Figure 1. Marks obtained vs Hours spent using the Blackboard site for the First Year course.
4. Conclusions and further work

Although the trendline for both courses show a correlation between increased usage of the
site and higher marks, the distribution of data points on graphs 1 and 2 suggest the
relationship between use of the online platform as a supplementary resource and achievement
of high marks is complex. This work can be expanded to a larger study where online habits
of students can be established with more courses and a bigger sample size. Due to the
individual nature of students preferred learning styles, qualitative research examining in more
depth the usage of the site and how it might correspond to lecture attendance (for this instance
it was not possible to do) can also be undertaken. It would also be useful to know if students
were using the platform out of choice or because they were unable to attend lectures due to
an understandable reason such as doing paid work or caring responsibilities. It might also be
of interest to distinguish any differences between, for example, arts and sciences students.
452
Robert A. Phillips
90
80
70
60
Marks %
50
40
30
20
10
0
0 5 10 15 20 25 30 35
Hours Spent using the Blackboard Site
Figure 2. Marks obtained vs Hours spent using the Blackboard site for the Third Year course.
References
Adedoyin, O. B., & Soykan, E. (2020). Covid-19 pandemic and online learning: the
challenges and opportunities. Interactive learning environments, 1-13.
Ajzen, I. (1980). Understanding attitudes and predicting social behavior. Englewood cliffs.
Adnan, M., & Anwar, K. (2020). Online Learning amid the COVID-19 Pandemic: Students'
Perspectives. Online Submission, 2(1), 45-51.
Bacigalupo, M., Kampylis, P., Punie, Y., & Van den Brande, G. (2016). EntreComp: The
Entrepreneurship Competence Framework. Luxembourg: Publication Office of the
European Union; EUR 27939 EN; doi:10.2791/593884
Bolliger, D. U., Supanakorn, S., & Boggs, C. (2010). Impact of podcasting on student
motivation in the online learning environment. Computers & Education, 55(2), 714-722.
Chapman, P., & Phillips, R.A. (2022). Entrepreneurial fear of failure: An international
comparison of antecedents and impact on venture creation. Journal of the International
Council for Small Business 3(4), 281-29.
Davis, F.D. (1989). Perceived usefulness, perceived ease of use and user acceptance of
information technology. MIS Quarterly, 13(3), 319-339.
Dhawan, S. (2020). Online learning: A panacea in the time of COVID-19 crisis. Journal of
educational technology systems, 49(1), 5-22.
DiRamio, D., & Wolverton, M. (2006). Integrating learning communities and distance
education: Possibility or pipedream? Innovative Higher Education, 31, 99-113.
Duval‐Couetil, N. (2013). Assessing the impact of entrepreneurship education programs:
Challenges and approaches. Journal of small business management, 51(3), 394-409.
453
Ferri, F., Grifoni, P., & Guzzo, T. (2020). Online learning and emergency remote teaching:
Opportunities and challenges in emergency situations. Societies, 10(4), 86.
Kuratko, D. F. (2005). The emergence of entrepreneurship education: Development, trends,
and challenges. Entrepreneurship theory and practice, 29(5), 577-597.
Liguori, E. & Winkler, C. (2020). From offline to online: Challenges and opportunities for
entrepreneurship education following the COVID-19 pandemic. Entrepreneurship
Education and Pedagogy, 3(4), 346-351.
Owston, R., Lupshenyuk, D., & Wideman, H. (2011). Lecture capture in large undergraduate
classes: Student perceptions and academic performance. The Internet and Higher
Education, 14(4), 262-268.
Papadopoulou, K.A., & Phillips, R.A. (2019). A comparison of on-curricular and off-
curricular activities in enterprise education for postgraduate students. 5th International
Conference on Higher Education Advances (HEAd’19), 55-64.
Phillips, R.A. (2022). Comparing Online and Physical Social Entrepreneurship Training:
Lessons Learned Moving Online. 8th International Conference on Higher Education
Advances (HEAd'22), 977-984.
Phillips, R.A. (2020). A Comparison of students’ attitudes and attainment on an enterprise
module for scientists and engineers. 6th International Conference on Higher Education
Advances (HEAd’20), 17-24.
Phillips, R.A. (2017). Measuring Entrepreneurial outcomes from a residential enterprise
school for postgraduate researchers. Journal of Asia Entrepreneurship and Sustainability
13(2), 62-89.
Phillips, R.A. (2010). Encouraging a more enterprising researcher – the implementation of
an integrated training program of enterprise for Ph.D. and postdoctoral researchers.
Research in Post Compulsory Education 15(3), 289-299.
Phillips, R.A. (2008). The use of personal journals for assessment of an undergraduate
Enterprise module. Journal of Asia Entrepreneurship and Sustainability, 4(4), 77-89.
QAA (2018). Enterprise and Entrepreneurship Education: Guidance for UK Higher
Education Providers Available Here: https://www.qaa.ac.uk/docs/qaas/enhancement-
and-development/enterprise-and-entrpreneurship-education-
2018.pdf?sfvrsn=15f1f981_8
Sanchez-Romaguera, V. & Phillips, R.A. (2018). A comparison of a first and final year UG
enterprise unit: lessons from experiential learning and interdisciplinarity. 4th
International Conference on Higher Education Advances (HEAd’18), 899-907.
Traphagan, T., Kucsera, J. V., & Kishi, K. (2010). Impact of class lecture webcasting on
attendance and learning. Educational technology research and development, 58(1), 19-
37.
Watermeyer, R., Crick, T., & Knight, C. (2022). Digital disruption in the time of COVID-
19: Learning technologists’ accounts of institutional barriers to online learning, teaching
and assessment in UK universities. International Journal for Academic
Development, 27(2), 148-162.
454
Digital multitasking during academic lectures: did the Covid-19

lockdown change the students’ behavior?
Rebeca Macedo Mendes1, Ana Isabel Veloso2
1
Department of Education and Psychology and Department of Communication and Art,
University of Aveiro, Portugal, 2DigiMedia, Department of Communication and Art,
University of Aveiro, Portugal.
Abstract
This paper explores digital multitasking during lectures for university students,
attending to the impact of the Covid-19 pandemic. Digital multitasking is
conceptualized in this research as engaging in a digital task (T1) during
academic lectures (T2). This paper has three main goals: 1. Analyze the
variations in digital multitasking in three different moments – pre-pandemic
(M1), during the lockdown (M2), and after the Covid-19 pandemic (M3); 2.
Explore the purpose by which students engage in digital multitasking –
exclusive educational (ED), exclusive non-educational (NED), and combined
(COMB), combining educational and non-educational purposes; 3. Explore
the digital resources – between 16 categories – used in multitasking during
lectures and their relationship with the different purposes – ED, NED, and
COMB. This quantitative research appeals to the survey and descriptive,
comparative, and correlational analysis. Results show that multitasking during
lectures is expected among university students and has changed with Covid-19
to a more educational purpose.
Keywords: Multitasking; digital resources; digital multitasking; Covid-19
pandemic; formal educative context; university students.

Digital multitasking during academic lectures
1. Introduction
The present article focuses on the good practices driven by the experience acquired during
the Covid-19 pandemic to bring some light to educational changes. As technological
development and digital resources have been the primary support during the pandemic and
demanded the development of teaching and learning processes at a distance worldwide, it
seems relevant to understand better how this might have changed individual habits in the
formal educative context.
2. State-of-the-art
Technological integration requires a diversified collection of learning strategies supported by
a learning management system (Tran & Meacheam, 2020), having multiple advantages –
enhanced communication, network collaboration, quick and easy sharing, permanent access
to materials, alternative types of resources, learning support, and technical skills (Legaree,
2015). Learning happens when the subjects establish connections between the information
acquired from their learning networks, encompassing numerous sources and technological
resources (Siemens, 2004). Without educational guidance and support during distance
learning, regulating the learning process becomes essential for achieving learning goals
(Kizilceca et al., 2017). Self-regulated learning skills – conscious thoughts, feelings, and
behaviors regarding management and guidance to achieve personal goals (May & Elder,
2018) – are traduced by the relationship between cognitive, metacognitive, and motivational
components during learning (Karlen, 2016). Systematical differences in students’ efficiency
are related to self-regulation strategies traduced by the understanding and application of
metacognitive resources, appealing to active control over the learning process (Karlen, 2016).
With the spread of access to media, concerns have been growing about the potential negative
impact of digital multitasking on students’ academic performance (Alvarez-Risco et al.,
2020). Multitasking involves simultaneously engaging in more than one task (Lau, 2017),
and when appealing to technological devices, it’s known as digital multitasking (Judd, 2013).
Leysens et al. (2016) and May & Elder (2018) found negative relations between digital
multitasking during lectures and academic performance. Wu (2017) found negative
relationships between perceived attention problems and self-regulation strategies. During the
Covid-19 lockdown, digital multitasking negatively influenced self-efficacy (Alvarez-Risco
et al., 2020). In opposition, Legaree (2015) found no significant relationship between media
multitasking and academic performance. Xu et al. (2016) noticed that digital multitasking
could positively influence students’ well-being, acting as a stress management strategy
regarding the demandingness of academic responsibilities (Xu et al., 2016). The nature of the
tasks encompassed might explain the divergent evidence regarding digital multitasking, the
motivational factors of each student, the perceived ability to achieve goals, moderators of
456
attention, digital multitasking, and studying forms (May & Elder, 2018). In sum, digital
multitasking relies mainly on context, individual metacognitive skills, and self-regulation
abilities (May & Elder, 2018; Xu et al., 2016).
3. Methodology
This paper explores digital multitasking during lectures among university students, attending
to the impact of the Covid-19 pandemic. Digital multitasking is conceptualized in this
research as engaging in a digital task (T1) during academic lectures (T2).
This paper has three main goals:
1. To analyze digital multitasking between three moments – before Covid-19 (M1),
during lockdown (M2), and after the pandemic (M3);
2. To explore the purpose by which students engage in digital multitasking – exclusive
educational (ED), exclusive non-educational (NED), and combined (COMB),
combining educational and non-educational purposes;
3. To explore the digital resources (16 categories) used in multitasking during lectures
and their relationship with the different purposes – ED, NED, and COMB.
The quantitative study developed had data collected through the survey by an online
questionnaire where the university students self-assess themselves in three moments,
followed by descriptive, comparative, and correlational analysis regarding digital
multitasking during academic lectures.
Three main dimensions are considered:
A. Digital resources: 16 categories of digital resources that can be appealed as parallel
tasks (T1) during lectures (T2);
B. Moments: three different moments where students have assessed themselves – pre-
pandemic (M1), lockout period (M2), and post-pandemic (M3);
C. Purpose: three different purposes (intentions) underlying the use of a parallel digital
task during lectures – exclusive educational (ED), exclusive non-educational
(NED), and combined (COMB), combining educational and non-educational
purposes.
The categories of digital resources (A) encompassed were the following: (A1) store & share
– store and share data; (A2) actuality – newspapers, journals, or others; (A3) commerce –
buying and selling products and services; (A4) audio communication – exclusive audio
communication; (A5) written communication – exclusive text communication; (A6) video
communication – exclusive real-time video communication; (A7) education – online
educational resources; (A8) entertainment – digital games, entertainment channels, and
applications; (A9) online office tools – document creation resources; (A10) management &
457
organization – calendars, agendas, notes, checklists, and similar; (A11) language – online
translators, dictionaries, text correctors, and similar; (A12) multimedia – photo, music, and
video edition tools, programs, or applications; (A13) music – listening to music online; (A14)
orientation – maps, guides, or similar; (A15) knowledge & research – browsers, wikis, and
databases; (A16) social networks – personal and professional social networks.
The study hypothesizes the following:
H.1. Students use (U) more (frequency) categories of digital resources during lectures in
the Covid-19 lockdown (M2) moment than before (M1) and after (M3) this period
(U: M1<M2>M3);
H.2. Digital multitasking (T1) during lectures (T2) has a more educational purpose (ED
and COMB) in the lockdown moment (M2) and after the Covid-19 pandemic (M3)
than in the pre-pandemic (M1) moment (ED: M1<M2 and M1<M3; COMB:
M1<M2 and M1<M3).
The quantitative data was submitted for descriptive, comparative, and explorative analysis.
The analyses focused on: i) the frequency of use (U); ii) the purpose (ED, NED, and COMB)
of using (U) each digital resource category (A) at each moment (M1, M2, and M3). It was
also intended to verify any relationships between the categories (A) and the purposes (ED,
NED, and COMB).
The population is university students. Participation criteria included: being an adult (≥18
years old); informed consent; currently a university student; having been a student since
2019/2020; having internet access since before the Covid-19 pandemic (2019); Portuguese
or English language. Exclusion criteria relied on the non-observance of one of the inclusion
criteria. This study collected quantitative data through the survey by appealing to an online
questionnaire, ensuring participation and ethical criteria, and providing data security. The
instrument encompassed five parts: informed consent; sociodemographic characterization;
before Covid-19 (M1); during the lockdown (M2); and after the pandemic (M3). Participants
assessed themselves regarding the use – did use (U) versus did not use (NU) – and the purpose
– ED, NED, and COMB – of each of the 16 categories of digital resources at each moment –
M1, M2, and M3. The data was collected between the 1st and 10th of December 2021. Data
treatment appealed to SPSS from IBM and Excel from Microsoft.
4. Results
The sample of this study has 44 university students (N=44). Regarding the sociodemographic
characterization of the sample: gender identification – 63.6% women, 34.1% men, and 2.3%
other; age range – between 18 and 47 years old (M=23.48); nationality – most of them are
Portuguese (61.4%), with (4.4%) Iranian, Brazilian and Mexican, and the rest (38.14%) not
458
responding; current academic level – 2.3% pre-graduation, 63.6% graduation, 25% master
degree, and 9.1% doctoral degree.
In the first analysis, the frequency of engaging with a digital resource during lectures – did
use (U) versus did not use (NU) in each category – was observed between the study’s
moments (M1, M2, and M3). Before Covid-19, 83% of university students used (U) digital
resources during lectures. During lockdown (M2), digital multitasking increased by 3%
(86%). After the pandemic, the use (U) decreased by 4% (82%). Interestingly, digital
multitasking was frequent before the Covid-19 pandemic (M1). Digital multitasking was
more frequent during the lockdown period, as expected (H.1). However, it was less frequent
in M3 than in M1, only confirming part of this hypothesis. A detailed analysis regarding the
frequency variation of each category (A) between the moments is presented in Figure 1.
Figure 1: Graphical (line chart) and descriptive (table) analysis of the percentual differences in the use (U) of
each category of digital resources (A) between the three moments – before Covid-19 (M1), during lockdown (M2),
and after the pandemic (M3).
When comparing the frequency of digital resources’ use in multitasking behavior during
lectures, higher variations occurred between the moments in written communication (A5),
video communications (A6), store & share (A1), commerce (A3), education (A7), and office
tools (A9). In a second analysis, we have focused on the purpose by which university students
engage in digital multitasking during lectures (Figure 2).
459
Figure 2: Percentual distribution (circular graphics) and absolute frequencies (table) of the purpose – exclusive
educational (ED), exclusive non-educational (NED), and combined (COMB) – regarding digital multitasking
during lectures, within the different moments – before Covid-19 (M1), during lockdown (M2), and after the
pandemic (M3).
The frequency of university students using (U) digital multitasking during lectures has
increased between M1 and M2 – with a lower percentage of “did not use” condition (NU) in
M2. This percentage is altered again between M2 and M3, with M3 showing a higher
frequency of not using (NU) digital multitasking during lectures. Educational purpose (ED)
frequencies of digital multitasking during lectures remained constant during the three
moments (M1, M2, and M3). The non-educational purpose was higher in digital multitasking
during lectures before Covid-19 (M1=25%) than during lockdown (M2=19%) or after the
pandemic (M3=17%). Combined purpose has an increased frequency in M2 (53%) when
compared to M1 (44%), which seems to remain in M3 (51%). These results confirm the
second hypothesis (H.2) only partially – although exclusive educational purpose (ED) didn’t
seem to variate over time, the combined purpose (COMB) has increased during the pandemic,
and exclusive non-educational purpose (NED) has decreased, with that trend being
maintained after the pandemic (M3). It was also intended to test if the variance observed was
significant. To see which categories (A) contribute to the different purposes, correlations
between them and the purposes (ED, NED, and COMB) were conducted (Table 1).
460
Table 1: Non-parametric correlations between the categories of digital resources (A) and the
purpose of engaging in digital multitasking during lectures – exclusive educational (ED),
exclusive non-educational (NED), and combined (COMB).
Purposes
Categories (A) ED NED COMB
1 Store & share np=-.028 p=.819 np=-.198 p=.103 np=.244 p=.041*
2 Actuality np=-.273 p=.024* np=-.175 p=.145 np=.340 p=.004**
3 Commerce np=-.089 p=.459 np=-.327 p=.006** np=.263 p=.025*
4 Audio communication np=-.253 p=.051 np=-.196 p=.101 np=.399 p<.001***
5 Written communication np=-.184 p=.140 np=-.161 p=.193 np=.332 p=.006*
6 Video communication np=-.050 p=.679 np=-.084 p=.487 np=.287 p=.015*
7 Education np=.089 p= .483 np=-.221 p=.079 np=.219 p=.076
8 Entertainment np=.029 p=.812 np=-.278 p=.021* np=.323 p=.006**
9 Office tools np=.006 p=.965 np=-.376 p=.003** np=.342 p=.006**
10 Organization np=-.078 p=.517 np=-.287 p=.017* np=.302 p=.011*
11 Language np=.003 p=.980 np=-.246 p=.046* np=.418 p<.001***
12 Multimedia np=-.192 p=.144 np=-.174 p=.148 np=.482 p<.001***
13 Music np=-.083 p=.494 np=-.277 p=.023* np=.384 p=.001**
14 Orientation np=-.069 p=.566 np=-.252 p=.034* np=.397 p<.001***
15 Research & Knowledge np=-.156 p=.211 np=-.297 p=.017* np=.441 p<.001***
16 Social Networks np=-.128 p=.290 np=-.342 p=.005** np=.437 p<.001***
Note: * (p<.050), ** (p<.010), *** (p<.001).
These results are evidence regarding the educative use of digital resources during lectures
(H.2) because they show that most of the categories have an inverted relation with the non-
educational purpose (NED), meaning that the increase in the use (U) of those digital resources
during lectures determines the reduction of non-educational purpose (NED) in a significant
way. Besides, significant positive correlations were found between all categories (except for
educational resources) and combined purpose (COMB).
5. Discussion
No study seems to have previously attended to the purpose of digital multitasking behavior
during lectures among university students. It was verified that digital multitasking is a
common trend and that research in this field is relevant as the learning processes are changing
to a more demanding, versatile, and continuous search for information in our daily lives
(Siemens, 2004). Besides, digital multitasking has been related to poorer academic
performance in some studies (Alvarez-Risco et al., 2020; Leysens et al., 2016) and seems to
have changed since the Covid-19 pandemic, supporting the need for more research in this
field. During lectures, university students seem to use that behavior not only as a distraction
factor, as suggested by Leysens et al. (2016), but also with a more educational purpose,
although combined, as we have observed. However, it is important to notice that digital
multitasking might only be productive if the demands are not competing for the same
cognitive resources to respond to both tasks in parallel, with digital multitasking requiring
461
individual self-regulation abilities and high executive processes to be effective (Karlen, 2016;
May & Elder, 2018).
6. Conclusion
This study provides exciting results. Although university students seem to resort more to
digital multitasking during the lockdown, this behavior has decreased to inferior levels to the
pre-pandemic moment more recently. The categories of digital resources that contribute to
reducing the non-educational purpose of digital multitasking during lectures might be the
ones that contribute to a more educational purpose, including commerce, entertainment,
office tools, organization, language, music, orientation, research & knowledge, and social
networks. The main limitations of this study were the small sample size (N=44) driven by
the high dropout rate (78%), mainly related to its length, and the method of data collection –
a self-assessment measure requiring the appeal to memory. Further research should introduce
self-management assessment measures and analyze digital multitasking between learning
contexts.
Acknowledgments
This paper was supported by the Digital Media and Interaction Research Center (DigiMedia),
Department of Communication and Art, University of Aveiro, Portugal. The data was
collected in collaboration with two colleagues – Francisca Lourenço and Sandra Paiva –
during the course of Multimedia and Cognitive Architectures, lectured by Prof. Dr. Luís
Pedro in the Ph.D. Program in Multimedia in Education at the University of Aveiro. The
credits from the data presented in this paper belong to the digital designer Tânia Ribeiro.
There are no conflicts of interest to declare.
References
Alvarez-Risco, A., Estrada-Merino, A., de las Mercedes Anderson-Seminario, M.,
Mlodzianowska, S., Politécnica Estatal Del Carchi, U., Cesar Villagomez-Buele, E., &
Espiritu Santo, U. (2020). Multitasking behavior in online classrooms and academic
performance: case of university students in Ecuador during COVID-19 outbreak.
Interactive Technology and Smart Education. DOI: 10.1108/ITSE-08-2020-0160.
Judd, T. (2013). Making sense of multitasking: Key behaviours. Computers and Education,
63, 358–367. DOI: 10.1016/j.compedu.2012.12.017.
Karlen, Y. (2016). Differences in students’ metacognitive strategy knowledge, motivation,
and strategy use: A typology of self-regulated learners. The Journal of Educational
Research, 103(3), 253–265. DOI: 10.1080/00220671.2014.942895.
462
Kizilceca, R. F., Pérez-Sanagustínb, M., & Maldonado, J. J. (2017). Self-regulated learning

strategies predict learner behavior and goal attainment in Massive Open Online Courses.
Computers & Education, 104, 18–33. DOI: 10.1016/j.compedu.2016.10.001.
Lau, W. W. F. (2017). Effects of social media usage and social media multitasking on the
academic performance of university students. Computers in Human Behavior, 68, 286–
291. DOI: 10.1016/J.CHB.2016.11.043.
Legaree, B. A. (2015). Considering the changing face of social media in higher education.
FEMS Microbiology Letters, 362(16), 128. DOI: 10.1093/FEMSLE/FNV128.
Leysens, J.-L., le Roux, D. B., & Parry, D. A. (2016). Can I Have Your Attention, Please?
An Empirical Investigation of Media Multitasking during University Lectures. DOI:
10.1145/2987491.2987498.
May, K. E., & Elder, A. D. (2018). Efficient, helpful, or distracting? A literature review of
media multitasking in relation to academic performance. In International Journal of
Educational Technology in Higher Education (Vol. 15, Issue 1). Springer Netherlands.
DOI: 10.1186/s41239-018-0096-z.
Siemens, G. (2004). Connectivism: a Learning theory for the digital age.
http://itdl.org/Journal/Jan_05/article01.htm.
Tran, T. P., & Meacheam, D. (2020). Enhancing Learners’ Experience Through Extending
Learning Systems. Transactions on Learning Technologies, 13(3), 540–551.
https://doi.org/10.1109/TLT.2020.2989333
Wu, J. Y. (2017). The indirect relationship of media multitasking self-efficacy on learning
performance within the personal learning environment: Implications from the mechanism
of perceived attention problems and self-regulation strategies. Computers & Education,
106, 56–72. DOI: 10.1016/J.COMPEDU.2016.10.010.
Xu, S., Wang, Z., & David, P. (2016). Media multitasking and well-being of university
students. Computers in Human Behavior, 55, 242–250. DOI:
10.1016/J.CHB.2015.08.040.
463
The role of financial aid in college performance: the importance of

class attendance, aid amount and type of aid
Isabel de Sivatte1, Patricia Gabaldón2
1
Department of Accounting, IE University, Spain, 2Department of Economics, IE University,
Spain.
Abstract
We use Social Identity Theory (SIT) and Tinto’s (1975) theoretical framework
on college dropout to argue how the provision of financial aid relates to
undergraduate students’ performance. Financial aid enables economically
less favored students to pursue high quality university education achieving
upward social mobility. We conduct this study using archival data of 4 cohorts
of business administration undergraduate students of an international, elite
university in Europe. We find that financial aid recipients obtain a higher first-
year GPA than non-recipients. This positive relationship is partially mediated
by class attendance. Financial aid recipients attend more classes, which also
enhances their GPA. Moreover, unexpectedly, we find that the relationship
between the amount of aid received and GPA is non-linear, and that merit-
based aid and need-based aid increase students’ GPA in a similar manner.
Keywords: Student performance in higher education; class attendance;
management of education; amount of financial aid; need- vs merit-aid.

The role of financial aid in college performance
1. Introduction
The role of financial aid in determining academic outcomes has been widely discussed and
researched. Previous research has focused on the impact of financial aid on academic
outcomes such as attendance to college, persistence in college, grade completion, or student
dropout (Cabrera et al., 1992; Allen, 1999; Kuh et al., 2008; Goldrick-Rab et al., 2016;
Bettinger et al., 2019). A few studies have examined the effect of financial aid on students’
grades (Kuh et al., 2008; Stater, 2009). However, grades are very important as they are strong
predictors of college persistence and completion (Stater, 2009). Also, while many theorize
that financial aid can improve student outcomes in college, there is still little evidence on
how aid influences the building blocks of student success. We attempt to fill this gap in the
literature by researching the impact of financial aid on students’ grade point average (GPA),
trying to understand the dynamics behind it. Relatedly, we add to prior work by examining
the distinct effects of two different types of aid, need-based aid and merit-based aid, on
college GPA.
Our main results indicate a positive impact of financial aid on students’ academic
performance – a result partially mediated by students’ attendance to class -, independently of
the type of financial aid received (merit versus need). Furthermore, we find a non-linear
relationship between the amount of financial aid received and GPA.
2. Theoretical framing and hypotheses development

Social identity, and more specifically social self-categorization, refers to the part of
individuals’ self-perception related to their membership to a social group (Tajfel, 1982).
Social Identity Theory (SIT) suggests that groups categorize their members between in-group
and out-group, based on salient differences, such as gender, race, ethnicity, the organization
they work for (Ashforth & Mael, 1989), or the college they study in (Lund Dean & Jolly,
2012). Individuals strive for upward social mobility, in their search for a more positive social
identity (Tajfel & Turner, 1979). Through financial aid, students coming from less
economically favored backgrounds may become members of the in-group of students at high
quality institutions, achieving upward social mobility.
Once membership into the in-group is achieved, it is important to maintain membership.
Tinto’s (1975) theory on students’ dropout from higher education, which has been widely
used to understand students’ performance and persistence in college, is useful to understand
the dynamics of membership maintenance. Positive social and academic experiences and
adequate academic performance at university would reinforce perceptions about the benefits
that derive from the completion of studies. Financial support, and adequate academic and
group integration would positively influence the decision to remain in the institution.
466
The combination of these two theoretical frameworks, SIT and Tinto’s (1975) theory of
dropout allows us to understand the dynamics that financial aid triggers when college
students receive it. Financial aid allows students to be part of the mainstream group and
motivates them to remain in the new group.
Being able to study at the academic institution we examine, and to join its international and
privileged student body, creates among students that receive financial aid a sense of gratitude,
and an incentive for better college performance.
Moreover, financial aid helps students be less anxious about how to finance their college
education and be more focused on their studies. Scholars have suggested financial aid can
lower this anxiety, leaving students to devote more time and energy to academic pursuits
(Bean & Metzner, 1985; Cabrera, et al., 1990), and therefore achieving more academic
integration. GPA is often reported in the literature as a proxy indicator for academic
integration (Cabrera, et al., 1992). Therefore, GPA may be positively related to financial aid.
Empirically, financial aid has been shown to be positively related to GPA in one meta-
analysis (Robbins et al., 2004) and in several studies (Kuh et al., 2008; Cabrera, et al., 1992;
Allen, 1999). Therefore, our baseline hypothesis is the following:
Hypothesis 1: Financial aid is positively related to college performance.
Financial aid increases students’ attendance to class due to several mechanisms. First, the
gratitude felt by financial aid recipients to be part of this institution may help them make
more efforts, which translate into higher class attendance. Second, financial aid recipients do
not face the same financial restrictions as students who do not have this support. Students
with financial aid can devote extra financial resources to engage in academic activities such
as attending class. Third, students receiving financial aid are usually subject to specific
performance standards. For example, in the institution examined, it is mandatory to attend
70% of sessions. Attending fewer sessions in any given subject translates into failing that
course, and into losing part or all of the financial aid received.
Hypothesis 2: Attendance to class mediates the financial aid – college performance
relationship.
In the present study, the amount of financial aid given to students is calculated as a percentage
of the tuition. Higher percentages of aid are given to more economically needy students. For
students who receive more financial aid, it would be more difficult to achieve membership
in this institution without that aid. Therefore, students who receive a higher percent of aid
will feel more gratitude about it than those who receive less, as it enables them to be part of
this international and prestigious university, to which they would not be able to access
without the aid. Therefore:
467
Hypothesis 3: The amount of financial aid given is positively and linearly related to college
performance.
All scholarship recipients in the examined institution received financial aid for economic
reasons. Indeed, they could not afford attendance without this financial aid. However, the
institution’s financial aid team classified the scholarships between those based solely on
financial need, labeled need-based aid, and those that are given to economically needy
students who also have strong academic records. We label the latter merit-based aid.
Merit-based aid recipients are more motivated and prepared to perform better academically
and to be more integrated academically (Tinto, 1975), as this was already the case when they
attended school. Also, due to their past academic success, merit-based aid recipients perceive
they are more academically capable and performant, which have been found to be positively
related to college performance (Richardson et al., 2012; Schneider & Preckel, 2017).
Therefore:
Hypothesis 4: Merit-based aid recipients have higher college performance than need-based
aid recipients.
3. Methods
3.1. Context and data collection

We use students’ data from one private, international university in Spain. Data is retrieved
from the university’s records of all students who start an undergraduate business
administration degree from 2014 to 2017 (n=1776). 55% of students are foreigners. Data is
gathered at different times. Most of it is collected during students’ admission process into the
university. Class attendance is collected during the degree, and the cumulative GPA at the
end of the first year.
3.2. Measures
GPA. We use students’ first-year GPA using the Spanish grading system.
Financial aid. It indicates the student received institutional financial aid, 0 signals he/she did
not. 34% of sampled students are financial aid recipients.
Attendance. It corresponds to the percentage of class sessions the student attends.
Type of aid. Need-based aid is given to students with financial necessity, and merit-based
scholarships to students who have both financial necessity and high grades in secondary
school. This variable takes 3 values: 2 if the student is a merit-based aid recipient, 1 for
recipients of need-based aid and 0 when the student did not receive financial aid.
468
Percent of aid. It is the amount of financial aid given, and it corresponds to a percentage of
the first year’s tuition. Regarding the requirements for aid renewal, the first year’s percent of
aid is applied in subsequent years unless students fail one or more subjects.
Controls. We control for students’ education system in their last year of high school, their
secondary education grades, the age at which they started the bachelor, their sex, the year
they enrolled into the bachelor, the location of the campus, whether the language of
instruction is always English or not, whether the student is a transfer from another university,
and the degree they study.
3.3 Analyses and results

The first hypothesis states that financial aid has a direct and positive association with first-
year GPA and the second one that class attendance mediates that relationship. Both
hypotheses were tested using Structural Equation Models (SEM). The direct effect is tested
entering financial aid and the controls as predictors of GPA. The indirect effects were tested
by entering financial aid and the controls as predictors of attendance, and attendance as
predictor of GPA. As shown (Figure 1), financial aid is related directly and indirectly
(through attendance) to GPA. Hence, hypotheses 1 and 2 are supported. Financial aid
recipients have on average 0.24 GPA higher than non-recipients, of which 35% is indirect
through attendance.
Figure 1. Path analysis of direct and indirect effect of financial aid on GPA.
The third hypothesis states that the amount of financial aid is positively and linearly related
to GPA. We test it by regressing the GPA on percent of aid (as well as its squared and cubic
terms) and the controls. Results show that the relationship is non-linear, when the percent of
aid increases from 5% to 43%, so does the GPA. Onwards, the GPA depicts a plateau.
Therefore, H3 is only partially supported (see Figure 2).
Hypothesis four states that merit-based aid recipients would have a higher GPA than need-
based aid recipients. We tested this hypothesis by conducting a regression of the of GPA on
469
the control variables and the type of aid. Results indicate that merit-based aid recipients’ GPA
does not differ from that of need-based aid recipients. Thus, H4 is not supported.
Figure 2. Relationship between amount of aid and GPA.
4. Discussion
In this study, we research the importance of financial aid in students’ college performance,
as a tool to integrate less privileged students into the main body of students, who do not
receive any financial aid. Our results confirm previous findings showing a positive
relationship between financial aid and college GPA (e.g., Kuh et al., 2008; Cabrera et al.,
1992; Allen, 1999). We have argued that thanks to the financial aid received, students
experience upward social mobility, and are especially grateful for being able to be part of an
international, prestigious, and selective university, which could explain the positive
relationship between financial aid and GPA. Although we have not conducted formal
qualitative research to understand the reasons for this relationship, in several informal
conversations with financial aid recipients, they express their gratitude for being able to study
at this university. This gratitude may translate into more commitment to the institution, which
has been found to be positively related to college performance (Schneider & Preckel, 2017).
Building on the second hypothesis, our results suggest that the financial aid-GPA relationship
is partially mediated by attendance to class, as 35% of the total effect is indirect. That is,
financial aid recipients attend more classes, which enhances their GPA. This mediation
suggests that financial aid recipients exert more effort (as they attend more classes) which
increases their performance. Previous research has shown that both effort and attendance to
class are related to performance in college (Schneider & Preckel, 2017). Also, thanks to the
scholarships, these students do not need to find alternative financial resources outside of
college, which may enhance their performance, as working during college is negatively
470
correlated to GPA (Kuh et al., 2008), and having more time to study is positively related to
college performance (e.g., Kuh et al., 2008; Schneider & Preckel, 2017).
Moreover, interestingly, and unexpectedly, we find that the amount of aid received is non-
linearly related to GPA. Financial aid and GPA are positively related for students who receive
from 5% to 43% of aid, which is the majority of students (80%). For the remaining 20%, who
receive a percentage of financial aid higher than 43%, the amount of aid seems unrelated to
GPA. The positive part of the relationship can be due to the gratitude these students feel to
be part of this group, and to the efforts they exert to renew the scholarship (so, to maintain
group membership). The part of the relationship where the amount of aid is unrelated to GPA
may be because students are already part of the main group, and that their level of gratitude,
effort or their grades cannot increase more. For example, scholarship is renewed when
students pass all subjects, that is, when they obtain a grade of at least 5 (out of 10) in each
course. The GPA at the turning point is 7.7. Once students realize how much effort is needed
to pass a course, they may figure out that it is not worth making more efforts to increase their
GPA, as these increases are not needed for financial aid renewal. Alternatively, a GPA higher
than 7.7 is hard to obtain in the Spanish college standards. In this data, only 30% of students
obtain a first-year GPA higher than this value. Therefore, even if they wanted to, financial
aid recipients may simply be unable to obtain a GPA higher than this value.
Also, contrary to our hypothesis, we find that the average GPA of merit-based aid recipients
is not different from that of need-based aid recipients. We expected that merit-based aid
recipients, as they are more prepared academically, would integrate academically and
socially better, and would therefore obtain better grades in college. However, we do not find
evidence for this. One potential argument can be that because their pre-college academic
ability is controlled for in the analyses by including their grades in secondary school, these
effects may be reduced. Furthermore, we can also claim that any student that qualifies for
financial aid, either merit-based or need-based, is somehow economically disadvantaged.
Research has found that family income is positively related to college performance (Kuh et
al., 2008; Richardson et al., 2012; Schneider & Preckel, 2017), and so, aid recipients’ GPAs
would be pulled downward. Hence, merit-based aid recipients’ GPAs would be pulled
upward by their stronger academic standing but downward by their lower income, which may
explain the non-significant GPA difference with need-based aid recipients.
This paper makes important contributions to theory. Framed within SIT, it shows the
importance of financial aid to students as a tool to reduce income inequality and enhance
social mobility by allowing grant recipients to be part of the main student body group. To
receive financial aid and reduce these income differences, students are incentivized to be
efficient in class, attending more sessions and achieving a level of GPA that allows them to
renew their grants every year. Financial aid is a key university tool to reduce students’
471
differences and to make less privileged students less salient in terms of income, which
incentives them to excel academically, and to attend more to class.
4.1. Implications for universities

The implications of these results for universities are of great relevance. First, as financial aid
is positively related to GPA, both students and institutions benefit from financial aid. For
institutions, providing scholarships is equivalent to investing in better academic performance
outcomes. Furthermore, class attendance is positively related to GPA. Therefore, monitoring
and encouraging class attendance will ameliorate students’ performance. Providing this
financial aid to students incentivizes them to attend more classes, which is beneficial for class
dynamics, faculty, and the whole student body.
Also, as shown, the amount of financial aid is non-linearly related to GPA. Universities need
to consider that beyond a certain amount of financial aid (beyond 43% of tuition in this study)
students do not exert more effort, and the GPA is not affected positively.
Finally, as noted before (Stater, 2009), need-based aid promotes more social class equality,
and merit-based aid promotes higher academic achievement at the universities. This study
suggests that both types of scholarships have a similar effect on GPA. Given this result, we
suggest that universities choose need-based over merit-based aid, as the university’s
academic performance will not be affected.
References
Allen, D. 1999. Desire to finish college: An empirical link between motivation and
persistence. Research in Higher Education, 40: 461–485.
Ashforth, B. E., & Mael, F. 1989. Social identity theory and the organization. Academy of
Management Review, 14: 20-39.
Bean, J. P., & Metzner, B. S. 1985. A conceptual model of nontraditional undergraduate
student attrition. Review of Educational Research, 55: 485-540.
Bettinger, E., Gurantz, O., Kawano, L., Sacerdote, B., & Stevens, M. 2019. The long-run
impacts of financial aid: Evidence from california’s cal grant. American Economic
Journal: Economic Policy, 11: 64–94.
Cabrera, A. F., Nora, A., & Castañeda, M. B. 1992. The role of finances in the persistence
process: A structural model. Research in Higher Education, 33: 571-593.
Cabrera, A. F., Stampen, J. O., & Hansen, W. L. 1990. Exploring the effects of ability to pay
on persistence in college. The Review of Higher Education, 13: 303-336.
Goldrick-Rab, S., Kelchen, R., Harris, D. N., & Benson, J. 2016. Reducing income inequality
in educational attainment: Experimental evidence on the impact of financial aid on
college completion. American Journal of Sociology, 121: 1762–1817.
472
Kuh, G. D., Cruce, T. M., Shoup, R., Kinzie, J. & Gonyea, R. M. 2008. Unmasking the effects
of student engagement on first-year college grades and persistence. Journal of Higher
Education, 79: 540–563.
Lund Dean, K., & Jolly, J. P. 2012. Student identity, disengagement, and learning. Academy
of Management Learning & Education, 11: 228–243.
Richardson, M., Abraham, C., & Bond, R. 2012. Psychological correlates of university
students’ academic performance: A systematic review and meta-analysis. Psychological
Bulletin, 138: 353–387.
Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. 2004. Do
psychosocial and study skill factors predict college outcomes? A meta-analysis.
Psychological Bulletin, 130: 261–288.
Schneider, M., & Preckel, F. 2017. Variables associated with achievement in higher
education: A systematic review of meta-analyses. Psychological bulletin, 143: 565–600.
Stater, M. 2009. The impact of financial aid on college GPA at three flagship public
institutions. American Educational Research journal, 46: 782-815.
Tajfel, H., & Turner, J. C. 1979. An integrative theory of intergroup conflict. In W. G. Austin,
& S. Worchel (Eds.), The social psychology of intergroup relations: 33-37. Monterey,
CA: Brooks/Cole.
Tajfel, H. 1982. Social psychology of intergroup relations. Annual Review of Psychology, 33:
1-39.
Tinto, V. 1975. Dropout from higher education: A theoretical synthesis of recent research.
Review of Educational Research, 45: 89-125.
473
Changing higher education governance in Latin America: the

cases of Chile and Ecuador
Centre for Comparative Education Policy, Faculty of Education, Diego Portales University,
Chile.
Abstract
This article examines recent changes in the governance of national higher
education systems in Chile and Ecuador. It focuses on the changes associated
with the different roles of the State/Government as a coordinating mechanism.
An analytical model is proposed that distinguishes between five roles of the
State/Government: system designer, principal, regulator, evaluator, and
funder. Recent reforms introduced in the sector by both countries are analysed
and their impact on the State/Government profile is compared. The results
show that the roles of system regulator, designer and evaluator are changing
in opposite directions because of the respective reforms. On the contrary, the
results show that both countries are moving in the direction of strengthening
the state/government roles as funder.
Keywords: Higher education; governance; Chile; Ecuador.

Changing higher education governance in Latin America: the cases of Chile and Ecuador
1. Introduction
Over the last few decades, the governance of national HE systems has been subject to
significant changes at a global level (Capano & Pritoni, 2020). These changes have redefined
roles and power relations between state, market, HE institutions (HEIs) and various external
stakeholders (Facchini & Fia, 2022). Indeed, HE governance reforms are acknowledged to
be influenced by global patterns associated with the main public sector transformation
streams: new public management (NPM), networked governance and neo-Weberian (Donina,
Meoli & Paleari, 2015). Moreover, the growing interest in studying HE governance has led
to increasing complexity in the conceptual and analytical approaches used to understand its
changing dynamics. Over the last two decades, the so-called governance equaliser approach
(de Boer, Enders & Schimank, 2008) - inspired by Clark's (1983) coordination triangle - has
gained increasing legitimacy among researchers in the field and has been used for the study
of HE governance transformations in several countries. See, for example, Donina, Meoli &
Paleari (2022). The results of these studies confirm the strong dominance of local contexts in
the direction of change, while showing a trend towards configuring hybrid HE governance
systems (Capano & Pritoni, 2019). In Latin America (LA), national governments of different
political tendencies have introduced adjustments in the governance of HE aimed at
strengthening the state-market axis. In this sense, a process of policy accumulation
(González-Ledesma & Álvarez-Mendiola, 2019) is recognised as a recent trend that, on the
one hand, increases the role of the state through increased public funding of the sector and
greater control over private HEIs, and, on the other, maintains public policies and regulations
that are recognised within the framework of academic capitalism (Brunner et al., 2021).
Indeed, it can be argued that changes in the governance of national HE systems in LA seek
to strengthen the role of the state as a counterbalance to the strong historical influence of
academic autonomy and self-governance on the one hand, and to the power of market forces
in deregulated contexts on the other. Recently, however, some national governments in LAC,
such as Chile and Ecuador, have undertaken important reforms that are changing the
trajectory of HE governance followed over the last decade. On the other hand, given the
growing diversity and complexity of national HE systems in LA, existing typologies -
especially those developed in the global North - have limitations in organising the complex
reality of their governance. Thus, their application in the region faces the challenge of
converging towards models of analysis capable of addressing the heterogeneity of national
HE systems and their ongoing processes of transformation. This article therefore has two
central objectives. First, to propose a conceptual and analytical approach to examine HE
governance from the perspective of the state's role as main system coordinator. Second, to
analyze the changes in the governance of national HE systems in Chile and Ecuador, focusing
on the similarities and differences between the two cases, based on the respective reforms
adopted in recent years. Its content is divided into four sections. First, it shows the conceptual
476
and analytical framework used to examine the case studies. Next, the methodology used is
displayed, including the data collection mechanisms and sources of information. It then
presents the major findings and finally draws the main conclusions.
2. Conceptual framework
To approach the complexity of the state's role as a coordinating force in HE, we turn to the
concept of multi-governance (Chou et al., 2017). This concept consists of four dimensions.
The first is related to its multilevel quality, which refers to the degree of concentration or
distribution of authority at different vertical levels of State/Government: national, regional,
and local. The second dimension relates to the multiple actors or stakeholders interacting in
the sector, such as state or governmental bodies (ministries, agencies, and others) and non-
governmental organizations, including HEIs associations, academics, business and labour,
students and their families. The third dimension relates to the multiple issues competing for
priority space on the public agenda. A fourth dimension is the multiple organizational and
coordinating arrangements that result from the interaction between the various forces of
State/Government, markets, and HEIs. Amid this complexity, the State/Government can play
different roles depending on the nature and emphasis of the public policies it promotes.
Indeed, we distinguish five different roles in the LA context, based on the orientation of the
oversight (behavior and/or outcomes) and various arrangements that the State/Government
uses: system designer, principal, regulator, evaluator, and funder. In the role of system
designer, the State/Government establishes the policies and norms that regulate the behavior
and relationships between the different actors that are part of the HE system. Indeed, it is
within this framework that the traditional academic autonomy and self-governance widely
recognized in LA take form. It is also through this design that the State/Government sets the
conditions for the participation of the private sector in the provision of HE. Next, in the
context of agency theory, the State/Government acts as a principal vis-à-vis HEIs, which act
as agents. In this model, the State/Government defines objectives and expected outcomes,
control mechanisms and accountability. As principal, the State/Government seeks to align
the performance of HEIs with the public policy priorities of the sector. This format has been
used in LA to guide the production of public goods in HE by both state and private
institutions. It is also argued that this coordination modality has installed a new form of
regulated autonomy. The third role adopted by the State/Government is that of regulator. In
this case, the State/Government defines, enforces, and monitors compliance with rules aimed
at regulating the functioning of markets in HE. Currently, a new regulatory landscape is
emerging in the global North after several decades of a trend towards deregulation (Capano
et al., 2020). Similarly, in LA, Rama (2006) observed this emerging phenomenon early on,
characterizing it as the latest wave of regulatory reforms in national HE systems. Fourth, we
distinguish the evaluative role of the State/Government, a function that the European
477
literature refers to as steering at a distance. By focusing on measurement, the

State/Government aims to increase the productivity, efficiency, quality, and effectiveness of
academic work at both institutional and individual levels. To this end, it deploys a set of
indicators and metrics that seek to align the outcomes of HEIs with the public policy priorities
of the sector. In fact, this role is increasing in LA countries, as part of national policies aimed
at the measurement and evaluation of outcomes. Finally, the fifth State/Government role is
associated with its role as funder. In this schema, funding is understood as more than a
mechanism for allocating resources to HEIs and students. Rather, it is a governance tool to
change behavior and maximize certain outcomes with limited resources. As such,
State/Government is oriented towards seeking greater effectiveness and efficiency in the use
of public resources by HEIs (Capano & Pritoni, 2019). In LA, there is a perceived shift from
block grants (allocated in an inertial, automatic, and discretionary manner) to formulas that
seek to influence the behavior and results of HEIs (García-Fanelli, 2019).
3. Method
To develop this study, we used the methodology of documentary research on secondary
sources, with two foci. Firstly, the literature on HE governance published over the last two
decades by the global north and LA scholarly communities was reviewed, serving as a basis
for the elaboration of the conceptual framework presented in the previous section. Secondly,
legislation, regulations, policy reports, and academic and grey literature related to the
national HE systems of Chile and Ecuador published over the last 20 years, and relevant for
the purposes of this study, were reviewed. Of particular interest were the recent HE reforms
introduced by the national governments of both countries. In the case of Chile, the Law on
HE (Nº20.091), published on 29 May 2018, and the Law on State Universities (Nº21.094),
published on 5 June 2018. Regarding Ecuador, the Organic Law published on August 2,
2018; and the Decree (Nº494) amending the Organic Law on HE (LOES) of Ecuador,
published on July 14, 2022. Each of these reforms was analysed through the lens of the five
roles of the State/Government. Next, each of the authors separately assessed the changes in
the roles of the State/Government in each country since the implementation of their
respective reforms. Both authors then shared the results of their analyses. Differences were
discussed until a consensus was reached.
4. Findings
In both countries, the State/Government has played an important role as a system designer,
albeit with different approaches and emphases at each stage and in each political context. In
the case of Chile, the State/Government has played an active role in the design of the HE
system, especially since the 1990s, seeking a balance between the State/Government, the
478
market and HEIs. However, after the recent reform of 2018, adopted during the
administration of Michelle Bachelet, the role of the State/Government as a system designer
became less important, along with a strengthening of its roles as regulator, evaluator and
financier, and the maturation of the system itself. Indeed, the State/Government has
strengthened and increased its influence in the management of public and private HEIs, which
has led to greater regulation and supervision of HE. Despite these changes, the
State/Government continues to be involved in setting up bodies and formulating policies and
strategies to guide the sector’s development. This is the case with the Under-Secretariat of
HE as the sector’s governing body, the Superintendence of HE as the supervisory body, and
the Quality Assurance Council as the guarantor of HE quality. In Ecuador, the
State/Government has gone through different stages in its role as system designer. During
the Rafael Correa administration, the State/Government assumed a strong regulatory role,
reducing the autonomy of HEIs and centralizing decisions in the central government
(Benavides et al., 2018). However, in subsequent reforms under the administrations of Lenín
Moreno and Guillermo Lasso, the State/Government has shifted its focus towards a more
diversified profile, strengthening its role as a designer of the rules of the game (policies,
norms, and instruments) that condition the relationships between different HE actors. In
particular, the changes have granted greater autonomy to HEIs and have also strengthened
the institutional framework of HE, creating inter-institutional and regional consultative
planning committees with greater participation of the various stakeholders and less influence
of the central government (Andrade, 2021). This indicates a more active role of the
State/Government as a designer of the system.
Similarly, the State/Government plays an active role as principal in both countries, but with
different degrees of relevance. In Chile, the State/Government has increased its influence in
the management of both State/Government and private HEIs through performance-based
contracts in priority areas. A notable example is the program to strengthen State universities
with 10-year development plans based on objectives and expected results approved by the
Ministry of Education. Although the State/Government remains an important factor in
defining objectives and accountability, its role as principal seems to have lost importance
compared to its roles as regulator, evaluator, and funder. Indeed, the State/Government's
influence on the objectives of HEIs has been complemented by other financial, evaluative,
and regulatory mechanisms, such as formulas for the distribution of resources, compliance
with quality criteria and standards, and new norms. In Ecuador, recent reforms have
strengthened the leading role of the State/Government, giving greater importance to the
definition of objectives and the accountability of HEIs and academics. An example is the
introduction of a new regulation to financially incentivize academics to carry out research
and development work, establishing monitoring and accountability mechanisms like a
contract between a principal and an agent. In addition, the reforms have put in place financial
incentives to improve the productivity of individual researchers.
479
Regarding the role of the regulatory State/Government, the two countries show different
trends. In Chile, it has increased its relevance in recent years, especially after Law
21.091/2018. This reform introduced a series of new regulations that explicitly establish the
mission of HE; it sets up new rules that increase the State/Government 's capacity to supervise
HE in terms of compliance with the law, especially the use of public resources; it establishes
new rules for access to and use of public funds for free education; and it determines the
obligation to accredit quality based on more demanding criteria and standards. On the other
hand, in Ecuador, the regulatory State/Government played a very strong role in the 2010
LOES reform, which established a high level of public regulations and reduced the autonomy
of HEIs through the implementation of regulations and the oversight function by bodies under
the control of the executive branch. However, subsequent reforms in 2018 and 2022 have
weakened the role of the State/Government as regulator and granted greater autonomy to
HEIs; for example, by simplifying procedures for creating new programs and making
curricular changes.
In addition, in both Chile and Ecuador, the State/Government has taken an active role as
evaluator in HE, albeit with different approaches. In Chile, the State/Government focuses on
compliance with criteria and standards, while in Ecuador it has moved towards a focus on
continuous improvement and greater involvement of HEIs in the evaluation process. In fact,
the Chilean State/Government has strengthened its role as evaluator with the creation of the
National Quality Assurance System, which is composed of different public bodies that should
establish a coordination plan, define, and coordinate criteria and standards for improving HE
quality. Law 21.091/2018 also introduces a new quality assessment model that emphasizes
the fulfilment of criteria (18 in the case of the university subsystem), each with three
performance levels. In effect, it creates an integrated and mandatory institutional
accreditation system that covers all academic functions, campuses, and programs. Thus,
according to the results obtained, HEIs are classified into three levels: basic, advanced and
excellence. Furthermore, new regulations also make mandatory the accreditation of doctoral
programs. In Ecuador, the role of the evaluative State has evolved from an approach based
on compliance with metrics to one focused on continuous improvement during the last decade
(Ponce & Intriago, 2022). The 2018 LOES reform established a new quality evaluation model
that distinguishes between accrediting and non-accrediting evaluation, ensures greater
involvement of HEIs in the evaluation process, and gives greater importance to self-
evaluation. It also emphasizes qualitative aspects and removes the previous categorization
based on compliance with metrics. The new model is binary, i.e., it determines whether a
HEI is accredited or not. Similarly, the latest reform of 2022 makes the annual evaluation of
the performance of individual academics in HEIs mandatory.
On the other hand, the funding State/Government role has been strengthened in recent years
in both countries. In Chile, Law 21.091/2018 increases public funding and introduces free
480
tuition for students from the six lowest income deciles. It also regulates fees per
undergraduate education program and limits the annual growth of new students. The Chilean
Law 21.094/2018 also institutes a specific amount of public funding to strengthen State
universities. These measures make HEIs more dependent on State/Government, thereby
restricting institutional decision-making autonomy in the use of resources. In Ecuador, the
LOES reform of 2018 introduced a formula mechanism to distribute public resources
according to results, based on teaching, research, extension and administrative indicators
(Ponce & Intriago, 2022). In addition, the new regulations allow universities to generate
private income through technical assistance and consultancy to strengthen their academic
projects.
5. Conclusions
This article contributes to the academic international debate on analytical models for
understanding changing State/Government roles in the governance of highly diverse and
complex national HE systems. A new model is proposed and applied to the Chilean and
Ecuadorean HE national systems, proving its usefulness for understanding changes in the
governance of HE in the LA context. The findings related to both case studies suggest four
conclusions. First, the results are consistent with previous research showing the strong
influence of the political, social, and economic national contexts on changes in HE
governance over a given period. Second, the results show that the reforms adopted since 2018
in both countries led to opposing changes in the role of the State/Government in regulation,
design, and quality evaluation. On the one hand, Chile increased regulations, reduced HEIs’
autonomy and moved from a quality assurance model focused on continuous improvement
to a compliance and classification system for HEIs. On the other hand, Ecuador reduces state
regulation, increases HEI autonomy and shifts its quality evaluation model from a logic of
compliance and categorising HEIs to a model of continuous improvement and binary
(accredited/non-accredited). Thirdly, in general, both countries maintain regulations and
policies aimed at strengthening the funding role of the State/Government and its role as
principal. Also in both countries, a formula is used as part of the allocating mechanism for
public resources and performance contracts are employed as a tool for aligning HEIs with
the objectives and results expected from public policies. Fourth, the analysis of changes over
the last decade shows that Ecuador has been more radical than Chile in its reforms related to
the role of the evaluative State/Government. While Ecuador has swung from one model to
another like a pendulum, Chile has done so gradually. Lastly, we suggest that the application
of this analytical approach can be useful for studying other Latin America and Caribbean
national HE systems.
481
References
Andrade, E. (2021). Universidad y Estado. La política estatal de reforma de la universidad
ecuatoriana (2007-2017). Tesis doctoral. Doctorado en Estudios Latinoamericanos.
Universidad Andina Simón Bolivar.
Benavides, M., Arellano, A. & Zárate Vásquez, J.S. (2019). Market- and government-based
higher education reforms in Latin America: the cases of Peru and Ecuador, 2008–2016.
High Educ 77, 1015–1030. doi: 10.1007/s10734-018-0317-3
Brunner, J. J., Labraña, J., Rodríguez-Ponce, E., & Ganga, F. (2021). Varieties of academic
capitalism: A conceptual framework of analysis. Education Policy Analysis Archives, 29
(January - July), 35. doi:10.14507/epaa.29.6245
Capano, G. & Pritoni, A. (2019). Varieties of hybrid systemic governance in European
Higher Education. Higher Educ Q. 73: 10– 28. doi: 10.1111/hequ.12180
Chou, Meng-Huan, Jungblut, J., Ravinet, P. & Vukasovic, M. (2017). Higher education
governance and policy: an introduction to multi-issue, multi-level and multi-actor
dynamics. Policy and Society 36:1, 1-15. doi: 10.1080/14494035.2017.1287999
Clark, B.R. (1983). The Higher Education System. Academic Organization in Cross-National
Perspective. Berkeley and Los Angeles: University of California Press.
de Boer, H. F., Enders, J., & Schimank, U. (2008). Comparing Higher Education Governance
Systems in Four European Countries. In N. C. Soguel, & P. Jaccard (Eds.), Governance
and Performance of Education Systems, 35-54. Springer. doi: 10.1007/978-1-4020-6446-
3_3
Donina, D., Meoli, M., & Paleari, S. (2015). Higher education reform in Italy: Tightening
regulation instead of steering at a distance. Higher Education Policy, 28(2), 215–234.
Facchini, C. & Fia, M. (2021). Public Sector Reform in Italian Higher Education: The
Governance Transformation of the Universities — A Comparison Among Perceptions of
Rectors and Department Chairs. High Educ Policy 34, 560–581.
García de Fanelli, Ana. (2019). El financiamiento de la educación superior en América
Latina: tendencias e instrumentos de financiamiento. Propuesta educativa, (52), 111-126.
González-Ledesma, M. & Álvarez-Mendiola, G. (2020). Neoliberals versus postneoliberals
in the formation of governance regimes in Latin America’s higher education. In Jarvis,
D. & Capano, G. (Eds.). Convergence and Diversity in the Governance of Higher
Education: Comparative Perspectives, 426–454. Cambridge University Press.
Ponce J., & Intriago, R. (2022). Capítulo Ecuador. En Informe de Educación Superior en
Iberoamérica 2022 (Unpublished manuscript).
Rama, C. (2006). La tercera reforma de la educación superior en América Latina y el Caribe:
masificación, regulaciones e internacionalización. Revista educación y pedagogía,
18(46), 11-24.
482
Resource orchestration and the higher education programme

director
Hans Frederik
Organization Sciences department of the Faculty of Social Sciences, VU University
Amsterdam, the Netherlands.
Abstract
With approximately half a million students, universities of applied sciences are
the largest providers of highly educated professionals in the Netherlands. The
programmes offered by these universities have a certain degree of policy
freedom when it comes to connection and knowledge exchange with
professional practice. Programme directors (PDs) are responsible for the
organization and development of the education programmes. In this role, they
have, within the institutional framework of the university, powers in the areas
of personnel, finance, quality assurance and planning. They have become the
central pivot in the organization of these programmes, and they are
responsible for optimizing the work processes and guaranteeing the quality of
the graduates. In this study, we interviewed 25 PDs how they use their
resources for innovation within their managerial frameworks in educational
organizations. This research shows that innovation requires room for
experimentation.
Keywords: University of applied sciences, resource orchestration, agency,
knowledge exchange, professional field, programme director.

Resource orchestration and the higher education programme director
1. Introduction
Universities are the driving force behind innovation and knowledge development in today’s
Western European societies. However, academics need an incentive to actively realize
knowledge sharing (Sormani & Rossano-Rivero, 2023). Stimulating innovation and
knowledge exchange are strongly influenced by the opportunities offered by the educational
institution (Cabrera & Cabrera, 2005). Programme directors (PDs) within a university are
responsible for the organization and development of the educational programmes. They have
become the central pivot in this, and they are responsible for optimizing the work processes
and guaranteeing the quality of the graduates (Westerheijden, 2022). In this study, we look
at how PDs use their resources for innovation within their managerial frameworks in
educational organizations.
2. Theory
Twenty-five directors were interviewed, spread over eleven educational institutions and
seven educational sectors. Nine women and sixteen men, all with more than five years of
experience in the management of a higher education institution, These PDs were interviewed
with open-ended questions about their experiences as managers, their successes, and their
failures over the past 4 to 5 years. In these interviews, attention was paid to topics such as
innovation in education, the relationship with the professional field, and the possibilities and
limitations that the university gave them as PDs. All higher educational sectors were
represented, with the exception of the art education sector.
We looked at awareness of institutionalized habits and routines (Emirbayer & Mische, 1998),
ability to identify problems in current institutional arrangements (Battilana et al., 2009) and
the ability to conceptualize alternative outcomes (Emirbayer & Mische, 1998). Central were
the institutional aspects of formal authority, including the actor’s right to make decisions
(Hardy & Phillips, 1998) and access to the financial costs of change (Greenwood et al., 2002).
We addressed resource orchestration by linking value creation in dynamic environmental
contexts to management resources (Sirmon et al., 2011). The components of the resource
management model included structuring the resource portfolio, bundling resources to build
capabilities, and leveraging capabilities to provide value.This document could be used as a
template for formatting the papers. All texts, figures and tables must be included within the
document margins.
2.1. Resource orchestration

A review of the literature showed that most papers that measured resource orchestration were
limited to commercial entrepreneurship and shed little light on the dynamics of commercial
enterprises (Ghalwash & Ismail, 2022). Resource orchestration processes that explain how
484
Hans Frederik
opportunities are created, operationalized and legitimated in the institutional environment of

universities have received less attention (Owusu & Janssen, 2013). The goal of this study
was to explore these concepts to show how in higher education, middle managers in an
institutional environment overcome resource constraints and achieve value through
innovative resource orchestration (Ireland et al., 2003; Sirmon et al., 2011).
2.2. Institutional entrepreneurship

Institutional entrepreneurship theory highlights how actors with sufficient resources see
opportunities to realize interests that they value highly” (DiMaggio, 1988, p. 14). Institutional
theory is characterized by adherence to a given way of acting – of doing things. This is
streamlined within predetermined frameworks. When actors want to act in a change-oriented
way, they will act outside these bandwidths. Thus, a contradiction arises. The question here
is how, within the usual course of action, change-oriented action takes place or, even better,
is conceived of at all? We used this change-oriented attitude to subdivide the ways in which
PDs use their powers and capabilities to act innovatively or to adopt a more administrative
attitude.
3. Method
The 25 interviews, each at least an hour, were transcribed verbatim, stripped of social talk,
anonymized and then coded based on Sirmon et al. (2011). ATLAS.ti was used for the coding
(Figure 1). Where the quotes are from an interview, reference is made to a letter and a
number. The letter stands for an education sector: E for education / teacher training, H for
health care, A for agriculture, M for management, S for social sciences and T for technology
/ IT. The numbering is consecutive.
The following concepts were coded: structuring, bundling or leveraging available resources,
looking for new opportunities and combining them with development opportunities, and
integrating identified resources to increase effectiveness or efficiency. In addition, search
terms such as profession, professional field, knowledge exchange, company, contacts,
knowledge, knowledge innovation, and relationship were used. This coding was arranged in
paragraphs to give an initial picture of the possibilities that PDs saw for maintaining and
renewing their study programmes.
485
30 Structuring Bundling Leveraging

25
20
15
10
5
0
g
ge
es
e
l
oo
st
tie
ce
tie
rin
iliz
iti
ve
ra
lp
ur
ni
ni
i
ab
n
qu
ve
Di
so
tu
rtu
rtu
na
St
le
Ac
or
re
er
po
po
to
p
t
e
op
op
In
op
at
oy
gr
e
h
te
pl
Us
ric
te
De
ea
In
En
Cr
Administrative-oriented Innovation oriented

Figure 1. Resource orchestration categories based on Sirmon et al. (2011).
4. Results
These results provide a picture of how PDs used their duties and powers. The numbers on the
y-axis are percentages of the total number of statements made by the PDs, ranked according
to the reasoning of Sirmon et al. (2011).
The left (blue) columns reflect the answers of the PDs whom we recognized as
administrative-oriented, in accordance with DiMaggio (1988). The right (green) columns
reflect the statements of PDs recognized as aiming for innovation. This distinction was the
result of the first coding in which the following question was distilled from the interviews:
How do the relevant PDs view the possibilities and limitations that were given by their
educational institute? (Tiberius et al., 2020).
All PDs indicated that innovation or modernization in relation to the professional field was
of great importance to them (H6: “I am manager of Education & Innovation, and that word
says it all – to really look from that point of view. What is needed? What is that dot on that
horizon?” S3: “We are doing several innovations – curriculum revisions anyway – making
crossovers in the context of interprofessional learning and working.” T6: “And whatever we
started doing, that was also quite innovative – was a kind of open maker space type of thing”).
Differences became visible in the extent to which directors saw opportunities to actually get
started. The connection with the professional field was often guiding them (S3: “We have
486
Hans Frederik
done all kinds of sessions with the professional field: What do you think it should go to? So,
we had a professional field committee there, which was closely involved there, but we also
had professional field sessions around it to reach a wider group.”) or at least stimulating them
(H4: “And on a professional level, we went together very well.”). A limited number of
directors related innovation and development to putting the standing organization in order.
In one case, a director who showed their year plan received feedback, but where was the
innovation? (M3: “And I remember very well that I once gave a faculty-wide presentation in
which I thought I was giving a reassuring message to everyone. And at that presentation,
there was a lecturer who was like, OK, now this is the message, and where is the
innovation?”)
A second notable feature was that directors received little support from the institutional
organization for their development and renewal activities, although there was sufficient
expertise available (S2: “The institutional environment, the policy documents that were
available were of a high level. But then policy was not binding or directive. And do you just
have almost a complete mandate to relate to it according to your own insights?”). It was noted
that the slowness and lack of clarity of institutional decision-making unnecessarily limited
the success of the innovation (E3: “Well, and then the way in which leadership is given. Just
to name a few things, I have a new study programme, actually a merger of three study
programmes, which is still very small, but then it will be half a year before a decision is made
about it.”).
It was striking that a similar approach, but without external support sources, was described
as very successful in one case (H1: “I noticed the new concept was really successful. That
meant that we really had a very high student satisfaction during the first 2 years. And also,
that employee satisfaction was very high, and absenteeism decreased.”) but as failed in
another case (T1: “At a certain point, the resistance is no longer manageable, and at that
moment, you also see that the gentlemen’s agreement between the university of applied
sciences and the business community no longer works.”), although the ambition was
supported at the highest institutional level. In terms of the latter, the lead time, due to a change
of personnel at the higher decision-making levels, clearly played a role.
Truly innovative and more or less disruptive innovation occurred in one case where, based
on the research input of a researcher, the educational vision was prescribed and adopted for
the entire university of applied sciences over time (E1: “Our educational concept is being
introduced in phases. I’ve been to expert meetings, and it’s much more a matter of time. If
you look at what most of the discussions are about, what does it mean for our education?”).
Another distinction that became visible was the focus on innovation. In a number of
situations, the innovation turned out to be aimed at the content of the curriculum, especially
updating the curriculum. In four cases, the update aimed at combining existing programmes
487
into a few strongly up-to-date programmes, as the connection with the professional field
required this. In this relatively limited research, it turned out that two very special educational
innovations were involved. At one educational institution, care training was combined with
social training, which together organized a new form of practical experience by advising their
starting students during openly accessible consultation hours (H2: “HU Healthy & Well
Centre, that is in the district. People have the opportunity for an interprofessional
introduction, and then we will see what is relevant and what we can offer you. So now it’s in
and out with eye measurements or skin consultation or whatever. We want to look much more
holistically at the people who come in.”). Another institution was able to combine the
innovation demands of a number of large companies by having final-year students from
different study programmes and universities of applied sciences work in teams on the
research questions (T4: “Are they going to experiment with the companies? They have
development and innovation questions, and we connect students to them: a mini hub with
students who spend half a year doing a research assignment or completing a graduation
assignment.”).
In this study, we looked at how PDs used their resources for innovation within their
frameworks as higher education programme managers. The results showed that structuring
and bundling were used for innovation regardless of the administrative or innovative
orientation of the PDs. The PDs first looked at the internal resources and then stabilized and
enriched these resources. Another remarkable result was the leveraging of resources. The
administrative-oriented PDs were active in just this area, looking for opportunities,
integrating resources and deploying them where possible. By contrast, innovation-oriented
PDs looked for success by structuring and bundling resources. In general, we can conclude
that these middle managers in higher education looked for resources they could handle and
used these whenever possible.
5. Discusion
Much is fixed in higher education, which has a high rule density. Almost everything – price,
place, naming, content, etc. – is meticulously monitored and controlled. This is particularly
the case at universities of applied sciences because of the attitudes of most lecturers and the
intertwining of management and education. So how do we bring about innovation? This is
clearly very difficult within the existing environment. Even small incremental innovations
quickly run into rules (“it can’t be done,” “it's not allowed,” “it costs too much,” and “what
are you going to do now?”). In practice, it appears that to realize innovation, it is best to set
up a new project (e.g. a new course). Old, long-serving teachers and young flamboyant
teachers come to life when they are allowed to come up with something new. Content is then
enthusiastically created out of the box, contradictions are easily bridged, and cooperation
flourishes. It is not surprising that the successful PDs almost all first tried to create their own
488
Hans Frederik
space in which they wanted innovation to take place. They then asked for a mandate / freedom
from their superiors. Only then did they get to work. They still have to fight back against
superstars, but at least the atmosphere is set. There are many examples of new study
programmes created in this way, such as the privatization of original tracks/specializations,
separate innovation projects, and private–public partnerships, which are carried out either by
the university of applied sciences itself or in cooperation with a number of fellow universities
of applied sciences.
References
Battilana, J., Leca, B., & Boxenbaum, E. (2009). How actors change institutions: Towards a
theory of institutional entrepreneurship. The Academy of Management Annals, 3(1), 65–
107; http://doi.org/10.1080/19416520903053598.
DiMaggio, P. J. (1988). Interest and agency in institutional theory. Institutional patterns and
organizations, 3-21.
Cabrera, E. F., & Cabrera, A. (2005). Fostering knowledge sharing through people
management practices. The international Journal of Human Resource Management,
16(5), 720–735; https://doi.org/10.1080/09585190500083020
Emirbayer, M., & Mische, A. (1998). What is agency? American Journal of Sociology,
103(4), 962–1023; https://doi.org/101086/231294
Ghalwash, S., & Ismail, A. (2022). Resource orchestration process in the limited-resource
environment: The social bricolage perspective. Journal of Social Entrepreneurship., 1-
28; http://doi.org/10.1080/19420676.2021.2020152
Greenwood, C. R., Horton, B. T., & Utley, C. A. (2002). Academic engagement: Current
perspectives on research and practice. School Psychology Review, 31(3), 328–349;
https://doi.org/10.1080/02796015.2002.12086159.
Hardy, C., & Phillips, N. (1998). Strategies of engagement: Lessons from the critical
examination of collaboration and conflict in an interorganizational domain. Organization
Science, 9(2), 217–230; https://doi.org/10.1287/orsc.9.2.217.
Ireland, R. D., Hitt, M. A., & Sirmon, D. G. (2003). A model of strategic entrepreneurship:
The construct and its dimensions. Journal of Management, 29(6), 963–989; https://
doi:10.1016/S0149-2063(03)00086-2.
Owusu, W. A., & Janssen, F. (2013). Social entrepreneurship: Effectuation and bricolage
approaches to venture establishment in West Africa. 4th EMES International Research
Conference on Social Enterprise.
Sirmon, D. G., Hitt, M. A., Ireland, R. D., & Gilbert, B. A. (2011). Resource orchestration to
create competitive advantage: Breadth, depth, and life cycle effects. Journal of
Management, 37(5), 1390–1412; https://doi: 10.1177/0149206310385695
Sormani, E., & Rossano-Rivero, S. (2023). Facilitating academic engagement with society:
A bonding social capital approach to self-determination. Triple Helix, 1, 1–29; https://doi-
org.vu-nl.idm.oclc.org/10.1163/21971927-bja10036.
489
Tiberius, V., Rietz, M., & Bouncken, R. B. (2020). Performance analysis and science
mapping of institutional entrepreneurship research. Administrative Sciences, 10(3), 69;
https://doi.org/10.3390/admsci10030069.
Westerheijden, D. F. (2022). Gatekeepers on Campus: Peer Review in Quality Assurance of
Higher Education Institutions. Peer review in an Era of Evaluation: Understanding the
Practice of Gatekeeping in Academia, 79-106; Palgrave Macmillan Cham;
https://doi.org/10.1007/978-3-030-75263-7.
490
Non-formal faculty development. Conceptual considerations and

implementation in practice
Center for University and Academic Teaching, University of Regensburg, Germany.
Abstract
Non-formal faculty development offerings might reach those lecturers who, for
a variety of reasons, are reluctant to attend formal faculty development
programs. It can also provide an opportunity to supplement informal
workplace learning with evidence-informed knowledge. A centre for teaching
and learning at a German university has implemented non-formal learning
opportunities through an open learning portal. Lehrblick.de offers lecturers
from all departments access to a bilingual, accessible service that provides
them with concise, quality-assured content on innovative topics in teaching
and learning in higher education, regardless of time or location. With this
approach, it reaches nearly 10 times as many lecturers every month as the
university's formal faculty development program.
This contribution describes theoretical considerations of non-formal faculty
development, its design and implementation in faculty development practice.
Keywords: Faculty development; non-formal training; open learning portal.

Non-formal faculty development. Conceptual considerations and implementation in practice
1. Introduction
In the past 20 years, high quality teaching at universities and thus faculty teaching
competencies have become increasingly important. As a consequence, German universities
have systematically established Centers for Teaching and Learning that provide various
faculty development programs.
2. Formal continued training and development

Typically, Centers for Teaching and Learning offer workshops for faculty development
(Centeno García, 2021). In these workshops, participants acquire theoretical and evidence-
informed knowledge about teaching, learning, and assessment. Other workshops focus on
improving observable teaching skills (e.g. designing slides for academic presentations) or
mastering a specific teaching method (e.g. problem-based learning) (Amundsen & Wilson,
2012). During the workshops, experts in faculty development deliver theoretical knowledge
based on didactic principles that encourage participants to apply this knowledge outside of
the workshops to their own teaching practices.
In recent years, alternative formats for faculty development have been established (Franke,
Sekyra & Vöing, 2021). Working groups, peer discussion groups, and other formats are
centered around reflecting on one's own teaching style. They aim to impart practical
knowledge (e.g. designing laboratory exercises) and tips on how to deal with specific
problems (e.g. classroom interruptions). To meet these objectives, these courses are often
facilitated by faculty from a variety of domains with extensive teaching experience.
The formats described above are all events lasting a few hours to a few days at a fixed time
and location. The courses have clearly defined intended learning outcomes and are anchored
in a curriculum.
3. What prevents university instructors from attending workshops

Although the number of faculty development programs has increased both in terms of
quantity and quality, Fleischmann, Schroeber & Tuschak (2017) estimate that only 5% to
25% of faculty make use of these offerings. The authors analyzed several studies to identify
three main reasons for this reluctance.
(1) They are unaware of the course offerings. Some lecturers are unaware that their university
offers faculty development courses. Most likely, they discover it by chance, e.g. by hearing
about it from a colleague.
(2) They cannot attend courses. Although many Centers for Teaching and Learning have
significantly increased their capacity: Workshop places are limited and tend to fill up quickly.
492
As a result, some people are simply unable to enroll in these courses. Some lecturers are also
unable to participate in those courses due to conflicting class schedules. A second barrier to
entry seems to be long-term planning: registration for workshops usually begins 3 to 6 months
in advance, and some instructors are reluctant to commit that far ahead of time.
Language barriers can also make it difficult for some people to attend courses. The number
of international instructors who teach in English and speak little German is on the rise as
universities become more international. However, only a handful of faculty developers at
German universities are able to offer courses in English.
(3) They do not want to attend courses. A final reason is that some educators do not even
want to take part in faculty development workshops. They do not want to take time as other
academic commitments (e.g. research, committees, networks) take precedence over attending
courses. Some instructors also might find the atmosphere uncomfortable and feel that these
courses are beneath them. The fear of losing face might prevent them from discussing their
own shortcomings with colleagues. Professors are used to being in control, so they might be
afraid to give up their authority and status: Participating in faculty development workshops,
however, requires them to give up this role and take on the perspective of the participants.
4. Workplace learning
Even though only a small percentage of instructors attend faculty development workshops,
does not mean the majority of faculty do not develop their own set of skills. However, for the
most part, they do so in an informal setting, directly at workplace (Smith, 2019). In a study
by Oleson & Hora (2014), nearly 80% of university instructors reflect on their previous
teaching experiences and monitor student reactions. Approximately 20% of instructors use
feedback about their teaching performance and 13% discuss their teaching performance with
colleagues.
Those informal learning settings, however, run the risk of passing on outdated knowledge or
even myths about teaching and learning without reflection. Smith (2019) therefore
emphasizes the importance of faculty development to “assure access to research-based
knowledge and skills to mediate the potential misconceptions that arise within social
networks of varying expertise“ (p.16).
There is strong empirical evidence for both, formal and informal approaches, to be effective
in promoting the development of teaching expertise (Myllykoski-Laine et al., 2022; Van
Geyte & Hadjianastasis, 2022). Therefore, there is little point in comparing the different
approaches (Amundsen & Wilson, 2012). Instead, it is worthwhile to consider which offering
may provide the best of both worlds.
493
5. Non-formal faculty development
5.1. Conceptual considerations

Between these two poles, non-formal education occupies a middle position:
“Like formal education (but unlike informal, incidental or random learning), non-formal
education is education that is institutionalised, intentional and planned by an education
provider. The defining characteristic of non-formal education is that it is an addition,
alternative and/or complement to formal education within the process of lifelong learning of
individuals” (UNESCO Institute for Statistics, 2012, p. 11).
Non-formal training opportunities might reach those lecturers who, for a variety of reasons
(see section 2), are reluctant to attend formal faculty development programs. It can also
provide an opportunity to supplement informal workplace learning with research-based
knowledge.
Web-based asynchronous environments seem to be well suited for this purpose. They enable
access to information regardless of time or place, and allow for the combination of different
media. The content and pedagogical principles must, however, meet the same standards in
terms of content and didactic quality as formal training.
5.2. Lehrblick.de as an example for non-formal academic development

The Centre for University and Academic Teaching at the University of Regensburg has
decided to implement this through an open learning portal (https://lehrblick.de/en/).
Lehrblick.de provides a non-formal educational offer that extends the established formal
formats of faculty development (see section 1).
5.2.1. Approach
As to formal workshops, the editorial team at lehrblick.de largely relies on three different,
valid sources of information when writing articles: Theoretical knowledge, expert
knowledge, and practical tips. The different sources of knowledge are reflected in the portal’s
individual categories:
(1) Theoretical knowledge. A crucial source of evidence-informed knowledge are articles
published in peer-reviewed journals. In the Teaching Concepts Section, the current state of
research on teaching and learning in higher education is reviewed and discussed. As an
example, the article 'blended learning' (Hawelka, 2022) summarizes a meta-analysis on this
topic from the Educational Research Review.
(2) Expert knowledge. Professors with extensive experience in teaching can be an invaluable
resource for their less experienced colleagues. In spite of this, expert knowledge (e.g. from
award-winning professors) often goes lost because these experts have too few opportunities
494
to share their knowledge (Shim & Roth, 2009). In the Teaching Practice Section, instructors
from various disciplines present methods and approaches they have found to be successful in
practice, not just in theory. In this way, faculty members are able to gain insight into teaching
culture beyond the bounds of their usual community of practice. For instance, in an article
by Jossberger (2022), the author describes the implementation and benefits of learning
journals in seminars.
(3) Practical Tips. Even the smallest changes can have a noticeable impact on how lecturers
approach learning processes. In the category Teaching Tips, there are a wide variety of tips
and tools provided to help instructors make their daily lives easier and to improve teaching
performance. For example, in one article on lehrblick.de, Bachmaier (2022) provides
recommendations on how physical signals can be translated into the digital space.
To ensure that the contributions actually address issues pertinent to developing teaching
competence, our selection of topics is guided by the Framework for Teacher Expertise in
Higher Education (Van Dijk et al., 2020).
5.2.2. Didactic and Media Design
The articles are short stand-alone, usually single-objective units that include text combined
with video and graphics. In order to maximize the effects of combining text and images
("multimedia principle"; Mayer, 2021), we follow the basic principles of multimedia design,
such as spatial proximity between graphics and text, a reader-friendly design of embedded
graphics and diagrams, and a deliberate use of colour.
To reduce extraneous load (Paas & Sweller, 2021), the articles are designed in a way that
makes them understandable for the target audience. In order to achieve this, the structure of
the texts plays a crucial role: a consistent order of content and concise headings structure
them visually as well as linguistically. Reader-friendly layouts with easy-to-read fonts and
highlighting of important text components ("signaling principle"; Fiorella & Mayer, 2021)
help maintain reading motivation.
Particularly in articles in the Teaching Concepts Section, we take care to adapt the content of
the original journals to the readers' knowledge and to always explain technical terms.
5.2.3. Accessibility
By using a keyword index, readers can quickly locate articles and a search function makes it
possible to easily find specific topics. Lehrblick.de is a responsive portal, which means that
it is accessible from any internet-enabled device. In order to facilitate access for people with
disabilities, alternative text is provided for embedded images and graphics, and transcripts
are added to all embedded audio files.
495
The teaching portal is designed to accommodate bilingual access. About a third of articles
are already bilingual, and all articles will be available in English and German starting in May
2023.
5.2.4. Quality assurance
Quality assurance is an essential part of the entire editorial process: The editorial team works
with an academic advisory board to select topics for the blog. Authors committed themselves
to follow the guidelines for ensuring good academic practice (DFG, 2019) when writing
articles. Internal guidelines for content structure and style promote consistent content
creation - regardless of whether the contributions are written by members of the editorial
team or by guest authors. Prior to publication, all articles are peer-reviewed for form and
content.
5.2.5. Use and Reception
The portal has been publishing articles every 14 days since March 2021. Through the
University's landing page, various social media channels (especially Twitter, LinkedIn) and
RSS feeds, faculty are kept up-to-date on current issues.
An average of approximately 480 lecturers read the posts each month, with up to 580 readers
per post. This means that lehrblick.de posts reach nearly 10 times as many lecturers every
month as the University's formal faculty development program.
6. Conclusion
Lehrblick.de offers lecturers from all departments access to a bilingual, accessible service
that provides them with concise, quality-assured content on innovative topics in teaching and
learning in higher education, regardless of time or location. Through this non-formal program
for faculty development, lecturers who are reluctant to attend formal offerings as well as
informal communities of practice get evidence-informed impulses for designing their courses
and assessments.
References
Amundsen, C., & Wilson, M. (2012). Are we asking the right questions?: A conceptual
review of the educational development literature in higher education. Review of
Educational Research, 82(1), 90–126. doi: 10.3102/0034654312438409.
Bachmaier, R. (2022, May 19). Digital Body Language 101. Lehrblick – ZHW Uni
Regensburg. doi: 10.5283/ZHW.20220519.EN.
Centeno García, A. (2021). Workshopgestaltung in der Hochschuldidaktik. Agieren in
komplexem Bedingungsgefüge [Designing Workshops for Academic Development.
Acting under complex conditions] In R. Kordts-Freudinger, N. Schaper, N., A.
496
Scholkmann, B. Szczyrba, R. Krempkow, P. Salden, P., I. Ulrich, I. van den Berk, I. &
M. Wiemer (Hrsg.). , Handbuch Hochschuldidaktik (S. 207-223). wbv.
Deutsche Forschungsgemeinschaft. (2019). Leitlinien zur Sicherung guter wissenschaftlicher
Praxis: Kodex. DFG.
Fiorella, L., & Mayer, R. (2021). Principles for Reducing Extraneous Processing in
Multimedia Learning. In R. Mayer & L. Fiorella (Eds.), The Cambridge Handbook of
Multimedia Learning (Cambridge Handbooks in Psychology, pp. 185-198). Cambridge
University Press. doi: 10.1017/9781108894333
Fleischmann A., Schroeber, J. & Tuschak, J. (2017). Nichtweiterbildung. Vorbehalte,
Hemmschwellen und Einwände gegen hochschuldidaktische Weiterbildung - und wie
man damit konstruktiv und strategisch umgehen kann. B. Berendt (Hrsg.), Neues
Handbuch Hochschullehre. Hochschuldidaktische Aus- und Weiterbildung. L 1.34-
L.1.35.
Franke, K., Sekyra, A. & Vöing, N. (2021). Besondere Formate der hochschuldidaktischen
Weiterbildung. Stand der Praxis inklusive Good-Practice-Beispiele [Special formats of
teaching and learning enhancement in higher education. State-of-the-art in current
practice and examples of good-practice]. In R. Kordts-Freudinger, N. Schaper, N., A.
Scholkmann, B. Szczyrba, R. Krempkow, P. Salden, P., I. Ulrich, I. van den Berk, I. &
M. Wiemer (Hrsg.)., Handbuch Hochschuldidaktik (S. 283-297). wbv.
Hawelka, B. (2022, October 27). Blended learning: Quality despite flexibility? Lehrblick-
ZHW Uni Regensburg. doi: 10.5283/ZHW.20221027.EN.
Jossberger, H. (2022, July 28). Good Reasons to Keep a Learning Journal. Lehrblick - ZHW
Uni Regensburg. doi: 10.5283/ZHW.20220728.EN.
Mayer, R. (2021). The Multimedia Principle. In R. Mayer & L. Fiorella (Eds.), The
Cambridge Handbook of Multimedia Learning (Cambridge Handbooks in Psychology,
pp. 145-157). Cambridge University Press. doi: 10.1017/9781108894333.
Myllykoski-Laine, S., Postareff, L., Murtonen, M. et al. (2022). Building a framework of a
supportive pedagogical culture for teaching and pedagogical development in higher
education. Higher Education. doi: 10.1007/s10734-022-00873-1.
Oleson, A., & Hora, M. T. (2014). Teaching the way they were taught? Revisiting the sources
of teaching knowledge and the role of prior experience in shaping faculty teaching
practices. Higher Education, 68(1), 29–45. doi: 10.1007/s10734-013-9678-9.
Paas, F., & Sweller, J. (2021). Implications of Cognitive Load Theory for Multimedia
Learning. In R. Mayer & L. Fiorella (Eds.), The Cambridge Handbook of Multimedia
Learning (Cambridge Handbooks in Psychology, pp. 73-81). Cambridge University
Press. doi: 10.1017/9781108894333.009.
Shim, H.S. & Roth, G. (2009). Expert Teaching Professors: Sharing Their Expertise,
International Journal for the Scholarship of Teaching and Learning, 3(2). doi:
10.20429/ijsotl.2009.030213.
Smith, G. A. (2019). Framing faculty development as workplace learning. Journal on Centers
for Teaching and Learning, 11, 3-23.
497
UNESCO Institute for Statistics (2012). International Standard Classification of Education.

ISCED 2011. http://uis.unesco.org/sites/default/files/documents/international-standard-
classification-of-education-isced-2011-en.pdf
van Dijk, E. E., van Tartwijk, J., van der Schaaf, M. F., & Kluijtmans, M. (2020). What
makes an expert university teacher? A systematic review and synthesis of frameworks for
teacher expertise in higher education. Educational Research Review, 31, 100365. doi:
10.1016/j.edurev.2020.100365.
Van Geyte, E., & Hadjianastasis, M. (2022). Quality and qualifications: The value of
centralised teaching courses for postgraduates who teach. International Journal for
Academic Development, 27(1), 4–16. doi: 10.1080/1360144X.2020.186381.
498
Teaching olfaction at the time of the 'sensual turn'. The case of

Pierre Bénard
Sandra Cadiou
Sciences of education and formation, UCO Niort, France.Université catholique de l'Ouest de
NiortFrance.
Abstract
Teaching olfaction is changing these days as the olfaction professional world.
The 'sensual turn' studied by Howes (2003) reveals the regain of interest in
this badly loved sense. We focus on teaching olfaction in higher education with
the case of Pierre Bénard: a perfumer, an expert in raw materials, an olfactory
manager, and a teacher. This case gives an overview of an unknown field:
teaching in olfaction and opens a new research field: education sciences in
olfaction. We used a semi-directive interview in a clinical orientation and
different exchanges in a cooperative methodology to collect new knowledge
transmitted in higher education. By his way of considering olfaction, teaching,
and art as a unique way to live, the singular course of Pierre Bénard seems
symptomatic of this ‘sensual turn’.
Keywords: already there sensory olfactory; teaching in olfaction; sensual turn.

Teaching olfaction at the time of the 'sensual turn'. The case of Pierre Bénard
1. Introduction
Higher education establishes privileged links with the professional world by preparing
students for the world of work. This is the case for olfaction. This teaching was done on the
job and in the family for a long time and now has its schools. The Institut Supérieur
International du Parfum, de la Cosmétique et de l'Aromatique alimentaire (ISIPCA) was
created in 1970 by Jean-Jacques Guerlain, and other prestigious schools have followed: the
Ecole Supérieure du Parfum (ESP), Cinquième Sens, Grasse Institute of Perfumery,
Montpellier University, etc., not to mention the professional schools that are integrated into
the companies in the sector. They train famous noses, and also professionals in detergent and
industrial perfumery.
At the same time, interest in this sense is growing. The creation of the Osmothèque1, a
memory of odors, and the magazine "Nez2 " bear witness to this, as does the field of
contemporary creation in olfaction (Castel, 2018).
We explore the higher education devoted to olfaction by considering this current interest in
odors. While some knowledge persists, other knowledge is emerging. What new knowledge
related to olfaction is created? What conceptual tools are used? For this, we look at Pierre
Bénard: a perfumer-creator, a specialist in natural raw materials, an olfactory scenographer,
and a teacher.
2. Theoretical framework
2.1. Olfaction
Olfaction remains a mysterious field. Interest in this field has increased, particularly
following the 2004 Nobel Prize of Linda Buck and Richard Axel (Buck & Axel, 1991), which
brought to light further research on this sense, often denigrated because of its animal and
sexual connotations.
This work is therefore often physicochemical, but the point of view of the human and social
sciences is gaining ground, particularly since the 'sensual turn' (Howes, 2003, p.29) of the
1990s, which marks the interest of anthropology in odors: 'sensory perception is a cultural,
as well as a physical, act' (Classen, 2010, p.401).
This research focuses on the cultural variability of olfactory perception, the knowledge and
skills of cultures, the place and meaning of odors in everyday life, and their significance
1
The Osmothèque is a perfume conservatory based in Versailles.
2
The website auparfum.com, the magazine Nez and the publication of books have largely contributed to the popularisation of
olfaction. https://shop.bynez.com/marques/nez/
500
Sandra Cadiou
(Candau, 2016). Cognitive sciences underline the importance of memory in sensory

perception, the first encodings in childhood in cognitive psychology (Schaal et al., 1980;
Schaal & Kontar, 1998, Urdapilleta & Dubois, 2003), or sensory categorization processes
(Urdapilleta & Dubois, 2003). And we underline an important characteristic of olfaction: its
resistance to classifications (Dubois, 2006) and to words, which impacts the teaching of a
highly verbal profession.
2.2. Olfaction teaching and clinical orientation

The teaching of olfaction is in this sense an even less well-researched field. Schools are
recent, students are few, and companies also train internally. Also, given the financial stakes,
discretion is preferred and doors are not easily opened. The sciences of olfaction education
are almost nonexistent, some few researches exist in clinical orientation under the hypothesis
of the Freudian unconscious.
The case of Marcel (Cadiou, 2021) is the first research in this sense: how does this olfaction
trainer in a luxury brand train the saleswomen? In his thesis on the olfactory part of the
sensory analysis of wine (2019), Alvarez identifies an already there sensory, a component of
the already there, i.e. the latent history of the subject (Combis-Carnus, 2001) which filters
the didactic action. And this already there sensory would be common to the teaching of
perfumery sales training and the teaching of sommelier training in hotels and restaurants
(Cadiou & Alvarez, 2021).
2.3. Teaching: between constructed knowledge and knowledge to be constructed

Teachers of olfaction do not come from the national education system, they generally do not
have didactic or pedagogical training and come from the professional world of which they
possess the knowledge and know-how. The anatomical-physiological apparatus, the olfactory
neuroreceptor system, extraction methods, the composition and facets of odors, etc., are all
taught in a more or less stable manner.
Teaching is also a discursive practice of professional know-how. In this sense, with teaching
experience, these professionals re-elaborate or construct new knowledge (Vanhule, 2009),
that is transmitted.
This meta-professional activity is coupled with a reflexive activity of research-action-
creation (Caumon, et al. 2016). Here, the knowledge linked to olfaction is also linked to
artistic practice: how to construct a perfume, how to smell, and how to take advantage of
odorous materials are questions that the aesthetic practice of the professionals has confronted.
501
3. Research questions and issues

How is olfaction taught in higher education? We aim to give an insight into a poorly
documented field. The teaching of olfaction is relatively young and uncommon. This varied
knowledge: conceptual, technical, know-how... is established for some, sometimes discussed,
and for others still to be constructed. What new knowledge is being constructed and
transmitted? On the other hand, we are interested in the subjects as actors of knowledge from
the educational sciences in a clinical orientation. We dedicate this text to Pierre Bénard, a
teacher (among others) in the world of perfumery.
4. Methodology
Our clinical approach presents a case. This singularity sheds light on the object taught,
particularly on the creation of knowledge in this teacher, in an informative and
comprehensive approach, and on the intimate and personal way of considering teaching.
Pierre Bénard did not grow up in Grasse, the land of perfume, but in the South-West. He
wanted to go to the Beaux-Arts but decided to study biochemistry and during an internship
at the Association Asquali, he discovered the world of olfaction.
This association envisages olfaction in a new way: "they were the pioneers in seeing odors;
it is more the world of odors in a way [...] not perfumistic at all". It is part of an approach to
cultural development, through the vector of smell, and olfactory mediatization.
The internship turns into six years of training with Asquali and he becomes a professional in
the sector. Later on, he teaches at the ISIPCA, the ESP, the University of Montpellier, and
the Grasse Institute of Perfumery. He teaches in the schools of the profession but also in art,
design, fashion, gastronomy, communication, marketing, and engineering schools. Its
teaching is organized both face-to-face and remotely. Pierre Bénard is also interested in
olfactory awakening and education. He works in nurseries, kindergartens, and colleges, and
addresses more vulnerable groups: ULIS and UAM.3
We had several exchanges with Pierre Bénard in various ways. I conducted a semi-structured
interview on 14 October 2022, which allowed me to collect the first set of information while
letting the speaker follow his train of thought. The extracts are therefore verbatims, proposing
a speech, an oral discourse with its syntax twisted by the thread of the subject, bringing out
chains of signifiers. In this sense, I am relying on groping speech rather than composed
speech.
3
ULIS : Localized school inclusion unit. UAM: UnAccompanied Minors.
502
Sandra Cadiou
On the other hand, we exchanged following this verbatim to collect precise information and,
in an afterthought, on martyr texts where Pierre Bénard validates or not the analysis of the
comments on the teaching set up by him.
This cooperative methodology seeks to keep transparency in the work of the perfumer and
teacher and to give easier access to the world of perfumery and its teaching. This article,
therefore, draws on Pierre Bénard's professional, personal, and creative contributions as well
as my own, which attempt to think about these from a theoretical perspective in educational
sciences and this sensual turn'.
5. The case of Pierre Bénard: olfaction teacher

Pierre Bénard's teaching practice is based on commonly shared knowledge. For example, he
starts any training course, whatever the audience, by reflecting on his olfactory system: its
anatomical functioning in particular. But we are only interested here in certain innovations
in terms of knowledge linked to this teacher.
5.1. Olfactory cone

The representation of odors and perfumes often involves geometry (Cadiou, 2021). In
marketing, the perfume is represented by a triangle which is called an olfactory pyramid. It
is divided into three parts: top notes, middle notes, and base notes.
Pierre Bénard uses a slightly different schematization: the cone. The three-plane dimension
is accentuated, but also a certain movement: seen from above it evokes the drop of water that
falls and spreads. The choice of the cone brings the notion of volume, waves, time, and
volatility into the perfume, and within its three theoretical times: head, heart, and wake. This
conceptual tool further emphasizes the continuity within the diffusing olfactory form and
thinks of perfume as an art of space and time.
5.2. Systematic of scents© or “chromatic code”©

There are various types of classifications of scents and perfumes. For example, the first
"Classification des Parfums" was produced in 1984 by the French Perfumers' Society and has
been reissued several times. Other classifications were created, such as Jean-Noël Jaubert's
"Champs des odeurs® ", a method used in olfaction teaching, but also tasting (Dratz, 2001).
Pierre Bénard proposes one: the "Systematic of scents"©, also called "chromatic code"©. It
is organized around the origin of the raw materials mentioned and comprises 7 areas: spicy,
fruity, floral, herbal, suave, silvan, and animalic. The classification allows us to think in terms
of creation by proposing connections between raw materials. Their similarities propose
possible replacements while keeping the assemblies specific to a certain classification of
perfume: cologne, chypre, fougère, oriental, woody, leathery.
503
It uses the term area which encompasses both smell and fragrance by uncovering
smell/fragrance structure combinations and thus drawing or re-drawing olfactory shapes. The
concept of area is of direct professional interest because these combinations offer frameworks
and proto-categories for olfactory projects. As an intellectual step, it facilitates the olfactory
thinking of (future) professionals.
The work on colors in this classification should also be highlighted. The expression
"chromatic code" is also a way of "speaking" about odors by relying on the synesthesia
strongly present with the world of odors, the linguistic incapacity to explain odors is replaced
by a color/odor equivalence that says something. This equivalence would be based on
conventions, also on a choice specific to its author, and it is evocative.
5.3. Sandalwood
Pierre Bénard is also an expert in rare materials, particularly sandalwood. His research work
produces knowledge that links material, smell, and perfume. This work is therefore a further
step in his work on the systematic of scents. Upstream, Bénard links with sandalwood
production.
This work is the result of a collaboration with a company producing high-quality and
sustainable Indian and Australian sandalwood.
Pierre Bénard has created a knowledge of the different olfactory interests of the various
sandalwoods, according to the species and the part of the trunk: from the heart to the bark.
This knowledge is pushed to its creative limits: the formulation of perfumes according to the
variety and parts of the various sandalwoods. For example, the sweet and milky side of
certain parts and essences forms an interesting combination with gardenia and tuberose.
5.4. Olfactory manager

Pierre Bénard is a perfumer: he creates perfumes, but he is also a manager and olfactory
scenographer. He perfumes concerts and streets. This work is technical: creation and
development of odorization systems, and artistic: place, volume, intensity, and moment of
the smell in a creation.
He developed a work on fluid mechanics which he teaches today. This knowledge concerns
the diffusion systems, the technical installations thought in the problematic of air design, that
is to say, what is an olfactory space.
It also integrates the basics of olfaction about its communicative and scenographic capacities.
It is therefore a field at its dawn, like the olfactory art detached from perfumery (alcoholic
intended for the body) which is at its beginning (Froger, 2016; Castel, 2018).
504
Sandra Cadiou
6. Discussion
With the case of Pierre Bénard, we can discuss the types of knowledge transmitted: if some
are of course always transmitted, others are transformed and some are completely created.
This may be due to the evolution of research and knowledge in olfaction, but also of the
professional world.
It is true that the case is a singularity and can not be generalized. Pierre Bénard's work touches
on various fields: perfumery, expertise in raw materials, mediation, olfactory management,
and, of course, teaching. These various facets are at the origin of new knowledge linked to
olfaction. His artistic and professional approach nourishes a singular look at this knowledge
and transforms it into a desire to live/teach odors that are part of this 'sensual turn' and
participate in the contemporary olfactory art scene.
Moreover, this teaching is also a matter for the actors. Sharing-affect-odor would be
intimately linked in him, and this psychic braiding would be inclined to set up a transferential
dynamic privileging the teacher/learner relationship. To the question: does teaching make
him happy? he answers "yes, very happy", "because I am made happy [...] by the presence of
[...] the feedback".
The methodology used is a mixture of journalism and clinical approach which sought to
report, in a spirit of transparency, on the often obscure work of olfaction professionals. This
point was close to Pierre Bénard's heart. Here again, this hybrid methodological approach
would also be symptomatic of the current position of olfaction. We regret not being able to
compare with other practices as they are little or not documented, and we are working on it.
7. Conclusion
Research in the teaching of olfaction is a little investigated field. The case of Pierre Bénard
gives access to current teaching in higher education: between usual, renewed, and new
knowledge. It seems to us symptomatic of the 'sensual turn'. Olfaction is regaining interest
and its teaching is opening up new paths crossing the contemporary art scene. His artistic and
professional approach nourishes a singular look at this knowledge and transforms it into a
desire to live/teach odors that are part of this 'sensual turn' and participate in the contemporary
olfactory art scene: "olfaction is intimately linked to breathing, to the body, to its vital
function, to smell is to survive, to live on. In this sense, to smell is to go beyond living”.
References
Alvarez, D. (2019). Enseigner l’analyse sensorielle du vin, ressorts didactiques et déjà-là
sensoriel de l’enseignant, études de cas en didactique clinique. Thèse, UT2J.
505
Buck L, Axel R. (1991) « A Novel Multigene Family may Encode Odorant Receptors: a
Molecular Basis for Odor Recognition» Cell;65:175-87. Consulté à
https://pubmed.ncbi.nlm.nih.gov/1840504/
Cadiou, S. & Alvarez, D. (avril 2021). Sciences de l’éducation et enseignement de l’olfaction:
le déjà-là sensoriel, un concept didactique heuristique. Savoir/pouvoir et singularité du
sujet. https://spsujet.hypotheses.org/167.
Cadiou, S. (2021, février 19). Au coeur des aspects économique, esthétique et intime du
parfum : Le cas d’un professionnel en vente de parfumerie de luxe. Le parfum, un
matériau pour la création en design. Journée d’étude LARA-SEPPIA, UT2J Toulouse.
Candau, J. (2016). L’anthropologie des odeurs : Un état des lieux. Bulletin d’études
orientales, 64, 43-61. DOI : 10.4000/beo.4642
Castel, M. (2018). Les Dispositifs olfactifs au musée. Le Contrepoint.
Classen, C. (2010). Foundations for an anthropology of the senses. International Social
Science Journal, 49(153), 401-412. DOI : 10.1111/j.1468-2451.1997.tb00032.x
Combis-Carnus, M.-F. (2001). Analyse Didactique du Processus Décisionnel de
l’Enseignant d’EPS en Gymnastique. Toulouse III Paul Sabatier.
Dratz, M.-T. (2001). The field of odors. Educational interest and application in descriptive
sensory analyse. Annales des Falsifications, de l'Expertise Chimique et Toxicologique
(France).
Dubois, D ( 2006). « Des catégories d’odorants à la sémantique des odeurs », Terrain [En
ligne], 47 | septembre 2006, mis en ligne le 15 septembre 2010, consulté le 19 décembre
2022. URL: http://journals.openedition.org/terrain/4263; DOI: 10.4000/terrain.4263
Froger, G. (2016). « L’Art olfactif contemporain ». Critique d’art. Actualité internationale
de la littérature critique sur l’art contemporain.
Houssaye, J. (1988). Le triangle pédagogique (2ème). Peter Lang.Caumon, C., Houssard, F.,
& Lecerf, G. (2016). Recherche action par la création artistique et design. Editions
Publibook.
Howes, D. (2003). Sensual relations: Engaging the senses in culture and social theory.
University of Michigan Press/2010.
Schaal, B. & Kontar, F. (1998) « Médiations olfactives de la socialisation précoce» in Rey-
Hulman, D., & Boccara, M. (Éds.). (1998). Odeurs du monde : Écriture de la nuit.
INALCO : Harmattan ( p. 293-319).
Schaal, B., Montagner, H., Hertling, E., Bolzoni, D., Moyse, A., & Quichon, R. (1980). Les
stimulations olfactives dans les relations entre l'enfant et la mère. Reproduction Nutrition
Développement, 20(3B), 843-858.
Urdapilleta, I. & Dubois, D. (2003). Evaluation sensorielle et psychologie cognitive.
Psychologie Française, 48(4), Psychologie française, 2003, Vol.48 (4).
Vanhule, S. (2009). Savoirs professionnels et construction sociodiscursive de l’agir. Bulletin
VALS-ASLA (Association suisse de linguistique appliquée), 90, 167-188.
506
Internal quality assurance systems in Namibian higher education:

Stakeholder perceptions and guidelines for enhancing the system
Department of Science, Mathematics and Technology Education, University of Pretoria,
South Africa.
Abstract
Namibian higher education institutions (HEIs) have been striving to enhance
quality assurance in the last decade. Building internal quality assurance (QA)
capacity has been challenging. We explored the perceptions and experiences
of internal QA stakeholders. This research is embedded in Margret Archer’s
social realism theory as a guide to improving internal QA systems. We adopted
a case study design based on an interpretive paradigm. Two purposively
selected HEIs with university status were selected, and we recruited
participants from the universities’ population of stakeholders based on their
roles. We conducted semi-structured interviews with stakeholders. The
findings showed that although both institutions had QA units, the institutions
were still facing challenges to attaining effective quality implementation and
administration. Challenges to implementing QA include slow implementation
of programme changes, mentorship programmes having an overemphasis on
early career academics and causing potential mistrust, a lack of financial
resources and students’ engagement in QA activities.
Keywords: Internal quality assurance; higher education institutions; quality
assurance; quality; stakeholders’ perceptions; social realism.

IQA systems in Namibia HEIs: Stakeholder perceptions and guidelines for enhancing the system
1. Introduction
Globally, most countries have established units within higher education institutions (HEIs)
responsible for ensuring the quality of higher education (HE). However, countries differ in
their priorities for implementing quality education. We explored the internal quality
assurance (IQA) implementation and administration of two Namibian HEIs. Participants
suggested how IQA systems, mechanisms and processes could be improved. According to
Kinser (2014), a historical perspective on effective standards emphasises how important it is
to embark on business operations. The concept of quality is not a unitary one; definitions are
relative to context and judgements. In the same vein, in modern education, the idea of quality
affirmation emerged in the HE fraternity as a borrowed concept derived from industrial and
commercial settings (Kinser, 2014). Given the variety of views, Goldenberg (2018)
emphasises that quality has no unequivocal definition but rather is constructed through
consensus and dialogue among the interested parties. Quality in HEIs is a somewhat elusive
and complex concept that requires a good regulatory framework that mandates, promotes and
ensures quality attainment. Globally, QA has become crucial to the extent that it is no longer
an option; but a requirement in the HE fraternity. HE plays a vital role in any country’s social
and economic development since it produces graduates who are highly qualified for socio-
economic development; furthermore, the global mode of living is constantly evolving, and
educational systems must keep up to be relevant to their economies (Balogun, Olajide, &
Adenagbe, 2022). There is a constant need to find new strategic ways to ensure continuous
improvement and accommodate contemporary needs in the HE environment. According to
Isaeva et al. (2020), the involvement of all the stakeholders improves the actual quality of
education; for instance, students’ engagement positively impacts their learning and increases
their sense of belonging, building trust and confidence between the institution and the
students. In the Namibian context, education was unequal before political independence from
the South African colonial regime, and native children attended schools based on their tribes
(Ministry of Education and Culture, 1993). At that time, the Namibian education system was
designed to strengthen the apartheid system instead of providing the required human
resources to ensure equitable economic and social development, and such practice
significantly impacted the standard of education offered in Namibia (Katjavivi, 2016). It was
fragmented along ethnic lines and racial groups in the Bantu Education System, enforced in
Namibia’s non-white communities (Katjavivi, 2016). Hence, the unequal system brought
about vast discrepancies in the allocation of financial and physical resources and the quality
of education offered to those ethnic groups. After independence, the desire to provide quality
education for everyone became a necessity. The government started initiating and
implementing measures to improve the quality of education, particularly in HEIs. According
to Katjavivi (2016), the Namibian government has developed initiatives to improve the
quality of education within the policy framework, namely the Education Sector Improvement
508
Programme (ETSIP). The ETSIP is a medium-term strategic plan aimed at improving quality
and efficiency in the education sector. The improvement programme was developed as a
remedial framework to enhance the educational weaknesses identified by the Namibian
government, such as the unsatisfactory performance of the students, poor quality teaching
and untrained teachers (Katjavivi, 2016). ETSIP aimed to align Namibia’s vision 2030 with
the education system to transform Namibia into an industrialised and knowledge-based
society. This vision can only be realised if HE provision is of acceptable quality (Kadhila &
Iipumbu, 2019). The Namibian government established statutory Quality Assurance
Agencies such as the National Council for Higher Education (NCHE) (HE Act No. 26, 2003
[Government Gazette of the Republic of Namibia, 2003]). The NCHE was launched in
November 2005 to promote and coordinate an effective HE system in Namibia and monitor
QA mechanisms of HEIs. According to Iipumbu and Kadhila (2020), different countries have
established QA mechanisms at institutional and national levels. Hence, quality matters are
handled by the Internal Quality Assurance Committee at the institutional level. In contrast,
an external QA panel deals with all the national-level matters on quality. IQA committees
evaluate various models for ensuring quality, such as institutional audits, programme
accreditation, and institutional accreditation. Given this background, it is evident that QA has
become a global phenomenon, and Namibian HEIs are finding it challenging to implement
well-established QA mechanisms effectively (Iipinge et al., 2020; Kadhila & Iipumbu, 2019).
We aimed to explore the perceived effectiveness of IQA implementation and administration
in HEIs.
The theoretical framework supporting this research is Margaret Archer’s (1995) theory of
social realism which was employed as an analytical tool to assist us in exploring the state of
the IQA in Namibian HEIs in the provision of quality HE. Such a theoretical perspective
enabled researchers to conceptualise how QA could improve IQA practices at Namibian
HEIs. Archer (1995) distinguishes three interrelated dimensions: structure, agent, and
culture, which co-exist in any social set-up. For analytical purposes, these dimensions are
isolated; however, in the real world, they are connected. Archer (1995) articulates that
studying these dimensions is crucial to understanding how the social setting functions. The
structure dimension is associated with material interest, recurring patterns of social
behaviours, or differentiating how aspects of society are related (Archer, 1995). Several
concepts are related to structure, such as race, gender, social class and education. In the
current study, the structural dimension consists of various aspects in place within the HEIs
context, such as QA policies, frameworks and regulations, committees responsible for
ensuring quality education, and QA divisions. Agents are the individuals such as QA
practitioners, students, administrative staff and academics; only through their actions can the
509
IQA structures be constructed, reconstructed and transformed. Regarding culture, QA units

in the Namibian HEIs should be highly valued and composed of dedicated and committed
individuals responsible for ensuring quality education within Namibian HEIs and the related
policies. It could become obsolete when HE keeps changing, but the IQA remains static
(morpho stasis), and these two systems will fail to live up to their purposes. We explored: (i)
how stakeholders perceive the effectiveness of the IQA implementation and administration
in Namibian HEIs, (ii) how stakeholders perceive the successes and challenges facing
Namibian HEIs when implementing and administering the IQA, (iii) how IQA systems of
Namibian HEIs could be improved from a stakeholder perspective.
3. Materials and Methods

We conducted a case study based on an interpretive paradigm in two selected HEIs with
university status: public and private, and they had established QA units. The sample was
purposefully selected to match the criteria of the type of subject under investigation. Semi-
structured interviews were conducted, and member checking was done to enhance the study’s
trustworthiness.
3.1. Sample and Instrument

Participants were purposively selected based on their positions, first-hand experiences and
strategic roles regarding the phenomenon under study from the two HEIs (12 participants; 6
from each HEI). The participants were deans of the two faculties in each university since
deans are responsible for hosting and managing the quality within their faculties. The QA
units and a QA coordinator were selected in each university since they are the custodians of
QA and coordinate QA systems and programmes. Moreover, participants from the Student
Representative Council were purposively selected to represent the student body since the
students are primary beneficiaries of quality education. The interview guide was developed
to gain an understanding of the participants’ current experiences and perceptions of the IQA
systems, mechanisms and processes.
3.2. Data Analysis and Ethical Considerations

Participants’ verbatim interviews were recorded and transcribed. Thematic analysis was used
for identifying, analysing and reporting themes from the data gathered (Vaismoradi, Turunen,
& Bondas, 2013). We enhance the trustworthiness through a special inspection conducted by
two researchers who evaluated the findings, interpretations and conclusions. We obtained
ethical clearance from all the relevant authorities and institutions involved. All the
participants received consent letters, participated voluntarily, and transcripts were
anonymised. The findings are presented in the next section based on the emerging themes.
510
4. Findings
4.1. Slow Implementation of Programme Changes Hinder Advancement and Negatively

Impacts Students
Some participants emphasised that courses and programmes are subject to slow change. “Few
programmes become obsolete in our institution when the relevant staff members delay and
fail to review it on time and identify the relevant and current programmes that assist in
responding to the national needs” (QA practitioner, HEIA). From participants’ perspectives,
the speed of updating courses was insufficient. Participants in both HEIs felt that the
departments took too long to implement changes and that slow implementation is a barrier to
advancement, negatively impacting students’ employability in the job markets. In the book
by Lim (2018) on QA in HE in developing countries, the need for HEIs to absorb new
information faster and to apply new processes faster is emphasised.
4.2. Mentorship for QA Purposes: Overemphasis on Early Career Academics and Potential
for Mistrust
Mentorship or peer review is a part of the QA of teaching in HEIs (Sachs & Parsell, 2014).
Although participants saw the value of mentorship programmes within their HEIs, the
pointed out that it reduced the trust between the students and the monitored lecturer (usually
early-career academics) “You may find a supervisor in the faculty that sits in the class where
a colleague is teaching and thereafter gives some hints and advice on improving the teaching;
however, this brings a problem of trust between a person who is being mentored and
students” (Dean, HEIB), and that it tended to be available only for early-career academics
“Our lecturers’ teaching strategies are being monitored and evaluated by a senior lecturer”
(Student, HEIB). Literature highlights the importance of continuing professional
development of all teachers in HEIs (novice or expert), which include peer observation, as
there are constant development and rapid changes in education (Ercan & Ivanova, 2020),
especially during the fourth industrial revolution. Literature also highlights the issues of
mistrust and suspicion that go with peer review but emphasis the need for transparent
processes for all to build trust (Sachs & Parsell, 2014).
4.3. Staff shortages, Lack of Adequately Trained Staff and Lack of QA Expertise
A lack of adequately trained staff and QA expertise and a shortage of staff within both HEIs
also emerged as a challenge. “Another challenge is that quality assurance is a new field in
Namibian HEIs; there is a lack of expertise in the country because not everybody has formal
training in the quality assurance course. Most of our quality assurance practitioners were
teachers before, we tend to think we are doing quality assurance, but we are doing it trial
and error the way we understand what it is and how it should be” (QA practitioner, HEIA).
“I think the whole issue of workload, the people who are supposed to review these policies
511
are the same people that supposed to carry out other duties, so, it then becomes a bit
burdensome” (Dean, HEIA). “Another strength is that we have a quality assurance dedicated
team. However, I indicated that one of the challenges is a lack of expertise” (QA practitioner,
HEIB). This study indicated that a lack of adequately trained staff and QA expertise and a
staff shortage are some challenges that hinder the effective implementation of QA initiatives
in Namibian HEIs. The findings of this research agreed with findings from the literature, such
as Lim (2018), who found that most universities in sub-Saharan African countries have a
shortage of qualified local staff and are forced to employ foreign academics, especially at
senior level, to provide the necessary expertise needed.
4.4. QA Progress Hindered by Resistance to Change in the HE Sector

The participants said that a challenge facing Namibian HEIs is resistance to change, which
hinders the implementation of QA improvements due to misconceptions about what QA can
do for an institution. “Some staff members have annual amnesia, and they must be reminded
all the time. So, it is a human factor, the human element in implementing the IQA
mechanisms, that create resistance to change. Like all dynamic institutions, resistance from
the stakeholders may arise when a new standard has been set. When stakeholders do not
comply for various reasons, the IQA mechanism faces a dilemma” (Academic, HEIB). Some
QA stakeholders regarded the QA system as a policing strategy. Furthermore, most
participants mentioned a lack of support from the top management. “One of the challenges
is also a lack of support from top management; sometimes, as a quality assurance division
would like to implement some of the new things that we think will add value to the well-being
of our institution. However, there is a lack of the senior management leaders’ blessings” (QA
practitioner, HEIA). In the book by Lim (2018) on QA in HE in developing countries, it is
mentioned that employees of HEIs are not necessarily against continuous quality
improvement and changes, but rather, they are “creatures of habit with a natural fear of the
unknown” (p. 35).QA management must educate members of the HEI so that QA is perceived
as a continuous improvement and maintenance channel. The findings indicate that it is
important for the QA stakeholders to develop ownership of the QA system to deal with the
negative perceptions about QA systems, mechanisms and processes. Deans of faculties and
QA practitioners need to know that QA promotes transparency and accountability of the
academic systems instead of policing strategies. At the same time, stakeholders need to know
that QA is vital for improving their educational equity and efficiency and supporting the
ongoing development of the T&L process.
4.5. Lack of Financial Resources for QA Systems

Some challenges to QA in Namibian institutions, such as a lack of financial resources and
budgetary constraints, contribute to ineffective IQA processes (Hoosen, Chetty, & Butcher,
2017). Institutions require sufficient financial resources and planning to support and sustain
512
institutional activities. Participants pointed out that limited financial resources were their
main stumbling block in attaining QA in education. “Challenges are limited financial
resources to cater for all activities in the teaching and learning processes. This will not allow
us to do regular visits to other campuses due to financial constraints” (Dean, HEIB).
4.6. Heavy Workload Reduces Opportunities for QA Implementation

Some participants stated that the heavy workloads of the QA stakeholders could be
hindrances in attaining quality education. “I think the whole issue of workload, the people
who are supposed to review these policies are the same people that supposed to carry out
other duties, so, it then becomes a bit burdensome” (Dean, HEIA). Heavy workloads
negatively impacting QA practices in HEIs, specifically in developing sub-Saharan African
countries, is not new information, as has been published, for example, by Uzhenyu (2015),
within a Zimbabwean context and Abe and Mugobo (2021) within a South African context.
4.7. Lack of Technological Resources as a Hindrance to QA Implementation

The participants mentioned a lack of advanced technology as an aspect that emerged from
the challenges perceived by QA stakeholders. Internet access plays a major role in the T&L
processes, research and community engagement required for quality education. The findings
indicated that student participants perceived a need to improve HEIs technology and internet
connectivity. “I would say the poor internet connections in our institution is a big problem
for us students because we only do our learning activities in class; we do not have any
platforms for doing things online learning or outside classes” (Student, HEIA). There is a
great need to improve their online courses and programmes, particularly during times of
crises such as the COVID-19 pandemic. The internet provides access to positive benefits on
educational attainment, and it has been utilised as a significant instrument for facilitating
effective academic activities in HEIs. However, limited internet connectivity in studies in
Namibian HEIs has listed poor internet connectivity as a challenge (Magesa & Josua, 2022).
The educational setbacks could have a notably negative impact on students’ academic success
since not all the students have access to computers with the internet, which is a concern,
particularly during times of crises such as the COVID-19 pandemic.
5. Conclusion and Recommendation

We aimed to explore the experiences and perceptions of the QA stakeholders in Namibian
HEIs. It is important to understand stakeholders’ perceptions, experiences, and attitudes
involved in facilitating QA systems to determine the roles their experiences and perceptions
play in executing that process successfully. This research provides recommendations based
on the findings that could improve the Namibian HE QA systems. It is worth noting that the
recommendations are somewhat aspirational and would require time and resources. This
513
research contributes to the crucial debates of IQA in Namibian HEIs and will hopefully
provide insights into the future direction of this vigorous topic in Namibian HEIs.
References
Abe, I. I., & Mugobo, V. V. (2021). Low research productivity: Transformation, institutional
and leadership concern at a South African university. Perspectives in Education, 39(2),
113-127. doi:10.18820/2519593X/pie.v39.i2.9
Archer, M. S. (1995). Realist social theory: The morphogenetic approach. Cambridge
University Press: Cambridge.
Balogun, B. N., Olajide, I. O., & Adenagbe, O. A. (2022). Empirical Evaluation of Higher
Education on Economic Development in Nigeria. KIU Interdisciplinary Journal of
Humanities and Social Sciences, 3(1), 200-212.
Ercan, B., & Ivanova, I. (2020). Language instructors’ perceptions and applications of
continuous professional development in higher education institutions. International
Online Journal of Education and Teaching, 7(2), 435-449.
Goldenberg, J. S. (2018). Quality in higher education: The view of quality assurance
managers in Chile. Quality in Higher Education. doi:10.1080/13538322.2018.1488395
Government Gazette of the Republic of Namibia. (2003). Higher Education Acts, Act No. 26
of 2003. Available at
https://www.moe.gov.na/files/downloads/153_Higher%20Education%20Act.pdf
Hoosen, S., Chetty, Y., & Butcher, N. (2017). State of play: A report of regional quality
assurance in Southern Africa (SADC). Available from https://haqaa.aau.org/wp-
content/uploads/2019/03/1.-2017.12.15.DAAD-QA-in-SADC-Study-Report-
Findings.pdf
Iipinge, S. M., Shimpanda, A., & Kadhila, N. (2020). Quality management and higher
education scenario in Namibia: A critical analysis. In N. Baporikar & M. Sony. (2020)
Quality management principles and policies in higher education (pp. 1-400). Hershey,
PA: IGI Global.
Iipumbu, N., & Kadhila, N. (2020). Strengthening internal quality assurance as a lever for
enhancing student learning experiences and academic success: Lessons from Namibia.
Quality in Higher Education, 25(1), 4-20.
Isaeva, R., Eisenschmidt, E., Vanari, K., & Kumpas-Lenk, K. (2020). Students’ views on
dialogue: Improving student engagement in the quality assurance process. Quality in
Higher Education, 26(1), 80-97. doi:10.1080/13538322.2020.1729307
Kadhila, N., & Iipumbu, N. (2019). Strengthening internal quality assurance as a lever for
enhancing student learning experiences and academic success: Lessons from Namibia.
Quality in Higher Education, 25(1), 4-20. doi:10.1080/13538322.2019.1597424
Katjavivi, P. (2016). Educational Transformation in Namibia. Forum of the Commonwealth
Council on Education, Westminster: London.
Kinser, K. (2014). Questioning quality assurance. New Directions for Higher Education, 168,
55-67. doi:10.1002/he.20113.
Lim, D. (2018). Quality assurance in higher education: A study of developing countries.
London and New York: Routledge.
514
Magesa, E., & Josua, L. (2022). Use of technology to morph teaching and learning in higher
education: Post COVID-19 era. Creative Education, 13(3), 846-853.
doi:10.4236/ce.2022.133055
Ministry of Education and Culture. (1993). Toward education for all: A development brief
for education, culture and training. Windhoek, Gamsberg: Macmillan.
Sachs, J., & Parsell, M. (Eds.). (2014). Peer review of learning and teaching in higher
education: International perspectives (Vol. 9). Dordrecht: Springer. doi:10.1007/978-94-
007-7639-5
Uzhenyu, D. (2015). Quality assurance: The driving force for effective service delivery in
open and distance learning. A case study of ZOU Harare Regional Office (2010-2015).
IOSR Journal of Business and Management, 17(10), 55-63. doi:10.9790/487X-
171025563
Vaismoradi, M., Turunen, H., & Bondas, T. (2013). Content analysis and thematic analysis:
Implications for conducting a qualitative descriptive study. Nursing & Health Sciences,
15(3), 398-405.
515
Lecturer language: EMI students’ experiences on first- and

second-cycle degrees
Jane Helen Johnson, Mariangela Picciuolo
Department of Modern Languages, Literatures and Cultures, University of Bologna, Italy.
Abstract
Previous research has long focussed on EMI lecturers’ English usage as an
aspect which might affect lecture comprehension. Fewer studies have directly
questioned EMI students about how their lecturers’ language competence
affects their experiences in the classroom, and while many studies have
focused on postgraduate students, less has been said about undergraduates.
We aim to provide further insight into students’ views of EMI lecturer
discourse by comparing the experiences of undergraduate and postgraduate
students, both local and international, at an Italian university. Data for this
study were collected from an online semi-structured survey of 128 students.
Findings indicated that it is students’ past experience of English-taught
courses as well as their familiarity with non-native English accented speech
that influences students’ opinion of EMI lecturers’ language performance and
their assessment of intelligibility in the classroom.
The findings will serve to highlight difficulties and critical points for further
development and pedagogical application.
Keywords: English-Medium Instruction; first and second-cycle study; survey;
lecturers’ language.

Lecturer language: experiences of EMI students on first- and second-cycle degrees compared
1. Introduction
Since the Bologna process, international courses at non-anglophone universities have
increased exponentially. This situation, mirrored the world over, has been accompanied by
an increase in research into all aspects of English-Medium Instruction (EMI). Previous
research has long focussed on EMI lecturers’ English usage as an aspect which might affect
lecture comprehension but fewer studies have directly invited EMI students to address the
issue of how far their lecturers’ language competence affects their experiences in the
classroom, and still fewer have compared the experiences of students of first- and second-
cycle degree courses in an EMI context.
Past research has suggested that greater experience with EMI might affect lecture
comprehension. For example, Clark (2017) investigated the language issues of both Italian
and international students on both years of second cycle degree courses1. While she found
differences between Italian and international students as regards evaluation of lecturer
language and their own competence, with the latter being harsher in self-evaluation and less
critical of NNS lecturers’ language competence, she also noted a difference between first-
year and second-year students, with the latter being more tolerant of lecturers’ English and
having fewer problems in comprehension, possibly because their greater experience with
EMI made them “able to reflect on the idea that language use is not just a question of the
linguistic capacity of one or both parties involved, but an interaction between parties” (Clark,
2017, p. 303).
A specific comparison between the two cycles is particularly relevant for local students
embarking on tertiary education in Italy, where students’ English language competence,
traditionally weak2, may have improved since the enforcement in 2014 of the Gelmini
Reform (240/2010), obliging Italian secondary schools to offer some English-taught courses
following the Content and Language Integrated Learning (CLIL) approach. This would imply
that Italian undergraduates are now embarking on university education with greater
familiarity with English-taught lessons and possibly better language skills than in the past.
However little research has been done to evaluate the effects of this reform.
Our paper aims to provide insights into students’ views of lecturer discourse by comparing
the experiences of local first- and second-cycle students attending EMI courses at the
1
Since 2005, the Bologna Agreement has provided European Union countries with a common framework for tertiary education
which is based on a three-cycle structure. The first cycle typically lasts 3 years and awards a Bachelor’s degree; the second cycle
lasts 2 years and awards a Master’s degree; the length of the third cycle or doctoral degree may vary across countries.
2
According to the EF English Proficiency Index (2022), Italy ranks 24th among European countries as regards general proficiency
in English as L2, https://www.ef.com/wwen/epi/
518
Jane Johnson, Mariangela Picciuolo
University of Bologna (UNIBO). The experiences of international students attending the

same courses are included as a benchmark for comparison.
Data for this study were collected via an online semi-structured survey of 128 local and
international students attending undergraduate and postgraduate courses. Students were
asked about their own English language competence as well as their impression of their
Italian lecturers’ language competence in English as to what extent this affects the lecturer's
intelligibility in the classroom, particularly as regards speech rate, accent, and pronunciation.
Learners’ opinions of their EMI lecturers’ language competence as well as of their own
linguistic ability in English are not the only variables likely to affect learners’perceptions of
the degree of intelligibility in the EMI classroom. Kamaşak et al. (2021) also found that
students with previous experience of EMI found lecture comprehension easier than those who
had none. Local Italian L1 students in general might also find comprehension easier since
they share the same NNS English variety with the lecturer (Fraser, 2006).
Students’ responses in our study were investigated and compared in relation to their degree
cycle: first or second. We also distinguish between Italian L1 and international students.
2. Methodology
Lecturers at UNIBO teaching first- and second-cycle courses in English in Economics or
Engineering were contacted as part of a wider project regarding lecture discourse (Johnson
& Picciuolo, 2022, 2020; Picciuolo & Johnson, 2020). Permission was asked to contact their
students for the purpose of gathering further information. Students choosing to participate
completed the survey via Google Forms. Participants were asked to select responses from a
preset list, express their agreement on a Likert scale, or give a brief written response.
Table 1 gives information about the students according to degree cycle and origin.
Table 1. Participant numbers according to degree cycle and origin.
International Italian Totals
1st cycle 23 53 76
2nd cycle 20 32 52
Total 43 85 128
We note slightly more first-cycle students overall (56%) than second-cycle (44%). While the
L1 of all local students was Italian, a variety of languages was declared by the international
students, with 65% of these belonging to one of three broad language sub-groups: Germanic
519
(e.g. German, English), Balto-Slavic (e.g. Russian, Polish), and Romance (e.g. Spanish,
Portugese).
Students were asked to what extent lecturers’ speed of delivery and pronunciation in general
interfered with lecture comprehension, and how students coped in such cases. Responses are
compared and discussed on the basis of degree cycle, as well as in relation to L1.
3. Findings
3.1. Students’ English language competence

While the literature is divided as to what constitutes the ideal requirement for benefitting
from courses in EMI (see Hultgren et al., 2022), providing documentation of English
language skills at a >B2 level is still a pre-requisite for admission in many European
universities. While most Italian L1 undergraduates declared a higher level than this (>C1),
most Italian L1 second-cycle students declared >B2. This suggests that the younger Italian
undergraduates have been able to benefit from enhanced English teaching including CLIL
since enforcement of the Gelmini Reform, thus raising their own language competence closer
to international language levels. By way of comparison, a greater proportion of international
students (63%) declared a higher overall language competence (>C1) than Italian L1 students
in general (53%).
3.2. Perceptions of lecturer’s English language competence

When asked to specify where EMI lecturers should improve their language competence in
order to perform effectively in lectures, both Italian L1s and international students indicated
pronunciation and oral communication in general, in line with Costa (2017).
3.2.1. Undergraduates
Despite their criticism however, only 12% of local undergraduates had difficulty
understanding, though this figure rose to 19% in the case of international students. Some
Italian undergraduates reported initial difficulties in comprehension but noted that
comprehension improved over time as they became accustomed to the lecturer’s speaking
style.
Most Italian L1 respondents and international students claimed that speed of delivery did not
hinder comprehension, in line with Ackerley’s (2017) findings. We suggest that problems
are exacerbated the further the students’ L1 language family from Italian, with Bangla,
Serbian, Bulgarian and Tamil being the L1 of those international students who declared
difficulties. We note also a possible lack of tolerance towards NNS on the part of the US
native English student, who also claimed that fast speech and distinctive accent impeded
520
comprehension. Other students mentioned the NNS accent as being problematic, drawing on
their experience with NS accents:
Italian's English accent is different from US and understanding it difficult [Persian speaker]
some Profesors using Italian accent when they teach in the class make me difficult to
understand, as I used to study in english-american accent during my previous degree.
[Bahasa Indonesian speaker]
Though, as Jenkins (2014) notes, the benefits of a NNS rather than NS English-speaking
lecturer are also understood:
Because it doesn't seem different from that which we speak back in my country (unlike in US
or UK) [French/English speaker from Cameroon]
In this regard, Derwing and Munro (2015) point out that if the learner is familiar with the
accent, s/he will also claim it is more intelligible. This suggests that the degree of accent
familiarity might not only be related to the shared L1 of lecturer and learners, but, rather, to
the degree of exposure students have had to varieties of English and particular accents.
82% of the international undergraduates said unclear pronunciation interfered with their
understanding, with only the Spanish and Portuguese L1 speakers claiming not to have
problems. Instead 58% of Italian L1 respondents said unclear pronunciation interfered with
their understanding. These results suggest that students who share the same L1 as the lecturer
or whose L1 is close to that of the lecturer have an advantage in understanding their lecturers’
pronunciation (Bent & Bradlow, 2003).
As regards strategies for coping with pronunciation-related difficulties, most Italian L1
undergraduates would prefer to guess from the context than ask the lecturer directly,
suggesting unwillingness to put themselves in the spotlight.
Students were also asked whether they would prefer the lecturer to use more simple language.
Past research has shown that NNS lecturers tend to slow down their speech rate and use
simpler sentence structures (Flowerdew & Miller, 2005), and this appeared sufficient for the
Italian L1 undergraduates, with only 6% wishing for any further simplification. However,
30% of international undergraduates would prefer further simplification. This perhaps
depends on the student’s own level of English competence, since it was those with a lower
level of English who wanted more simplification. However further simplification was also
required by the US English native speaker, suggesting that other factors are at play.
3.2.2. Second-cycle students
83% of Italian L1 second-cycle students had no difficulty understanding lecturers’
pronunciation, and indeed, whatever their L1, these students generally tended to have fewer
problems with lecturers’ pronunciation. This would suggest that maturity plays a more
521
important role in students’ lecture comprehension in the EMI classroom than shared language
or even language competence. Just one international second-cycle student (Pakistan/ English
home language) claimed pronunciation was not clear. This finding is in line with previous
research by Clark (2017) who found that international students tend to be less critical as
regards the quality of their NNS lecturers’ English.
As regards strategies for coping with pronunciation-related issues, like undergraduates,
Italian second-cycle students were also reluctant to ask the lecturer for clarification but prefer
other means. The responses of most international students however show they are more likely
to ask their classmates or even the lecturer directly in case of pronunciation problems. This
suggests greater confidence due to greater maturity or experience.
40% of international second-cycle students would prefer the lecturer to use more simple
language. Most of these students could not rely on similarities between their L1 and Italian
to help them, and they also declared lower English competence. It should be remembered in
fact that “EMI learners may experience greater language-related challenges according to their
English level” (Aiwaza et al., 2020, p. 6), and indeed even a higher level of English
proficiency “does not necessarily alleviate all of the challenges that students encounter”
(Aiwaza et al., 2020, p. 16).
3.3. Would students prefer the lecturer to switch to L1?

Local students are still often the majority in international courses at UNIBO, but only 10%
of Italian undergraduates and 21% of Italian second-cycle students would prefer the lecturer
to switch to the shared L1 in these cases. The higher percentage in the latter case could be
due to the lower language competence of second-cycle students.
4. Discussion and conclusions

Both Italian and international students said their EMI lecturers needed to improve
pronunciation. Nevertheless nearly all Italian and most international students claimed that
lecturer pronunciation and accent did not interfere with understanding, suggesting that it
could be learners’ familiarity with the lecturers’ speaking style – in terms of speech rate,
accented speech and pronunciation – that affects students’ perception of their lecturers’
language intelligibility. This contrasts with EMI lecturers’ perception of their own language
skills (Picciuolo & Johnson, 2020), where pronunciation is described as one of their major
worries.
Our findings have shown clear distinction between cycles as regards language proficiency,
and coping strategies for pronunciation-related issues. As regards language proficiency,
Italian undergraduates overall had higher levels than Italian second-cycle students, thus
confirming our hypothesis that the Gelmini Reform had led to better language skills among
522
the younger Italian cohort. In comparison, international students reported a higher English
language competence than the Italian students in general. As regards coping strategies,
postgraduate students are more willing than undergraduates to ask either their peers or the
lecturer directly for clarification. This could be due to their relative maturity when compared
to undergraduates, as well as suggesting they feel more at ease with the learning environment
of the EMI university classroom. Instead, students’ L1 seems to have more influence than
degree cycle as regards the desire for simplification, with those international students who
could not rely on similarities between their own L1 and the EMI lecturer’s L1 requiring
further simplification. Second-cycle students in general have fewer problems than
undergraduates with understanding due to pronunciation, speed of delivery and accent. As
regards undergraduates however, internationals who did not share the lecturer’s L1 or did not
speak a language close to Italian had greater problem understanding than the locals. Overall,
the majority of local students from both cycles would not switch to their L1 suggesting that
they appreciate the added value of EMI.
Findings from this study have several pedagogical implications. Prospective undergraduate
students, whatever their L1, would benefit more than second-cycle students from a content
preparation course to gain familiarity with the English spoken by the local lecturers, while
introducing them to their field of study and encouraging them to interact. Preparation courses
for both students and lecturers (who also need to gain familiarity with student speech) could
therefore also include audio-visual materials delivered in both synchronous and
asynchronous formats.
Indeed, although much research has focused on training lecturers to improve their
communicative skills in EMI settings, it is worth recalling that students are active participants
in learning, and that any action taken in relation to improve comprehension in the EMI
classroom should involve students and lecturers alike.
References
Ackerley, K. (2017). What the students can teach us about EMI and language issues. In K.
Ackerley, M. Guarda, & F. Helm (Eds.), Sharing perspectives on English-medium
instruction (pp. 257–284). Bern: Peter Lang.
Aizawa, I., Rose, H., Thompson, G., & Curle, S. (2020). Beyond the threshold: Exploring
English language proficiency, linguistic challenges, and academic language skills of
Japanese students in an English medium instruction programme. Language Teaching
Research 1–25. doi: 10.1177/1362168820965510.
Bent, T., & Bradlow, A. (2003). The interlanguage speech intelligibility benefit. Journal of
the Acoustical Society of America, 114(3), 1600–1610.
523
Clark, C. (2017). Perceptions of EMI: The students’ view of a Master’s degree programme.
In K. Ackerley, M. Guarda, & F. Helm (Eds.), Sharing Perspectives on English-Medium
Instruction (pp. 285–308). Bern: Peter Lang.
Costa, F. (2017). English-medium Instruction in Italian Universities: "If we're Gonna do it
Do it Right - Right?" In K. Ackerley, M. Guarda, & F. Helm (Eds.), Sharing Perspectives
on English-Medium Instruction (pp. 79–93). Bern: Peter Lang.
Derwing, T.M., & Munro, M.J. (2015). Pronunciation Fundamentals: Evidence-Based
Perspectives for L2 Teaching and Research. Amsterdam/Philadelphia: John Benjamins
Publishing Company.
Flowerdew, J., & Miller, L. (2005). Second Language Listening: Theory and Practice.
Cambridge: Cambridge University Press.
Fraser, S. (2006). Perceptions of varieties of spoken English: implications for EIL. In R.
Kiely, P. Rea-Dickins, H. Woodfield, & G. Clibbon (Eds.), Language, Culture and
Identity in Applied Linguistics (pp. 79-97). London: Equinox.
Hultgren, A.K., Owen, N., Shrestha, P., Kuteeva, M., & Mežek S. (2022). Assessment and
English as a medium of instruction: Challenges and opportunities. Journal of English-
Medium Instruction 1(1), 105–123. doi: 10.1075/jemi.21019.hul.
Jenkins, J. (2014). English as a Lingua Franca in the International University. Oxford:
Routledge.
Johnson, J. H., & Picciuolo, M. (2020). Interaction in spoken academic discourse in an EMI
context: the use of questions. 6th International Conference on Higher Education Advances
(HEAd’20), 211-219. Doi: 10.4995/HEAD20.2020.11787
Johnson, J. H., & Picciuolo, M. (2022). The EmiBO Corpus. A resource for investigating
lecture discourse across disciplines and lecture modes in an EMI context. Lingue e
Linguaggi 53, 253-272. doi: 10.1285/i22390359v53p253.
Kamaşak, R., Sahan, K., & Rose, H. (2021). Academic language-related challenges at an
English-medium university. Journal of English for Academic Purposes, 49. doi:
10.1016/j.jeap.2020.100945.
Picciuolo, M., & Johnson, J. H. (2020). Contrasting EMI lecturers’ perceptions with practices
at the University of Bologna. In Miller, D.R.(ed.), Quaderni del CeSLiC. Occasional
papers. ISSN 1973-221X. Bologna: Centro di Studi Linguistico-Culturali (CeSLiC) e
Alma Mater Studiorum, Università di Bologna. AlmaDL, p. 23. doi:
10.6092/unibo/amsacta/6399.
524
Learning analytics dashboard to support instructors: a literature

review
Computer Science Department, Colorado State University, United States.
Abstract
The purpose of this study is to conduct an evaluation of the literature on
learning analytics dashboards in order to address the following research
questions: "How do instructors use learning analytics dashboards to send
notifications?" and "How can a learning analytics dashboard help students get
reminded of the due dates for assignments, quizzes, and exams?". A total of 20
papers were analyzed. Although the majority of them discussed how students
utilize dashboards to track their progress or compare their progress with their
peers, none of them mentioned how dashboards may be utilized to
automatically notify students or send reminders, reducing the amount of work
instructors have to do. This can be taken into account when developing a
dashboard for learning analytics in the future.
Keywords: Learning analytics; learning analytics dashboard; learning
management system; higher education.

Learning analytics dashboard to support instructors: a literature review
1. Introduction
Learning Analytics (LA) is a growing field that draws on many different fields of study, such
as Data Analytics (DA) and Artificial Intelligence (AI), to support and enhance education
(Nouri, 2019). Among the several uses of LA, including supporting learners self-regulated
process (Winne et al., 2019), prediction of students’ success (Chen et al., 2020), and analysis
of students study behaviors (Sher et al., 2022), there are works on Learning Analytics
Dashboards (LADs).
LADs can automatically analyze large amounts of data and deliver the results to relevant
stakeholders. It consists of "a single display that aggregates several indicators regarding the
learner(s), learning process(es), and/or learning context(s) into one or multiple
visualizations" (Schwendimann et al., 2016). The stakeholders in this context could be
instructors, students, and/or academic advisors. We are particularly interested in
understanding how instructors could use LADs to improve feedback and communication with
their students. Previous research has demonstrated that instructor feedback positively impacts
students' performance both in-person (Baranczyk & Best, 2020) as well as in online classes
(Lee, 2020). Considering that, this study aims to answer the following research questions:
RQ1: "How do instructors use learning analytics dashboards to send notifications?"
RQ2: "How can a learning analytics dashboard help students get reminded of the due dates
for assignments, quizzes, and exams?"
To answer those research questions, we conducted a literature review of the last five years
on learning analytics and learning analytics dashboards.
2. Methodology
We conducted a literature study by searching for and reviewing recent five-year (2018 - 2022)
published articles written in English in order to gather the most recent work and identify any
limits or prospective future research from those articles. The databases that we used were
IEEE Xplore, ACM Digital Library, Wiley Online Library and Google Scholar. These
sources were selected because they are known for publishing works related to the learning
analytics area. In particular, the ACM Digital Library is the current publisher of the Learning
Analytics and Knowledge Conference proceedings.
The keywords used for the search were: “learning analytics” and “learning analytics
dashboards”. In some cases, the search resulted in a large number of results; the number of
total distinct results is not reported because searching different sources returned many
repeated results across all searches.
526
For the search carried on IEEE Xplore, there are 8,073 search results for “learning analytics”
and 106 for “learning analytics dashboard." We skimmed the abstract of those 106 papers
and selected five papers which were related to LAD for teachers or students. Most of the
other papers were related to other dashboards.
There are over 150,000 results for "learning analytics dashboard" in the ACM digital library.
However, among those there were non-learning analytics-related papers because the search
also included "learning," "analytics," and "dashboard". We manually screened papers which
were relavant to LAD. We went through titles of first 1000 latest results and found that none
of them were related to LAD. We have gone through 100 relavant papers because after 100
results there were no papers for LAD. Among the relavant papers, we read 5 papers and
skimmed through 10 papers based on the titles. The 10 papers were not included in the total
papers we have analyzed because they were not related with LAD for teachers and/or
students.
We looked at the first 50 results for the term "learning analytics" in the Wiley Online Library
and read the papers and literature reviews among them that were published after 2018 (the
previous five years). After that, we read the first 20 results in the Wiley online library for the
keyword "learning analytics dashboard". The majority of the results were related to general
dashboards and not specifically dashboards related to learning aspects.
When we searched in Google Scholar using keywords “learning analytics” and “learning
analytics dashboard” most of the results were the same as which we have already read from
IEEE Xplore, ACM Digital Library, Wiley Online Library. So we searched for journals using
the phrase “International Journals of Learning Analytics dashboards” over the range 2018 -
2023. We got 16,700 results. From those, we read the first three journal papers presented in
the first page results, since we skimmed through the remainder papers in the first page and
they weren’t related to learning dashboards.
Besides the database searches, we used the snowball technique and read four papers from the
previous papers found. We read all papers and summarized their purposes in order to look
for information that would respond our research questions.
3. Results
A total of 20 papers were analyzed. From those, three are literature review papers and 17
were either conferences or journal papers. One literature review paper is in the area of LA
and the other two are related to LAD.
527
3.1. Previous Literature Review on LA and LAD

Ahmad et al. (2022) looked at 161 learning publications to investigate the function of learning
design in learning analytics. By offering a reference framework where they suggest potential
linkages between learning analytics and learning design, they sought to better align learning
design with learning analytics and to demonstrate the prior study and applications of LA
indicators and metrics. They discovered that a few learning analytics articles did, in fact, take
into account learning design activities for gathering user data. They also identified an ongoing
rise in the quantity and quality of indicators, as well as an evolution of these indicators
through time.
By leveraging theoretical foundations including human cognition and perception, context
awareness, and technology, designers need to have a thorough understanding of how learners
see and think in order for LA to be effective (Few, 2009). The degree to which the target
objectives of LAD are consistent with the domain metrics used to assess their effectiveness
is investigated in a Valle et al. (2021) literature study. Additionally, it states that regardless
of the intended target audience, the dashboard's primary purpose is to support learners as end
users.
In order to evaluate the influence on teaching and learning, a thorough evaluation of the
literature was done on the LADs research that presents empirical evidence. Self-regulated
learning has been cited in a number of earlier research reviews as the main area of interest
for LADs. The literature on self-regulated learning and how self-regulated learning is
supported has, however, received considerably less attention than it deserves. They reviewed
empirical data on LADs using Winne and Hadwin's well-known model of self-regulated
learning (Winne & Hadwin, 1998) in order to solve this limitation. The findings demonstrated
that existing LADs have significant shortcomings in how their evaluation is carried out and
reported, are rarely based in learning theory, cannot be proposed to support metacognition,
do not provide any information about efficient learning tactics and strategies, and cannot be
suggested to support metacognition (Matcha et al., 2020).
3.2. Previous Papers on LA and LAD

From the 17 papers, four were related exclusively to learning analytics. Those will be
described first in this section, followed by the description of the 13 papers related to LAD.
In order to comprehend and improve learning, learning analytics makes extensive use of data
on learner interactions in digital learning environments. Although measurement is a key
component of learning analytics, there hasn't been much study looking at how learning
analytics and assessment are related. In order to fully utilize the potential of the connections
between learning analytics and assessment, interdisciplinary teams will be required to
address task design, analysis of learning progressions, trustworthiness, and fairness, as well
528
as other pressing scientific and practical challenges and opportunities (Gašević et al., 2022).
Concerns about fairness and bias may be advanced by integrating learning analytics and
assessment. The amount of study on fairness and bias in learning analytics, much alone in
assessment using analytics, is, nevertheless, lacking (Gašević et al., 2022).
The application of learning analytics could help curriculum committees reflect on their
curricula. According to Chou et al. (2018), competency-based learning analytics are
suggested as a way to analyze competency-based curricula and graduates' academic records
of coursework in order to provide systematic evaluation information that could help
curriculum committees reflect on curricula, faculty teaching, and student learning. According
to Whitelock et al. (2019), in order for future implementations of LA to be successful, it is
essential to gauge student expectations of LA services. The authors presented a descriptive
instrument to assess students' ideal and anticipated expectations of LA services. They created
and validated a descriptive questionnaire that offers a robust and methodologically sound
solution to measuring student expectations of LA services using the identified expectation
themes (ethics and privacy, agency, intervention, and meaningfulness) and expectation types
(ideal and predicted). They found that while some characteristics of LA services may be
useful (such as the implementation of early alert systems), they may not be the features that
students anticipate (such as LA services created to enhance academic abilities like self-
regulated learning).
Ten of 13 LAD papers mentioned giving students feedback to track their development or
compare it to peers. Those LADs display data that students need to track their progress, such
as grades and how they compare to others, number of times they've read the course materials,
and amount of hours spent watching lecture videos.
A paper discussed about the student’s ability to get acquainted with the LAD’s and focused
on helping students practice self-regulated learning (SRL) and to keep track of the learning
progress with the help of technologies (Park et al., 2022). Another paper that addressed
queries raised by students in virtual breakout rooms did so by creating a dashboard (Stanislav
et al., 2021) . The design approach consists of: interviews with teachers in which questions
were asked to teachers about their current practices while monitoring groups of students.
Based on open answers from teachers, an inductive analysis is performed to map the critical
challenges they face with the evidence that can be used to address them. Codes emerged
through the analysis are grouped thematically. Based on the teachers’ questions identified in
the previous steps, the most highly ranked questions (i.e., those that address the concerns of
the majority of the participants) are explicitly added to the dashboard or LA user interface
(Echeverria et al., 2018).
In a study by Wang and Han (2020), LADs are created using process-oriented feedback,
providing both visual and textual feedback for various learning processes. A virtual learning
529
system called iTutor was used for the study. They upgraded iTutor by including LAD to give
students process-oriented feedback based on the performance and behavioral data that are
automatically recorded during students' learning processes. Students used those visual
feedbacks to compare their performance to that of other students.
Rienties et al. (2018) used an Analytics4Action (A4AW) workshop to examine how
instructors in a two-hour training tried to make sense of LAD and whether their technology
use affected their satisfaction with A4AW. The study found that 89% of participants were
pleased with the A4AW program. 68% of participants thought open university LA tools were
useful. (i.e., the degree to which teachers believe the use of LAD and visualizations will, for
example, improve the quality of their teaching or increase academic retention).
Jaramillo et al. (2022), proposed a dashboard to gather information about students who are
suspected of academic dishonesty. The dashboard is composed of information about how
students have interacted with the course materials and videos, how many forum posts have
been made, how many tests have been turned in, and what was the learner’s behavior on a
specific day or during a specific lesson.
4. Discussion and Conclusion

We examined learning analytics and learning analytics dashboard papers from the past five
years to see whether any answered this paper's research questions. We found out that no
papers addressed the research questions. We want to solve this literature gap by designing
and implementing a LAD to help instructors improve feedback and communication with
students.
According to Bouchrika (2022), learning management systems (LMS) are considered
mandatory in higher education and institutions are using them to display course material,
assignments, and quizzes in their courses. Professors generally create an announcements
section, which will be used by them from time to time to send out announcements.
Announcements can be used to convey a wide range of information, including reminders for
impending tests, projects, exams, and assignments, as well as information on altered class
times. The time spent by professors in keeping track of their students is huge, especially if
we consider courses with a large number of students (Hernández-García et al. 2015). Even if
the professors use the learning analytics already provided by the LMS there is a considerable
number of selections that needs to be done in order for the professor to know if the students
did or not did the assignment and which messages to send in each case. By introducing a
LAD, which can automatically gather information regarding due dates from assignments and
display this information to professors along with a tool to personalize deliver alerts about
such deadlines, we aim to reduce the amount of time professors spend on doing such
feedback. Through this automation, professors will benefit from the time and effort savings.
530
This literature review helped us to identify that there is no proposals for a LAD which can
help teachers reduce their burden by automating and personalizing student’s feedback. So we
plan to design a LAD that can send notifications to students automatically by taking the data
from the assignments, quizzes, or any other sections included in the course which would
address our research questions.
References
Ahmad, A., Schneider, J., Griffiths, D., Biedermann, D., Schiffner, D., Greller, W., &
Drachsler, H. (2022). Connecting the dots – A literature review on learning analytics
indicators from a learning design perspective. Journal of Computer Assisted Learning, 1–
39. https://doi.org/10.1111/ jcal.12716
Baranczyk, M. C., & Best, C. A. (2020). How does instructor feedback help and hurt
students’ academic performance? Scholarship of Teaching and Learning in Psychology,
6(1), 1–14. https://doi.org/10.1037/stl0000169
Bouchrika, I. (2022). 51 LMS Statistics: 2023 Data, Trends & Predictions. Retrieved from
https://research.com/education/lms-statistics
Chen, F., & Cui, Y. (2020). Utilizing Student Time Series Behaviour in Learning
Management Systems for Early Prediction of Course Performance. Journal of Learning
Analytics, 7(2), 1-17. https://doi.org/10.18608/jla.2020.72.1
Chou, C-Y., Tseng, S-F., Wang, C-J., Chao, P-Y, Chen, Z-H., Lai, R., … Yu, L-C (2018).
Learning analytics on graduates’ academic records to reflect on a competency-based
curriculum. Computer Applications in Enginnering Education, 26, 2168–2182.
https://doi.org/10.1002/cae.22019
Echeverria, V., Martinez-Maldonado, R., Granda, R., Chiluiza, K., Conati, C., & Shum, S.
B. (2018). Driving data story-telling from learning design. Proceedings of the 8th
International Conference on Learning Analytics and Knowledge. Association for
Computing Machinery, 131–140.
Few, S. (2009). Now you see it: Simple visualization techniques for quantitative analysis.
Analytics Press.
Gašević, D., Greiff, S., & Shaffer, D. W. (2022). Towards strengthening links between
learning analytics and assessment: Challenges and potentials of a promising new bond,
Computers in Human Behavior, 134, https://doi.org/10.1016/j.chb.2022.107304.
Hernández-García, A., Gonzalez-Gonzalez, I., Jimenez-Zarco, A. I., & Chaparro-Peĺaez, J.
(2015). Applying social learning analytics to message boards in online distance learning:
A case study. Computers in Human Behavior, 47, 68–80.
Jaramillo-Morillo, D., Ruipérez- Valiente, J. A., Burbano Astaiza, C. P., Solarte, M.,
Ramirez- Gonzalez, G., & Alexandron, G. (2022). Evaluating a learning analytics
dashboard to detect dishonest behaviours: A case study in small private online courses
with academic recognition. Journal of Computer Assisted Learning, 38(6), 1574–1588.
https://doi.org/10.1111/jcal.12734
Lee, J. W. (2020). The Roles of Online Instructional Facilitators and Student Performance of
Online Class Activity. Journal of Asian Finance Economics and Business, 7(8), 723-733.
531
Matcha, W., Uzir, N. A., Gašević, D., & Pardo, A. (2020) A systematic review of empirical
studies on learning analytics dashboards: A self-regulated learning perspective. IEEE
Transactions on Learning Technologies, 13(2), 226-245.
Nouri, J. (2019). Editorial of the First Issue of the International Journal of Learning Analytics
and Artificial Intelligence for Education. International Journal of Learning Analytics and
Artificial Intelligence for Education (iJAI).1,1(Jul. 2019), 4–7.
Park, E., Ifenthaler, D., & Clariana, R. B. (2022). Adaptive or adapted to: Sequence and
reflexive thematic analysis to understand learners' self-regulated learning in an adaptive
learning analytics dashboard. British Journal of Educational Technology, 54(1), 98-125.
Pozdniakov, S., Martinez-Maldonado, R., Singh, S., Chen, P., Richardson, D., Bartindale, T.,
… Gašević, D. (2021) Question-driven learning analytics: Designing a teacher dashboard
for online breakout romos. International Conference on Advanced Learning
Technologies, 176-178, https://doi.org/10.1109/ICALT52272.2021.00060.
Rienties, B., Herodotou, C., Olney, T., Schencks, M., & Boroowa, A. (2018). Making sense
of learning analytics dashboards: A technology acceptance perspective of 95 teachers.
International Review of Research in Open and Distributed Learning, 19(5).
https://doi.org/10.19173/irrodl.v19i5.3493
Schwendimann, B. A., Rodriguez-Triana, M. J, Vozniuk, A., Prieto, L. P., Boroujeni, M. S.,
Holzer, A., … Dillenbourg, P. (2016). Perceiving learning at a glance: A systematic
literature review of learning dashboard research. IEEE Transactions on Learning
Technologies 10(1), 30–41.
Sher, V., Hatala, M., & Gašević, D. (2022). When Do Learners Study? An Analysis of the
Time-of-Day and Weekday-Weekend Usage Patterns of Learning Management Systems
from Mobile and Computers in Blended Learning. Journal of Learning Analytics, 9(2),
1-23. https://doi.org/10.18608/jla.2022.6697
Stanislav, P., Martinez-Maldonado, R., Singh, S., Chen, P., Richardson, D., Bartindale, T.,…
Gašević, D. (2021). Question-driven learning analytics: Designing a teacher dashboard
for online breakout romos. International Conference on Advanced Learning Technologies
(ICALT), 176-178, http://doi.org/10.1109/ICALT52272.2021.00060.
Valle, N., Antonenko, P., Dawson, K., & Huggins-Manley, A. C. (2021). A systematic
literature review on learner-facing learning analytics dashboards. British Journal of
Educational Technology, 52, 1724-1748. https://doi.org/10.1111/bjet.13089
Wang, D., & Han, H. (2020). Applying learning analytics dashboards based on process-
oriented feedback to improve students' learning effectiveness. Journal Computer Assisted
Learning, 37, 487–499. https://doi.org/10.1111/jcal.12502
Whitelock Wainwright, A., Gašević, D., Tejeiro, R., Tsai, Y. S., & Bennett, K. (2019). The
student expectations of learning analytics questionnaire. Journal Computer Assisted
Learning, 35, 633–666. https://doi.org/10.1111/jcal.12366
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. Metacognition
Educational Theory Pract., 93, 277–304.
Winne, P. H., Teng, K., Chang, D., Lin, M. P.-C., Marzouk, Z., Nesbit, J. C., … Vytasek, J.
(2019). nStudy: Software for learning analytics about processes for self-regulated
learning. Journal of Learning Analytics, 6(2), 95–106.
https://doi.org/10.18608/jla.2019.62.7
532
Which career should I choose? Application of a pre-university

vocational guidance platform based on the Ikigai methodology
Ángel Millán1, Jorge García-Unanue2, Marta Retamosa1
1
Department of Marketing, Universidad de Castilla-La Mancha, Ciudad Real, Spain,
2
Department of Physical Activity and Sport Sciences, Universidad de Castilla-La Mancha,
Toledo, Spain.
Abstract
Choosing a university degree is a crucial decision in a person's life. However,
many students have chosen their career without proper counselling. Ikigai
refers to a Japanese concept that tries to find a person's purpose. This research
explains the implementation and results of an online platform based on the
Ikigai methodology. The Ikigai methodology was implemented through a
sequential process. A total of 172 schools were surveyed, involving 1,878
students. The predictive phase asked about: what participants study, interests,
hobbies, the social causes that motivate them and the future professions they
are interested in. Subsequently, an individual graph is generated with scores
in 15 subject areas. Based on the reports, the university sends a personalised
proposal of possible university degrees to each student by e-mail. The Ikigi
methodology is therefore a very useful tool that allows students to improve
their self-knowledge and the effectiveness of their decision-making processes,
among others.
Keywords: Ikigai; career counseling; career choice; career guidance
programs.

Aplication of a pre-university vocational guidance platform based on the Ikigai methodology
1. Introduction
Career building has major implications for both the individual and society. An economically
sustainable career involves costs during the various stages of education, which should be
transformed into an investment and provide benefits by means of the employee’s professional
activity.
Choosing a career is one of the most difficult tasks students have to confront when
completing secondary school, as it will largely determine their plans for the future. At this
stage, it is very important for students to receive adequate information about their career path
(Ehigbor and Akinlosotu, 2016). Many students have selected their careers without receiving
adequate advice from professional services, and this can result in discrepancies between their
academic performance, personality, interests and abilities. In order to provide
recommendations to students on how to choose the right career, it is important to develop a
referral system with which to guide them in their career choices (Razak et al., 2014).
The relevance of providing career guidance to students has been examined extensively in
several works (Loan and Van, 2015; Gordon and Steele, 2015; Zunker, 2015). Career
guidance counsellors can help individuals who are confronting career challenges (Whiston et
al., 2017). Their experience in career development and labor markets allows them to analyze
a person's qualifications, experience, strengths and weaknesses from a broad perspective,
while taking into account the desired salary, personal hobbies and interests, location, labor
market and educational possibilities (Savickas, 2019).
When evaluating career guidance programs, it is relevant to examine the methodologies used
to deliver those programs. In this respect, Watts and Sultana (2004) examined the vocational
guidance policies of 37 middle-income countries and found that that career guidance services
were provided in various formats, such as individually, in groups, face-to-face or remotely
(e.g. helplines and internet-based services).
Finally, Whiston et al. (2003) used meta-analytic techniques to compare different forms of
intervention (e.g. individual career counselling, career guidance classes) and found that
interventions that involved a counsellor were significantly more effective than interventions
without a counsellor. In particular, they found that the combination of a computer system and
a career guidance counsellor was more effective than allowing people to use only a
computerized counselling system.
Technology in the form of websites, social media and apps has now extended the resources
available to career counsellors. In this context, this work explains the implementation and
results of an online platform based on the Ikigai methodology. More specific, this platform
has been used as a vocational guidance tool for pre-university students in a region in Spain.
534
Ángel Millán, Jorge García-Unanue, Marta Retamosa
This project is the result of a strategic partnership between the educational administration
and a public university.
2. Theoretical background
2.1. Ikigai and career guidance

Ikigai is a Japanese construct that is concerned with finding one's purpose in life or reason
for being. Kondo (2007) states that Ikigai is “a sense of purpose and willingness; an
awareness that one is helping others, a sense that one has something to live for” and that
“this can also be felt when one is being acknowledged by others”. With regard to this
conceptualization, Kumano (2006) carried out a principal component analysis and found that
the central concepts of Ikigai were life affirmation, goals/dreams, the meaning of existence,
a sense of fulfilment and commitment.
Several pieces of research regarding the application of Ikigai to pre-university students have
been carried out, and studies have shown that it may be a useful tool with which to help
students identify their interests and goals, and plan their future careers (Eller, 2016; Hamzaid
et al., 2022).
It is possible to state that the use of Ikigai is helpful for many reasons. It provides the student
with a goal to strive for, and something more important than just momentary success to focus
on. Alternatively, it could even be used simply as a means to guide oneself toward obtaining
a degree or as a form of checklist for the education a student seeks to attain. In this context,
while it is not absolutely necessary for one to explore, or even create, one's own Ikigai, it
might be useful for students everywhere to look introspectively at why their education is
relevant and why they are obtaining it (Eller, 2016).
In the context of career counselling, the Ikigai methodology is frequently represented as a
Venn diagram comprising four circles: what you love, what you are good at, what the world
needs and what you can get paid for. Finding the Ikigai is, therefore, believed to bring a sense
of fulfilment and satisfaction.
3. Method and results
3.1. Methodology
In our work, the Ikigai methodology has been been implemented by employing a sequential
process in collaboration with secondary school counsellors, as described in Figure 1 (see
Figure 1). The students completed 4 questionnaires in the predictivie analytics phase, as
describe in Table 1 (see Table 1).
535
Figure 1. Ikigai implementation process. Source: The authors.
Each of the answers of these 4 questionnaires are linked to one or more of the 17 areas of
knowledge that have been established, based on the 5 main branches of knowledge, after a
process of research and co-design with specialists in the field of education and guidance. The
calculation of the final result is carried out automatically in the tool through a mathematical
algorithm that relates all the answers with the different percentage weights of each of the
questionnaires and the areas associated with each of the answers, assigning a score from 0 to
100 to each area. These scores are reflected visually in a spider graph and the area or areas
of knowledge that have achieved the highest score are presented in a table of vocational
suggestions.
536
Table 1. Description of the questionnaires applied in the predictive analysis phase.
Questionnaire Description Question example
Academic Specilaisation Questionnaire regarding Choose your current studies or the

what the participants specialisation that corresponds most
were studying closely with them. (limited list of
current studies in secondary school
and professional education).
What you love and what the Questionnaire on Studying new languages for fun and
world needs interests and hobbies that not as an imposition.
motivate them in their Thinking about the creation of new
daily life devices to be licensed and launched
on the market
Questionnaire on social Participating in campaigns to help
causes that motivate homeless people.
them Fighting bullying among your
classmates.
What they will pay for you Questionnaire concerning Analyzing data using technology.
(professional life) future professions of Functioning of the brain in the
interest to them learning process (neuroeducation).
Source: The authors.
3.2. Results
A total of 172 secondary schools were surveyed, with the participation of 1,878 students. As
for the characterisation of the sample, they are all first and second year students in the upper
secondary school. Figure 2 shows the percentage of respondents according to the area with
the highest score for each of them. It can be seen that Education is the most repeated
(54.01%), followed by computer technology (13.95%) (see Figure 2).
After completing the questionnaires, the online platform generates an individualized report
for each student called a predictive report. Specifically, a graph is generated with scores
related to 15 knowledge areas on a scale of 0 to 100 points. Examples of two students'
predictive reports are shown in Figures 3 and 4.
In the case of Student 1 (see Figure 3), the predictive report shows a clear vocational
orientation toward the field of computer engineering. As a second possibility, it indicates that
this student also has a clear orientation toward marketing and advertising studies. In the case
of Student 2 (see Figure 4), the predictive report indicates a clear vocational orientation
toward tourism studies, followed by arts and humanities. The predictive report also indicates
studies in education as a third vocational alternative.
537
The university uses the students' reports as the basis on which to create a personalised
proposal of possible university degrees that will best suit the vocational profile of the pre-
university student. This proposal is then emailed to each individual student.
60% 54,01%
50%
40%
30%
20% 13,95%
10% 4,65% 4,91% 6,20% 6,46%
0,52% 0,78% 1,03% 1,29% 1,55% 1,81% 2,84%
0%
Ed ce…
ism
es
rS n
n
y
s
um n
le
w
gy
gy
ce
tio
io
io
og
op
l T erin
iti
en
La
lo
lo
at
ot
en
ur
ol
tra
an
Pe
ci
uc
no
o
ci
To
hn
e
hn
s
in
ro
lS
ni
ch
ec
c
g
p
te
i
H
e
te
en
ta
dm
ot
pu
d
en
d
an
nd
Bi
nd
A
an
m
ia
nm
a
Co
la
str
rts
sin
ce
ro
nd
du
n
A
ti
vi
ie
la
In
er
En
Sc
dv
vi
Ci
od
A
Fo
Figure 2. Sample profile according to the preferred area. Source: The authors.
Arts and humanities

Language and
Health sciences 100
literature
Food science and 80
Tourism
technology 60
40
Architecture Law
20
Civil and land 0 Advertising and

engineering promotion
Computer science
Education
engineering
Industrial technology Administration

Environmental
Biotechnology
sciences
Figure 3. The predictive report of student 1. Source: The authors.
538
Arts and humanities

100 Language and
Health sciences
literature
Food science and 80
Tourism
technology 60
40
Architecture Law
20
Civil and land 0 Advertising and

engineering promotion
Computer science
Education
engineering
Industrial technology Administration

Environmental
Biotechnology
sciences
Figure 4. The predictive report of student 2. Source: The authors.
4. Conclusions
The experience gained after applying the Ikigi methodology to the vocational guidance
process of secondary school students makes it possible to conclude that it is a very useful and
operational tool. It particularly enables students to achieve the following objectives: improve
their self-awareness, improve the efficiency of their decision making processes, obtain
information of interest concentrated in a single report, obtain a brief and interactive
assessment, and achieve immediate results.
Furthermore, the application of the Ikigai methodology provides a number of benefits to
secondary school counsellors and improves the effectiveness of their work.. It particularly
allows the following objectives to be achieved: (1) carrying out quick analyses that facilitate
the counsellors’ work, (2) automating part of the guidance work while keeping the
importance of their mentoring intact, (3) monitoring progress in real time, (4) obtaining an
overview of thestudents’ preferences, and (5) using the data collected to generate statistics.
From the university's point of view, the main problem is that of correctly adapting its
provision of degree studies to the vocational profiles proposed by the Ikigai tool. This
adaptation requires intensive work that it is difficult to automate. Moreover, the increase in
fields of knowledge and specialization planned by the administration in secondary education
will imply a compulsory and complex process of readjustment in the areas included in the
Ikigai tool.
539
References
Ehigbor, B. O., & Akinlosotu, T. N. (2016). Parents’ occupation as correlate of students’
career aspiration in public secondary schools in Ekpoma Metropolis. AFRREV IJAH: An
International Journal of Arts and Humanities, 5(3), 197-212. doi: 10.4314/ijah.v5i3.17
Eller, R. (2016). Ikigai and higher education: A review of the literature. AU eJournal of
Interdisciplinary Research (ISSN: 2408-1906), 1(2).
Gordon, V. N., & Steele, G. E. (2015). The undecided college student: An academic and
career advising challenge. Charles C Thomas Publisher.
Hamzaid, N. A., Usman, J., Jewaratnam, J., Khuen, C. C., Rahim, S. S. A., & Azmi, M. F.
(2022). Parents Involvement in Young STEM Learners and Talent Development: A Pilot
Study. Kuala Lumpur International Conference on Biomedical Engineering, 243-250.
Kondo, T. (2007). Ikigai wo Hakaru (Measuring the reason to live). Nakanisiya syuppan.
Kumano, M. (2006). The structure of ikigai and similar concepts. Japanese Journal of Health
Psychology, 19(1), 56-66. doi.org/10.11560/jahp.19.1_56
Loan, D. T. B., & Van, N. T. (2015). Career guidance in secondary schools-A literature
review and strategic solutions for Vietnamese rural areas. American International Journal
of Social Science, 4(5), 135-143.
Razak, T. R., Hashim, M. A., Noor, N. M., Abd Halim, I. H., & Shamsul, N. F. F. (2014).
Career path recommendation system for UiTM Perlis students using fuzzy logic. 5th
International Conference on Intelligent and Advanced Systems (ICIAS), 1-5. IEEE.
Savickas, M. (2019). Career counseling (pp. xvi-194). Washington, DC: American
Psychological Association.
Watts, A. G., & Sultana, R. G. (2004). Career guidance policies in 37 countries: Contrasts
and common themes. International Journal for Educational and Vocational Guidance,
4(2), 105-122. doi.org/10.1007/s10775-005-1025-y
Whiston, S. C., Brecheisen, B. K., & Stephens, J. (2003). Does treatment modality affect
career counseling effectiveness?. Journal of Vocational Behavior, 62(3), 390-410.
doi.org/10.1016/S0001-8791(02)00050-7
Whiston, S. C., Li, Y., Mitts, N. G., & Wright, L. (2017). Effectiveness of career choice
interventions: A meta-analytic replication and extension. Journal of Vocational Behavior,
100, 175-184. doi.org/10.1016/j.jvb.2017.03.010
Zunker, V. G. (2015). Career counseling: A holistic approach. Cengage learning.
540
Quest-based Gamification in a software development lab course: a

case study
Nuno H. Flores1,2, Rui Pinto3
1
Department of Informatics, Faculty of Engineering of the University of Porto, Portugal,
2
INESC-TEC, Portugal, 3SYSTEC-ARISE, Faculty of Engineering of the University of
Porto, Portugal.
Abstract
Motivation and engagement play a crucial role in student success in a course.
Students may lose interest or underestimate courses that tackle non-core
learning outcomes to their specific curriculum or program. Gamification,
using game elements (e.g., rewards, challenges) in non-game contexts, is one
way to motivate and engage students. Some educational courses use project-
based learning, where students tackle problems, overcome obstacles, and gain
knowledge. Quest-based games are designed as systems of challenges that
players must complete to advance and win the game. They were linked with
education by applying specific game mechanics to a computing course unit.
This paper case studies the application of a quest-based gamification
approach in a mandatory software engineering course to boost engagement
among higher education students. Results were collected through
observational methods and surveying the students, indicating a tendency for
higher grades in course years implementing gamification while maintaining
satisfactory levels of motivation and engagement.
Keywords: Gamification; education; quest-based games; software
Engineering.

Quest-based Gamification in a software development lab course: a case study
1. Introduction
Motivation and engagement are crucial factors in determining a student's success in a course.
However, students often disengage or undervalue courses they perceive as not central to their
curriculum or program. This can negatively impact the achievement and quality of desired
learning outcomes. One way of increasing the levels of engagement is through gamification
(Alsawaier, 2018).
Gamification involves incorporating elements of games into non-game contexts, as explained
by Deterding, Khaled, Nacke, and Dixon (2007), such as education (Tsai, 2016). This can
include using rewards and challenges to promote intrinsic and extrinsic motivation, as
Surendeleg, Murwa, Yun, and Kim (2014) pointed out.
Based on their contents, specific educational courses adopt a project-based learning strategy.
Students engage in problem-solving activities: a series of actions to move from an initial state
to a desired end state while overcoming obstacles and building knowledge (Jonassen &
Rohrer-Murphy, 1999) (Ward, 2012).
Quest-based games are intentionally built as systems of problem-solving tasks that players
must accomplish to progress and ultimately win the game (Howard, 2008). Gameplay
activities (completion of specific objectives related to quests) provide a logical narrative
structure, determining the story and progression within the game, as stated by Smith,
Anderson, Kopleck, Lindblad, Scott, Wardell, and Mateas (2011). Fabricatore and López
(2014) linked quest-based games and educational courses by identifying and implementing
five patterns of game mechanics in a computing course.
This paper presents a case study where gamification was used to increase the engagement of
higher education students in a mandatory informatics engineering-based course unit not
perceived as central to the overall curriculum. While revising the syllabus, a quest-based
system was applied, entitled “World of LPRO”, following the game mechanics stated by
Fabricatore and López (2014)). Results were collected through observational methods and
surveying the students. Research aimed at answering the following research questions:
• RQ1. Regarding student engagement, what was the impact of a Quest-based

Gamification on student grades?
• RQ2. Regarding student motivation and interest, how do students perceive the
Quest-based Gamification approach?
The paper is organised into four additional sections. Section 2 provides context on Quest-
based Gamification and outlines the core gameplay patterns. Section 3 showcases the case
study, a programming lab course taught in an Electrical and Computer Engineering master's
program. Section 4 details the student data collection methodology and presents the results
obtained. Finally, Section 5 concludes the paper with final remarks on the study.
542
2. Adopting Quest-based Gamification

Game mechanics are crucial in determining the player experience, as they are central to the
gameplay and must be mastered for success (Fabricatore, 2007). These mechanics make the
gameplay appealing and motivating for players (Polaine, 2005). According to Fabricatore
and López (2014), quest-based games' core mechanics are divided into five gameplay
patterns, as described in Table 1.
Table 1. Quest-based games' patterns of gameplay.
Pattern of Gameplay
Quest structure (QS): Quests consist of: (I) A goal that requires completing specific tasks to meet
the victory conditions, together with optional conditions for additional challenges and rewards; (II)
The resources or tools needed to achieve the objective and (III) The significance of the quest in terms
of progressing in the game. Quests are conveniently described (usually through a thematic narrative)
so the player knows the expected outcomes.
Strategic open-endedness (SOE): Quest objectives can be accomplished through multiple

strategies, giving players the freedom to "do more" or "do things differently." This enables players
to try different methods of play and encourages them to attempt increasingly complex strategies as
their skills improve.
Non-linear progression (NLP): Players are usually free to choose when they want to complete a
quest. When this is impossible, it is usually because of the quest's narrative or how it connects to
other quests. The quest's briefing and debriefing stages provide information about its availability
and how it is related to other quests.
Orientation (O): The game provides briefing information through resources such as maps,
accessible to the player at any time to aid in decisions about when, how, and what to engage.
Challenge-based reward (CBR): Rewards are typically given according to the principle of "the
more you achieve, the more you receive." Meeting the primary victory conditions earns a standard
reward. Accomplishing more difficult tasks earns additional rewards, such as extra resources and
recognition of increased skill (such as a promotion in the game). In quests involving teamwork,
rewards are based on each player's contributions.
Source: adapted from Fabricatore and López (2014).
When applying the gamification approach to the course, although not restricted to these, there
was an attempt to enforce all five gameplay patterns. For clarity, when describing a specific
gamification element in the next section, the pattern(s) acronym will be annexed between
italic box brackets [] (e.g. if an element incorporated the “Challenge-based reward” pattern,
the [CRB] notation would appear next to it), whenever relevant.
543
3. Programming Lab: A Case-Study

The presented approach was applied to a programming lab course (LPRO) in the 1st semester
of the 4th year of a 5-year Electrical and Computer Engineering integrated master programme
(Faculty of Engineering of the University of Porto [FEUP], 2013). Although not central to its
core syllabus, LPRO was a mandatory course, and students had basic programming skills
from a couple of courses in the 1st year. Its purpose was to endow the students with sufficient
software development aptitudes to seamlessly integrate a professional development team.
LPRO’s contents encompassed three main topics, namely i) JAVA: Object-Oriented
Programming (OOP), using Java as a supporting programming language and JUnit testing;
ii) UML: Unified Modeling Language (Kaur & Singh, 2011) for documenting the artefacts
of software systems; and iii) PROCESS: Scrum (Rising & Janoff, 2000) for software project
management and development. Lectures briefly overviewed the main topics, engaging
students in autonomous, self-paced learning. At the lab, teams of students developed a
software project (80% of the final grade) and hands-on exercises to consolidate the learning
outcomes. 20% of the final grade came from two individual tests.
The World of LPRO setting used a role-playing game-like medieval fantasy narrative where
the player character (student) starts as an inexperienced adventurer, clueless about the fine
arts of Software Development, and thrives to become a skilful master.
3.1. Main plot

The main plot (Figure 1a) that drives the adventurers comprises three challenges: a main
quest, side-quests and trials of passage. To be successful, the student must finish the main
quest, complete 3000 XPs (eXperience Points) worth of side-quests and overcome the two
trials of passage.
Figure 1. Main Plot elements (on the left); XP Levels (on the right).
Grouped in pairs or triads, adventurers prepare for the main quest by completing side quests.
Side quests [QS] are small-scale hands-on exercises to develop, improve and enhance
student skills, covering the three main topics (five per topic). Some are mandatory (assuring
544
minimum topic coverage), while others are optional (Figure 2) [O]. Growing in complexity,
each side-quest awards the adventurers with XPs [CBR], allowing promotion up a series of
levels (Figure 1b) until attaining "Developer" (3000 XPs).
Adventurers can tackle the side-quests in any order they choose [NLP]. However, following
the "Path of Enlightenment" [O, CBR] by completing certain side-quests until a specific
deadline awards enough extra XP for reaching the final level without needing an extra side-
quest.
Trials of passage are two individual tests. The first consists of a mid-course 2-hour unit-test-
driven JAVA programming challenge, and the second consists of a final course one-hour-long
multiple-choice questionnaire covering UML and PROCESS. Failing one means failing the
course.
The main quest [QS] is the course-long software development project, where adventurers
apply all their attained skills, grouped in a party (team) of four or five. Although having some
required outcomes, adventurers are free to tackle the intrinsic challenges as they see fit
[SOE].
Figure 2. Side-quests schema.
3.2. Rewards and Special Items

As motivators for completing more side-quests (thus further consolidating their skills),
adventurers are rewarded with special items [CBR] that will allow for grading bonuses. Here
is where "Magic" enters the narrative.
A Wand is rewarded if an adventurer completes all five side-quests regarding a specific topic.
Wands cast magical spells and can be used to compensate for faults or failures on the trials
545
of passage (Figure 3a). Moreover, adventurers can combine Wands into Staffs (Figure 3b) for
more potent spells, even allowing extra credits on the main quest.
Figure 3. Rewards and Special Items: Wands (on the left); Staffs (on the right).
4. Results
Although starting in 2007, the course had the gamification approach introduced in 2017/2018,
lasting for four more occurrences (as of 2021/22, the course's contents were redefined and
adapted to a new Master Programme).
Students' final grades data was collected and compared with the equal period before adopting
gamification to assess the impact on overall success. To answer RQ1., we considered the
quantitative data collected, represented in Figure 4. Final grades tend to improve in gamified
course instances (2017/2018 and beyond) compared to non-gamified instances (before
2017/2018).
Figure 4. LPRO final grade average before and after introducing gamification.
Also, students' feedback was surveyed to measure the perceived gamification's effects. The
survey (Figure 5) was about the new gamification-like learning experience, where students
had to rate statements using a five-point Likert scale: (1) Strongly disagree; (2) Disagree; (3)
Neither agree nor disagree; (4) Agree; (5) Strongly agree. They also had room to provide
further comments on the course.
546
Figure 5. LPRO student feedback survey results.
Analysing the survey results and answering RQ2., students considered that introducing
gamification and following the "Path of Enlightenment" performed well. Also, they
considered the system of rewards and special items highly motivating.
However, despite stating that side-quest complexity levels were adequate, they felt the "Path
of Enlightenment" was time-consuming and required extra work, thus preferring a more self-
paced approach. One critique focused on the lack of direct connection between the reward
system and actual weight in the student grade.
Overall, special items were valuable to help low-grade students not to fail the course while
enabling high-grade students to improve their final grades further. The fewer Wands needed
to pass the trials of passage (in case of borderline failure), the more one could improve the
final grade (by combining them into Staffs). One of the most popular (and coveted) items
used by students was the Staff of Divine Development since it increased by 1 point the main-
quest grade (on a grading scale between 0-20), translating into a 0.8-point increase in the
final grade (a plausible explanation for the increase in the average grade in gamified
instances).
5. Conclusion
This paper applied a quest-based gamification strategy to a non-core mandatory software
engineering course. We discussed how gamification can improve student engagement and
motivation by linking different quest-based games' core mechanics to the teaching and
learning component. Results indicate a tendency for higher grades in course years
implementing gamification. Student feedback suggests that the gamified-based course is
motivating and gripping while suitable for the assessment components.
References
Alsawaier, R.S. (2018), "The effect of gamification on motivation and engagement",
International Journal of Information and Learning Technology, Vol. 35 No. 1, pp. 56-79.
https://doi.org/10.1108/IJILT-02-2017-0009
547
Deterding, S., Khaled, R., Nacke, L., & Dixon, D. (2011, May). Gamification: Toward a
definition. CHI 2011 Gamification Workshop Proceedings. In 2011 Annual Conference
on Human Factors in Computing Systems (CHI’11).
Fabricatore, C. (2007) “Gameplay and game mechanics design: a key to quality in video
games”, in Proceedings of the OECD-CERI Expert Meeting on Videogames and
Education (Vol 14), OECD, July, Santiago, Chile
Fabricatore, C. & López, X. (2014, October). Using gameplay patterns to gamify learning
experiences. In Proceedings of the 8th European conference on game based learning (pp.
110-117). UK: Academic Conferences and Publishing International Limited, Reading.
Faculty of Engineering of the University of Porto [FEUP]. (2013, November 19). Master in
Electrical and Computers Engineering. Retrieved January 27, 2023, from
https://sigarra.up.pt/feup/en/CUR_GERAL.CUR_VIEW?pv_curso_id=741
Howard, J. (2008). Quests: Design, Theory, and History in Games and Narratives. CRC
Press.
Jonassen, D. H., & Rohrer-Murphy, L. (1999). Activity theory as a framework for designing
constructivist learning environments. Educational technology research and development,
47(1), 61-79. https://doi.org/10.1007/BF02299477
Kaur, H., & Singh, P. (2011). UML (Unified Modeling Language): standard language for
software architecture development. In International Symposium on Computing,
Communication, and Control (pp. 313-320).
Polaine, A. (2005) “The flow principle in interactivity”, in Y. Pisan (Ed.), Proceedings of the
Second Australasian Conference on Interactive Entertainment (IE2005) (pp 151–158) ,
Creativity and Cognition Studios Press, Sidney, Australia.
Rising, L., & Janoff, N. S. (2000). The Scrum software development process for small teams.
IEEE Software, 17(4), 26-32.
Smith, G., Anderson, R., Kopleck, B., Lindblad, Z., Scott, L., Wardell, A., ... & Mateas, M.
(2011). Situating quests: Design patterns for quest and level design in role-playing games.
In Interactive Storytelling: Fourth International Conference on Interactive Digital
Storytelling, ICIDS 2011, Vancouver, Canada, 2011. Proc. 4 (pp. 326-329).
Surendeleg, G., Murwa, V., Yun, H. K., & Kim, Y. S. (2014). The role of gamification in
education–a literature review. Contemporary Engineering Sciences, 7(29), 1609-1616.
https://doi.org/10.12988/ces.2014.411217
Tsai, M. J., Huang, L. J., Hou, H. T., Hsu, C. Y., & Chiou, G. L. (2016). Visual behavior,
flow and achievement in game-based learning. Computers & Education, 98, 115-129.
https://doi.org/10.1016/j.compedu.2016.03.011
Ward, T. B. (2012). Problem solving. In Handbook of organizational creativity (pp. 169-
187). Academic Press.
548
Retail design education. Designing new and reframed learning

tools for experience-based learning
Politecnico di Milano, Design Department, Milano, Italy.
Abstract
The retail industry is a fast-changing sector characterized by innovations’
openness to adoption dictated both by technological advancement, supply
chain management transformation and consumer behaviour evolution. As an
increasingly knowledge-intensive industry, updated retail skills and
competencies need to be investigated and improved, promoting new
educational and learning approaches.
The paper presents the results of an experience-based learning held within the
“Fashion Retail Experience Studio” course at the School of Design of
Politecnico di Milano and rooted in a project-based approach. Several
learning tools were developed and applied in order to investigate how design
can properly manage multi-level and multidisciplinary retail challenges
fueling and generating meaningful innovation. A transdisciplinary perspective
was also adopted in designing new design learning tools or adapting
management, marketing and IT retail tools to enhance design competencies
and skills with a holistic approach.
Keywords: Retail design; retail education; experiencial-based learning
experience; design learning tools; transdisciplinary perspective.

Retail design education. Designing new and reframed learning tools
1. Introduction to Retail Transformation

A radical transformation has characterised the retail sector over the last twenty years. On the
one hand, the progressive dematerialisation of goods and the consolidation of the so-called
service economy have mutually fuelled each other, driving the retail transformation from a
predominantly product-centric to a service-centric approach (Lusch & Vargo, 2006). Retail,
from being primarily focused on goods’ transactions, has become capable of catalysing and
promoting value co-creation within an interdependent system of service innovation, meaning
generation and consumer centrality (Kustrak Korper et al., 2021). On the other hand,
technological acceleration has pushed towards a radical change in the economic and social
sphere with significant effects on manufacturing and distributive systems and on collective
living and behaviours. In particular, new digital channels (social media and web platforms
from traditional online channels to the metaverse) and advanced technologies (AR/VR, IoT,
advanced 3D modelling technologies, adn Digital Twins, AI, etc.) have rapidly entered the
production and distribution systems profoundly impacting their processes and operations and
multiplying the touch points connecting the consumer with products/services in an
experiential and relational continuum (Shankar et al., 2021). The consumer experience is now
embedded within a mature omnichannel system that requires a seamless connection in a
complex and interwoven system of physical, digital and mixed channels, streamlining and
fast-tracking operations and the compliance with consumer expectations (Hoyer et al., 2020).
The retail system has thus proven to be a remarkably open and receptive sector to the adoption
and promotion of innovation with impacts on business models, supply chain management,
and consumer value delivery models (Mostaghel et al., 2022). Moreover, being a highly
transformational field, the compentence domains involved into the retail system are multiple
and constantly redefined their roles, hierarchies and intervention models. Retail design,
marketing, management and, more recently, IT converge within a system that requires
transdisciplinarity (Iannilli et al., 2019a) and design, in particular, needs to redefine its role
and develop new tools and approaches to bring value to the system. In this context, a
reflection on retail design education becomes necessary both to frame the new levers that
design can use to promote significant retail innovations and explore the most effective
competencies and skills that will be relevant in the future.
2. Insights from the Evolution of Retail Education

In light of this changing scenario, the research in retailing is prolific, with multiple scientific
researches aiming both at identifying the impacts and areas of application of advanced
technologies within retail processes and outlining and conceptualising the sector’s innovation
trajectories. However, the interest in retail education appears to be more limited (Pantano et
al., 2020) and does not have the same spread within all disciplinary fields that nowadays
interplay within the retail system. Assuming a disciplinary perspective, marketing education
550
appear to be the areas that first recognised how the relevance of retail transformations needed
updating in terms of both competencies and learning approaches. Grewal et al. (2018) address
the issue by examining major changes in retail and consumer landscapes, tying them to
changes in retail education. Taking a historical view on retailing education, the authors
highlight its past, its present, and the likely future of retailing technologies and education,
emphasising how technologies must become an opportunity to enhance active learning.
Within the editorial introducing the 2018 special issue “Educating the Retailers of
Tomorrow” in the Journal of Marketing Education, Roggeveen and Beitelspacher (2018)
provide insights on innovative teaching methods, innovative pedagogical tools, and an
overview of retailing education. The first two sections, in particular, showcase a variety of
educational experiences. On the one hand, they demonstrate the effectiveness of experiential,
situated and skill-based learning (Lange et al., 2018; Rhee, 2018), able to provide wins for
students, partnering retailers, and faculty when retailing curricula developed cross-functional
partnerships with retailers. On the other hand, these experiences demonstrate how innovative
pedagogical tools can be incorporated into classrooms to improve students’ knowledge and
skills to effectively integrate relevant retail issues (e.g. Mobile Retailing, Corporate Social
Responsibility) (Beitelspacher & Rodgers, 2018; Fischbach & Guerrero, 2018). The
experiential nature of all the described learning experiences is consistent within the special
issue’s articles. This approach results in a student’s active involvement with consequent
positive impacts on knowledge advancements through hands-on activities and reflective
practices. Similarly, faculty and retailers jointly benefit from exploring contemporary
challenges in retailing (Lange et al., 2018) and allowing retailers to reflect on the
competencies of young prospective practitioners.
The experience-based learning approach is also crucial from a retail management perspective
and, of course, a retail design perspective. Soft skill development, simulated navigation of
roles, support in pursuing success, and bridging the gap between textbook and real-world
emerge as the main positive outcomes for students in participating in comprehensive course
projects exposing them to work with retailers or companies to solve particular issues the firm
faces (Jones et al., 2021). Similarly, the need for collaboration in educational curricula
between educational institutions, here, in particular, Higher Educational Institutions (HEIs),
professionals and retailers, becomes crucial to provide students with up-to-date competencies
suited to the changing retail sector. Technology-driven changes, specifically, have rekindled
attention to retail education. The technological transformation’s speed indeed imposes a need
to provide digital skills (or hard skills). However, it simultaneously requires reinforcing soft
skills to understand and fully exploit the potential of new technologies (Woods et al., 2022).
This viewpoint has also been adopted by retail design. Research and theoretical
conceptualisations in this area are still scarce but sufficient to signal a renewed need both to
redefine the role of design in contemporary retail and to investigate the expected
competencies of future retail designers. Mainly assuming an interior design perspective,
551
Quartier et al. (2020) claim a need for a multi-dimensional competence model to frame all
the required contemporary competencies (in terms of knowledge, abilities, skills, and
attitudes) and to cope with future change dynamically. A set of meta-competences and multi-
level retail design competencies was framed, bringing together design, research, socio-
cultural studies, communication, branding, marketing, omnichannel and management. This
multi-dimensional perspective at the core of the design approach is also mirrored in retail
design research and learning experiences, thus both developing HEIs curricula capable of
increasing digital literacy in knowledge-intensive and creative sectors harmonising new
educational methods (e.g. Massive Open Online Courses MOOCs) within project-based
design studios (Iannilli et al., 2019a) and merging transdisciplinary, theoretical and
operational knowledge in design and experiential-based learning (Iannilli et al., 2019b).
3. The need of a holistic approach to contemporary retail design

Today, retail design can be considered as a specific transdisciplinary design discipline, which
concerns the design of virtual or physical spaces for selling products, services and/or brands
to consumers (Quartier et al., 2020). In their works Quartier et al., try to update the list of
competencies and skills needed in contemporary retail design, with an eye on both
educational and professional finalities. The authors (2017) set the new requirements for the
retail designer in the age of phygit, asserting that now more than ever designers should
assume a holistic approach and that trans-disciplinary work is necessary to manage the
complexity of customer experience. They further argue that the approach to the contemporary
retail designers’ education path should be aimed at: (i) understanding how digital
technologies can be applied and how they work; (ii) ability to generate creative ideas; (iii)
ability to think across channels starting from the customer journey and technology
integration, considering variables and conditional factors.
Architects’ and designers’ work is typically characterized by the use of tools. Some meet the
coordinative functions as objects of persuasive communication while others help to develop
a general understanding of an idea or a task and others still may work as recall of design
principles, approaches, methods or open questions. Still, some others help to keep control of
the activities and materials while others represent the design decisions to a predetermined
level of detail and technical precision (Lança & Loução, 2013). The new directions in retail
experience design raise questions about which educational and professional tools need to be
integrated in designers’ educational path. Firstly, the questions address which tools designers
use to generate creative ideas, to think across channels and, to integrate technology. Further,
it is to understand how these tools have been modified or need to be updated to be respondent
to the new requirements. Furthermore is important to understand how to foster trans-
disciplinary work; and facilitate the dialogue and the exchange of ideas and concepts between
the different, heterogeneous actors during the customer experience design process. This
552
discourse is a necessary premise to understanding the motivations underlying this work,

which describes the experience in the “Fashion Retail Experience Studio” course held within
the Design for the Fashion System master degree - Politecnico di Milano. Building on the
strong and established relationship between design and experiential learning theories (Iannilli
et al., 2022), several tools were involved in the teaching activities aiming at letting the
students gain “some intellectual concepts from the very beginning that become part of a
practical activity enriching it” (Dewey J., 1961). The course has been structured following
design-project phases, namely: “meta design”, the first project phase including a research and
analysis step and an early concept definition step; “concept development”, the phase in
which the concept starts to be shaped and its meaning defined; “project definition” is when
all the details are defined and the final project is clearly described in all its parts.
The tools map (fig. 1) correlates project phases and project tools, moreover, since retail
customer experience design is a transdisciplinary activity, and so are the tools used, a label
shows from which discipline each tool originates. It is interesting to notice the presence of
some overlapping of labels between the disciplines regarding some of the tools, that is where
transdisciplinary and holistic-approach work happens. Even if the map cannot still be
considered exhaustive, and has improvement margins, it has been a good starting point in the
choice-making process for the tools to use in the course as didactic tools.
Figure 1. Retail Design Experience Tools Map.
4. Fashion Retail Experience Studio – In-class experience

The “Fashion Retail Experience Studio” course addressed in this work dates back at spring
2022, at Politecnico di Milano. It involved sixty-eight international students attending
Fashion System mater degree, at their first year of study, a team of four retail experience
design professors with strong professional backgrounds and Deutsche Telekom, a
telecommunications company, providing solutions for business and corporate customers. The
553
students, divided into teams of five to six people, were asked to design the concept of a
phygital retail experience, able to create innovative and valuable relationships among
contemporary consumers and fashion products, services and physical/digital spaces.
The students were guided through the whole course, with frontal lessons and individual in-
class activities supported by specific tools, aimed at consolidating key concepts and/or having
a first-hand experience in applying them. In the design process, as group work, students were
guided to the use of specific tools aimed at clarifying and supporting the organization,
creation, communication and discussion of concepts and ideas. For the individual work, the
tools used were: the “24h inventory” tool and the “empathy map”. The 24h inventory activity
aimed at consolidating the ability to analyse consumers’ habits and preferences. The empathy
map, from the marketing field, helps to schematize knowledge about end-users in order to
create understanding of user needs in decision-making processes. In the group work, through
the design process, the students were supported by the following tools: “research boards”,
“scenario moodboard”, “fictional characters”, “personas” “customer journey map”,
“storyboard”. In the first project phase of research and analysis, students were asked to
organize their research using research boards. This tool was specifically adapted to the course
requirements, for helping students at categorizing their research findings on technology
applications by specific focuses. “Scenario moodboards” were used to visually describe the
first project’s concepts and directions. “Fictional characters” and “personas” were used to
understand and describe the ideal customer to whom the project is addressed, by describing
lifestyle, personal interests, and tastes, in a schematic and visual way. “Customer journey
map” belongs to the concept development phase, and aims at the description of all the steps
through which the customers go while approaching the purchasing experience, also
considering pre- and post-purchase steps, and possible variations. The “storyboard” tool, also
referred to the concept development phase, clarifies the details of “customer journey map”,
by the use of a narrative example. Following the briefing agreed with the company involved,
the course stopped at concept development phase, since students were not asked to build real-
size simulations or actual prototypes of their ideas.
5. Conclusions
Retail designers of the present and next generations need to develop a sharp view on the
transformations happening and an extensive set of competencies and skills, which will allow
them to understand and operate in the increasing complexity of contemporary retail design
and customer experience. In this work a teaching experience is presented, where
transdisciplinary tools were designed and/or reframed to help develop a holistic view and
comprehension while designing complex phygital retail systems. Withdrawing from related
literature, the tools adopted refer to the disciplines involved in retail esperience design,
namely: the “24h inventory” tool belongs to the realm of social studies on consumption, as
554
an observation tool; “empathy map”, “fictional characters” and “personas” are part of
marketing and user experience design disciplines; “research boards”, “scenario moodboard”,
and “storyboard” come from the design realm; “customer journey map” is a intersciplinary
tool used in the collaboration between management, marketing, information-technology and
design disciplines. The tools involved in the teaching experience described in this work were
specifically aimed at developing the ability to generate creative ideas (“research boards”,
“scenario moodboard”, and “storyboard”) and ability to think across channels considering
variables and conditional factors (“24h inventory”, “empathy map”, “fictional characters”,
“personas”, “customer journey map”). The requirement of understanding how digital
technologies can be applied and how they work, was covered during the course as well, with
the aid of frontal lessons delivered in the class, desktop and case study research conducted
by students. With this work, our contribution lies in gaining a better understanding of how
the developments of digitalisation and omni-channel retailing influence the education of
(future) retail designers, and of how tools borrowed from professionals’ experience and from
different disciplines can be integrated in the learning experience, with the aim of applying a
holistic approach to retail experience design. Although improvements can still be applied to
future versions of the course, specifically in the development of specific tools related to the
aims of the didactic activities, the learning outcomes were satisfying, and the students’
engagement was high.
References
Beitelspacher, L., & Rodgers, V. L. (2018). Integrating Corporate Social Responsibility
Awareness Into a Retail Management Course. Journal of Marketing Education, 40(1),
66–75. https://doi.org/10.1177/0273475318754933
Dewey J., (1961). Come pensiamo. Una riformulazione del rapporto fra il pensiero
riflessivo e l’educazione. Firenze: La Nuova Italia.
Fischbach, S., & Guerrero, V. (2018). Mobile Business Retailing: Driving Experiential
Learning on Campus. Journal of Marketing Education, 40(1), 56–65.
https://doi.org/10.1177/0273475317733509
Grewal, D., Motyka, S., & Levy, M. (2018). The Evolution and Future of Retailing and
Retailing Education. Journal of Marketing Education, 40(1), 85–93.
https://doi.org/10.1177/0273475318755838
Iannilli V.M., Di Iorio M. (2022), Making Ideas Visible: Design Approach In Teaching
Abstract Concepts Through Hands-On Activities, Edulearn22 Proceedings, pp. 7322-
7329. 10.21125/edulearn.2022.1712
Iannilli V.M., Linfante V., Spagnoli A. (2019a). Convergence and Integration between
Disciplinary Knowledge and Transversal Skills in the Fashion Area. ICERI2019
Proceedings, 7708-7717. https://doi.org/10.21125/iceri.2019.1831
555
Iannilli V.M., Penati A.V., Spagnoli A. (2019b). Integrated Design Learning Methods in
Fashion Retail Design Studio. EDULEARN2019 Proceedings, 4938-4947.
https://doi.org/10.21125/edulearn.2019.1231
Hoyer, W. D., Kroschke, M., Schmitt, B., Kraume, K., & Shankar, V. (2020). Transforming
the Customer Experience Through New Technologies. Journal of Interactive Marketing,
51, 57–71. https://doi.org/10.1016/j.intmar.2020.04.001
Jones, D., Childs, M., Frankel, S., & Loveday, M. (2021). Experience-based learning:
Exploring meaningful outcomes for students in retail and fashion. International Journal
of Fashion Design, Technology and Education, 14(1), 58–68.
https://doi.org/10.1080/17543266.2020.1859625
Kustrak Korper, A., Holmlid, S., & Patrício, L. (2021). The role of meaning in service
innovation: A conceptual exploration. Journal of Service Theory and Practice, 32(2),
179–198. https://doi.org/10.1108/JSTP-01-2020-0004
Lança, L., & Loução, M. D. (2013). Retail Design. Do we need a project instrument or a
project tool? Revista Lusófona de Arquitectura e Educação, 08–09, 595–619.
Lange, F., Rosengren, S., Colliander, J., Hernant, M., & Liljedal, K. T. (2018). Bridging
Theory and Practice in an Applied Retail Track. Journal of Marketing Education, 40(1),
6–16. https://doi.org/10.1177/0273475318755837
Lusch, R. F., & Vargo, S. L. (2006). Service-Dominant Logic: What It Is, What It Is Not,
What It Might Be”. In R. F. Lusch & S. L. Vargo (Eds.), The Service-Dominant Logic of
Marketing: Dialog, Debate, and Directions. Routledge.
Mostaghel, R., Oghazi, P., Parida, V., & Sohrabpour, V. (2022). Digitalization driven retail
business model innovation: Evaluation of past and avenues for future research trends.
Journal of Business Research, 146, 134–145.
https://doi.org/10.1016/j.jbusres.2022.03.072
Pantano, E., Pizzi, G., & Rogers, A. (2020). Who is interested in retail education? The
(mis)match between the leading universities’ offerings and job demand in the UK.
International Journal of Retail & Distribution Management, 49(3), 317–340.
https://doi.org/10.1108/IJRDM-04-2019-0139
Quartier, K., Claes, S., & Vanrie, J. (2020). A holistic competence framework for (future)
retail design and retail design education. Journal of Retailing and Consumer Services, 55,
101914. https://doi.org/10.1016/j.jretconser.2019.101914
Quartier, K., Claes, S., Vanrie, J. (2017). Rethinking the education of retail design. What are
the competences a retail designer needs in this fast evolving discipline?. Design Journal,
20 (sup1), S1285–S1292. https://doi.org/10.1080/14606925.2017.1352657
Rhee, E. (2018). Teaching Community-Based Learning Course in Retailing Management.
Journal of Marketing Education, 40(1), 47–55.
https://doi.org/10.1177/0273475317753680
Roggeveen, A. L., & Beitelspacher, L. (2018). Educating the Retailers of Tomorrow: An
Overview of Journal of Marketing Education Special Issue on Retailing. Journal of
Marketing Education, 40(1), 3–5. https://doi.org/10.1177/0273475318763667
556
Shankar, V., Kalyanam, K., Setia, P., Golmohammadi, A., Tirunillai, S., Douglass, T.,
Hennessey, J., Bull, J. S., & Waddoups, R. (2021). How Technology is Changing Retail.
Journal of Retailing, 97(1), 13–27. https://doi.org/10.1016/j.jretai.2020.10.006
Woods, R., Doherty, O., & Stephens, S. (2022). Technology driven change in the retail
sector: Implications for higher education. Industry and Higher Education, 36(2), 128–
137. https://doi.org/10.1177/09504222211009180
557
Analysis of the feasibility of investment projects in real assets with

PBL: a very real experience
Isabel Abinzano, Harold Bonilla, Pilar Corredor, Cristina Del Rio, Elena Ferrer, Ana
González, Jose Manuel Mansilla, Beatriz Martínez, Luis Muga
Department of Business Administration, Public University of Navarre (UPNA), Pamplona,
Spain.
Abstract
This paper studies the application of the PBL methodology in the Corporate
Finance I course. The project to be carried out consists of the realization of a
report on the feasibility of an investment project for a company that wants to
take advantage of a subvention to finance the renovation of the bus fleet
towards a more sustainable one. In addition to describing the implementation,
this paper analyzes the impact that the introduction of the PBL methodology
has in terms of class attendance and participation in the activity and also in
the rest of the course. A clear decrease in absenteeism in class and in exams is
observed. Moreover, the impact on grades is analyzed, with a significant
increase in marks for all the degrees under study. Finally, we interpret the
surveys that were passed to the students, showing that the students recognize
the value of applying PBL in the subject.
Keywords: PBL; corporate finance; involvement; qualification; assessment.

Analysis of the feasibility of investment projects in real assets with PBL: a very real experience
1. Introduction
Project Based Learning (PBL) is an active teaching methodology in which the student learns
the concepts of a course by carrying out a project. According to Blank (1997) and Harwell,
(1997), the project must be linked to a real-world issue whose purpose is to help the student
to learn through cooperative work in search of solutions or answers.
The PBL methodology relies on the active role of the student in the learning process, and the
starting point is the realization of a project that they want to carry out (Christie and de Graft,
2017). Students work in groups to identify what they need to learn to solve the project. The
teacher's role is to advise, mediate, and collaborate with students. According to Hmelo-Silver
(2004), “the objectives of PBL include helping students to develop flexible knowledge,
practical problem-solving, learning and collaboration skills, and intrinsic motivation”. This
way, PBL prepares students more effectively for the future.
Among the advantages of this methodology is the increased motivation and involvement of
the students. Thus, Robinson (2013) and Salam et al. (2016), among others, find that the
project-based learning approach results in a student-centered environment with higher
engagement and motivation. As suggested by Thijs and Verkuyten (2009), the more engaged
students are, the more successful they will be in their learning since the lack of motivation
and commitment can be barriers to learning if the importance of the work is not clear to the
student. A consequence of this higher involvement is the improvement of student scores, as
can be seen in Requies et al. (2018) and Abinzano et al. (2022), among others.
This methodology emerged at McMaster University (Hamilton, Canada) in the 1960s to
tackle the demotivation of Medicine students. As Servant-Miklosa (2019) points out, that
experience eventually became known as ‘project-based learning’ (PBL) and has since spread
throughout the world and not only to medicine but also to a number of fields of education.
Finance is one of the disciplines in which the PBL methodology has been applied. Thus,
Young and Legister (2018) apply PBL in the undergraduate subject of International Finance,
while Parrado-Martínez and Sánchez-Andújar (2020) apply this methodology to the subject
"Finance" of an MBA.
In this paper, we present the application of the PBL methodology in the subject Corporate
Finance I of the degree in Business Administration and Management (BA&M) and the double
degree in Business Administration and Law (BA&L) of the Public University of Navarre
(UPNA). Moreover, we analyze the change in the participation and results of the course with
respect to the previous year, in which the traditional methodology with lectures and partial
exams was applied, as well as the evaluation of the new methodology. This implementation
is part of a project of educational innovation in the Financial Economics area, supported by
560
Isabel Abinzano et al.
the Public University of Navarre, which consisted of the introduction of active methodologies
in the four subjects of the field “Basic Finance”.
2. Applying PBL in Corporate Finance I
2.1. Context
Corporate Finance I is one of the four subjects of the Basic Finance field, all of them
compulsory, which start in the first year of the Degree in BA&M with Financial Operations.
Then, the study of Basic Finance continues in the second year with Financial Markets and
Instruments, and ends in the third year with Corporate Finance I in the first semester and
Corporate Finance II in the second semester.
The aforementioned Educational Innovation Project started in the academic year 2021/2022
with the implementation of the PBL methodology in Financial Markets and Instruments.
Following the success of this implementation and the assessment of its strengths and
weaknesses (see Abinzano et al., 2022), it was decided to implement the PBL methodology
in Corporate Finance I.
Corporate Finance I is a course in the first semester of the third year of the degree in Business
Administration and Management, in three groups, and in the first semester of the fourth year
of the double degree in Business Administration and Management and Law, with a single
group. This course studies investment in financial assets (Unit 1) and real assets (Unit 2),
specifically the evaluation of investment projects. Given the suitability of the PBL
methodology, it was decided to implement it in the part dedicated to investment in real assets.
As for the contents taught in the subject prior to the implementation of PBL, Table 1 details
the units included in the PBL.
Table 1: Contents of Corporate Finance until academic year 21/22.
UNIT 1: INVESTMENT IN FINANCIAL ASSETS. THE VALUATION OF RISK

1. PORTFOLIO THEORY
2. THE FINANCIAL ASSET PRICING MODEL (CAPM)
UNIT 2: INVESTMENT IN REAL ASSETS
3. INVESTMENT DECISIONS IN A CERTAINTY ENVIRONMENT (I): STAND-ALONE
INVESTMENT
4. INVESTMENT DECISIONS IN A CERTAINTY ENVIRONMENT (II): INVESTMENT
MANAGEMENT
5. INVESTMENT DECISIONS IN A RISK ENVIRONMENT
Both Unit 1 and Unit 2 were taught until 2021/2022 year with lectures, computer practices,
partial tests, and a final exam. The former evaluation can be found in Table 2.
561
Table 2: Former assessment system.
Assessment system Weight Character

Final written exam covering the knowledge acquired. 60% Recoverable
Minimum mark: 5/10
Individual theoretical-practical test of Unit 1 15% Non-recov.
Individual theoretical-practical test of Unit 2 15% Non-recov.
Group practices in the computer room 10% Non-recov.
2.2. The project

In order to better engage students in PBL, it was decided to start the course with the real asset
investment part, i.e., to invert the order of Units 1 and 2. The content of the former Unit 2
was mostly taught with the PBL methodology, with the exception of some issues, which were
taught using the traditional methodology after finishing the project.
Following the steps necessary to build a project that ensures correct learning, suggested by
Blank (1997), Harwell (1997), and Dickinson et al. (1998), among others, the project was
defined as follows. The project consists of the preparation of a financial feasibility study
report for the acquisition of a low-carbon bus by the Navarre company Sociedad de
Automóviles Río Alhama S.A. (ARASA) in order to take advantage of the aid of the
Government of Navarre within the Spanish aids published in the BOE on November 16, 2021,
for the transformation of the passenger transport fleet. The product to be submitted is the
report that our consultancy would deliver to the company. This report is composed of a pdf
document (6 pages and annexes) and an Excel file with all the calculations and analysis.
The students must carry out the project through the completion of three versions, 0, 1 and 2.
For this purpose, the work is established in seven weeks, organized as shown in Table 3.
Table 3: Organizatión by weeks of the PBL.
Week Resultado de aprendizaje

1 Presentation, group formation, puzzle, distribution of topics, individual study.
2 Work in groups of specialists. Exercise resolution, individual study.
3 Work in groups of specialists. Exercise resolution, self-assessment.
4 Clarification of doubts. Beginning of Version 0.
Presentation and delivery of Version 0. Individual extensions and distribution.
5
Incorporation of the extensions to Version 0.
6 Presentation and delivery of Version 1
7 Presentation and delivery of Version 2. Final individual exercise + basic knowledge test.
The students were organized into groups of three and each member was a specialist in one of
the pieces of the puzzle shown in Table 4. Through the virtual platform, the material for each
of these pieces was provided, consisting of notes, some basic exercises, and some more in-
562
depth exercises. In addition, the competing groups had to carry out a more extensive case
similar to the project evaluation.
Table 4: Distribution of contents through the puzzle.
Puzzle piece Contents

A Initial investment and cost of capital
B Cash-flow calculation
C Decision criteria
The assessment of the full course is shown in Table 5.

Table 5: Current assessment of the subject Corporate Finance I.
Weight Method of Assessment Character

On-time delivery (20% PBL) Non-recoverable
Version 1 of the project (15% PBL) Non-recoverable

Unit1 Version 2 of the project (30% PBL) Non-recoverable
50%
(PBL)
Individual exercise (15% PBL) Non-recoverable
Test of basic knowledge (20% PBL) Recoverable
Unit 2 50% Exam Recoverable
As for the evaluation of the versions of the project, the students have the rubric of the versions
at their disposal from the beginning of the course, so that they can take into account the
contents under evaluation and, if necessary, the punctuation.
3. Results
3.1. Students’ involvement

In line with the results obtained by Robinson (2013) for laboratory Science students, and by
Salam et al. (2016) for ICT students, we obtain that student involvement improves
considerably with the introduction of PBL in the course. On the one hand, class attendance
in the four groups analyzed was practically 100% during the time in which the project that
addresses Unit 1 is being developed, with a drop of approximately 30% when switching to
the traditional methodology in Unit 2. On the other hand, participation in exams was higher
compared to participation in the evaluation in the previous course. In Table 6 we appreciate
that for the BA&M groups, the participation in the unit studied with PBL is higher in the
course where the PBL methodology is applied, and also the participation in the whole course.
563
In addition, we notice that the increase in participation is higher for the part seen with PBL
than for the whole course. In the case of the double degree in BA&L, there are no differences,
since participation was total before applying this methodology, which is in line with the
results obtained for these two groups in Abinzano and Gonzalez (2015).
Table 6: Students’ non-attendance before and after PBL.
2021/2022 2022/2023
Non-attendance Non-attendance
Group Students Former Students
Whole PBL Whole
Unit 1
BA&M 155 14.19% 1.94% 170 1.76% 1.18%
BA&L 29 0.00% 0.00% 21 0.00% 0.00%
All the groups 184 11.96% 1.63% 191 1.57% 1.05%
3.2. Qualifications
In terms of scores, Table 7 also shows a considerable improvement with the implementation
of the PBL.
Table 7: Mean and standard deviation of marks before and after PBL.
2021/2022 2022/2023
Unit 1 (former
Whole subject Unit 1 with PBL Whole Subject
Unit 2)
Group Mean SD Mean SD Mean SD Mean SD
BA&M 4.8305 1.9024 5.1171 1.8860 7.3050 1.2880 6.2385 1.3890
BA&L 7.3401 1.7051 7.2688 1.4916 8.9350 0.3770 8.6108 1.2011
All the groups 5.2797 2.1018 5.5303 2.0049 7.4870 1.3241 6.5694 1.5353
While the mean for the overall group of Unit 1 (former Unit 2) was 5.2797 before
implementing the PBL, with the implementation it rises to 7.4870, this difference being
significant at 1%. In addition, the standard deviation has also been reduced. As for the overall
results of the course, the mean has also increased by more than one point out of 10, and with
lower dispersion. Individually, we can see that although the involvement has not changed for
the students of the double degree (see Table 6), their achievement, measured by their marks,
has changed, which implies a better use of active methodologies by the students. These results
are consistent with the ones obtained by Requies et al. (2018) and Abinzano et al. (2022).
3.3. Students’ survey

After finishing Unit 1, we passed to the students a survey with different questions, in order
to know their opinion and thus to detect possible problems to be fixed. Table 8 shows the
aggregate responses to some of the questions.
564
Table 8: Results from the survey of all the groups.
Quite To a large
Not at all A little
a lot extent
To what extent do you think PBL outperforms
5.93% 24.58% 46.61% 22.88%
traditional teaching as a learning method?
To what extent do you think that PBL has helped
you learn about the evaluation of investment 0.85% 22.03% 49.15% 27.97%
projects?
To what extent do you think that PBL has allowed
you to get in touch with the financial reality of 3.39% 20.34% 55.93% 20.34%
companies?
Yes No
Would you recommend the PBL methodology in
80.51% 19.49%
university studies?
We observe that students recognize the additional contribution of PBL, and most of them
would recommend the methodology.
3.4. Feasibility of the idea

We must emphasize that the idea of the project was very well received by the students, as it
was something identifiable as real. In fact, a few days after the end of the PBL activity, a
story appeared in the press about the launch of new electric bus lines from the company
analyzed. In addition, the characteristics of the investment to be made by the company were
easy to understand for the students, as it is a sector they know closely as the users they are.
4. Conclusions
In this paper, we describe the application of the PBL methodology to the part of investment
projects of the Corporate Finance I course. In addition, we analyze the impact of this
implementation in terms of class attendance, and participation in exams, resulting in a
reduction in the number of absences. Moreover, we analyze the effect on marks, observing a
significant increase in scores for all the degrees studied, not only at the PBL level, but also
with effects on the entire course. Finally, we interpret the surveys and find the contribution
of PBL valuable and that students would recommend this methodology for other courses.
References
Abinzano, I., González, A. (2015). La implantación del acuerdo de Bolonia y su efecto en las
calificaciones finales de los estudiantes: El caso de Dirección Financiera I. In: Ballester,
L. y Carchano, O. (coord.), V Jornada de Intercambio de Experiencias de Innovación
Educativa en Finanzas. Ed. Tirant lo Blanch. ISBN: 9788490867679.
565
Abinzano, I., Corredor, P., del Rio, C., Ferrer, E., González, A., Mansilla, J.M., Martínez, B.,
Muga, L. (2022). Enhancing Learning in the Finance Classroom. 8th International
Conference on Higher Education Advances (HEAd’22), 355-362.
Blank, W., 1997, Authentic instruction, in W.E. Blank and S. Harwell (Eds.), Promising
practices for connecting high school to the real world, (pp. 15–21), Ed. Tampa.
Christie, M., de Graaff, E. (2017). The philosophical and pedagogical underpinnings of active
learning in engineering education. European Journal of Engineering Education, 42(1),
5–16.
Dickinson, K.P., Soukamneuth, S., Yu, H.C., Kimball, M., D'Amico, R., Perry, R., Kingsley,
C., & Curan, S.P. (1998). Providing Educational Services in the Summer Youth
Employment and Training Program. Technical Assistance Guide.
Harwell, S., 1997, Project-based learning, in W.E. Blank and S. Harwell (Eds.), Promising
practices for connecting high school to the real world, (pp. 23–28). Ed. Tampa.
Hmelo-Silver, C.E. (2004). Problem-Based Learning: What and How Do Students Learn?
Educational Psychology Review, 16(3), 235-266.
Parrado-Martínez, P., Sánchez-Andújar, S. (2020). Development of competences in
postgraduate studies of finance: A project-based learning (PBL) case study. International
Review of Economics Education, 35, 100192.
Requies, J. M., Agirre, I., Barrio, V. L., Graells, M. (2018). Evolution of project-based
learning in small groups in environmental engineering courses. Journal of technology and
science education, 8(1), 45-56.
Robinson, J.K. (2013). Project-based learning: improving student engagement and
performance in the laboratory. Analytical and Bioanalytical Chemistry, 405, 7–13.
Salam, F., Mailok, R., Ubaidullah, N., Ahmad, U. (2016). The effect of project-based
learning against students’ engagement. International Journal of Development Research,
6 (02), 6891-6895.
Servant-Miklosa, V.F.C. (2019). Fifty Years on: A Retrospective on the World's First
Problem-based Learning Programme at McMaster University Medical School. Health
Professions Education, 5, 3–12.
Thijs, J., Verkuyten, M. (2009). Students’ Anticipated Situational Engagement: The Roles
of Teacher Behavior, Personal Engagement, and Gender. The Journal of Genetic
Psychology, 170, 3, 268-286.
Young, J.H., Legister, A.P. (2018) Project-Based Learning in International Financial
Management. Journal of Teaching in International Business, 29(1), 76-87.
566
A framework for developing mathematical tasks for automatic

formative assessment in higher education
Clara Horvath, Andreas Körner, Lana Međo
Institute of Analysis and Scientific Computing, TU Wien, Austria, Strategic Education
Development Center, TU Wien, Austria.
Abstract
Mathematics education in STEM studies poses a challenge that students must
practice the methods they learn to use them accurately. The mapping of
teaching exercises is an important step towards creating a more engaging,
interactive learning experience for students. An automatic assessment system
can support such a process and help overcome obstacles towards a better
learning environment for students. Developing effective support in form of
high-level mathematical tasks for this type of assessment is challenging and
requires careful consideration of the goals, objectives, and content of the task.
With this paper, we aim to provide practical insights and references for
teachers looking to develop various mathematical tasks that can provide
meaningful feedback and support learning, while taking into account the
limitations of automatic assessment in higher education. Additionally, this
paper addresses the possibilities and challenges that arise in this educational
process and provides examples of different applications.
Keywords: Automatic assessment; e-learning methodology; innovative
teaching; algorithmic randomization; STEM education.

A framework for developing mathematical tasks for automatic formative assessment
1. Introduction
The integration of technology into education has brought numerous benefits to the teaching
and learning process. Computer-Assisted Instruction (CAI) has been shown to have a positive
impact on student achievement and attitude in general as well as in higher education, but its
effect size has been limited, see Kulik and Kulik (1991), Schmid et al. (2014). The effects'
average size for achievement as reported by these meta-analyses ranges between 0.26 and
0.27 and therefore remains medium-low throughout the decades. Kulik and Kulik (1991)
identify the developments of CAI at higher education levels to be less successful than those
in elementary and secondary schools, whereas Schmid et al. (2014) identified subject matter
as the most influential predictor of effectiveness and noted that the effects of technology-use
were higher in non-STEM subjects.
In order to address these findings and reinforce the potential of technology-enhanced learning
environments, it is important to understand the benefits and limitations of various
instructional tools and assessment methods.
One such method that has gained popularity in recent years is Automatic Formative
Assessment (AFA), which has the potential to support higher student achievement.
Automatic assessment can be used to provide individual real-time feedback to students,
allowing for an immediate response and helping to create a more engaging and interactive
learning experience, addressing the conclusion of Schmid et al. (2014) that “learning is best
supported when the student is engaged in active, meaningful exercises via technological tools
that provide cognitive support” (p. 285).
Schmid et al. (2014) define cognitive support as “the category which encompasses various
technologies that enable, facilitate, and support learning by providing cognitive tools” (p.
274). The development of effective cognitive support in form of high-level mathematical
tasks for AFA is challenging and requires careful consideration of the goals, objectives, and
content of the task. For example, an automatic assessment system which does not provide
automatic grading beyond algebraic expressions constrains the type of tasks that can be
developed. In order to maintain the benefits of AFA while still allowing for a wide selection
of tasks, the teacher must, as Fisher (2006) warned, assume the role of an agent of change.
To support this role, this paper explores the development of mathematical tasks for AFA in
higher education and presents a framework for its adaptation and optimization. Through this
framework, we aim to provide practical insights and references for lecturers looking to
develop various high-level applied mathematical tasks. By emphasizing the goal of achieving
comparable difficulty for exam tasks and maximizing the randomization potential for
exercise tasks, this framework helps to create meaningful feedback for students and supports
their learning process, while taking into account the limitations of automatic assessment.
568
2. Exploring the Potential of Automatic Assessment Systems to Enhance Math

Learning
According to the literature, an effective way to study mathematics at the university level is
through active learning, i.e., independent solving of mathematical problems and hands-on
contact with mathematical methods, see Rosenthal (1995). The challenge for teachers is to
create a learning environment that enables students to solve these problems on their own and
is suitable to guide and support them throughout the semester to achieve the goals of the
course.
Automatic Assessment Systems (AAS) describe computer-aided applications that provide
information on mathematical tasks and automatically evaluate student responses. AAS offer
a conveniently accessible platform through which students can receive learning materials as
well as grading and feedback services (Ihantola, Ahoniemi, Karavirta, & Seppälä, 2010).
As discussed by Barana, Marchisio, and Sacchet (2021), high-quality interactive feedback
has a particularly positive effect on students, helping to improve their performance and
support them in class preparation. Teachers are supported in the creation of learning materials
by AAS, as these programs provide tools for the randomization of tasks, thus continuously
offering new learning materials to students, as noted in the findings of Ihantola et al. (2010).
In the following subsections, we will present approaches that support the process of task
development for AAS, specifically, Möbius (formerly known as Maple T.A.).
2.1. Enhancing Mathematical Tasks through Randomization

Randomization is a powerful tool that can be used to create a wide variety of mathematical
objects, such as numbers, vectors, and matrices.
In Möbius, the generation of randomized variables is accomplished through the use of Maple
commands. These commands offer the possibility to assign certain properties to the variables,
such as restricting a parameter to be in a certain range or specifying the shape of the matrix.
For example, rand(1..10) restricts a random number to be within the range of 1 to 10.
Similarly, LinearAlgebra[RandomMatrix](3,3, generator=-3..3, shape=diagonal) generates
a 3x3 matrix with random integer entries on its diagonal, ranging from -3 to 3. Using such
Maple commands, available in libraries for Möbius, improves the precision of developed
examples, allowing for more specific yet versatile tasks.
Furthermore, the possibility of drawing from a pool of predefined variables enables the
randomization of text by assigning single words up to entire sentences to these variables.
Consequently, the same assignment provides different tasks for the students at different
attempts; for instance, specifying on continuity of rational functions or on the differentiability
of logarithmic functions.
569
It is important to note that while the randomization approach introduces variability, it does
not necessarily guarantee the comparability or solvability of the tasks. This topic will be
explored in more detail in Section 3.
2.2. Enhancing Grading of Mathematical Expressions
Another essential component of any AAS is grading student responses. In Möbius, the
computer algebra system Maple is utilized to efficiently and reliably grade equivalent
algebraic expressions. Nonetheless, grading any other types of mathematical expressions,
such as trigonometric equations, intervals, and general solution of linear systems, is an area
of difficulty. To overcome this and enhance the grading functionalities, custom grading codes
are developed.
A sustainable solution to manage and grade these more complex mathematical expressions
conveniently would be to compile the custom grading codes in form of a library. Handily,
Möbius supports the integration of Maple libraries through the repository in their so-called
Maple-graded question types, allowing for more effective integration of the self-developed
Maple procedures. When writing procedures, incorporating functionalities such as grading
with partial credit can provide more finely-tuned feedback on the task level. This kind of
grading system offers a more precise way to score a question than default scoring methods,
allowing for a more nuanced understanding of a student's knowledge and skill. As an
example, Figure 1 shows a problem of an under-determined system of linear equations, where
the student is tasked with determining the general solution.
Figure 1. Example of Möbius question with an illustrated partial-credit evaluation in the response area of the
answer field.
570
In this case, the provided answer is incomplete since a particular solution is missing.
Typically, this would result in the response being marked incorrect; however, instead of a
full deduction, partial credit is awarded.
The library allows for a more efficient and consistent approach to grading, as the same
grading code can easily be reused. It can also serve as a basis for a custom grading system
incorporating further functionalities and design choices, which can go as far as optimizing
the library for custom syntax. In turn, this allows teachers to focus on other quality aspects
of their tasks.
Furthermore, the library concept can be extended to randomization as well. Zimmermann et
al. (2010) provide a concrete example of library implementation to improve the grading and
randomization aspect of AAS. This publication was the starting point of the development of
customized grading and randomization libraries for the AAS Möbius at TU Wien. These
libraries enabled greater flexibility in designing mathematical tasks and opened new
possibilities for their creation, further emphasizing the importance of custom grading and
randomization as enhancement methods in designing mathematical tasks.
3. Task Differentiation Based on Use Case Scenario

AAS can be used to provide teaching material for students, as well as for examinations. These
different use case scenarios require specific adaptations and considerations in the
implementation of the tasks. We propose a differentiation of tasks based on their respective
use cases and discuss them in the following subsections.
3.1. Exercise Tasks or Encouraging Learning and Proficiency through Practice

By using randomization with a wide range of possible parameters, lecturers can provide a
large variety of examples for students to study and practice the required skills and abilities.
The difficulty of a task might vary due to the parameters, which is desirable in practice
scenarios as students learn to master the same skill on different levels of complexity.
According to Barana et al. (2021), “interactive feedback can be effective for the development
of Mathematical knowledge … ” (p. 17) and is more elaborate than simple right or wrong
feedback. It should consist of an instructive feedback guide to engage students to solve
mathematical problems after failed attempts, see Marchisio et al. (2020). With this in mind
and in line with Leikin's (2014) definition of the mathematical challenge as a mathematical
difficulty that a person is motivated to overcome, we further endorse the use of so-called
adaptive questions.
Möbius offers a predefined adaptive question type that incorporates randomization
capabilities and an automated grading system, similar to standard questions. However, it also
571
guides students to the correct path for solving the given problem after a predefined maximum
number of unsuccessful attempts. After an initial failure, the difficulty of the problem is
reduced and dissected into smaller and simpler sub-problems and the students' answers in
this adaptive section are graded. Depending on the correctness of the answer, more hints or
instructions are given. The next problem definition to achieve the correct solution is displayed
accordingly. Therefore, an adapted path to the solution of the mathematical problem is
provided.
Although the proposed steps are one of usually many possible solution paths and the level of
individualization and allowed creativity is limited, adaptive assignments offer students
valuable first-level support when tackling tasks. Another valuable feature of adaptive
assignments is that individual intermediate steps change in accordance with the initial
problem, therefore every student receives a personalized guide at every attempt.
3.2. Exam Tasks or Ensuring Fairness and Consistency in Assessment

Using AAS during examinations provides the possibility to conduct exams in locations other
than lecture halls. Before the global COVID-19 pandemic, this feature may not have seemed
necessary; however, it is now indispensable. Stowell and Bennett (2010) report that students'
anxiety can be reduced when having the opportunity of taking online exams compared to
exams in classrooms or lecture halls. Additionally, AAS tests can be used to provide feedback
to students away from regular exams, for example in preparation for a midterm or final test
that counts towards the final grade.
Providing mathematical assignments via AAS during exams prevents students from copying
solutions from neighboring students during in-person exams as well as exchanging solutions
via messaging services such as WhatsApp during online exams due to randomized and thus,
individualized exercises. Therefore, e-learning examinations enhance the credibility of exam
results, since they address security concerns such as identifying students with their work and
threats of cheating, see Karim and Shukur (2016).
However, the high variability of possible values and texts in questions poses difficulties for
teachers since unwanted problematic realizations of the randomization cannot be easily
prevented when using standardized randomizing Maple functionalities. Especially in exam
situations, undefined values which cause the overall task to become unsolvable need to be
prevented. An exemplary problematic realization would be the difference of two parameters
in the denominator of a term leading to division by zero if equality is not manually excluded.
Above all, the comparability of the tasks also must remain given for reasons of fairness.
Teachers can use the same task repeatedly in exam situations since the randomization
guarantees different values for each attempt. By drawing from a pool of predefined variables,
cheating can still be minimized, and tasks can be reused in future exams without greatly
572
increasing the effort required to create them. For instance, consider a task in which a student
is asked to compute eigenvalues and eigenvectors of a given matrix. In this case, the switch
command can be used to randomly select from a predefined set of matrices that impose an
equivalent level of difficulty to the task's given context.
Furthermore, Sangwin (2013) suggests using reverse engineering as an advanced approach
to address the challenge of comparable task difficulty in exam situations, while also allowing
for greater task versatility.
4. Discussion
With the proposed framework, we are addressing the comparatively low effectiveness of
technology use in STEM subjects. In doing so, we focus on advancing the process of
development of various tasks relevant to applied mathematics in higher education. We also
emphasize the need to differentiate and adapt tasks according to their specific use cases.
Automatic assessment of tasks which are more relevant to pure mathematics, e.g. proof
examples, require different approaches and are out of the scope of this framework.
Nonetheless, AAS pose a suitable tool for lecturers to provide learning material for students.
Conveniently, AAS, such as Möbius, can be integrated, e.g., into Moodle courses easily
because of their Learning Tools Inoperability (LTI). Feedback from students in the various
courses showed that the pool of examples as learning material was particularly praised.
Before incorporating course material for AAS, teachers need to take into careful
consideration the responsibility, time and effort involved. Contrary to static questions on
paper, computer-algorithmic questions with incorporated randomization are more susceptible
to errors as malfunctions can occur on several parts of the questions. Apart from undefined
solutions originating from randomized variables, errors concerning grading algorithms or the
display of variables in the text of the questions need to be addressed. One solution, we
presented in this contribution, is to create custom libraries with advanced grading and
randomization algorithms, as they can be reused conveniently. As a result, the possibilities
of task design have advanced leading to higher engagement rates of students.
The benefits and challenges of AAS in conjunction with examinations were also discussed.
In this context, the issue of available resources needs to be addressed, as students must access
the exam using a computer. To prevent discrimination against certain social groups, the
infrastructure and technical equipment of the university or educational building must be
reviewed beforehand by lecturers to ensure access to all students. Furthermore, universities
need to review their facilities providing such infrastructure.
Over the past few years, the development of randomization and grading libraries has been
advanced and the related approach of designing examples has been improved and optimized
573
to address specialized tasks in the areas of analysis and linear algebra. A future goal is to
analyze assignments and exams in one course simultaneously, providing detailed statistical
correlation between them. With this statistical analysis, we plan to identify questions with a
high potential for incorrect answers and adapt the framework to better meet the needs of
students.
References
Barana, A., Marchisio, M. & Sacchet, M. (2021) Interactive feedback for learning
mathematics in a digital learning environment. Education Sciences, 11 (6). doi:
10.3390/educsci11060279.
Fisher, T. (2006). Educational transformation: Is it like "beauty" in the eye of the beholder,
or will we know it when we see it? Education and Information Technologies, 11,293-303.
doi: 10.1007/s10639-006-9009-1
Ihantola, P., Ahoniemi T., Karavirta V. & Seppälä O., (2010). Review of recent systems for
automatic assessment of programming assignments. Koli Calling '10 Proceedings of the
10th Koli Calling International Conference on Computing Education Research. doi:
0.1145/1930464.1930480.
Karim. N.A. & Shukur Z. (2016). Proposed features of an online examination interface design
and its optimal values. Computers in Human Behavior, 64. doi:
10.1016/j.chb.2016.07.013.
Kulik, C.C., & Kulik, J.A. (1991). Effectiveness of computer-based instruction: An updated
analysis. Computers in Human Behavior, 7, 75-94. doi: 10.1016/0747-5632(91)900305
Leikin, R. (2014). Challenging Mathematics with Multiple Solution Tasks and Mathematical
Investigations in Geometry. In: Li, Y., Silver, E., Li, S. (eds) Transforming Mathematics
Instruction. Advances in Mathematics Education. Springer, Cham. doi: 10.1007/978-3-
319-04993-9_5
Marchisio M., Remogna S., Roman F. & Sacchet M. (2020). Teaching mathematics in
scientific bachelor degrees using a blended approach. 2020 IEEE 44th Annual Computers,
Software, and Applications Conference (COMPSAC), 190-195, doi:
10.1109/COMPSAC48688.2020.00034.
Mario, P. (2018). Online assessment, adaptive feedback and the importance of visual learning
for students. The advantages, with a few caveats, of using MapleTA. International Review
of Economics Education, 28, 11-28, doi: 10.1016/j.iree.2018.03.002.
Sangwin, C. J. (2013). Computer aided assessment of mathematics. Oxford: Oxford
University Press.
Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R. M., Abrami, P. C., Surkes, M.
A., Wade, C. A., & Woods, J. (2014). The effects of technology use in postsecondary
education: A meta-analysis of classroom applications. Computers & Education, 72, 271–
291. doi: 10.1016/j.compedu.2013.11.002
Stowell, J., Bennett, D. (2010). Effects of online testing on student exam performance and
text anxiety. Journal of Educational Computing Research, 42 (6). doi:
10.2190/EC.42.2.b.
574
Rosenthal, J. S., (1995). Active learning strategies in advanced mathematics classes. Studies
in Higher Education, 20 (2), 223-228. doi: 10.1080/03075079512331381723.
Zimmermann, A., Urbonaite, V., Körner, A., Winkler, S., Krause, S., & Kleinert, M. (2010).
Advanced randomization and grading in the e-learning system Maple T.A. In M. Snorek,
M. Cepek, Z. Buk, & J. Drchal (Eds.), Proceedings of the 7th Congress on Modelling and
Simulation (p. 1209–1214).
575
Insights from a transgender student in the EFL classroom: from

individual perspective to institutional change
Universitat Jaume I, Spain.
Abstract
Among other minority groups, the trans community is systematically repressed
by a hegemonic heteronormativity. In the English as a Foreign Language
(EFL) teaching area, the perspective of this collective is equally absent. This
study aims at gaining insight from a transgender student on his impact and
perception within the EFL classroom to make secondary-school EFL contexts
more inclusive regarding transgender. An exploratory case study constituted
the basis of this article via an open interview with a transgender student. By
exposing him to input stereotyping gender and marginalizing the trans
community, their perception of the danger of the LGBTQIA+ community
beyond the classroom was elicited. Different measures were discussed as
potential improvements whose implementation could ensure safer
environments and a more inclusive teaching experience. Further research
could obtain insight beyond the trans community to assess the awareness and
compromise towards the collective from individuals who are not part of it.
Keywords: EFL textbooks, transgender, inclusive education,
heteronormativity, positioning theory.

Insights from a transgender student in the EFL classroom
1. Introduction
Striving for survival has been a primary urge within the human species leading to numerous
conflicts throughout history. In dominating one community over another, not only does life
prevail, but also other aspects entailing the foundations upon which societies are constructed:
language, culture, and ideology, to name a few (van Dijke & Poppe, 2006). If today there is
an aggressive spirit of making the trans community visible, it may have been summoned by
the apparently complete triumph of its antithesis. This collective has been protractedly
repressed by a hegemonic cisheteronormativity; nonetheless, recent accounts are evidencing
a readjustment in most ideologies across cultures, aiming at increasing the visibility of the
lesbian, gay, bisexual, trans, queer, intersexual, and asexual (LGBTQIA+) collective
(Currier, 2012; see also Chen et al., 2023). The limitedness of representing this community
in the field of English as a Foreign Language (EFL) leads to the purpose of this study.
Together with a literature review and proposal of improvements that can be made in the EFL
teaching area, a case study approach has been followed through an online open interview
conducted with a member of a trans community itself so as to elicit areas of improvement
within the EFL learning field.
2.1. Power dynamics and investment

Catering for the needs of queer learners and asking questions about how classroom materials
should be tailored to their profiles are issues that the field does not accustom itself to address
(Kappra & Vandrick, 2006). Along with these concerns interweave the notions of power
dynamics and learner investment. If identities are perceived as sites of struggle based on
relations of imposing or submitting forces, and they are inscribed in discourse practices
within a particular domain, it is because of this inseparable connection that texts are reflective
of underlying systems of beliefs of varying typologies (Fairclough, 1995).
The tripartite model of investment is orchestrated through the interrelation of the following
elements: identity, capital, and ideology. Following the emergence of the field of applied
linguistics, the notion of identity has been discussed following a poststructuralist approach.
Under this theory, identity is envisioned as “the attempt to differentiate and integrate a sense
of self along different social and personal dimensions such as gender, age, race, occupation,
gangs, socio-economic status, ethnicity, class, nation-states, or regional territory” (Bamberg,
2012, p. 1) and characterized by being “fluid, multiple, diverse, dynamic, varied, shifting,
subject to change and contradictory” (Kouhpaeenejad & Gholaminejad, 2014, p. 200). To
claim a particular identity involves facing three dilemmas, mainly the “sameness of a sense
of self over time in the face of constant change,” the “uniqueness of the individual vis-à-vis
578
others faced with being the same as everyone else,” and “the construction of agency as
constituted by the self and the world,” posits Bamberg (2012, p. 1).
2.2. Positioning theory

From the interrelation between identity and ideology, the positioning theory is developed.
This theory was approached by Davies and Harré (1990), who understood it as “the discursive
process whereby selves are located in conversations as observably and subjectively coherent
participants in jointly produced storylines” (p. 48). When a speaker positions themselves and
other partakers in a given interaction, the following dimensions are taken into account: the
words chosen by the speaker are inevitably synecdochic to the way the participant perceives
the world; participants may not be aware of these assumptions nor the power of what they
invoke particular ways of being, therefore downplaying the importance of what is said; the
variation from speaker to speaker when interpreting these positionings depending on a wide
range of factors; the positions created are the cumulative fragments of a lived autobiography
rather than a linear non-contradictory one; and the positions may be perceived as “roles” or
shifts in power, access, or blocking of access to features related to identity (p. 49).
2.3. EFL materials and ideology

So far, research has shown that students are fully capable to engage in LGBTQ+-related
issues and content as these are inscribed in their daily lives, either by identifying as members
of that community, having LGBTQ+ acquaintances, or through encounters in popular media
(Day et al., 2019). As revealed by Moore in some of his studies (2013, 2016), among other
issues, this was illustrated in how desirable it was for Japanese men to be able to perform a
gay identity and find a welcoming community to acquire English in a motivating way.
Furthermore, he also highlighted the dangers that could originate if critical reflection is not
promoted by recognizing the conflicted nature of LGBTQ+ individuals within the classroom.
Developing literacy in a language learning environment is as paramount as acquiring any
other skill. However, when presenting students with different resources, teachers tend to
dismiss LGBTQIA+-related topics within their methodologies (Pawelczyk, 2022). On this
issue, Paiz (2015) suggested that it can lead to a “perpetuation of a heteronormative discourse
in the classroom, which can be seen as detrimental to the learners and instructors in the class”
(p. 78).
As seen in this literature review, although attempts have been made to widen the visibility of
the LGBTQIA+ spectrum in the (E)FL teaching field, the vindication has not reached its
objectives. Since the cisheteronormative system is not likely to be redesigned more
inclusively, improvements can be made from other areas. For this study, the ideas
aforementioned are adapted to a case study conducted with a transgender male student, who
will reflect upon his own perception of the EFL context, the way his environment positions
579
him, and the improvements that can be made to ensure a more inclusive educational
experience.
3. Method
The previous literature review leads to the purpose of this study, which is twofold: first, to
gain insight from a transgender student on the impact and perception he has in the EFL
classroom; second, to suggest ways in which secondary-school EFL contexts can become
more inclusive regarding transgender issues. These objectives result from an area that has not
been addressed in the literature review, viz. the opportunity to shed light on the topic of trans-
visibility from the community itself with the expectation of knowing more realistically what
the situation is, and what can be done to improve it. The Research Questions (RQs) which
will be commented on in the Results and discussion section are the following:
• RQ1: How does a transgender language learner perceive trans-visibility and

inclusion in the EFL classroom?
• RQ2: What steps can be taken to widen trans-visibility and inclusion on an
institutional scope beyond the EFL subject?
3.1. Participant
One participant was selected for the procedure of this study, whose identity will remain
anonymous by referring to them with the pseudonym of X. He identifies as a male
transgender who is currently undertaking a Bachelor of Arts in a public university and
combines their studies with extra English lessons at an official school of languages. The
selection process was followed without the application of any inclusion or exclusion criteria.
3.2. Research instrument

For this study, an open interview was designed as the research instrument, which was divided
into three main sections. These sections consisted in obtaining information about the
student’s vision of trans-related issues, exposing him to input regarding views on transgender
issues in the EFL classroom, and proposing different measures to take action and make a
difference to gain visibility of the LGBTQIA+ community at an institutional level.
3.3. Data collection and procedure

In order to meet this paper’s objectives, Yin’s approach to case studies (1984) was followed
to opt for an exploratory case study. This type of analysis constituted the basis to design the
interview which was carried out via online means (i.e. Google Meet). The interview was
recorded so that active listenership would not become an issue in the process of asking
questions to the student and to avoid excessive reliance on field notes. It must be noted that
580
the interview has been translated from Spanish into English and labeled with an “Excerpt”
subheading with the corresponding numbering. If needed, some comments have been added
and signaled through italic typography and in square brackets. One intervention in particular
was implemented which consisted in searching for examples of how dominant ideologies
were imprinted in EFL textbooks so that the interviewee felt comfortable enough to share his
thoughts on it.
4. Results and discussion
4.1. Transgender language student’s perception of trans-visibility and inclusion in the

EFL classroom
The first section of the interview was centered around obtaining data about Steven’s
perception of himself and his environment. The results related to RQ1 (How does a
transgender language learner perceive trans-visibility and inclusion in the EFL classroom?)
indicate that he exerts a neutral attitude about how his condition is portrayed and included in
the EFL classroom, in the sense that his environment, the instructor, and the written materials
do not seem to impede his performance as a foreign language student.
Excerpt 1:
No, the truth is that since I got enrolled in this course, I have been feeling quite
comfortable with my group mates and I have not found it difficult to socialize with
everyone else.
The next step in the study consisted in reviewing EFL materials to comment on how gender
diversity is represented in the Foreign Language Learning (FLL) classroom. From the
beginning, the interviewee realized he had not reflected upon this issue, as he considered that
he had other interests when he was taking part in the FL classroom.
Excerpt 5:
Every student after all has their interests when learning a language, and I have to admit
that I have completely left out this crucial part [referring to the inscription of
ideologies into the materials made available for students], not only for me, you know,
but also for the younger generations.
As seen in the literature review, the role of the instructor in the classroom context is as
paramount as the materials used as a means for the presentation of the TL community to the
students. In X’s words, deciding on the types of materials is a responsibility that ought to be
assumed by the instructors themselves, but making this seemingly objective and ideology-
free decision can become challenging when the teacher is already a product of that ideology,
therefore biasing the selection process:
581
Excerpt 7:
Whom do these decisions rely on? The English department? Then it should be them
who ought to decide what materials are most fitting to meet the requirements of the
class and which ones to discard […] Although that could be troublesome because
sometimes you do not really know if these textbooks are coming across as sexist,
homophobic, transphobic and stuff, or let me rephrase that, sometimes you do not
know if they are hiding a discriminatory ideology underneath.
The previous excerpt confirms X’s acknowledgment of the role of other points of view to
make the EFL classroom more inclusive. He also seems to be aware of the fact that curricula
are not impartial in the ideologies they manifest, which can become a challenge when
deciding on the type of materials that are presented to students.
4.2 Potential measures to widen LGBTQIA+ perspective and inclusion at the institutional
level
After covering the previous items, the last section of the interview was devoted to proposing
a series of actions that could be taken to increase the visibility of the LGBTQIA+ community
at a more institutional level (RQ2: What steps can be taken to widen trans-visibility and
inclusion on an institutional scope beyond the EFL subject?). In order to approach this last
section of the interview, X shared the following comment about classroom materials:
Excerpt 9:
What I sincerely believe should be done is a systematic review of the English subject
materials to see which areas are deficient -such as the exclusionary discourse rooted
in the materials themselves- and therefore need to be reviewed, and from there start
working to improve not the form of the materials, but the content.
The interviewee assured that not only should the scope of the topical areas of the units be
broadened in EFL textbooks, but it should be extended to other relevant issues. What seems
to be implied in Excerpt 9 is that curricular materials tend to explore different identities for
language learners, but there may be a reinforcement of a particular discourse, (viz.
heteronormativity).
Even though X does not currently feel threatened or unsafe in their classroom environment,
he is aware that this may not be the case for every trans individual, hence the following action.
To ensure a safe environment, as this privilege is not always granted to minority groups,
institutions should become part of the inclusivity process by raising the corresponding
awareness through different campaigns, like making LGBTQIA+-related workshops,
organizing seminars, and promoting more tolerant attitudes towards those who are
discriminated against daily. Referring to the literature review, these implementations may
582
not take place unless there is an active intention from the perspective of the instructors to
redirect those materials whose perspectives might reify a particular ideology.
Following the previous idea, gender diversity was another aspect highlighted by X which
was suggested to be promoted among students to keep on widening the scope and reach those
non-conforming positions which oppose the traditional binary division of gender:
Excerpt 11:
Another thing I would consider when implementing all these things in the classroom
materials is the gender spectrum. Ever since they [referring to the students] learn how
to walk, they just do not care about gender roles, which is something they learn over
the years as society includes them. So, although there might be a heteronormative
discourse overruling and suppressing everything else, these materials should bring
these topics up so that students may engage in interactive debates to make something
useful out of those English lessons that often ignore these issues.
Once more, X mentions the issue of inclusivity concerning gender diversity and how the
cisheteronormative ideology persists in today’s society. Excerpt 11 reveals his awareness of
how heterosexuality is presented as the only accepted sexual identity, confirming the
previous findings of the literature review (Paiz, 2015), but despite the presumed dominance
of this sexuality, he favors the position of including gender-related topics in the classroom
and designing activities that allow for students to discuss openly on these topics. The
aforementioned measures would go in line with the literature review’s evidence that catering
for the needs of queer learners and adapting classroom materials to their profiles is a question
that is not frequently addressed in the teaching field (Kappra & Vandrick, 2006).
5. Conclusion
The purpose of this paper was to explain the circumstances of a transgender student in terms
of his impact and perception of the EFL classroom as well as suggest ways through which
inclusivity regarding transgender issues could be widened. Through an open interview, more
information has been drawn from a member of the trans community itself, which can pave
the way for a chain of changes toward realistic inclusivity. The implications of this study are
centered on the pedagogical domain, specifically written materials, and the role of the
instructors. Potential areas of improvement include those with dated underlying ideologies
which can potentially become harmful for minority groups and contents that can be
dynamized and presented as real-world issues.
It must be clarified that the current study has its limitations, as a case study may not have
provided representative evidence to draw the conclusions stated in the article. However, it
has served as a platform for a hidden voice to express his views regarding the transgender
583
community in the current EFL education system. Further studies could broaden the scope of
the current paper and conduct different interventions to larger groups of students, gain a more
representative insight of the status quo of transgender outside the community, and get a
glimpse of their attitudes towards the collective and the role and/or degree of implication they
believe they should have. Ultimately, the aim is to ensure safe environments where the
learning experience of each individual is not hindered by the dominance of oppressive
ideological forces.
References
Bamberg, M. (2012). Identity and narration. The Living Handbook of Narratology.
https://www.lhn.uni-hamburg.de/node/29.html
Chen, J. M., Joel, S., & Castro Lingl, D. (2023). Antecedents and consequences of LGBT
individuals’ perceptions of straight allyship. Journal of Personality and Social
Psychology. https://dx.doi.org/10.1037/pspi0000422
Currier, A. (2012). Out in Africa: LGBT Organizing in Namibia and South Africa. University
of Minnesota Press. http://www.jstor.org/stable/10.5749/j.cttttdkw
Davies, B., & Harré, R. (1990). Positioning: The discursive production of selves. Journal for
the Theory of Social Behaviour, 20(1), 43–63. https://doi.org/10.1111/j.1468-
5914.1990.tb00174.x
Day, J. K., Ioverno, S., & Russell, S. T. (2019). Safe and supportive schools for LGBT youth:
Addressing educational inequities through inclusive policies and practices. Journal of
School Psychology, 74, 29–43. https://doi.org/10.1016/j.jsp.2019.05.007
Fairclough, N. (1995). Critical discourse analysis: The critical study of language. Routledge.
Kappra, R., & Vandrick, S. (2006). Silenced Voices Speak: Queer ESL Students Recount
Their Experiences.
Kouhpaeenejad, M. H., & Gholaminejad, R. (2014). Identity and language learning from
post-structuralist perspective. Journal of Language Teaching and Research, 5(1).
https://doi.org/10.4304/jltr.5.1.199-204
Moore, A. R. (2013). The ideal sexual self: The motivational investments of Japanese gay
male learners of English. In P. Benson & L. Cooker (Eds.), The applied linguistic
individual: Sociocultural approaches to identity, agency and autonomy (pp. 135–151).
Bristol, CT: Equinox Publishing.
Moore, A. R. (2016). Inclusion and Exclusion: A Case Study of an English Class for LGBT
Learners. TESOL Quarterly, 50(1), 86–108. https://doi.org/10.1002/tesq.208
Paiz, J. M. (2015). Over the monochrome rainbow: Heteronormativity in ESL reading texts
and textbooks. Journal of Language and Sexuality, 4(1), 77–101.
https://doi.org/10.1075/jls.4.1.03pai
Pawelczyk, J. (2022). New Transnational Voices on Gender Diversity in English Language
Education: Moving Forward. Gender Diversity and Sexuality in English Language
Education: New Transnational Voices, 211.
584
van Dijke, M., & Poppe, M. (2006). Striving for personal power as a basis for social power
dynamics. European Journal of Social Psychology, 36(4), 537–556.
https://doi.org/10.1002/ejsp.351
Yin, R.K., (1984). Case Study Research: Design and Methods. Beverly Hills, Calif: Sage
Publications.
585
English for engineering: intercultural formal letter writing

Adrián Pla-Ángel, Begoña Bellés-Fortuño
English Studies Department, Universitat Jaume I/ IULMA, Spain.
Abstract
Writing is a productive skill highly addressed in the ESP classroom. Students
learning a foreign language have to learn how to create accurate structured
texts with organised ideas to convey relevant meaning. The aim of this study is
to analyse and categorize written mistakes produced by 1st cycle foreign
students from different nationalities (i.e.: Romanian, Moroccan and
Bulgarian) in the English for Engineering module at a Spanish university. The
objective is to examine written language accuracy in formal letter writing such
as enquiring and complaint letters, to detect any intercultural interference as
to mother-tongue influence and other language varieties in the learning of
curricular and linguistic knowledge. We believe that studies such as the one
presented here allow teachers to provide students with written corrective
feedback which may have a positive impact on students’ writing skills (Li,
2000), in this case, foreign students enrolled in an ESP module at a Spanish
university.
Keywords: Formal writing; ESP; written corrective feedback, intercultural
interference.

1. Introduction
Writing is a productive skill highly addressed in the ESP classroom. Students learning a
foreign language have to learn how to create accurate structured texts with organised ideas
to convey relevant meaning. Written production is a preferred and demanded task in class to
assess language learning progress since it is one of the most widespread forms to record
students’ performances (Whittaker et al. 2011). Other language elements that are of interest
in the ESP classroom have to do with language accuracy, creativity, and corrective feedback.
Some studies have analysed written corrective feedback from British spelling samples to be
able to describe the lexico-grammatical errors committed by students for the assessment of
their written outcomes (Bellés-Calvera & Bellés-Fortuño, 2018; Zhang, 2011). Error analysis
and later correction is fundamental for the systematic study of the learner’s language (Corder,
1981), benefiting all participants in the learning process, that is, teachers, students and
researchers.
The aim of this study is to analyse and categorize written mistakes produced by 1st cycle
foreign students from different nationalities (i.e.: Romanian, Moroccan and Bulgarian) in the
English for Engineering module at a Spanish university. The objective is to examine written
language accuracy in formal letter writing such as enquiring and complaint letters, to detect
any intercultural interference as to mother-tongue influence and other language varieties in
the learning of curricular and linguistic knowledge.
In this pilot study, an initial number of ten texts have been analysed and corrected to provide
a classification of errors committed by foreign students attending the English for Engineering
module at a Spanish university. Spell-checker software and Grammarly (2009) have been
used to spot mistakes. Errors such as: “I would be fine with removing shipping coste”, “and
for this se need tour” “un cash” “this are my informations”, “will safe you money”, “we
would spend knowing what your options are to make the payment” or “standarts” have been
detected among a range of different error typologies. Some of the errors encountered may
have been originated by intercultural language interference, considering the foreign students
mother tongue is other than Spanish or English. Wrong order choice, grammatical
correlations or lexical errors are the most recurrent ones. The classification of errors
generated in this study and committed by intercultural students is later compared with the
typology of common errors produced by Spanish learners of English as a foreign language
with the aim of generating a number of written corrective feedback strategies, namely direct
feedback, indirect feedback, or reformulation among others (Ellis, 2009).
2. Methodology
To carry out this study we have gathered a selection of 10 texts corresponding to 67 writing
assignments of formal letters. The formal letter writing is a task the students participating in
588
the study need to accomplish in the practical section of the subject (1st year Mechanical
Engineering Degree). Specifically, the selected writing assignments consisted of an evaluated
compulsory task included in the curriculum of the subject “English for Engineering”. The
formal letters had to follow the structure and topic of ‘inquiry letter’ and ‘letter of complaint’.
The instructions and standardization of ‘inquiry’ and ‘complaint’ formal letters are explained
in the subject. Students have to hand in the letters as part of the final subject evaluation.
2.1. Participants
The participants taking place in this study are 1st year Mechanical Engineering students
enrolled in their ESP English for engineering module at a Spanish university. A total number
of 67 undergraduate students writing assignments were gathered, 62 of them were male and
5 female. As for their nationalities, their origin countries varied: Spain, Romanian, Moroccan
and Bulgarian, being Spanish the majority of them (see Table 1).
Table 1. Participants.
Participants
Gender Nationality
62 male 60 Spanish
5 female 5 Moroccan
1 Bulgarian
1 Romania
TOTAL 67 67
The identity of students has been protected and the writings analysed in this study have been
anonymised for a more objective analysis.
2.2. Materials and tools

The corpus of ten formal letters used in this study was analysed in search of different basic
errors. To do so a previous error classification model was considered (Bellés-Calvera &
Bellés-Fortuño, 2018). Aside from that, a spell-checker software and Grammarly (2009) have
been also used to detect those errors. This classification model included errors such as:
Lexical, grammatical or spelling errors as well as wrong order choice. As to grammatical
errors Bellés-Calvera and Bellés-Fortuño include “verb tense, subject-verb agreement,
fragments, word order and punctuation” (2018:110). Regarding lexical errors, these include:
“articles, nouns, pronouns, verbs, adjectives, prepositions, word choice and spelling” (Bellés-
Calvera & Bellés-Fortuño, 2018: 110).
589
Table 2. Errors classification model (Bellés-Calvera & Bellés-Fortuño, 2018).
Grammatical errors Lexical error
Subject-verb agreement Spelling
Punctuation Capitalisation
Fragment Prepositions
Verb tense Word choice
Word order Adverbs
Pronouns
Articles
Verbs
Nouns
This previous classification (Table 2) was initially used to analyse the corpus of this study in
order to detect and classify the students’ errors found. However, a closer look at the writings
revealed that some of the categories included in the classification model were not found in
the corpus. The relevant results are presented in the following section.

In this section, the results from students’ writing performance on the formal letter task are
analysed and discussed. Therefore, a classification of students’ grammatical and lexical
errors has been generated, based on the classification from the study carried out by Bellés-
Calvera & Bellés-Fortuño (2018). The errors have been analysed according to their frequency
of occurrence and the percentage over the total number of grammatical or lexical errors and
lastly, in Table 3 below, some examples from each type of error have also been provided.
The analysis of 10 samples of formal letters led us to the identification of students’ most
commonly committed errors. A difference in the use of grammar and lexis can be observed
amid the Mechanical Engineering students. Grammatical errors embraced verb tense,
subject-verb agreement, fragments, word order and punctuation, while lexical errors included
articles, nouns, pronouns, verbs, adjectives, prepositions, word choice and spelling.
From the results it can be observed that the greatest difficulties for the participants lie on
punctuation, fragment, spelling, word choice and verbs. Fragment errors are understood as
unfinished utterances as well as with grammatically and lexically incorrect utterances as a
590
whole. Nonetheless, it should be stressed that both grammatical and lexical categories show
similar tendencies in the sense that students commit nearly the same percentage of errors.
Table 3 below shows the most common grammatical errors committed by the participants of
this study. Surprisingly, the most frequently-made error types correspond to punctuation
(30,5%) followed by fragments (10%). On the other hand, the number of errors is
considerably lower when it comes to word order (4,3%), and subject-verb agreement (2%).
Table 3. Grammatical errors: Frequency, percentage and examples.
Grammatical errors Frequency Percentage Examples
Subject-verb agreement 2 8,7% “...and this are my

information”
Punctuation 7 30,5 % “I look forward to

hearing from you
soon!
Yours sincerely
Fragment 10 43,5% “After five years

of close I need a
good discount”
Verb tense 5 13,04% “We would like to

knowing”
Wrong word 2 8,7 “We would spend

knowing what
your options are
to make make the
payment”
Total 23
Focusing on lexical errors, it can be noticed that the most prominent error is that of spelling
(32,7%). Another category to be highlighted is the failure of students to select verbs (16,3%)
and the choice of words (14,3%). In the same line, prepositions (10,2%) are often substituted
by other prepositions or adverbs.
Some of the errors encountered have been originated by intercultural language interference,
since for some of the participants Spanish is not their mother tongue. Henceforth, most
grammatical and lexical errors -which are the two major errors typologies-, may have a
connection with the intercultural language interference. Lexical (spelling with a 32,7%),
grammatical correlations (punctuation with a 30,5%) followed by fragments (10%) are the
most recurrent ones. The classification of errors generated in this study and committed by
591
intercultural students is later compared with the typology of common errors produced by
Spanish learners of English as a foreign language with the aim of generating a number of
written corrective feedback strategies, namely direct feedback, indirect feedback, or
reformulation among others (Ellis 2009).
Table 4. Lexical errors: Frequency, percentage and examples.
Lexical error Frequency Percentage Examples
Spelling 16 32,7 % “The consigment of plugs was

succesfully sent to you
adress”.
Capitalisation 5 10,2% “Thank you for you letter of

12 november..”
Prepositions 5 10,2% “payment of delivery”
“payment un cash”
Word choice 7 14,3% “If you want we can send you

one show, you can test it”
“that will safe you money”
“On all prime order”
Adverbs 4 8,2% “Also the procomo400 has

raised our standarts…”
Pronouns 3 6,12% “We write you regarding the

consigment for their
company”
“We would spend knowing

what your options are…”
Verbs 8 16,3% “What kind of discount could

you give me for stays and
keep you in mind for next
purchases”.
Nouns 1 2,04% “the possibility of

upgradation”
Articles 1 2,04% “I need a available good

discount”
Total 49
592
Up until this point, one can construe that first year university students, and in this specific
case, Mechanical Engineering students in their English module, have not been trained on the
proper use of punctuation symbols in previous years at high school. English and other foreign
language codes have many differences in the way punctuation symbols are used, that is to
say, the usage of commas and semi-colon or colon in combination with cohesive markers.
For this reason, more specific training on punctuation symbols may be convenient for first
year university students. Furthermore, the participants of this study have also displayed errors
in spelling and wrong verb choice, hence, more training on these aspects should then be
implemented in the classroom syllabuses.
We believe that studies such as the one presented here allow teachers to provide students
with written corrective feedback which may have a positive impact on students’ writing skills
(Li, 2000). In this case, foreign students enrolled in an ESP module at a Spanish university.
Thus, detecting and indexing these intercultural errors to create classifications of identified
wrong uses of literary translations or language correspondences in order to later explain them
in class would definitely support students in their FL learning process.
References
Bellés-Calvera, L., & Bellés-Fortuño, B. (2018). Written corrective feedback with online
tools in the Medicine classroom: Bombay TV. Transforming education for a changing
world, Edited by Camilo López-García and Jesús Manso, Adaya Press, 106-119.
Corder, S. P. (1981). Error analysis and interlanguage. Oxford: OUP.
Ellis, R. (2009). Typology of written corrective feedback types. ELT Journal Volume, 63(2),
97-107.http://dx.doi.org/10.1093/elt/ccn023
Grammarly, Inc. 2009. English language writing-enhancement platform
Li, Y. (2000). Linguistic characteristics of ESL writing in task-based e-mail activities.
System, 28(2), 229-245
Whittaker, R., Llinares, A., & McCabe, A., (2011). Written discourse development in CLIL
at secondary school. Language teaching research, vol. 15, no. 3, 343-362.
Zhang, M. (2011). Error analysis and interlanguage. Focus 1, 85-93.
593
The explicit teaching of vocabulary in French L2: theoretical

models, teaching practices and experimental itineraries
Department of Economics, Law, Cybersecurity, and Sports Sciences, University of Naples
Parthenope, Italy.
Abstract
This contribution presents the results of a recent experimental research aimed
at investigating the potential of explicit teaching of French L2 vocabulary in
the university context of Langues pour Spécialistes d’Autres Disciplines
(LANSAD) in Italy. After introducing the research background, the tools,
methodologies and teaching materials are outlined. The qualitative and
quantitative evaluation of the results obtained from an experimental test (ET)
is then presented. The test consisted of four tâches concerning the
identification and use of paraphrastic rules, the analysis of lexical relations
through the use of LF and a written production on one of the specialised topics
addressed during the course (with the use of LU and PU) in order to measure
the students’ acquisition of metalessical knowledge, as well as their ability to
spontaneously and correctly use PU in specialised communication. Finally,
the results that emerged during the experimental course are outlined, with
reference to the significance of the tools and methodologies tested and the
promotion of teaching activities focusing on the explicit teaching of vocabulary
even in areas such as LANSAD.
Keywords: Explicit vocabulary teaching; LANSAD context; French L2;
paraphrase; lexical functions.

The explicit teaching of vocabulary In French L2: theoretical assumptions, teaching practices
1. Introduction: theoretical premises and research aims

In recent years, the development of lexical competence in L2 learning processes has gained
considerable importance in the field of linguistic research, fostering a multi-voiced dialogue
between different fields of investigation, such as lexicology, terminology, educational
linguistics, corpus linguistics and foreign language teaching. According to the descriptors of
the Common European Framework of Reference for Languages (CEFRL), a B2 level learner
should have “a good lexical repertoire related to his field and to many general topics” (CoE
2020: 143). He/she should also be able to “vary formulation to avoid excessive repetition
[...]; understand and use a large proportion of the vocabulary related to his or her specialist
field; [...] insert appropriate words/signs in most contexts in a fairly systematic way” (CoE
2020: 143). However, recent studies on lexical competence and its combinatorial properties
(better defined as “phraseological competence”1) have shown that learners at both
intermediate and advanced levels have significant lexical deficits, especially with regard to
the use of locutional and idiomatic phraseological units (PU) of low and/or medium
frequency (Hamel et al. 2007; Tsedryk et al. 2019). In order to avoid linguistic errors,
learners often resort to the mechanisms of substitution and repetition, relying on a very high
frequency general lexicon (use of the hypernym → poisson, instead of the hyponym →
requin, loup de mer) and making excessive use of simple locutions, sometimes generated by
L1 linguistic interference (i.e. avec tout le cœur rather than de tout mon cœur; profit manqué
instead of manque à gagner). In addition to the possible interferences with the mother tongue,
further elements come into play in the process of learning the lexicon of an L2: the time
needed to store new LU, the ability to reuse them spontaneously and correctly according to
the various communicative contexts, motivation and, lastly, the use and experimentation of
didactic tools, methodologies and materials produced ad hoc according to the learners’
linguistic and educational needs. One need only think of the approaches adopted for the
teaching of Français sur Objectifs Spécifiques (FOS) at university level (Raus et al. 2020) or
of the methodologies employed in the area of “Langues pour Spécialistes D’Autres
Disciplines” (LANSAD) (Chaplier et al. 2019). In these contexts, a purely thematic approach
(Binon et al. 2004) is usually preferred for teaching vocabulary through activities based on
the creation and mnemonic learning of lists of names, inventories and specialised glossaries
taken from textbooks, paper and electronic dictionaries, thesauri, digital corpora, etc. The
result of this approach is undoubtedly the expansion of encyclopaedic knowledge relating to
a domain, but the development of knowledge and lexical skills associated with the function
of LU within a specialised discourse is compromised. Therefore, there is the risk of moving
from the specificity of a lexicon in its technical-scientific and professional use to the likely
aseptic labelling of world objects in specialised communicative situations. According to the
1
Although the CEFRL makes no reference whatsoever to the notion of “phraseological competence”, there are numerous references
to the concept of “paraphrase”. For theoretical insights into the concept of 'phraseological competence', please refer to the studies by
González-Rey (2016).
596
attentional model suggested by Picoche (1992) and the theoretical framework of Meaning-
Text Theory (MTT) and Explanatory Combinatorial Dictionary (ECD) (Mel'čuk et al. 1995),
the aforementioned problems can be solved, at least in part, through theoretical-applicative
didactic paths centred on the explicit teaching of the lexicon. More specifically, this approach
emphasises the explanation of theoretical notions, the description of lexical structures and
reasoned exercises in reformulation and written production, with the help of two purely
linguistic tools: lexical functions (LF)2 and paraphrase rules.
In the light of the above-mentioned theoretical linguistic models, which have possible
didactic implications, this contribution presents the results of a recent experimental research
aimed at investigating the potential of the explicit teaching of French L2 vocabulary in the
university context of LANSAD in Italy. More specifically, the experimentation took place in
a French language course aimed at Italian-speaking students of Bachelor’s Degree
Programmes in Statistics and Business Management Sciences at the University of Naples
Parthenope. The objective of the experimentation was dual: on the one hand, it fosteredthe
development of basic lexical and metalexical skills through theoretical-applicative, reasoned
and structured training of the main concepts of ECD and through reformulation, paraphrase
andwritten production excercises; on the other hand, it taught students the effective use of
dictionaries and other lexical resources, such as corpora or other lexical databases for the
study of specialised lexicons related to the dominant disciplines of their study paths. After
introducing the research background and the minimum language requirements (§ 2.1.), the
tools, methodologies and teaching materials characterising the operational protocol of the
theoretical-applicative training are outlined (§ 2.2The qualitative and quantitative evaluation
of the results emerging from an experimental test (ET) is then presented (§ 2.3.). The test
consisted of four tâches concerning the identification and use of the paraphrastic rules, the
analysis of paradigmatic lexical relations and collocations through the use of LF, and a
written production on one of the specialized macro-themes tackled during the course (with
the compulsory use of the LU and PU studied), in order to measure the students’ acquisition
of metalexical knowledge, as well as their ability to spontaneously and correctly use PU in
specialized communicative contexts. Lastly, some of the main themes dealt with during the
experimental course are outlined, with particular reference to the significance of the tools and
methodologies tested and the promotion of teaching activities focused on the explicit teaching
of vocabulary even in areas such as LANSAD.
2
In the context of LF, the term “function” is used in its mathematical sense, as emerges from the following technical definition,
which we have translated: “From a formal point of view, a Lexical Function f is a function that associates with a given expression L
– the argument, or base, of the function f – a set (L) of expressions – the value of the function f – that express, in relation to L, a
specific meaning associated with f” (Mel'čuk et al. 2021: 75). For more on the concept of FL, please refer to a recent study by Mel'čuk
et Polguère (2021).
597
2. Didactic experimentation
2.1. The context: target group and language requirements

The experimentation was carried out during the second semester of the academic year
2021/2022, as part of the course in French Language Skills. The course was divided into two
modules of 36 hours each for a total of 72 hours and aimed at Italian-speaking students of the
Bachelor’s Degree Programmes in International Business Management and Statistics of the
University of Naples Parthenope. These are two professional training courses that aim, on
the one hand, to train specialists capable of governing the internationalisation processes of
companies and marketing policies and, on the other, to qualify future statisticians and
analysts in the management of big data in a corporate context, through mathematical-
computational methods and new information technologies. Within the curriculum of the two
above-mentioned degree programmes, particular attention is paid to the teaching of foreign
languages, whose macro-objectives include the attainment of at least a B2+ level of the
CEFRL and a methodological basis for learning the specialised languages of the various
disciplines studied over the three-year period. With regard to the above-mentioned teaching
programme, the contents and objectives were designed for a particularly numerous (n= 300
students in total, among which 196 attending lectures) and markedly heterogeneous class, as
fair as their French language knowledge and skills are concerned. An analysis of the
numerical data shows that 20% of the students hold a C1 level DALF language certificate;
55% hold the B2 level EsaBac and/or EsaBac Techno Italian-French upper secondary school
dual diploma; 15% claim to hold a B1 level DELF certificate and, lastly, 10% claim to have
studied French at lower and upper secondary school but do not hold any language certificate.
2.2. The operational protocol: methodologies, tools and teaching materials

As outlined above, the course was divided into two modules of 36 hours each. During the
first module entitled “The Language System of the French Language”, basic knowledge was
provided on the phonetic, morphosyntactic and lexical aspects of a B2+ level of French
according to the CERFL. The topics covered ranged from the revision of various grammatical
aspects of the analysis of the main syntactic, lexical and phrasal structures, and the study of
the major writing and specialised communication strategies representative of three
professional sectors, including international trade, blue economy and environmental
sustainability in the marine environment.
In the second module entitled “Lexicology, specialised lexicography and digital literacy”, a
detailed, albeit introductory, path was provided on the various word formation processes from
a contrastive perspective (Italian/French), thus directing learners to the study of specialised
languages as well as the traits differentiating them from the general language in their lexical,
terminological, pragmatic and translation aspects, with the aid of digital tools such as corpora,
598
online databases and term extractors. A series of introductory seminars aimed at learning the
basic metalexical notions, such as lexeme, vocabulary, LU, PU, as well as their
interconnection with terminological studies (L’Homme 2020; Bonadonna 2020) was also
proposed. Referring to the actantial model proposed by Picoche and to the overall theoretical
background of ECD, a clear examination of the main syntagmatic and paradigmatic lexical
relations through the use of LF as well as of the concepts of semantic actant, collocation
(base+collocate) and locution was presented. Through a brief summary of the various types
of LF, a wide range of criteria to classify them (idiomaticity, non-compositionality, semantic-
conceptual transparency/opacity, polylexicality and fixity) was illustrated and, in addition,
the difficulties faced by those studying them were outlined. Depending on the perspective
one takes, it is possible to encounter terms that differ in the various languages and traditions
of phraseological studies. An attempt has therefore been made to offer as schematic a view
of the main PU as possible, based on the classification proposed by Mel'čuk (2013). With
regard to the basic theoretical aspects concerning reformulation operations, the definitions of
sense, semantic equivalence and paraphrase were commented upon, preferring the criteria of
lexico-syntactic paraphrase and focusing especially on the lexical criteria of synonym,
antonym and conversive substitution. Once the seminar training was completed, a practical
activity was designed to link the aforementioned paraphrastic processes to the more general
operations of text composition, in order to encourage students to use the vocabulary studied
in their written productions. Starting from two short expository texts on two very topical
subjects, such as the internationalisation and business organisation sector and the marine
bioeconomy for the correct business management of marine protected areas, the students
were asked to reformulate some parts and fill in the blanks, using appropriate collocations
and/or phrases with the help of dictionaries and open access digital terminology databases
(GDT, TERMIUM Plus, FranceTerme, TLFI, DES, IATE and LOTERRE). In addition, they
were shown the importance and usefulness of lexical resources such as Spiderlex (RL-fr) and
IdiomSearch, as well as some linguistic corpora (frTenTen20, TermiTH and ScienQuest) to
identify PU through the use of the “co-occurrences” or “word combination” entry, as well as
to analyse their frequency and linguistic properties from context. At this stage, ways of
searching for and modelling lexical relations through the use of LF were defined. Particular
interest was paid to certain paradigmatic relations (hypernymy, hyponymy, synonymy,
antonymy, conversion, syntactic derivation and semantic derivation). As far as syntagmatic
LF (or collocations) are concerned, one adjectival FL expressing intensity, two adverbial LF
representing place and medium and, lastly, two verbal LF composed of a supporting verb and
a realisation verb were selected3. The choice of the aforementioned LF is to be correlated to
their potential didactic value in the class context under investigation, assuming that it would
have been undoubtedly easier for learners to understand the above formalisms through
3
https://drive.google.com/file/d/1-8me2ZwTltePIiXp6LnkfjE5sgh329d3/view?usp=share_link
599
examples relating to nouns, actions, agents and places related to the specialised domains of
their interest. Although brief, this theoretical-applicative examination proved particularly
useful from a didactic point of view: on the one hand, it contributed to the students’ mastering
of metalinguistic language; on the other hand, the paraphrasing and reformulation exercises
enabled them to learn how to organise the syntactic structures of a sentence (becoming aware
of their errors and self-correcting), to explore some of the paradigmatic and syntagmatic
relations between several LU belonging to the lexicons studied and, lastly, to develop a good
metalexical competence.
2.3. The experimental test (ET): activity design and results

In order for the learners to familiarise with both the theoretical and methodological concepts
outlined so far, an experimental test (ET) lasting 4 hours (2+2) was proposed, consisting of
four tests concerning the identification (ET1) and use (ET2) of the paraphrastic rules
examined. The aim of the test was to verify the acquisition of lexical knowledge and skills;
the analysis of paradigmatic lexical relations and collocations through the use of LF (ET3)
and a written production (ET4) involving the use of the LU and PU studied, with the aim of
proving their correct use in written text production. The first test included an exercise
consisting in the identification of the rules of paraphrasing (syntactic and/or lexico-syntactic)
in four pairs of sentences, which were given a maximum point of 44. By way of example, for
the pair of sentences (1a. Les températures de l’eau de mer montent considérablement; 1b.
Les températures de l’eau de mer connaissent une forte hausse), the paraphrase is based on
the equivalence between the verb monter and the equivalent lexico-syntactic structure
involving the noun of monter, i.e. its nominalisation – S0(monter)=hausse – preceded by the
corresponding supporting verb: connaître. In this case, it is a lexico-syntactic paraphrase
since there is not only a lexical substitution, but also a modification of the syntactic
construction of the sentence. In contrast, in the pair of sentences (2a. L’économie bleue
garantit la sécurite alimentaire sur le marché mondial; 2b. La sécurité alimentaire sur le
marché Mondial est garantie par l’économie bleue), only a syntactic transformation from
active to passive sentence was made, without any lexical change. Students were then asked
to rephrase six sentences of one point each, using the criteria of synonymous, antonymic and
conversive substitution (ET2)5. The results of the ET1 total averages show that the concept
of paraphrasing was well understood by the majority of the students with a minimal
percentage difference between correct identification of syntactic paraphrasing criteria (75%)
and lexico-syntactic paraphrasing criteria (73%), while approximately 27% failed the test due
to lack of adequate metalexical knowledge and skills. The scores obtained for ET2 confirm
almost the same success rates as ET1: around 74% of the students were able to rephrase the
sentences in the test, except for 27% of the participants who experienced problems. From a
4
https://drive.google.com/file/d/1gUGmiN8eHQCbOBuMKLDyQ9OFZsf2y6Jc/view?usp=share_link
5
https://drive.google.com/file/d/1DVAzf-q1ZzUpZtBWLLL1LGP4nWsZMJdH/view?usp=share_link
600
more detailed analysis of the data collected, one could hypothesise that these shortcomings
can be traced back to elements such as non attendance of classes (most of the students, in
fact, work), a lower language level than the other members of the class and difficulties in
metalinguistic thinking starting from one’s mother tongue. As regards usage, the total number
of reformulations produced by the students is 19, among which 6 are synonymic
substitutions, 3 antonymic substitutions, 2 collocations, 1 locution and 7 cases of non-total
semantic equivalence which, instead of being evaluated as errors, we preferred to consider as
quasi-synonyms. In the second part of ET, we focused on the recognition of paradigmatic
lexical relations, collocations and/or locutions (ET3). After having submitted a list6 of LU
and PU to the students they were asked to identify the paradigmatic relations by means of LF
and, in the case of LU composed of more than one element, to indicate the type of phraseme
(between locution or collocation). We were not asked to carry out a detailed linguistic
analysis, but rather to make a distinction between locution and collocation, illustrating, in
summary, that – by way of example – PU payer quelqu’un en monnaie de singe is a locution
because its meaning is not compositional; while chimie verte (AB) is a collocation because
it is composed of the base chimie (A) freely chosen by the speaker for its meaning and the
collocate verte (B) which expresses the meaning of the expression, i.e. a concept of chemistry
that aims to promote sustainable approaches to chemical industry by eliminating the use of
hazardous procedures and substances. Test ET3 was undoubtedly the most successful
compared to all the others: 80% of the class was able to identify the lexical relations and
classify the proposed PU, except for 20% of the participants who misinterpreted the above
concepts in several examples and only answered correctly in four cases (économie bleue,
chimie verte, pollution sonore, aire marine protégée). Lastly, the last activity consisted of a
free production (ET4) on one of the macro-themes tackled during the course, with the
necessary usage of LU and PU correctly, among those belonging to the specialized lexico-
semantic fields studied in the class. The ET4 results were also particularly interesting. The
average total score for written production was 78%, which corresponds more or less to the
average of the ET3 results on metalexical knowledge. 17 out of the 30 points awarded for
correct answers were for examples of LU and PU studied in class and reused, consisting
mostly of collocations – adjectival type (Bon: énergie verte, chimie verte; économie bleue;
AntiBon: marché noir), verbal type (Real: defender/sauver/preserver/détruire/polluer/nuire
à l’environnement; Oper: pratiquer une politique soutenable, mener un audit, etc.) – and a
few locutions (n=4) (col blanc, voie maritime, marché réglementé, agrobusiness mondial,
obstacle commercial, etc.).
6
https://drive.google.com/file/d/1cRuZlrewqd1lT50Yll6Bib6RfpUVIG65/view?usp=share_link
601
3. Conclusions
The ET results suggest that students have acquired good metalexical skills. More than 70%
of the participants benefited from the explicit teaching of the lexicon and from the exercises
to identify paraphrastic rules (ET1) and lexical relations with the help of LF (ET3), as well
as from the resources made available, through which the learners learnt to search for
information on the combinatorial properties of the lexicon. As far as the use of LU and PU is
concerned, it was high in both tâches (ET2 and ET4): more than 70% of the correct answers
contained examples of LU and PU. The percentages were on average higher in written
production (ET4: 78%) than in paraphrasing (ET2: 74%). This result can be explained as a
consequence of the fact that the reformulation exercise presented more constraints due to the
search for synonymic expressions and patterns to follow, unlike ET4 whose only constraint
was the use of LU and/or PU studied in class. With regard to collocation patterns, it was
found that the “supporting verb+noun” pattern was more frequent in the rephrasing exercise,
while the “noun+adjective” pattern was more frequently used in written production. As for
the locutions, the percentage of use remained very low in both tests, a result that was expected
for two reasons: due to their non-compositionality, learners found it difficult to translate and
use them; moreover, more attention was given to collocations than to locutions in class. Even
for the weaker learners and/or those with a scarce knowledge of lexical resources, cognitive
effort was noted during the paraphrase and production exercises: for example, rephrasing or
the use of compound collocations with inexact collocates caught our attention. The fact that
a learner used an incorrect support verb highlights the need to teach broader syntactic
structures in order to include the corresponding support verb in a noun phrase, especially if
its use is limited to one or two verbs or well-defined syntactic patterns.
The theoretical and methodological reflections and empirical data outlined here allow us to
state that the explicit and structured teaching of L2 vocabulary, combined with numerous
reformulation and lexical network identification exercises, can contribute to the enhancement
of the various lexico-syntactic constructions – and, more generally, to the development of
metalexical competence – albeit with varying success rates. Although the choice of linguistic
modelling tools tested may, at first sight, seem ambitious for a university context such as
ours, we believe that even an audience of non-specialists in language sciences can benefit
greatly from this approach. Promoting explicit and reasoned vocabulary training in French
L2 aimed at specialists in other disciplines at university level appears, therefore, necessary
both for the maintenance of the French language in the specialist fields and the quality of
linguistic exchanges – oral and/or written – between technicians and experts in the field.
However, it is necessary to ensure the creation of manuals and pedagogical-didactic tools for
teaching/learning the lexicon in the LANSAD sector.
602
References
Binon, J., & Verlinde, S. (2004). L’enseignement/apprentissage du vocabulaire et la
lexicographie pédagogique du Français sur Objectifs Spécifiques (FOS) : le domaine du
français des affaires. ÉLA. Études de linguistiques appliquées, 135(1), 271-283.
Bonadonna, M. F. (2020). L’enseignement des notions métaterminologiques en français L2 :
un parcours fondé sur les corpus spécialisés. Cahiers de lexicologie, 117, 11-34.
Chaplier, C., & O’Connell, A.-M. (Eds.). (2019). Épistémologie à usage didactique. Langues
de spécialité (secteur LANSAD). Paris : L’Harmattan.
Council of Europe (2020). Quadro comune di riferimento per le lingue: apprendimento,
insegnamento, valutazione. Volume complementare. Italiano LinguaDue, 12(2), 8-285.
González-Rey, M. I. (2016). Une approche analogique à la compétence phraséologique : une
double compétence, intégrale et intégrée. Language Design. Journal of Theoretical and
Experimental Linguistics, Special issue, 165-188.
Hamel, M.-J., & Milićević, J. (2007). Analyse d’erreurs lexicales d’apprenants du FLS :
démarche empirique pour l’élaboration d’un dictionnaire d’apprentissage. Revue
canadienne de linguistique appliquée, 10(1), 25-45.
L’Homme, M.-C. (2020). La terminologie : principes et techniques. Montréal : Les Presses
de l’Université de Montréal.
Mel'čuk, I. (2013). Tout ce que nous voulions savoir sur les phrasèmes mais…. Cahiers de
lexicologie, 102(1), 129-149.
Mel'čuk, I., Clas, A., & Polguère, A. (1995). Introduction à la lexicologie explicative et
combinatoire. Louvain-la-Neuve : Duculot.
Mel'čuk, I., & Polguère, A. (2021). Les fonctions lexicales dernier cri. S. Marengo (Ed.). La
Théorie Sens-Texte. Concepts-clés et applications, Paris : L’Harmattan, 75-156.
Picoche, J. (1992). Précis de lexicologie française : l’étude et l’enseignement du vocabulaire.
Paris : Nathan.
Raus, Rachele, & Froeliger, N. (Eds.) (2020). La terminologie et l’enseignement du français
de spécialité, du français langue professionnelle ou sur objectifs spécifiques, Le Langage
et L’Homme, 542.
Tsedryk A., & Doe, C. (Eds.). (2019). The description, measurement and pedagogy of words.
Newcastle : Cambridge Scholars Publishing.
603
Mathematics in economic and business science: how to reach the

top without a pathway
Yeray Rodríguez, Ana Munárriz, María Isabel Goicoechea, María Jesús Campión
Department of Statistics, Computer Science and Mathematics, Public University of Navarre.
Abstract
The level of mathematical knowledge with which students enter Economics and
Business Administration Degrees varies according to the Baccalaureate
pathway taken. The Zone of Real Development of students coming from the
Social Sciences Baccalaureate differs from the Necessary Zone of Real
Development for the correct acquisition of academic competences, which is
close to that developed in the Scientific-Technological pathway.
In order to characterise the existing mathematical deficit, a quantitative and
perceptive analysis of the mathematical deficiencies of new students in those
Degrees coming from the Social Sciences pathway is proposed. The results
allow the identification of various areas where training deficiencies exist,
which are corroborated by the students' perception.
To solve them, the creation of a virtual platform that identifies, in a
personalised way, the deficiencies of each student is proposed. Through guided
gamification, the complementary course would allow students to reach the
Necessary Real Development Zone before starting University.
Keywords: Mathematics, economics, business, digital innovation project.

Mathematics in economic and business science: how to reach the top without a pathway
1. Introduction
The White Paper on the Bachelor's Degree in Economics and Business Administration
prepared by the National Agency for Quality Assessment and Accreditation (ANECA, 2005)
states that one of the chief problems of the current design of the degrees in this branch is "the
inadequate training in mathematics of this Baccalaureate (Social Sciences) compared to that
which will be required at the University" due to the existing divergences between the
mathematical curricular content of both pathways. Furthermore, it points out that "the content
in this subject should be the same as in the Scientific-Technological Baccalaureate".
Although the White Paper was published almost two decades ago, neither successive
educational laws nor the adaptation to undergraduate studies that took place with Spain's
entry into the European Higher Education Area (EHEA) have managed to solve these
deficiencies, making Mathematics one of the most failed subjects in the first year of
Economics and Business degrees and, consequently, students have to cope with higher-level
subjects (e.g., Economic Analysis, Finance, Statistics, Econometrics) without the required
mathematical basis (Anderson et al., 1994; Swope & Schmitt, 2006; Kara et al., 2009).
The mathematical deficit observed can be corroborated in various articles such as Adillón et
al. (2013) and Carrillo Fernández et al. (2016), which detail the differences between the
mathematical knowledge acquired during the Secondary Education stage depending on the
type of Baccalaureate studied. In this regard, articles such as Opstad (2018) show that this is
a widespread problem in other European countries, such as Norway, for example.
The main objective of this study is to determine whether the subject of Mathematics taken by
students in the Social Sciences pathway of the Spanish Baccalaureate provides an adequate
mathematical basis to take and pass the subject of Economics and Business Administration
degrees to which it leads, considering the mathematical level with which they enter university
and their perception after having taken the university subject.
Therefore, in accordance with Vygotsky (1978), the aim is to characterise the Zone of Real
Development (ZRD) of students entering university and to evaluate it with respect to the
Necessary Zone of Real Development (NZRD) required for the correct acquisition of
university competences, with the purpose of detecting the existence of conceptual
deficiencies that prevent the appropriate learning of new knowledge.
2. Methodology
The study consists of two clearly differentiated parts. On the one hand, mathematical aspects
are analysed by means of a level test that quantifies the knowledge with which the students
of Economics and Business Administration degrees of the Public University of Navarre
(UPNA) enter university studies. On the other hand, the perception of the level of
606
mathematics they have and the demand and adequacy of the level in the Social Sciences
pathway of the Spanish Baccalaureate to study university degrees is evaluated.
2.1. Evaluation of mathematical knowledge according to access type

Firstly, an evaluation of the mathematical knowledge acquired by students entering the
degrees in Economics and Business Administration during their time in Secondary Education
in the Spanish Education System is proposed. To accomplish this, new students were
subjected to a compulsory multiple-choice test on the first day of the academic year 2022/23,
in order to obtain an objective assessment of the mathematical level with which they access
the university and to observe whether this level is sufficient to satisfactorily complete the
mathematics subjects of both degrees.
The questionnaire consists of eight control questions, which were used to characterise the
entry profile of each student (e.g., access model, Mathematics subject taken in the
baccalaureate, average access mark), and eight mathematical questions on various basic
aspects of algebra and calculus that should be mastered, as these are the basis for the subjects
of both degrees, for instance: a simplification of a radical fraction by means of notable
equalities (Q1) or a 2𝑥2 matrix multiplication (Q8).
2.2. Evaluation of student perception

Secondly, an evaluation of the students' perception of the mathematics subject in the first
semester of the Economics and Business Administration degrees is proposed, with the aim
of analysing the students' feelings before and during this subject, as well as detecting any
possible difficulties encountered.
This information is obtained through the previous survey and with an additional voluntary
survey that students can fill out on the virtual platform of the subject from the last day of the
semester. On this occasion, the survey consists of fourteen questions, where the first four
questions correspond to control questions and the last ten are focused on obtaining the
students' perception of the subject, both in relation to the subject taken and to the level of
mathematical knowledge with which they have entered the university.
3. Results
In order to carry out the analysis of mathematical knowledge in university entrance and the
students' perception, the responses of 299 and 164 students, respectively, are evaluated. Both
questionnaires were conducted in an unpaired manner to ensure the anonymous nature of the
responses and the difference between the number of responses to the questionnaires is due to
the mandatory and voluntary basis of each of them.
607
3.1. Analysis of mathematical knowledge by access type

As can be seen in Table 1, 87% of the people surveyed got five questions or less right, out of
the eight questions answered. These results obtained in the initial questionnaire are
considerably poor, especially bearing in mind that the eight questions assessed are about basic
mathematical concepts that are essential for the correct acquisition of the competences
corresponding to university subjects.
Table 1. Distribution of hits in the initial survey.
Number Right Answer 0 1 2 3 4 5 6 7 8
Student Percentage 1% 9% 14% 20% 21% 21% 7% 5% 1%
Analogously, as shown in Figure 1, evaluating the results obtained in each of the questions,
only two of the eight answers (Q6 and Q8) are correctly answered by three out of every four
students surveyed. In this sense, relevant mathematical deficiencies are found in questions
Q2, Q3, Q4, and Q5, as, more than just having a low rate of correct answers (less than 40%),
the percentage of wrong answers is higher than the correct answers, which implies that not
only have they not acquired this knowledge but also they use it wrongly and based on
inadequate mathematical criteria.
Figure 1. Overall distribution of answers per question.

The lack of mathematical knowledge is more evident when the answers obtained from
students coming from the Scientific-Technological Baccalaureate (STB) and the Social
Sciences Baccalaureate (SCB) are differentiated, as can be seen in Figure 2.
Although it is obvious that students coming from the STB branches have mathematical
deficiencies in the knowledge relating to questions Q2 and Q4 (Figure 3), they enter with
much more complete training and much closer to the minimum knowledge required for the
correct acquisition of university contents in those degrees. However, the entry profile of
608
students coming from the Social Sciences pathway shows highly notable deficiencies in
practically all the basic aspects required.
Figure 2. Distribution of answers by question and entry profile.
Figure 3. Comparison of mathematical knowledge according to entry profile.

Finally, in order to corroborate that the difference in knowledge between students with
different income profiles is statistically significant and that it is not the result of the difference
in the number of surveys answered by one group and the other (27 and 272, respectively), a
Welch's t-test was performed, with which the hypothesis corresponding to the existence of
equality of means between samples can be accepted or rejected; in other words, that
mathematical knowledge in both income profiles is identical.
Considering that the p-value obtained in the test is 0.00136, lower than the limit marked by
the significance level α, the null hypothesis is rejected and the existence of significant
differences between the levels of the mathematical knowledge of both profiles is confirmed.
Therefore, this result shows that the mathematics of the STB generates more competent and
609
related profiles than those of the SCB for an adequate understanding of mathematics and the
acquisition of the competences of those degrees.
3.2. Analysis of student perception

Student perception is studied, both in relation to their level of knowledge on entering
university and in relation to the overall perception of their learning at the end of the
Mathematics lessons in the first semester at university, before taking Mathematics exam.
Firstly, analysing the data displayed in Figure 4, more than 90% consider their mathematical
base to be between adequate and satisfactory. However, these perceptive data contrast
completely with the results obtained in the previously analysed questionnaire.
Figure 4. Students’ perceptions of their level of mathematics at university entrance.
Nevertheless, at the end of the Mathematics subject, the percentage of students who consider
their mathematical base to be between adequate and satisfactory is approximately 30% lower.
In any case, according to the results obtained in the perception survey, there is a large
majority of students who consider that they enter university with notable deficiencies in
mathematics, as pointed out in Figure 5. Students identify the existence of a structural deficit
in mathematics training in the Baccalaureate: 69% of them consider that it is essential to
improve the level of mathematics in Secondary Education specialisations leading to degrees
in Business Administration and Economics and 63% consider that the introduction of a
complementary course to cover the shortcomings detected between the exit-level from
Secondary Education and the entry-level required for those degrees would be positive.
Figure 5. Student Perception of the suitability of access level and the need to implement a complementary course.
610
In both cases, there is a notable difference in perception depending on the entry profile. While
the answers of students who have studied the SCB show a need to improve the level of the
Mathematics subject in their speciality and the opinion of implementing the complementary
course prevails, with 71% and 63% of positive responses respectively, students in the STB
speciality do not consider such a structural deficit in the training received in the mathematics
of their speciality nor so essential the implementation of the complementary course.
4. Discussion
The results obtained highlight the existence of significant differences between the level of
mathematical knowledge acquired in the STB and that of SCB, in line with previous academic
literature (Adillón et al., 2013; Carrillo Fernández et al., 2016). In this sense, the STB
provides a considerably more appropriate mathematical basis for entry to Business
Administration and Economics degrees, which should not be the case, given that the
recommended Baccalaureate for these degrees is that of Social Sciences.
Precisely, students coming from this pathway are aware of the existence of a mathematical
deficit in their Baccalaureate modality. This lack of basic training affects all university
subjects based on mathematics, as mentioned before (Anderson et al., 1994; Swope &
Schmitt, 2006; Kara et al., 2009); thus, corroborating the existence of a difference between
the current ZRD and the NZRD (Vigotsky, 1978). In fact, this may be a key reason for the
worrying failure and no-show rates in the first year, and even for the high drop-out rates.
This problem is hard to address. The ideal situation for this type of students would be to give
them the opportunity to take both SCB and STB Mathematics in the Social Sciences pathway,
as a modality and optional subjects, respectively. Unfortunately, this is not possible in most
Spanish schools due to organizational reasons. Therefore, the recommended solution to the
problem would be to be able to choose STB mathematics as a subject of modality within the
Social Sciences pathway, a matter that the current legislation does not consider.
611
Figure 6. Necessary and current model of the students' Zone of Proximal Development.
Until the education system solves the training problem detected, it is recommended to
alleviate this deficit through a virtual platform that allows students to reach the NZRD in a
guided manner, prior to the start of the first-year university subjects. In this way, depending
on the individual ZRD of each student, characterised progressively as they answer the check
questions of each concept to be reinforced in the course, the tool will generate a personalised
Zone of Proximal Development (ZPD); that is, an individualised and guided complementary
course, which, depending on the deficiencies observed, will lead each student to the target
NZRD, as can be seen in Figure 6.
This recommendation is in line with what was observed in the carried-out survey: students,
especially those coming from the SCB, are clearly in favour of implementing a virtual
platform to cover their conceptual deficiencies. The development of this complementary
course at the UPNA, which is the next step of current research, is subordinated to the
educational innovation project Digital Gamification to address pre-university mathematical
training deficiencies in students of Economics and Business Degrees of the UPNA.
Finally, note that the study was performed in a single year and exclusively with UPNA
students. It would be interesting to carry out this study over a larger number of academic
years, as well as in other universities.
References
Adillón, R., Jorba, L., Purroy, P., Ribas Marí, C., & Tarrío Reboredo, J. A. (2013). Los
Conocimientos matemáticos de los nuevos universitarios: el caso de Economía y Empresa
de la UB [The mathematical knowledge of new university students: the case of Economics
and Business at the University of Barcelona (UB)]. Univest 2013.
612
Anderson, G., Benjamin, D., & Fuss, M. A. (1994). The Determinants of Success in
University Introductory Economics Courses. The Journal of Economic Education, 25(2),
99–119. https://doi.org/10.2307/1183277
Agencia Nacional de Evaluación de la Calidad y Acreditación, ANECA. (2005). Libro
blanco: Título de Grado en Economía y en Empresa [White Paper: Degree in Economics
and Business Studies]. Madrid.
Carrillo Fernández, M., Cruz Báez, D. I., Fariña, G., González Concepción, C., Pestano
Gabino, C., & Sosa Martín, D. N. (2016). Análisis de la formación matemática
preuniversitaria con datos españoles y consecuencias en los estudios de Economía y
Empresa: el caso de la Universidad de La Laguna [Analysis of pre-university mathematics
education with Spanish data and consequences for economics and business studies: the
case of the University of La Laguna]. Revista Internacional de Economía y Gestión de
las Organizaciones, 3(1), 59-81.
Kara, O., Bagheri, F., & Tolin, T. (2009). Factors Affecting Students Grades In Principles Of
Economics. American Journal of Business Education (AJBE), 2(7), 25–34.
https://doi.org/10.19030/ajbe.v2i7.4581
Opstad, L. (2018). Success in business studies and mathematical background: The case of
Norway. Journal of Applied Research in Higher Education, 10(3), 399.
Swope, K. J., & Schmitt, P. M. (2006). The Performance of Economics Graduates over the
Entire Curriculum: The Determinants of Success. The Journal of Economic Education,
37(4), 387–394.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes.
613
The role of education in fostering entrepreneurial intentions

among business students
Faculty of Business, Management and Economics, University of Latvia, Latvia.
Abstract
Entrepreneurial intention is a critical aspect of the creation of new businesses
and the development of an entrepreneurial culture. It is considered to be the
strongest predictor of an individual's behavior towards entrepreneurship. The
present study aims to examine the role of education in promoting
entrepreneurial intentions among business students and understand how
educational programs, initiatives and experiences can impact the development
of an entrepreneurial mindset. The study was conducted using focus group
discussions to gather views and attitudes of individuals. The results suggest
that in order to cultivate entrepreneurial intentions among business students,
education must provide exposure to successful entrepreneurs, encourage
hands-on experience, offer entrepreneurship-focused classes and workshops,
provide access to funding and mentorship, foster a supportive and inclusive
environment, emphasize the development of critical thinking and problem-
solving skills, and encourage continuous learning. The findings of this
research provide valuable insights for educators, policymakers, and
stakeholders to enhance entrepreneurial education and foster a supportive
environment for the development of entrepreneurial skills and mindsets among
business students.
Keywords: Entrepreneurial intention; business education; leadership skills;
business students.

The role of education in fostering entrepreneurial intentions among business students
1. Introduction and theoretical background

Entrepreneurs are instrumental in fostering innovation and boosting economic expansion in
the current volatile business landscape. Entrepreneurship is an important skill that students
can learn from universities, which can help them create their own businesses and contribute
to the economy. Universities play a crucial role in preparing students for the future. Exploring
what drives students perceived as potential entrepreneurs to engage in entrepreneurial
pursuits has been a widely studied topic for over three decades (Dragin et al., 2022). Surveys
suggest that between 15% and 20% of students who participate in the Minicompany
Programme in secondary school will later start their own company, a figure that is about three
to five times that for the general population (Jenner, 2012). It means universities should
become more entrepreneurial (Gibb et al., 2013). This not only creates new jobs and
contributes to economic growth, but also provides students with a sense of autonomy and
independence.
It cannot be denied that entrepreneurship is a dream of many people. An entrepreneurial
dream can exist for a person regardless of their current business success, opportunities,
desires or hopes of becoming an entrepreneur (Li et al., 2022). An individual who associates
their professional identity with unfulfilled values, will not simply forget about them, but
instead, continually strive to remember them and explore ways to make these values a reality
(Obodaru, 2017). Research indicates that many individuals fail to achieve their
entrepreneurial dream due to a lack of business knowledge or fear of risk-taking. Despite
this, the unfulfilled dream does not fade away and may eventually turn into a feasible
intention for the future (Berg et al., 2010).
Own business gives a person a certain independence and autonomy and allows him to prove
to himself and others what he is capable of.
2. Entrepreneurial intention in the aspect of entrepreneurial culture

Entrepreneurial intention is a crucial aspect in the formation of new companies and the
development of an entrepreneurial culture, yet existing literature primarily addresses the
challenges faced by entrepreneurs during the start-up and implementation phase, neglecting
research on the identification of Entrepreneurial Intention. Entrepreneurial intentions refer to
the desire and willingness to start a new business venture. Business students, as future
business leaders, are potential entrepreneurs who can contribute significantly to the economy.
It will also improve economic factors, which will positively affect the working environment
in Latvia (Saksonova & Jansone, 2021). Thus, it is essential to foster entrepreneurial
intentions among them.
616
Testing Entrepreneurial intention using university students is appropriate as it can inform

education policies, particularly those related to entrepreneurship education, and have
implications for those responsible for creating support programs in this area. Several studies
have targeted business and engineering university students as potential entrepreneurs (Liñán
et al., 2011). According to the 2021/2022 data of the Central Statistics Bureau of Latvia on
higher education in Latvia, the highest number of enrolments is still found in Social sciences,
business and law, with 26.1 thousand, or one third of the total number of enrolments (CSB,
2022).
The proportional distribution of bachelor's level graduates by thematic groups of education
reflects the distribution of students in these groups. The largest group of graduates are those
who studied in the "Social sciences, commercial sciences, and law" thematic group. In a
survey of young people in Latvia, 63% expressed a desire to become entrepreneurs who
employ others, 8% preferred entrepreneurship in a family business, 16% preferred self-
employment, while only 12% chose the status of an employee (MES, 2015). Studies in Latvia
indicate that over half (52%) of Latvian residents have considered starting their own business,
showing that entrepreneurship is a highly popular employment option. However, few have
actually put their ideas into action, with only 14% of those who have considered starting a
business actually doing so. It's worth noting that in recent years, residents have become more
likely to consider starting a business; in 2012, this figure was 37% (DNB, 2015).
Intention is the strongest predictor of an individual's behavior (Ajzen, 1991). Entrepreneurial
intention refers to an individual's belief and plan to start a business in the near future
(O’Reilly et al., 1991) and create new and competitive innovations (Batraga et al., 2019).
Scientific literature suggests that people's self-perception is complex and constantly evolving,
rather than simple and constant (Caza et al., 2018). The likelihood of starting a business is
influenced by attitudes towards the potential outcomes and the perceived risks and benefits.
Multiple studies on the motivations for entrepreneurship have confirmed that there is no
direct correlation between external environmental factors and entrepreneurial activity.
Instead, external factors are seen as merely providing a more favorable environment for
entrepreneurship, serving as a "trigger" rather than a direct cause (Janse van Rensburg &
Tjano, 2020). The likelihood of choosing entrepreneurship as a career is significantly
impacted not only by external factors such as incentives, but also by personal values and
orientation. Personal values refer to the beliefs and principles that guide an individual's
behavior and decision making.
617
3. Strengthening of identity in the studies process

Research indicates that while young business management graduates exhibit strong decision-
making skills, they often lack the comprehensive management abilities needed to effectively
handle the numerous critical factors involved in problem solving (Ding et al., 2020). Several
scholars (Gould & Voelker, 2010; Jones & Lavallee, 2009) have identified leadership as an
important life skill for young people to develop and that all young people have the potential
to be leaders. Researchers have argued that intentionally offering young people leadership
roles can help them develop quality-of-life skills such as communication and decision-
making (Shaikh et al., 2019).
Based on the evidence presented, it appears that there is a need for young business
management graduates to develop comprehensive management abilities beyond strong
decision-making skills. This could be achieved by intentionally offering them leadership
roles to develop quality-of-life skills such as communication and decision-making.
4. Methods. Research design

The objective of this research is to understand how educational programs, initiatives and
experiences can impact the development of entrepreneurial mindset and encourage students
to pursue entrepreneurial careers. The study aims to provide valuable insights and
recommendations for educators to enhance the entrepreneurial education and foster a
supportive environment for the development of entrepreneurial mindsets among business
students. The focus group discussions were conducted to find out the views and attitudes of
individuals. In addition, focus groups are an appropriate method for constructivist studies.
The study was conducted on 84 business management students from the "Social sciences,
commercial law and law" group, who were recruited from a specific course on business
management. It consisted of 5 focus group discussions. The participants were recruited using
of convenience sampling method. The sample was chosen based on the relevance of students’
academic background to the research question and because they had already demonstrated a
commitment to studying business-related topics.
5. Data collection
The discussions covered three main topics: the impact of the academic environment on the
students' identification with entrepreneurship, factors promoting or hindering the realization
of entrepreneurial intentions, and solutions to the intrapersonal conflict faced by business
students in realizing their entrepreneurial goals.
The research results are summarized through content citations, highlighting the key factors
in fostering Entrepreneurial intention among business students. When discussing the number
618
of times issues/outcomes were identified in a focus group discussion, Table 1 indicates how
many times certain topics or issues were mentioned or discussed by the participants. This
provides an indication of the frequency and importance of certain issues or topics within the
group, and can also help identify areas where there is consensus or disagreement among
participants.
Table 1. Focus group results.
General Theme Specific Issue/Outcome Number of Times

Issues/Outcomes
Were Identified
The impact of the academic Practical experience 36
environment on the students' Mentorship 32
identification with Entrepreneurial education/training 32
entrepreneurship Networking opportunities 24
Curriculum design 21
Research opportunities 18
Entrepreneurial ecosystem 18
Total outcomes identifided by groups 181
Factors promoting or Self-efficacy beliefs 28
hindering the realization of Fear of failure 26
entrepreneurial intentions Perceived barriers 26
Personal circumstances 23
Lack of knowledge/experience 17
Positive attitude/mindset 13
Solutions to the intrapersonal Goal-setting 24
conflict faced by business Established decision-making style 24
students in realizing their Pressure to pursue traditional careers 21
entrepreneurial goals. Personal development 18
Action planning 16
Mindset shift 16
Resilience building 8
The results of the study show that business students may lack hands-on experience in
entrepreneurship, making it difficult for them to understand the realities of starting and
running a business. Content citations, e.g., incorporating more practical tasks that foster
business thinking or business simulation games into the study process would be beneficial.
Many business students may not have access to an entrepreneurial ecosystem that can help
them understand the benefits and challenges of starting a business. Content citations, e.g., It
helped me a lot to change my thinking from "what would I like to do?" to "what problem
could I solve?"… ; not to perceive studies as an obligation, but as an opportunity. One of the
important problems revealed by the research is that business students may struggle to find
experienced mentors who can guide and support them as they develop their entrepreneurial
ideas. Business students may lack a comprehensive understanding of the various business
619
models and strategies necessary for success in entrepreneurship. Content citations, e.g.,
practical experience and hands-on learning can be just as valuable as formal education. By
starting small and gaining experience through internships, side projects, or even by starting
our own businesses, we can gain a deeper understanding of the various business models and
strategies that are most effective; a limited knowledge of business models and strategies
should not deter us from pursuing entrepreneurship.
Also, focus group participants emphasized that they feel pressure from friends, family, or
society to pursue traditional career paths, which deter them from pursuing entrepreneurship.
Business students, who are the future entrepreneurs, often face intrapersonal conflicts while
realizing their entrepreneurial intentions. These conflicts arise from within the individual and
can lead to a barrier in realizing their entrepreneurial intentions.
6. Conclusion and implications

Summarizing the results of the interviews, it can be concluded that not only lack of finance,
but also lack of creative ideas, insecurity and competition - these are just some of the reasons
that prospective, young entrepreneurs, having doubts about starting their own business,
should replace with purposefulness, confidence and daring. Intrapersonal conflict refers to
the conflicting thoughts, feelings, and behaviors that arise within an individual. In the context
of entrepreneurial intention, these conflicts can arise from various internal factors such as
self-doubt, fear of failure, lack of confidence, and uncertainty about the future. The
intrapersonal conflict faced by business students can significantly impact their
entrepreneurial intentions. The fear of failure can lead to a lack of motivation, which can
result in students giving up on their entrepreneurial aspirations.
Since many behaviors create performance difficulties that limit volitional control, it is useful
to pay increased attention to promoting students' self-control abilities. An individual's control
over behavior can be influenced by his learned decision-making style. It refers to the
approach or method of decision-making that an individual has developed through past
experiences and learning. These decision-making styles can be influenced by factors such as
personality, values, and cognitive biases. Business decision-making is a complex process that
involves choosing the best course of action among several alternatives. It is important to
include in the study process the evaluation of various situations for developing critical
thinking. Experience is a very important factor in the business intent aspect. Two types of
business experience can be distinguished: experience gained before starting a business and
experience gained during the course of business. Previous experience is very valuable in
terms of business intent.
The research findings suggest that in order to foster entrepreneurial intentions among
business students, the incorporation of specific actions into the study process is crucial. These
620
actions comprise the following: Providing exposure to accomplished entrepreneurs and their
stories, which encompasses both triumphs and setbacks, to prompt students' identification
with entrepreneurship - an influential catalyst for the commencement of new businesses.
Furthermore, hands-on experience via project-based or business incubator activities can aid
in the realization of delayed professional identity and self-expectations, as focusing on
unrealized identities tends to create a strong desire for them (Lyubomirsky et al., 2010).
Additionally, the provision of courses or workshops focusing on creativity is the preliminary
step toward innovative thinking.
The identification of an individual's entrepreneurial intention is a critical task, as it aligns
with activities that have the potential to create significant value for both the individual and
society, fostering job creation and innovative breakthroughs. Concurrently, based on the
principle of sustainability, it is also essential to identify decision-making models that enable
effective resolution of intrapersonal conflicts. Such models play a key role in enhancing
psychological and mental resilience of individuals, thereby reinforcing their capacity to
confront challenges in a sustainable manner. Fostering a culture of continuous learning and
encourage students to seek out new knowledge and skills to help them grow as entrepreneurs.
Entrepreneurship is an important skill that universities can impart to students, helping them
create new businesses and contribute to the economy.
Acknowledgements
This research is supported by the project "Strengthening of the capacity of doctoral studies
at the University of Latvia within the framework of the new doctoral model”, identification
No. 8.2.2.0/20/I/006.
References
Ajzen, I. (1991) The theory of planned behavior». Organizational behavior and human
decision processes, vol. 50, 179-211.
Batraga, A., Salkovska, J., Braslina, L.,Legzdina, A., Kalkis, H. (2019). New innovation
identification approach development matrix. Advances in Intelligent Systems and
Computing,783, pp. 261 – 273.
Berg, J.M., Grant, A.M., Johnson, V. (2010). Your callings are calling: Crafting work and
leisure in pursuit of unanswered occupational callings. Organ. Sci., 21, 973–994.
Caza, B.B., Vough, H., Puranik, H. (2018). Identity work in organizations and occupations:
Definitions, theories, and pathways forward. J. Organ. Behav. vol. 39, 889–910.
Ding, N., Xu, X., Yang, H., Li, Y., van Heughten, P. (2020). Journal of Education for
Business, Aug-Sep 2020, vol. 95, iss. 6, pp. 351-358.
DNB banka, SIA "SKDS" (2015). DNB Latvijas barometrsNr. 83: Uzņēmējdarbība. Rīga.
621
Dragin, A.S., Mijatov, M.B., Munitlak Ivanović, O., Jovičić Vuković, A., et al. (2022).
Entrepreneurial Intention of Students (Managers in Training): Personal and Family
Characteristics. Sustainability, vol. 14, iss 7345
Gibb, A., Haskin,s G., Robertson, I. (2013). Leading the Entrepreneurial University: Meeting
the Entrepreneurial Development Needs of Higher Education Institutions, Innovation,
Technology, and Knowledge Management, in: Andreas Altmann & Bernd Ebersberger
(ed.), Universities in Change, edition 127, chapter 0, pages 9-45, Springer.
Gould, D., Voelker, D.K. (2010). Youth sport leadership development: Leveraging the sports
captaincy experience. Journal of Sport Psychology in Action, vol. 1(1), pp. 1-14.
Janse van Rensburg, L.J., Tjano, R.N. (2020). The relationship of self-efficacy and
entrepreneurial intentions on the commitment of the next generation in family-owned
agribusinesses’, Acta Commercii, vol. 20(1), 742 p.
Jenner, C., Business and Education: Powerful Social Innovation Partners, Stanford Social
Innovation Review (Aug. 27, 2012).
Jones, M. I., Lavallee, D. (2009). Exploring the life skills needs of British adolescent athletes.
Psychology of Sport and Exercise, vol.10, pp. 159-167.
Li, M., Li, J., Chen, X. (2022). Employees’ Entrepreneurial Dreams and Turnover Intention
to Start-Up: The Moderating Role of Job Embeddedness. Int. J. Environ. Res. Public
Health, vol. 19, p. 9360.
Liñán, F., Urbano, D., Guerrero, M. (2011). Regional variations in entrepreneurial
cognitions: Start-up intentions of university students in Spain. Entrepreneurship &
Regional Development, 23:3-4, 187-215.
Lyubomirsky, S.; Boehm, J.K.; Kasri, F.; Zehm, K. The cognitive and hedonic costs of
dwelling on achievement-related negative experiences: Implications for enduring
happiness and unhappiness. Emotion 2011, 11, 1152–1167.
Martinek, T., Hellison, D. R. (2009). Youth leadership in sport and physical education, New
York, NY: Palgrave Macmillan.
Ministry of Education and Scienc, Report on Youth Policy Monitoring 2015, SIA “Analītisko
pētījumu un stratēģiju laboratorija” (2015). Jaunatnes politikas monitorings. Rīga.
O’Reilly, C.A., Chatman, J., Caldwell, D.F. (1991). People and Organizational Culture: A
Profile Comparison Approach to Assessing Person-organization Fit. Acad. Manag. J.,
vol. 34, 487–516.
Obodaru, O. (2017). Forgone, but not forgotten: Toward a theory of forgone professional
identities. Acad. Manag. J., 60, 523–553.
Saksonova, S., Jansone, M. (2021). Economic Factors of Labor Migration Analysis. Lecture
Notes in Networks and Systems, 195, pp. 649–659.
Shaikh, M., Bean, C., Forneris, T. (2019). Youth Leadership Development in the Start2Finish
Running & Reading Club 2019, Journal of Youth Development, vol. 14 (1) , pp.112-130.
622
The learning motives of business students and postponement

Tarvo Niine, Jelena Hartšenko, Merle Küttim, Anna Kornijenko, Juhan Loorits
Department of Business Administration, Tallinn University of Technology, Estonia.
Abstract
This paper examines the motives of students’ program and specialization
choice on the example of business students in TalTech, Estonia. The program
applies postponement strategy – students choose between five specializations
of more direct vocational relevance during their second year. This study aims
to distinguish extrinsic and intrinsic motivation factors of students who have
weighed their specialization choice for longer from students who have made
that decision in an earlier phase.
The findings suggest that the level of intrinsic learning motivation is not
significantly different between the two groups. However, an earlier decision
indicates that the student is more specialization-driven, whereas the
postponement approach is connected to a generalist / entrepreneurial profile.
This proposes that when a programme targets cross-functional skills and
entrepreneurial spirit development, the postponement strategy is
recommended as a motivational tool. To a degree, postponement brings about
benefits and being undecided should not be treated as a weakness.
Keywords: Business school; undergraduate studies; career choice motives,
learning motivation, postponement, intrinsic motivation.

1. Introduction
Student learning motivation and motives of career choices are not only diverse in a static
moment but also possibly significantly changing over time, both before starting college as
well as during undergraduate years. Therefore, on a personal level, there is a lot on the stake
if one needs to make this decision in relative haste somewhere around the final stages of
secondary education. Luckily, in modern business education, this risk is often lowered by
program design that expects students to do multiple step-by-step choices instead, for example
starting a broad-based business program initially, then choosing a specialization branch and
later as well as alongside facing a wide pool of elective courses or modules. Learning
business in this way builds a T-shaped profile: a sum of a birds-eye view augmented with
deeper, often more functional and specialization-specific competences. In such
configuration, it often makes sense to decouple the specialization choice from the initial
choice of starting university studies. Simply put, such postponement allows additional first
year experiences to drive students towards a more enriched perspective to allow more
meaningful choices as well as build confidence.
Motivation has received considerable attention over the past two decades. Central to this
debate is self-determination theory, which differentiates between two types of motivation:
intrinsic and extrinsic (Ryan & Deci, 2000; Ryan & Deci, 2017; Ryan & Deci, 2020). While
there is a plethora of factors that can possibly impact the student learning motivation and
principal career choices, they can be broadly viewed in the same framework, defined by
whether the factor is mostly controlled by some external power or actor that applies pressure
through punishments and rewards, or is instead coming from inside the person and is more
carried by the enjoyment of the process rather than any desired outcome. In the context of
career choice motives, some factors are connected to principal expectations to outcomes,
such as the perceived value of competences, range of accessible jobs, wages, lifestyle,
reputation, or even having proved oneself to the society. Such points are relatively common
themes in the marketing communication of business schools aimed at prospective students.
Rather more intrinsic would be factors such as a program matching with personal interests,
hobbies, personal development goals and even with character traits.
Ideally, one could expect the freshman year to boost both extrinsic as well as intrinsic
motivation of students, which would result in a more analytical decision. However, for a
section of students, the postponement opportunity is irrelevant altogether as they have already
committed in their hearts to a specialization earlier, some even before entering the program.
This by itself would suggest a higher degree of intrinsic motivation. However, it can also be
viewed as a potential risk in hindsight. Our study observes the perceived motives of program
and specialization choice of business students as perceived amidst their studies, contrasting
the “predetermined” and “postponed” student segments. The aim of the study is to understand
the differences in motives of choosing a career and to evaluate the favor or disfavor of
postponing the specialization decision. The paper includes some brief comments on
624
motivation aspects relevant for business student career choice. We then present the
methodology of our study and limitations. The findings allow debate on the preferable model
of student choice from the viewpoint of program management.
According to the self-determination theory, amotivation (lack of competence and value,
nonrelevance) and intrinsic motivation (interest, enjoyment, satisfaction) are the two ends of
a spectrum. In between resides extrinsic motivation consisting of four regulatory styles:
external regulation (external rewards or punishments), introjection (approval from others and
self), identification (personal importance, valuing, goals) and integration (consistency of
identifications) (Ryan & Deci, 2020).
The activity is intrinsically motivated when it’s purposeful, creates interest and provides
satisfaction (Laran & Janiszewski, 2011; Wasserman & Wasserman, 2020). According to
self-determination theory, autonomy, competence and relatedness are needed to satisfy
persons’ basic needs and create the environment for personal growth and integration (Ryan
& Deci, 2020). Extrinsic motivation is characteristic of a situation when the activity itself is
less important than its social or material consequences (Fischer et al., 2019). The stronger the
extrinsic motivation, the smaller the effort undertaken (Ryan & Deci, 2000). People can,
however, have simultaneously multiple motivations for their actions (Ryan & Deci, 2020).
Students’ study and career motivation can be also studied in the intrinsic and extrinsic
motivation framework (Kornijenko, 2022; Akosah-Twumasi et al., 2018). Extrinsic
motivation is related to more shallow study methods, while intrinsic motivation is
characteristic more of a deep study (Lucas & Meyer, 2005). The former is driven by external
factors necessary to get the grade or fulfil the exam requirements, while the latter leads to
understanding the study content, engaging actively and critically with it, in order to solve
problems and implement new ideas, and find associations with existing knowledge (Duff &
McKinstry, 2007). Study motivation is not fixed, but can change over time, influenced by the
person, situation and important others (Boström & Bostedt, 2020).
According to Iacovou et al. (2011), there are five main work attributes: promotion potential,
type of work (interesting, challenging), safety at work, remuneration and co-workers. In
terms of youths’ career motivation, Akosah-Twumasi et al. (2018) have found the extrinsic
factors to include financial remuneration, job security, professional prestige and job
accessibility; and the intrinsic factors to consist of personal interests, self-efficacy, outcome
expectations and professional development opportunities. For business students, their career
is mainly related to three factors: earning money for personal needs, self-fulfillment and
growth, and making one's dreams come true (Frankowska et al. 2015). In choosing business
as a field of study, the strongest motives are personal interests, but also compatibility with
one’s lifestyle and talents (Loorits, 2022).
625
In addition to internal and external motivators, family members, teachers and peers can
impact youth’s career decision-making. According to Polenova et al. (2018), students
indicated parental influence on a specific career choice. Moreover, Akosah-Twumasi et al.
(2018) reported that interpersonal factors and emergent bicultural influence play a role in
career choice and that parental influences can be significant in collectivist cultural settings.
3. Research methods
The study employed quantitative research design. Data was collected from undergraduate
business students of TalTech, Estonia with an online survey in early 2022. The Business
program is characterized as a broad foundation to business, where students choose between
five specializations on their second year: finance, entrepreneurship and management,
logistics and supply chain, accounting and business intelligence, and marketing. Still, roughly
⅔ of the program is shared across specialization branches. The questions had roots in
previous studies on study and career motivation (Ryan & Deci, 2000, 2020; Akosah-Twumasi
et al. 2018). The survey included demographic profile, evaluation of motives for students’
initial choice of a program and of their specialization, and the time of the latter choice. The
choice motives were evaluated on a 5-point Likert scale [strongly disagree; strongly agree].
The timing of the decision provided choice between five options, which were later condensed
into two categories: “predetermined” (choice made before studies or during the first
semester), and “postponed” (from second semester up until the last moment on the third).
The study population was a cohort of 2020 intake with 113 students. 69 responses were
received (61%), 75% female, average age 21. Two respondents were excluded due to
incompleteness. The response rates were comparable across gender and across
specializations – the data is representative of both genders and we could not identify any
major sample bias (other than the most typical “lowly motivated subjects might not
respond”). Data was analysed with Spearman’s and Wilcoxon correlation analysis and linear
regression, conducted by SPSS v.28.
4. Results
The survey was firstly validated with exploratory factor analysis, yielding three factors:
intrinsic, extrinsic and interpersonal. Table 1 demonstrates factor loadings exceeding 0.4
(Hair et al., 2009). Cronbach’s alpha shows the internal consistency of the study. The
reliability estimates of scaled items ranged [0.63 to 0.82]. Table 1 includes 13 motives, while
eight others in the original survey were omitted, following the result of factor analysis. This
paper could not fit some of auxiliary and more specific data, but the authors are willing to
share it on demand. Table 2 indicates the means of the three motivational factors (extrinsic
motivation, intrinsic motivation, interpersonal factor), according to the independent sample
Mann-Whitney test.
626
Table 1. Mean and standard deviation of the scale items.
Factor Cronbach's
Mean SD loading Alpha
Intrinsic factor
I am interested in the topic area 4.19 1.02 0.886 0.824
I aim to develop my entrepreneurial skills 3.88 0.90 0.794
I aim to be a part of social and business network 3.96 1.02 0.765
I can combine my future job with my hobby 3.54 1.11 0.699
My vision is to become an entrepreneur 3.37 1.22 0.637
The curriculum is attractive and suitable for me 4.51 0.59 0.563
Extrinsic factor
The university and program are modern and practical 4.49 0.59 0.781 0.633
I think TalTech School of Business and Governance
offers the best business education in Estonia 4.01 0.79 0.687
Graduating the program is a way to prove my worth
to the society 3.45 0.99 0.633
The program has positive public reputation 3.67 0.94 0.615
Ability to make money 3.88 0.90 0.415
Interpersonal factor
Parents 2.45 1.26 0.835 0.706
Friends and peers 2.33 1.15 0.783
Source: authors’ calculations
Surprisingly, most motives did not vary between predetermined and postponed groups.
Seemingly slightly, but statistically insignificantly, the postponed group leaned towards a
stronger motive of broad career perspective, whereas the predetermined group was more
motivated by “relevant specialization available”, which suggests that the predetermined
group have a slight leaning towards seeing themselves as specialists, while generalism
appeals to the postponed group, as they were also, on average, more inspired by
entrepreneurship. This also contributes to why overall intrinsic motivation is slightly higher
in the postponed group. It can be seen as modest positive feedback to the faculty – arguably
the entire learning process is now more meaningful.
The data showed that the postponed group is slightly more motivated by wages. It can be
interpreted that the students more driven by future wages would indeed postpone their
decision, as such direct pragmatism would allow a more calculated decision. However, for
both groups, it was s secondary motif. The overall extrinsic component was slightly lower in
postponed group because of such pattern of other extrinsic elements. Between the two groups,
one difference was in the interpersonal factor - the influence of parents and friends had played
a stronger role in the predetermined group and less so in the postponed. This was expected,
but the overall low position of interpersonal factor alongside other elements was a bit
surprising. Still, this is in line with previous studies (Akosah-Twumasi et al., 2018),
considering high level of individualism in local cultural context.
627
Table 2. Differences in the mean scores for choice motives.
Mean
Factor Career decision Mean difference p-value
Extrinsic Predetermined group 3.88 0.15 0.292

Postponed choice group 3.73
Intrinsic Predetermined group 3.84 -0.06 0.766
Interpersonal Predetermined group 2.61 0.52* 0.084
Note: * significant difference at the 0.1 level.
Table 3 provides the results of the correlation analysis. There is a strong positive correlation
between intrinsic and extrinsic factors for a “predetermined” group of students (choice made
before or during the first semester), which is not the case for postponed group. Furthermore,
there is negative correlation between intrinsic and interpersonal factors for postponed group.
Table 3. Spearman’s correlation for career decision-making and motivation factors.
Intrinsic Extrinsic Interpersonal

factor factor factor
Postponed choice group Intrinsic 1
Extrinsic 0.077 1
Interpersonal -.382** 0.037 1

Predetermined group Intrinsic 1
Extrinsic .559*** 1
Interpersonal 0.101 0.218 1

Note: *** correlation is significant at the 0.01 level; ** correlation is significant at the 0.05 level.
Furthermore, it appears worthwhile to mention that the learning motives were rather similar
across specialization options. The data indicated minor differences for local reasons (such as
certain specializations having higher reputation perhaps because of visibility of local faculty
activities). In addition, the motifs of societal contribution and being attracted by rapid
development of the area applies to logistics students more than for other branches, but still
the overall role of these motifs is secondary. In comparing the specialization choice with the
timing aspect, there was only one significant finding – the share of marketing students in the
628
predetermined group stood out from the overall sample. We speculate it is a result of various
local education landscape reasons, as different specializations have different supply patterns.
Not only were the motives of choice similar across specializations, but also various other
specializations were showing to be still rather close “in competition” in terms of their overall
attractiveness. While we had assumed that giving students time to weigh their options makes
it a clear choice for most, the data indicated that for most students, there is at least one other
option almost similarly favorable. While it is positive in the sense that all options are seen
worthy to be considered (suggesting there are no major local quality bottlenecks) it is also a
sign that students (at least on their second year of studies) are still diverse mostly in similar
ways between specializations. Even though many colleagues in academia might feel that
there are clearly “students better-suited for marketing” and others for finance or accounting,
the individual choices of students seem mostly to disregard such stereotypes.
5. Discussion and conclusion

One possible conclusion is that postponement does not significantly boost overall motivation
because locally our faculty is not good enough. Still, fortunately, there is no significant
motivational decline, suggesting we are at least managing to avoid student discouragement.
As the average scores of more favorable motives (from faculty viewpoint) appear moderate-
to-high, they don’t indicate a major alarm. Postponement as a motivational strategy appears
a sound suggestion for business programs. Even if not significantly impacting student
motivation, it appears relevant as supporting student satisfaction. Ideally it could also boost
both extrinsic and intrinsic motivation. We still recommend business programs to increase
the number of elective courses as well as other ways of meaningful choices. This wouldn’t
be even mainly for vocational reasons (such as to better cater for the direct needs of labor
market), but for expecting the students to further practice self-reflection based decision-
making, which we see as a key foundation of modern entrepreneurship competence.
In conclusion, a balanced mix of student motivation appears most desirable. If intrinsic

motivation is lacking, the student might be treating the effort input as qualifier – just enough
to cross a certain threshold. Extrinsic motivation should not be treated as a substitute for
intrinsic, rather than complementary. One aim of the faculty would be to ensure that students
would continuously develop a stronger perception of relevance of each and every topic area.
Coupling that with practicing ongoing self-evaluation would be a way to create a machine
that further refines the elements of relevance towards the level of individual identification.
Finally, we propose that a driving force of this machine ought to be curiosity, the role of
which is sometimes underestimated in faculty teaching practice. A teacher should never give
up aiming to spark and fuel student curiosity with aims to develop this into a habit. This
might be perhaps even more important than any formulated learning outcome of a program.
629
References
Akosah-Twumasi, P., Emeto, T. I., Lindsay, D., Tsey, K., & Malau-Aduli, B. S. (2018). A
systematic review of factors that influence youths career choices—the role of culture.
Frontiers in Education, 3(58), 1–15.
Boström, L., & Bostedt, G. (2020). What about study motivation? Students´ and teachers’
perspectives on what affects study motivation. International Journal of Learning,
Teaching and Educational Research, 8(19), 40–59.
Duff, A., & McKinstry, S. (2007). Students’ approaches to learning. Issues in Accounting
Education, 2(22), 183-214.
Fischer, C., Malycha, C. P., & Schafmann, E. (2019). The influence of intrinsic motivation
and synergistic extrinsic motivators on creativity and innovation. Frontiers in psychology,
10, 1–15.
Frankowska, A., Głowacka-Toba, A., Rasińska, R., Prussak, E. (2015). Students entering the
labour market, their hopes, expectations and opportunities in the context of sustainable
economic development. Journal of International Studies, 3(8), 209–222.
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2009). Multivariate Data Analysis:
A Global Perspective. 7th ed. Upper Saddle River: Prentice Hall, 2009.
Iacovou, C. L., Shirland, L., & Thompson, R. L. (2011). Job Selection Preferences of
Business Students. Journal of Applied Business Research, 1(20), 87–98.
Kornijenko, A. (2022). Ärinduse bakalaureuseõppes peaeriala valik, üliõpilaste rahulolu ja
seonduvad karjääriootused [Choice of major in undergraduate business students, students
satisfaction and related career expectations on the example of TalTech business students].
Bachelor Thesis. Tallinn University of Technology.
Laran, J., & Janiszewski, C. (2011). Work or fun? How task construal and completion
influence regulatory behaviour. Journal of Consumer Research, 6(37), 967–983.
Loorits, J. (2022). Tudengite õppesuuna valikuid motiveerivate tegurite analüüs ärivaldkonna
näitel [Analysis of factors motivating students study field on the example of business
studies]. Bachelor thesis. Tallinn University of Technology.
Lucas, U., & Meyer, J. H. F. (2005). Towards a mapping of the student world: the
identification of variation in students’ conceptions of, and motivations to learn,
introductory accounting. British Accounting Review, 2(37), 177–204.
Polenova, E., Vedral, A., Brisson, L., & Zinn, L. (2018). Emerging between two worlds: A
longitudinal study of career identity of students from Asian American immigrant families.
Emerging Adulthood, 6(1), 53–65.
Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions
and new directions. Contemporary educational psychology, 25(1), 54–67.
Ryan, R. M., & Deci, E. L. (2017). Self-determination theory: Basic psychological needs in
motivation, development, and wellness. Guilford Publications.
Ryan, R. M., & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-
determination theory perspective: Definitions, theory, practices, and future directions.
Contemporary educational psychology, 61, 1–11.
Wasserman, T., & Wasserman, L. (2020). Motivation, Effort, and the Neural Network Model
(3rd ed). Springer.
630
Online Repository for Facilitating Teaching and Learning of

Undergraduate Statistical Modeling Tools
Qing Wang1, Xizhen Cai2
1
Department of Mathematics, Wellesley College, USA, 2Department of Mathematics and
Statistics, Williams College, USA.
Abstract
This paper presents an online repository created for facilitating teaching and
learning statistical modeling tools at the undergraduate level. Statistical
models and modeling techniques have always been considered the backbone
in data analysis and statistical learning. Over the past decade, teaching of such
topics has also gained an increasing attention in the undergraduate statistics
and data science curricula. The developed online repository aims at improving
the teaching and learning of statistical modeling tools in various
undergraduate statistical modeling courses. We present the four core
components of the repository, showcase some of its functionalities, and exhibit
available resources online. Through our informal assessments after
incorporating the online repository into our classrooms, our students seemed
to have a uniformly better understanding of the related concepts and methods,
which was reflected during the in-class discussions as well as in the subsequent
tests in the courses.
Keywords: Interactive web applications; linear regression; online repository;
statistical models.

Online Repository for Statistical Modeling Tools
1. Introduction
Statistical models play a vital role in data analysis and have been applied to an abundance of
practical problems (Cox, 1990). Teaching statistical modeling tools with an emphasis on
regression analysis has always been an essential part of the undergraduate statistics
education. In the American Statistical Association report on Curriculum Guidelines for
Undergraduate Programs in Statistical Science (Horton et al., 2014), statistical modeling,
including regression analysis, has been identified as one of the core topics in undergraduate
statistics curriculum. Moreover, in the current Guidelines for Assessment and Instruction in
Statistics Education (Carver et al., 2016), learning and using statistical models is listed as one
of the important learning goals, even for introductory statistics. Here the goal is not only to
ensure students’ mastery of the general knowledge of statistical models, but also to facilitate
the development of students' statistical maturity, which serves as a crucial building block for
them to acquire some more advanced topics in their statistics curriculum. Given the rapid
development and growing interest of data science over the past decade, regression analysis
has also been universally recognized and widely adopted as a central component in the data
science curriculum (De Veaux et al., 2017; Donoghue et al., 2020).
Even though students often get exposed to the topic of (simple) linear regression in a first
course in statistics, such as introductory statistics, taking some more advanced modeling
courses is necessary to fill them in the technical details, expose them to other modeling
techniques, introduce them to inferential procedures of different models, and improve their
computational skills through the use of certain statistical software or programming
languages. These more advanced courses in statistical modeling are especially vital for
students who are considering pursuing a graduate degree in statistics or a related field. Most
undergraduate programs in the US offer applied statistics modeling courses in addition to
introductory statistics. Although slightly different in content, such courses often require
students to have certain mathematical background, and their prerequisites usually include
calculus or linear algebra. The mathematical components of such courses further solidify
students' understanding of the fundamental concepts and properties of various statistical
models. However, at the same time, they may unfortunately place a barrier in the learning
process for students with a relatively weaker quantitative background, especially for under-
represented minorities, first-generation college students, students of color, and indigenous
students. Consequently, those students may feel discouraged from further pursing statistics
or a field in STEM in general.
When teaching abstract or challenging concepts, it is a common practice for statistics
educators to take advantage of visualization in order to break each concept into several pieces
of information for ease of understanding. It is also crucial to connect each concept with real-
world applications through concrete yet interesting examples. Some recent pedagogical
efforts on this front often involve active learning (Gelman and Nolan, 2017; Green et al.,
632
2018; Cai and Wang, 2020) and interactive web applications (Tintle et al., 2020). Although
the use of those applications saves students from undertaking the hard coding of simulations,
there are only a small number of such online applications available for teaching or learning
statistical modeling at the undergraduate level.
Another important learning goal we often emphasize in our classrooms is the introduction
and clarification of formal inferential procedures of statistical models. Since the true
population model is not observable, one of the critical steps in the modeling process is to
draw conclusions of the true model based on its sample estimate. This procedure is referred
to as statistical inference. Due to a lack of mathematical preparations for some students, one
of the challenges in teaching such concepts is to convey the fundamental idea behind
statistical inference and explain intuitively why a certain distribution is assumed in each of
the inferential procedures. To overcome this hurdle, one popular approach nowadays is to
adopt the so-called simulation-based inferential procedures (Tintle et al., 2018; Hildreth et
al., 2018). There are a number of statistics textbooks with an emphasise on simulation-based
methods, especially for introductory statistics (Tintle et al., 2020). However, there seems to
be insufficient resources available that are thoughtfully designed for more advanced
modeling topics in statistics.
Given all the challenges discussed above, we believe that there is a pressing need to develop
an online repository to engage students in active learning, help students build connections
and identify differences between various modeling tools, as well as to incorporate simulation-
based inference into the teaching of statistical modeling at the undergraduate level. In what
follows, we will detail the components of our designed online repository.
2. Components of the Proposed Repository
Figure 1: Overview of the four components in the repository.
Figure 1 presents the four core components of our designed online repository, namely,
Comprehension of Modeling Concepts, Modeling Process, Statistical Inference, and
Presentation and Communication. In the following, we will provide more details for each of
these four components.
633
2.1. Comprehension of Modeling Concepts

The objective of the first component, Comprehension of Modeling Concepts, is to facilitate
the introduction of statistical models, discuss important elements and assumptions of each
model, and demonstrate the differences and similarities between a statistical model that
describes the relationship of variables in the target population and the corresponding fitted
model estimated based on a sample of data drawn from the population.
Figure 2. Example of an interactive application for simple linear regression.
Interactive web-based applications can well meet this objective. Over the past several years,
we have been working on developing a number of interactive applications for our statistics
modeling courses, using the R Shiny package (R Core Team, 2022). For example, Figure 2
showcases one application we created when first introducing the simple linear regression
model in our classes. In this application, users can input values of the true parameters of the
model, and then generate a random sample of data based on the underlying true model. Given
the simulated data set, the estimated regression coefficients, 𝜷 "𝟎 and 𝜷 "𝟏 , obtained by the
Ordinary Least Squares Criterion are displayed on top of the scatterplot. Moreover, both the
true regression line and the estimated regression line are presented in the plot to clarify the
difference between the population relationship and its corresponding sample estimate.
A set of thoughtfully designed interactive applications are available in the first component of
our online repository, ranging from topics on multiple linear regression, analysis of variance
(ANOVA) models, logistic regression, receiver operating characteristic (ROC) curve, to
support vector classifiers. We plan to continue our efforts in this direction, and develop more
interactive tools that are useful for teaching and learning more challenging concepts in
statistical modeling.
2.2. Modeling Process

The second component of our repository focuses on providing more details of the modeling
process of a given model. It provides students with a road map of mastering different
634
modeling tools in a more systematic way and aims at enhancing students' understanding of
various statistical modeling techniques.
Figure 3. Diagram used to identify a proper statistical model.
The Modeling Process component includes an overview of various types of statistical models,
typically introduced at the undergraduate level. Students start with a diagram, as displayed
in Figure 3. By following the diagram, students are guided through the process of selecting
an appropriate model given a real data set and some research question of interest. In addition,
the diagram is interactive, i.e. each model is linked to a module, where students can further
explore and review some relevant concepts for the selected model.
In order to illustrate the order, connection, and cyclic nature between steps of the modeling
process, we designed another diagram to reflect the modeling steps in a sequential manner.
The diagram also offers a head-to-toe comparison between different models. With the help
of the hyperlink feature, when selecting one of the models the user will be directed to a new
web page that presents the comparison across various models. For example, Table 1 presents
an example that compares multiple linear regression model, two-way ANOVA model, and
multiple logistic regression model. By comparing these models side by side, it is easy for
students to recognize that linear regression and ANOVA models share similar model
assumptions, while linear regression and logistic regression have more in common in their
formulations, as both relate the (transformed) mean response with a linear combination of
predictors.
635
Table 1. Sample Comparisons of Different Models.
2.3. Statistical Inference

Over the past decade, simulation-based inference has gained much attention and become a
popular approach for introducing statistical inferential tools to undergraduate students.
However, most existing work and interactive applications were built for introductory
statistics. To bridge the gap, we construct the following modules in order under the Statistical
Inference section in our repository: Module 1. Sampling distribution and computing p-value
for a hypothesis test of the population mean based on a simulation; Module 2. Commonly
used distributions and their properties; Module 3. Sampling variation and inference for model
parameters and predictions; Module 4. Other simulation-based methods and their
applications to statistical models.
Figure 4. Sample Applications of Statistical Inference.
As an example, Figure 4 presents one of the developed interactive applications for Module
3. It visualizes the variability of the estimated coefficients in a simple linear regression model,
which also echoes back to the application displayed in Figure 2. By increasing the sample
size as well as the number of samples drawn from the true model, the histogram will display
636
a distributional shape that is approaching a bell-shaped curve. This observation naturally

guides students to the topic of hypothesis testing of model parameters.
2.4. Presentation and Communication

One of the important learning goals for a statistical modeling course is to improve students'
communication skill in delivering statistical results to a broad audience across different
disciplines. This skill is usually honed by having students work on a course project, give a
oral presentation, and/or complete a written project report. We have noticed from our
previous teaching experience that some students have limited prior experience in working on
data analysis projects, such as students who are not traditionally well represented in the
STEM fields. Hence, they often face more challenges during the completion of such class
assignments.
To overcome this issue, we offer useful resources in our repository on how to present
statistical results and write a professional project report. We include useful tips on best
practices and related literature. Moreover, with our former students' permission, we created
a library that showcases past students' work (e.g., papers or presentation videos).
3. Summary and Conclusion

In this paper, we discussed an online repository designed for teaching and learning statistical
modeling tools at the undergraduate level. In particular, we presented the four core
components of the repository and showcased some examples among its functionalities. We
anticipate that the four components in our repository will complement each other to engage
students in the classroom, facilitate their understanding, and make the learning of statistical
modeling more enjoyable and effective. Through our informal assessments when integrating
some of the applications in the repository to our own classes, we received uniformly positive
feedback from our students. Furthermore, by incorporating the online repository over the past
semesters, our students seemed to have a generally better understanding of the related
concepts and methods, which was reflected during the in-class discussions as well as in the
subsequent tests of the courses. We plan to continue our efforts in this direction, enriching
the available tools and web applications in the online repository. In addition, we hope to make
the respository publicly available in the near future so that it can benefit the broader statistics
and data science education community across the globe. As a future project, we plan to
conduct formal analysis on the feedback received from students and educators who have
experimented the developed online repository.
637
References
Cai, X. & Wang, Q. (2020). Educational tool and active-learning class activity for teaching
agglomerative hierarchical clustering. Journal of Statistics Education, 1–9.
Carver, R., Everson, M., Gabrosek, J., Horton, N., Lock, R., Mocko, M., Rossman, A.,
Roswell, G. H., Velleman, P., Witmer, J., & Wood, B. (2016). Guidelines for assessment
and instruction in statistics education (GAISE) college report.
Cox, D. (1990). Role of models in statistical analysis. Statistical Science, 5(2):169–174.
De Veaux, R., Agarwal, et al. (2017). Curriculum guidelines for undergraduate programs in
data science. Annual Review of Statistics and Its Application, 4:15–30.
Donoghue, T., Voytek, B., & Ellis, S. E. (2020). Teaching creative and practical data science
at scale. Journal of Statistics Education, 1–22.
Gelman, A. & Nolan, D. (2017). Teaching statistics: A bag of tricks. Oxford University Press.
Green, L. B., McCormick, N., McDaniel, S., Rowell, G. H., & Strayer, J. (2018).
Implementing active learning department wide: a course community for a culture change.
Journal of Statistics Education, 26(3):190–196.
Hildreth, L., Robison-Cox, J., & Schmidt, J. (2018). Comparing student success and
understanding in introductory statistics under consensus and simulation-based curricula.
Statistics Education Research Journal, 17(1).
Horton, N., Chance, B., Cohen, S., Grimshaw, S., Hardin, J., Hesterberg, T., Hoerl, R.,
Malone, C., Nichols, R., & Nolan, D. (2014). Curriculum guidelines for undergraduate
programs in statistical science. American Statistical Association.
R Core Team (2022). R: A Language and Environment for Statistical Computing. R
Foundation for Statistical Computing, Vienna, Austria.
Tintle, N., Chance, B. L., Cobb, G. W., Rossman, A., Roy, S., Swanson, T., &VanderStoep,
J. (2020). Introduction to statistical investigations. John Wiley & Sons.
Tintle, N., Clark, J., Fischer, K., Chance, B., Cobb, G., Roy, S., Swanson, T, & VanderStoep,
J. (2018). Assessing the association between precourse metrics of student preparation and
student performance in introductory statistics: Results from early data on simulation-
based inference vs. nonsimulation-based inference. Journal of Statistics Education,
26(2):103–109.
638
The MILAGE LEARN+ app on higher education

Mauro Figueiredo1, Custódia Fonseca2, Paula Ventura3, Marielba Zacarias3, José
Rodrigues1
1
Higher Institute of Engineering, University of Algarve, Portugal, 2Department of Chemistry
and Pharmacy, Faculty of Sciences and Technology, University of Algarve, Portugal,
3
Department of Electronic Engineering and Informatics, Faculty of Sciences and Technology,
University of Algarve, Portugal.
Abstract
This paper presents the use of the MILAGE LEARN+ app in Higher Education,
which takes advantage of gamification to motivate students and implements a
self and peer assessment scheme. There are tasks with three different levels
of difficulty to include all students. Teachers are responsible for producing
assignments which once done are uploaded to MILAGE LEARN+ app, thus
making them available to students. When students solve tasks they have
immediate feedback, with the solution available with criteria for assessment
and an educational video that explains the task solution, which can be
revisited as many times as needed for the student to learn. MILAGE LEARN+
is available for free and can be used for face to face, online, blended or flipped
learning. Its use in Higher Education shows that it promotes pedagogical
differentiation and increases the autonomy of students.
Keywords: Gamification; self and peer assessment; m-learning; pedagogical
differentiation; autonomy.

1. Introduction
The MILAGE LEARN+ app was developed by the University of Algarve, with the initial
objective to improve the learning of mathematics in elementary and secondary schools in
Portugal. This project was also implemented in Spain, Germany, Norway, Turkey and
Cyprus. After this period this tool start to be used in other disciplines such as, biology,
languages, etc. (Figueiredo, Bidarra, González-Pérez, & Godejord, 2017; Figueiredo,
Godejord, & Rodrigues, 2016).
This app is available for free on Windows and OSX computers as well as iOS and Android
devices which to take advantage of a wide availability of mobile devices that students carry
with them.
Mobile devices give students flexibility and individualization of their learning experiences,
as well as allow them to expand the time spent learning outside the classroom. This offers a
great opportunity for teachers and students to take advantage of technologies to facilitate
learning. It is important to note that these students are the generation of digital games and
social networks. They are no longer the same for whom the education system was designed
a few decades ago. A generation whose expectations are different from those that preceded
it.
The MILAGE LEARN+ app uses features of games to motivate students and implements an
approach with a self and peer assessment that promotes pedagogical differentiation and
increases the autonomy of students.
This paper presents several examples about the use of the MILAGE LEARN+ app for
learning on Higher Education.
2. The MILAGE LEARN + app

MILAGE LEARN+ was conceived as a game where the players are students and the main
goal is to solve tasks uploaded by the teacher (Figure 1). These tasks are grouped in
subchapters and chapters and this organization makes MILAGE LEARN+ structurally
similar to a task book.
After login into the app, the game start, with the following steps:
1. Student select the “game” (task).
2. Find the game solution (solve the task).
3. Submit the solution.
4. Self-assessment of the task solution submitted based on the criteria for assessment.
640
Mauro Figueiredo, Custódia Fonseca, Paula Ventura, Marielba Zacarias, José Rodrigues
5. Watch the educational video explaining the task solution if needed for better understanding
of the task.
6. Peer-assessment of a colleague's solution.
Figure 1. App MILAGE LEARN+ functioning/usage diagram.
In case of a multiple choice task, after the student selects his option, the app automatically
identifies it as correct or wrong
In case of an open question task, the student need to solve the task using a pen and paper and
after with a mobile device takes a photo and upload it to the server. In this case, the MILAGE
LEARN+ app combines the analog and technology. It is also possible to submit the answer
using the keyboard.
After submitting the answer, the student gets the access to the instructions for self-
assessment. In this way, the student gets immediate feedback and understands if what was
done was correctly or not and feeds forward to know what to do to improve. At the same time
the student can see small video with the solution.
Thus, the student can study solving exercises or review the course content while evaluating
his work or the work done by a colleague.
Inclusion of self and peer assessment contributes to promoting formative learning, learners’
independence and taking responsibility for the learning process.
3. Gamification Learning Scenario

Gamification is defined as the incorporation of game mechanics in other situations, with the
aim of increasing motivation in a given product or service, facilitating behaviors that promote
learning (Hsu, Chang, & Lee, 2013).
In the game it is possible for each player to choose their own path and choose different levels
of difficulty according to their abilities, which ensures fun and new learning in case of
corrective feedback (Chen, 2007).
641
The use of educational games allows the student to put in practice contents and ideas that are
topics of a certain area of knowledge. Chairs(Winter, Wentzel, & Ahluwalia, 2016),
Chirality-2(Jones, Spichkova, & Spencer, 2018) and Nomenclature Bets(Silva Júnior et al.,
2018) are examples of applications for teaching chemistry based on gamification.
The MILAGE LEARN+ implements a gamification scheme, which is being used in
teaching/learning “Organic Chemistry'' course, where gamification is a used strand. This
subject, organic chemistry, has been classified by the students as difficult to understand,
which leads to a high failure rate in its curricular units.
The objective of this work is using gamification elements from App MILAGE LEARN+, to
motivate the learning of this area of knowledge, promote autonomous work and facilitate
fixation concepts through self and peer assessment.
The practice involving App MILAGE LEARN+ was carrying out organic chemistry tasks to
do outside of the class and after there is a discussion in the class. These reinforce learning
the concepts. The topics involved were, mechanisms of reaction, stereochemistry and
structure of compounds. MILAGE LEARN+ works like a task book and is structured in
chapters and subchapters with exercise sheets. This approach follows the strategy described
above. The teacher uploads the tasks including the solutions with criteria for assessment and
the student solves the tasks using the app MILAGE LEARN+ app (Figure 2).
Figure 2. Students use the App MILAGE LEARN+ to solve tasks.
642
4. Class Discussion Scenario

MILAGE LEARN+ allows the application of active learning strategies, which aim to focus
on the student in the creation of new knowledge, with the teacher as a facilitator of this
process. The pedagogical practice of the “System Analysis and Design” (SAD) course aims
to promote new ways of reasoning to students in order to facilitate the application of acquired
knowledge, and aims to:
• Active learning in the classroom (through MILAGE) and outside the classroom
(through a problem-based learning scenario)
• Plurality of points of view.
• The use of devices with network access to replace notebooks and the blackboard,
incorporating technologies into everyday classroom life.
The SAD course encompasses theoretical and practical classes. Theoretical classes last one
hour, taught twice a week. The content is provided in two 15-minute segments, alternating
with tasks on the topics taught, using the MILAGE LEARN+ app. In this application, the
statement of the task is presented and the students submit their solution. After submission,
the student is able to view the video explaining the solution and self-assess its solution using
the criteria instructions for self-assessment provided by the teacher. Upon completing the
self-assessment, the student can proceed with the assessment of a peer’s solution (randomly
assigned by the application) using the same instructions for assessment. Students accumulate
points by self and peer assessment taking advantage of the gamification to motivate students
and promoting autonomy with the self and peer scheme implemented in the MILAGE
LEARN+ app.
In practical classes, tasks are carried out to reinforce the knowledge acquired in theoretical
classes and in preparation for class discussion learning scenarios. In this context, the
application is used in a different way, with the aim of discussing the various submitted
solutions in the classroom. The teacher does not upload videos with detailed solutions to the
MILAGE LEARN+ app, but criteria and good practices to apply in the solution. Figure 3
shows the tasks performed during this process:
1. Students submit solutions in groups using the MILAGE LEARN+ app.
2. Teacher shows all the student solutions.
3. Student groups discuss solutions.
4. At the end the teacher comments.
643
Figure 3. Class discussion learning process using the MILAGE LEARN+ app.
5. Evaluation and results

The practice of using the MILAGE LEARN app was measured through an online
questionnaire with students attending the “Organic Chemistry”, consisting of the following
questions:
1. Are points results important?
2. Is the existence of leaderboard score motivating?
3. Doing the tasks as part of the game makes learning more fun?
4. Carrying out the tasks helps to understand the theory?
5. Carrying out the tasks motivates students to study the theory?
6. Self-assessment of the tasks helps to understand the solution?
7. Evaluating colleagues' tasks helps to understand the solution?
These questions aimed to assess the influence of the various gamification elements
introduced in pedagogical practice through MILAGE i.e. the importance of points and the
ranking of students as well as performing and correcting exercises as part of a game. The
students had, as answer options, a 4-point Likert ordinal scale: (1) Totally Disagree, (2)
Partially Disagree, (3) Partially Agree, (4) Totally Agree. Results are illustrated in figures 4.a
and 4.b. The former shows the results of questions 1-3, and figure 4.b shows the results for
questions 4-7.
The analysis of the 48 answers to the questionnaire shows how the majority of the students
totally or partially agreed with all the questions asked. In this sense, it is possible to conclude
that the gamification elements significantly contributed to the motivation of the students and
to the understanding of both the theoretical contents and the practical tasks.
644
Figure 4a. Results of questions 1-3 of the gamification evaluation questionnaire with the MILAGE LEARN+ App.
The Y axis represents the number of responses obtained from a maximum of 48 respondents.
Figure 4b. Results of questions 4-7 of the gamification evaluation questionnaire with the MILAGE LEARN+ App.
The Y axis represents the number of responses obtained from a maximum of 48 respondents.
However, it was observed that the degree of agreement was variable, with question (6) "Self-
assessment of the tasks helps to understand the solution" the one that obtained the highest
agreement (97.62%) and question (2 ”Self-assessment as an aid to understanding the solution
with the lowest agreement (56.25%). It should be noted that in terms of the different game
activities, there was greater agreement on the value of carrying out the tasks than on the value
of the evaluation activities.
6. Conclusions
This paper shows the use of the MILAGE LEARN+ app to implement gamification and class
discussion scenarios on Higher Education explored for the teaching of “Organic Chemistry”
and “System Analysis and Design” courses.
645
The main element for analyzing the results is the appreciation of the MILAGE LEARN+
application by the students, which, being positive, motivated the teachers to continue with
this teaching approach by applying the MILAGE LEARN+ app.
After analyzing the questionnaires, it can be seen that the appreciation on the part of the
students was, in general, quite positive. From the perspective of teachers, active learning
strategies were applied extensively. Gamification motivated students for the learning.
Carrying out activities centered on students allowed promoting the autonomous exploration
of solutions and the ability to explain them. Strategies such as self-assessment and peer
assessment promoted the development of their critical thinking capabilities and autonomy.
References
Chen, J. (2007). Flow in games (and everything else). Communications of the ACM, 50(4),
31–34. doi:10.1145/1232743.1232769
Figueiredo, M., Bidarra, J., González-Pérez, A., & Godejord, B. J. (2017). Promoting
autonomous work of students with the Milage Learn+ app (pp. 7660–7667).
doi:10.21125/inted.2017.1778
Figueiredo, M., Godejord, B., & Rodrigues, J. (2016). The development of an interactive
mathematics app for mobile learning. Retrieved from http://hdl.handle.net/10400.1/9141
Hsu, S. H., Chang, J.-W., & Lee, C.-C. (2013). Designing Attractive Gamification Features
for Collaborative Storytelling Websites. Cyberpsychology, Behavior, and Social
Networking, 16(6), 428–435. doi:10.1089/cyber.2012.0492
Jones, O. A. H., Spichkova, M., & Spencer, M. J. S. (2018). Chirality-2: Development of a
Multilevel Mobile Gaming App To Support the Teaching of Introductory Undergraduate-
Level Organic Chemistry. Journal of Chemical Education, 95(7), 1216–1220.
doi:10.1021/acs.jchemed.7b00856
Silva Júnior, J. N. da, Sousa Lima, M. A., Nunes Miranda, F., Melo Leite Junior, A. J.,
Alexandre, F. S. O., de Oliveira Assis, D. C., & Nobre, D. J. (2018). Nomenclature Bets:
An Innovative Computer-Based Game To Aid Students in the Study of Nomenclature of
Organic Compounds. Journal of Chemical Education, 95(11), 2055–2058.
doi:10.1021/acs.jchemed.8b00298
Winter, J., Wentzel, M., & Ahluwalia, S. (2016). Chairs!: A Mobile Game for Organic
Chemistry Students To Learn the Ring Flip of Cyclohexane. Journal of Chemical
Education, 93(9), 1657–1659. doi:10.1021/acs.jchemed.5b00872
646
Leveraging AI to instruct architecture students on circular design

techniques and life cycle assessment
Toktam B. Tabrizi1, Ozgur Gocer1, Arash Sadrieh2, Anastasia Globa1
1
School of Architecture, Design and Planing, The University of Sydney, Australia, 2AI
Engineering at Ninjatech.ai (Conversational AI), Australia.
Abstract
The aim of this study is to examine the use of AI as a tool for educating
architecture students in circular design principles and life cycle assessment
(LCA). A theoretical research approach is proposed to identify current
challenges and solutions, and to provide insightful predictions and
explanations from various perspectives. The paper emphasises the importance
of sustainability in architecture education and assesses the difficulties of
teaching circular design and LCA, and how AI can simplify the process. The
role of AI in promoting cradle-to-cradle thinking in the design stage is
demonstrated through the use of a standard LCA framework to make the
assessment process more accessible to students. Key factors such as the
building's climatic zone, location, type, service life, energy consumption and
CO2 emissions are considered part of the system boundary. The study
concludes with limitations, recommendations, and guidelines for architecture
students.
Keywords: Artificial intelligence (AI); circular design techniques; life cycle
assessment (LCA); sustainability education; cradle-to-cradle thinking;
environmental impact factors.

Leveraging AI to instruct architecture students on circular design techniques
1. Introduction
In recent years, the architecture profession has been facing the challenge of creating buildings
that are not only aesthetically pleasing but also environmentally responsible. Circular design
techniques and life cycle assessment (LCA) are two key approaches that aim to promote
sustainable design practices, by reducing waste and maximising resource efficiency
throughout the lifecycle of a building (Eberhardt, Birkved, & Birgisdottir, 2022; Trusty &
Horst, 2002). However, despite their importance, these concepts are often not taught in depth
in traditional architecture education, leaving students with limited exposure to their practical
applications (Gomes, da Silva, & Kowaltowski, 2022).
Artificial intelligence (AI) has the potential to bridge this gap, by providing students with
interactive, data-driven tools that help them to understand the principles and benefits of
circular design and life cycle thinking. AI-based educational tools can simulate the impacts
of different design choices on the environment and help students to evaluate their designs
from a life cycle perspective, in real-time. This type of feedback can be instrumental in
promoting the adoption of circular design techniques and life cycle thinking in architecture
education, by providing students with a deeper understanding of the trade-offs and
opportunities associated with different design choices (Ewa; Gilner, Adam; Galuszka, &
Tomasz Grychowski, 2019).
The aim of this study is to address the following questions:
• Why is the inclusion of LCA and circular design strategies in architecture education
important and what makes it challenging?
• How can AI be utilized to streamline the implementation of LCA and circular design
techniques in architecture education?
Then, the possibility of the development of a method that allows architecture students to
easily apply the logic of LCA and circular design strategies at the early design stage is
discussed.
2. Why is the inclusion of Life Cycle Assessment and Circular design strategies
in architecture education important and what makes it challenging?
The inclusion of LCA in architecture education is important because it equips future
architects with the tools and knowledge necessary to create sustainable and environmentally
conscious buildings. LCA is a method for evaluating the environmental impact of a product,
building, or system over its entire life cycle, from raw material extraction to disposal. This
allows architects to consider the environmental impact of their designs and make informed
decisions that minimize harm to the environment (Fnais et al., 2022). Circular design
strategies, on the other hand, promote the idea of closed-loop systems, where waste is reduced
648
Toktam B. Tabrizi, Ozgur Gocer, Arash Sadrieh, Anastasia Globa
and resources are conserved. These strategies encourage architects to design buildings that
can be easily disassembled, recycled, and repurposed, reducing the amount of waste
generated and contributing to a more sustainable future (Attia & Al-Obaidy, 2021).
However, incorporating these principles into architecture education can be challenging for
several reasons. Firstly, the complexity of building LCA, as it involves multiple interrelated
parameters, makes it difficult for students to base decisions on during their design process.
The difficulty of LCA application is related to various areas. Performing LCA requires a
significant amount of data and expertise, including well-defined building plans, inventory
assessment of materials, and knowledge of production and demolition processes. The
environmental impact assessment also faces difficulties such as non-standardized material
production and limited access to information about environmental impacts of the production
and manufacturing of construction materials plus the actual process of construction and
demolition (Ramesh, Prakash, & Shukla, 2012). Furthermore, embodied energy and
equivalent carbon emissions can vary due to the energy mix, transformation processes,
efficiency of the industrial and economic system of the country, and the variability of these
factors over time (Sartori & Hestnes, 2007), making it challenging to make informed
decisions. Previous studies on life cycle energy requirements have also shown that a
material's performance can vary in different situations. It is crucial to make decisions from a
life cycle perspective that does not worsen the overall environmental impact, even if they
reduce it for one stage. Previous studies on life cycle energy requirements (Crawford, 2011;
Utama & Gheewala, 2009) have shown that the performance of a material or assembly can
vary based on the context in which it is used. For example, a material with low initial
embodied energy may not necessarily have low life cycle energy. The lack of generalizable
design principles and the use of different assumptions, databases, and analysis methods in
LCA studies further complicate the application of LCA in architecture education.
LCA and circular design require a multidisciplinary approach, involving expertise from fields
such as engineering, materials science, and environmental science. This can make it difficult
for architecture schools to provide the necessary training and resources to students. Despite
these challenges, the benefits of incorporating LCA and circular design into architecture
education far outweigh the difficulties, as they equip future architects with the knowledge
and skills necessary to create a more sustainable future.
3. How can AI be utilized to streamline the implementation of Life Cycle

Assessment and Circular design techniques in architecture education?
Large Language Models (LLMs) are powerful machine-learning models that have been
trained on vast amounts of text data, allowing them to understand and generate human-like
language (Devlin, Chang, Lee, & Toutanova, 2018). They have many applications, including
649
data retrieval and educational use cases. In data retrieval, LLMs can be used to extract
information from large databases, such as medical records or legal documents. They can also
assist in chatbots, customer service, and search engines by understanding natural language
queries and providing relevant responses. In education, LLMs can facilitate personalized
learning experiences by generating customized quizzes, exercises, and feedback based on
student responses (Brown et al., 2020). However, despite the significant potential of these AI
models there is no application report in the field of architecture education. AI has the
potential to revolutionize the way LCA and circular design techniques are implemented in
architecture education. These techniques play a critical role in ensuring that buildings are
designed and constructed in a sustainable and environmentally responsible manner.
However, their implementation can be complex, time-consuming, and challenging for
students. In this article, several ways that AI can be utilized to streamline the implementation
of LCA and circular design techniques in architecture education are discussed (Gomes et al.,
2022; Ji, Lee, & Yi, 2021).
Automated Data Collection: LCA requires a significant amount of data collection and
analysis, including information on embodied energy, carbon emissions, and other
environmental impacts of building materials, products, and systems. AI can automate this
process by collecting data from various sources and processing it in a way that is fast and
accurate. This can reduce the time and effort required for manual data collection, allowing
students to focus on the analysis and interpretation of the results.
Predictive Modelling: AI algorithms can be trained to predict the environmental impact of
different building materials, systems, and assemblies. This can help students to make
informed decisions about the materials and systems they use in their designs, based on the
predicted environmental impact. Predictive modelling can also be used to optimize the design
based on specific sustainability and environmental criteria, allowing students to find the most
sustainable and circular design solutions.
Optimization: Optimization is a critical component of circular design and LCA. AI can be
used to optimize the design based on specific sustainability and environmental criteria, such
as embodied energy, carbon emissions, and waste reduction. AI can be used to optimize
product designs for circularity, by identifying opportunities for materials reuse, repair, and
recycling This can help students to find the most sustainable and circular design solutions
and understand the trade-offs involved in different design decisions.
Visualization: The results of LCA and circular design analysis can be complex and difficult
to interpret for students. AI can be used to visualize and communicate these results in a way
that is accessible and intuitive, improving students' understanding of the concepts. This can
include visualizations of the environmental impact of different building materials, systems,
and assemblies, as well as visualizations of the circular design process and its outcomes
650
(Abioye et al., 2021; E. Gilner, A. Galuszka, & T. Grychowski, 2019; Ji et al., 2021; Oluleye,
Chan, & Antwi-Afari, 2023; Xue et al., 2021).
In conclusion, AI has the potential to revolutionize the way LCA and circular design
techniques are implemented in architecture education. By automating data collection,
enabling predictive modelling, optimizing designs, and improving visualization, AI can make
these techniques more efficient, effective, and accessible for students. As a result,
architecture students will be better equipped to design and construct buildings that are
sustainable and environmentally responsible.
4. Methodology: Leveraging AI to instruct architecture students on circular

design techniques and LCA
The methodology of this study leverages a theoretical model to examine the potential of AI
in educating architecture students on circular design techniques and LCA. The research
process involves analysing the current challenges and solutions in this field, exploring
various perspectives, and providing explanations, predictions, and proposed solutions.
To achieve this, the study aims to:
• Minimize data requirements and identify the most effective parameters with the greatest
potential for impact
• Define a system boundary that is specific to each building, limiting the criteria for LCA
• Offer recommendations and rules of thumb for environmentally friendly design
decisions based on the results of previous LCA studies.
In this study, a Knowledge-based System (KBS) was utilized to introduce a method for
application of simplified LCA by architecture students. KBS is a computer system that
leverages various forms of AI for problem-solving across different domains. It consists of a
database of expert knowledge that is tailored for specific queries, along with learning and
justifications capabilities. The system provides data and information from multiple sources
and is designed to support human decision-making through the use of knowledge-based
techniques. To simplify usage of the KBS system for students we propose to integrate a Large
Language Model (LLM) such as chat-gpt3-turbo (Ouyang et al., 2022) with KBS. The LLM
model can assist in processing student queries and questions in natural language, which can
be challenging for traditional KBSs. By leveraging the LLM's ability to understand context
and generate relevant responses, the KBS can provide more accurate and tailored results to
student queries which have always proven to be very difficult and time consuming for the
students to grasp. several implementations for such a system have been proposed (Izacard &
Grave, 2020; Izacard et al., 2022; Khattab et al., 2022). In this study we will adapt the method
proposed by (Yao et al., 2022).
651
The tool is structured into three main parts: inputs, system functions, and outputs. The inputs
section requires essential information about the building to be inputted, such as climate,
location, building type, and building lifetime, to investigate the most effective parameters on
the building's life cycle environmental performance (Tabrizi & Brambilla, 2019). System
functions are made up of a basic tool that utilizes KBS, which contains rules specific to the
domain. This enables the system to generate outputs based on the user's inputs, helping
students make more environmentally conscious decisions at the early stages of the design
process with minimal time, effort, and cost.
Outputs from the system provide recommendations for design and material selection
decisions for the building's envelope structure, which are expected to minimise the building's
life cycle environmental impact. These recommendations are generated in different groups
like one for window design and the other for solid envelope design, based on the specific
conditions of each building that the user inputs into the system (Figure 1).
In conclusion, the KBS-designed LCA method serves as a valuable resource for architecture
students, as it simplifies the process of understanding the complexities of circular design
techniques and LCA. The tool's outputs provide a foundation for thinking through problems
and understanding the life cycle of a building, from cradle to cradle.
Advice to Designer generated by the tool
· Optimal WWR is 30% to 50% (Avoid overuse of glazing)

Designer Inputs · Use DGU/ Clear glass
· Use high SHGC For northerly glazing
· Use low SHGC for east and west-facing windows
· Use low U-Value
· U-value and SHGC values are according to
AFRC requirements
· Use openable area for natural ventilation
· Use Aluminium skinned timber
Window Related · Use passive solar shading on north
Design · Use adjustable shading
· Use deep overhang shade for all east and west-facing
Considerations
glass in summer
· Use Concrete tile for roof

Solid Envelope · Use Concrete slab on ground
Related Design · Use light coloured roof materials
· Use lightweight materials for cladding
Considerations
· Use low energy and carbon intensive materials
· Use materials with minimum wastage rates
· Use local materials
· Use bulk and reflective insulation in ceilings
· Use bulk or reflective insulation for walls
· Minimum R-value of 1.5m2 K/W is for the walls,
2.5m2 K/W is for the ceiling (BASIX requirement)
· Optimised R-value for walls is 5.2 m2 K/W
· Optimised R-value for roof and ceiling is 6.5 m2 K/W
· Use under concrete slabs insulation
(if using in-slab heating)
· Use insulate elevated floors (concrete and lightweight)
· Use external insulation to all thermal mass.
· Use high mass construction on lower levels
(to stabilise temperatures)
· Use low mass on the upper levels
· Thermal mass must be protected from summer sun
& exposed to cooling night breezes
· Use timber frame for external walls and roof
· Use durable and low maintenance materials
· Use recycle contents materials (e.g. Reclaimed timber)
Figure 1. Inputs and outputs sample for a hypothetical case study in Sydney. Source: (Tabrizi & Brambilla,
2019).
652
5. Concusions and future research

The research aims to shed light on how AI can facilitate the integration of cradle-to-cradle
thinking into the design process of architecture students. The study aims to enhance the
instruction of LCA to architecture students by combining the standard LCA framework with
the power of AI. The research highlights the importance of considering sustainability in
architecture education and the challenges faced in teaching these concepts to students. The
use of AI in this context can simplify the process of sustainability education and provide a
quick early analysis of the building's life cycle environmental performance. The method aims
to overcome the difficulties faced by architecture students in understanding and applying
LCA, and provides a means for them to perform assessments and make informed decisions
on their designs. The output of this method helps students understand the environmental
impact of their design decisions and supports them in optimizing environmental performance
through multiple dimensions while considering the specific conditions of each building. The
immediate next step that this research team is going to take is to apply this theoretical model
on a real case study to investigate the applicability of this proposed model.
Future research will progress to the user studies and implementation of the proposed method
to the teaching and learning process. This stage of the project will systematically evaluate the
benefits and limitations when applied in the context of higher architectural education for
architecture. A series of 1–2-day workshops will be run at the University of Sydney – School
of Architecture, Design and Planning targeting architecture students as a test population.
A follow up offshoot of this study will aim to evaluate the benefits and implementation
methods of educational AI for conceptual design in architecture. This AI application will
focus on qualitative aspects of design thinking to accompany quantitative assessments of the
proposed AI for LCA . This holistic approach of using AI for both quantitative and qualitative
aspects of design solutions will allow to create a more informed feedback loop, allowing
learners to assess multiple parameters at the same time.
References
Abioye, S. O., Oyedele, L. O., Akanbi, L., Ajayi, A., Davila Delgado, J. M., Bilal, M., . . .
Ahmed, A. (2021). Artificial intelligence in the construction industry: A review of present
status, opportunities and future challenges. Journal of Building Engineering, 44, 103299.
doi:https://doi.org/10.1016/j.jobe.2021.103299
Attia, S., & Al-Obaidy, M. (2021). Design criteria for circular buildings.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., . . . Askell, A.
(2020). Language models are few-shot learners. Advances in neural information
processing systems, 33, 1877-1901.
Crawford, R. H. (2011). Life Cycle Assessment in the Built Environment. USA and Canada:
Spon Press.
653
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Eberhardt, L. C. M., Birkved, M., & Birgisdottir, H. (2022). Building design and construction
strategies for a circular economy. Architectural Engineering and Design Management,
18(2), 93-113. doi:10.1080/17452007.2020.1781588
Fnais, A., Rezgui, Y., Petri, I., Beach, T., Yeung, J., Ghoroghi, A., & Kubicki, S. (2022). The
application of life cycle assessment in buildings: challenges, and directions for future
research. The International Journal of Life Cycle Assessment, 27(5), 627-654.
doi:10.1007/s11367-022-02058-5
Gilner, E., Galuszka, A., & Grychowski, T. (2019, 26-29 Aug. 2019). Application of Artificial
Intelligence in Sustainable Building Design - Optimisation Methods. Paper presented at
the 2019 24th International Conference on Methods and Models in Automation and
Robotics (MMAR).
Gilner, E., Galuszka, A., & Grychowski, T. (2019). Application of Artificial Intelligence
Methods in Sustainable Building Design. In (pp. 408-417). Cham: Springer International
Publishing.
Gomes, V., da Silva, M. G., & Kowaltowski, D. C. C. K. (2022). Long-Term Experience of
Teaching Life Cycle Assessment and Circular Design to Future Architects: A Learning
by Doing Approach in a Design Studio Setting. Sustainability, 14(12), 7355. Retrieved
from https://www.mdpi.com/2071-1050/14/12/7355
Izacard, G., & Grave, E. (2020). Leveraging passage retrieval with generative models for
open domain question answering. arXiv preprint arXiv:2007.01282.
Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., . . . Grave, E. (2022).
Few-shot learning with retrieval augmented language models. arXiv preprint
arXiv:2208.03299.
Ji, S., Lee, B., & Yi, M. Y. (2021). Building life-span prediction for life cycle assessment
and life cycle cost using machine learning: A big data approach. Building and
Environment, 205, 108267. doi:https://doi.org/10.1016/j.buildenv.2021.108267
Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., & Zaharia, M. (2022).
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-
intensive NLP. arXiv preprint arXiv:2212.14024.
Oluleye, B. I., Chan, D. W. M., & Antwi-Afari, P. (2023). Adopting Artificial Intelligence
for enhancing the implementation of systemic circularity in the construction industry: A
critical review. Sustainable Production and Consumption, 35, 509-524.
doi:https://doi.org/10.1016/j.spc.2022.12.002
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., . . . Ray, A. (2022).
Training language models to follow instructions with human feedback. Advances in
neural information processing systems, 35, 27730-27744.
Ramesh, T., Prakash, R., & Shukla, K. K. (2012). Life cycle approach in evaluating energy
performance of residential buildings in Indian context. Energy and Buildings, 54, 259-
265. doi:10.1016/j.enbuild.2012.07.016
654
Sartori, I., & Hestnes, A. G. (2007). Energy use in the life cycle of conventional and low-
energy buildings: A review article. Energy and Buildings, 39(3), 249-257.
doi:10.1016/j.enbuild.2006.07.001
Tabrizi, T. B., & Brambilla, A. (2019). Toward LCA-lite: A Simplified Tool to Easily Apply
LCA Logic at the Early Design Stage of Building in Australia. European Journal of
Sustainable Development, 8(5), 383. doi:10.14207/ejsd.2019.v8n5p383
Trusty, W., & Horst, S. (2002). Integrating LCA tools in green building rating systems. Paper
presented at the USGBC Greenbuilding International Conference and Expo.
Utama, A., & Gheewala, S. H. (2009). Indonesian residential high rise buildings: A life cycle
energy assessment. Energy and Buildings, 41(11), 1263-1268.
doi:http://dx.doi.org/10.1016/j.enbuild.2009.07.025
Xue, K., Hossain, M. U., Liu, M., Ma, M., Zhang, Y., Hu, M., . . . Cao, G. (2021). BIM
Integrated LCA for Promoting Circular Economy towards Sustainable Construction: An
Analytical Review. Sustainability, 13(3), 1310. Retrieved from
https://www.mdpi.com/2071-1050/13/3/1310
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). React:
Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
655
When a test-taking strategy is better? An approach from the

paradigm of scheduling under explorable uncertainty
Cristóbal Alfredo Mauricio, Sebastián Davila-Gálvez, Óscar Carlos Vásquez
University of Santiago of Chile (USACH), Faculty of Engineering, Program for the
Development of Sustainable Production Systems (PDSPS), Chile, University of Santiago of
Chile (USACH), Faculty of Engineering, Industrial Engineering Department, Chile.
Abstract
In this article, we adopt the paradigm of scheduling under explorable
uncertainty to explore test-taking strategies to solve standardized tests in terms
of maximizing the correct questions answered. From this approach, a test taker
considers a number of questions and has the possibility to read in order to
obtain the difficulty of the question. Later, he/she has the option, for example,
to answer the question or to skip the one that seemed difficult and read the next
question in the test. Specifically, we state the problem definition by considering
two test-taking strategies, formulate and implement a mathematical model, and
generate computational experiments in order to determine the dominance of
one strategy over another. The results show that the dominance depends
directly on the design of the test and the maximum time to perform it, so
knowing these parameters allow us to provide algorithmic insights to address
this problem.
Keywords: Test-taking strategies;scheduling; standarized test; optimization.

When a test-taking strategy is better?
1. Introduction
In the education setting, there are different types of standardized tests. These consider a wide
range of knowledge from the foreign language proficiency tests (e.g., Test of English as a
Foreign Language (TOEFL), Test of English for International Communication (TOEIC),
among others) to the mandatory admission test such as the Graduate Management Admission
Test (GMAT) in the United States of America or the Prueba de Acceso a la Educación
Superior (PAES) in Chile (DEMRE, 2023), whose results define the position in the admission
ranking of the university degree to be studied. In general, the students prepare to take these
standardized tests over many months, attending some specialized courses where several test-
taking strategy recommendations are provided to achieve the best possible result. In
particular, a test-taking strategy from a scheduling point of view distinguishes a series of
decisions to be made by the test-taker in sequence when faced with the questions in an
assessment. For instance, the test-taker would read and answer all the questions in order of
appearance or only answer the questions if he/she is sure of the correct answer. Considering
that assessments typically have a limited time frame for completion, the choice of test-taking
strategy can significantly impact the test results. Rapidly skipping challenging questions may
allow test-takers to attempt more questions and answer the easier ones first. Still, it may also
require them to revisit the skipped questions later, thus increasing the overall test completion
time. Therefore, there exists a threshold where it may be more advantageous to respond
immediately, regardless of the difficulty of the questions, rather than postponing difficult
questions and answering easier ones first.
In literature, several studies have been carried out in different areas. In psychology, most of
the strategies have been studied before taking the test in order to reduce the anxiety of the
students (Theobald, Breitwieser, & Brod, 2022), and it has been found that the application of
these, in conjunction with the application of other strategies has a direct correlation with
student achievement in college (Viet, 2022). From a practical perspective, classic books such
as "Test-Taking Strategies" (Kesselman-Turkel & Peterson, 1981) expose in a practical way
several strategies for different types of tests, such as strategies for multiple-choice tests, true-
false tests, matching tests, vocabulary tests, number tests, etc. However, the choice of these
strategies is vaguely explained. Finally, from an algorithmic point of view, the decision of
which question to answer first can be studied as a problem of allocating limited resources to
tasks over time such as (Wang & Zhang, 2006), which presents an iterative mathematical
model applied to a general case as well as a theoretical framework using nonlinear
optimization, considering a probabilistic perspective previously known by the test-taker.
In this paper, we adopt the paradigm of scheduling under explorable uncertainty (Retsef,
Magnanti , & Shaposhnik, 2018) (Dürr, Erlebach, Megow, & Meißner, 2020) (Dufossé, Dürr,
Nadal, Trystram, & Vásquez, 2022) to explore test strategies to solve standardized tests and
to analyze under what circumstances the application of these strategies will generate an
658
optimal result. From this approach, a test-taker considers a number of questions (set of
parameters) and has the possibility to read (to make a query) in order to obtain the difficulty
of the question. Later, he/she has the option, for instance, to answer the question or to skip
the one that seemed difficult and read the next question in the test. Clearly, a compromise
has to be found between the number of questions to be skipped and the test result obtained.
Formally, we state the problem definition by considering two test strategies, formulate and
implement a mathematical optimization model and generate computational experiments in
order to determine dominance properties of the strategies to obtain the best test result.
2. Problem definition and formulations
2.1. Problem description

Consider a test with N questions, of which M£ N are easy questions, and N-M are hard
questions. P is defined as the set of questions {1, … , 𝑁} where each question 𝑖 ∈ 𝑃 has a
reading time 𝑟! and an answer time 𝑝! . Each question appears in the test follows a particular
sequence S, e.g., 𝑆 ≔ {h, h, e, h, e, e, … , h}, where h denote the hard questions and e the easy
questions. These questions are differentiated by their resolution time, where the time to
answer a hard question 𝑝" will be longer than the time to answer an easy question 𝑝# .
Additionally, there is a deadline 𝑑 to finish the test. We assume that the test-taker will know
the number of total questions (N) and the easy questions (M) in the test.
We consider two test strategies for answering the test, which are described below:
• Read and answer (RA) strategy: This test-taking strategy consists of reading and
answering the question immediately, regardless of the difficulty of the question.
• Read, skip, and answer (RSA) strategy: This test-taking strategy consists of reading
in an exploratory way and analyzing the difficulty of the question. If the question is
easy, it is answered immediately, but if it is hard, then it is skipped and answered
once all the easy questions have been answered. Skip one question 𝑖 ∈ {1, … , 𝑁} has
an associated time cost ℓ! which will consist of re-reading the hard question before
answering it.
This problem aims to study the best strategy to answer as many questions as possible into the
deadline 𝑑. The choice of a strategy when facing a test can drastically affect the result. This
result depends on several factors; the test design is the most important to consider (such as
sequence, read, response, and re-read times), and the second factor is the deadline to perform
it.
659
2.2. Dominance
We define the dominance of strategy A over strategy B, when under certain conditions,
strategy A implies a better result than strategy B in terms of test results, i.e., more questions
are correctly answered. To illustrate the situation, we consider a reading time (𝑟! = 𝑟 = 1)
and a re-reading time (𝑙! = 𝑙 = 1) to be equivalent to one time slot, the response times for
the hard question (𝑝" ) and the easy question (𝑝# ) equals to 7 and 2 time slots, respectively;
and the test sequence given by S={h,e}. Figure 1 shows that in the case of a deadline 𝑑 of
four slots of time. On the one hand, if the RA strategy is used, no question will be answered
because it will answer the hard question in the first instance, whose resolution time is longer
than the deadline. On the other hand, if the RSA strategy is used, one question can be
answered because the first question must have been skipped, and the easy question will have
been read and answered. In this case, the RSA strategy dominates RA strategy. In a similar
way, Figure 2 shows the case of a deadline 𝑑 of ten slots, where the RA strategy will dominate
with two questions answered over the RSA strategy, which will have only managed to answer
a single question.
Figure 1. Example of dominance RSA over RA. Figure 2. Example of dominance RA over RSA.
2.3. Mathematical formulation

The problem can be formulated as integer linear programming. We adopt a upper bound on
|%|
the time to perform the test given by 𝐾 = ∑!&' 𝑟! + 𝑝! + 𝑙! . Consequeltly, we discretize the
time into K unit time slots. We then define three sets of binary decisión variables: 𝑋!,) which
is equal to 1 if the reading of question 𝑖 starts at time 𝑡 and 0 otherwise; 𝑌!,) which is equal to
1 if the answering of question 𝑖 starts at time 𝑡 and 0 otherwise; and 𝑍!,) which is equal to 1
if the re-reading of question 𝑖 starts at time 𝑡 and 0 otherwise. The problem can be set as
follows:
|𝒫| $%&'! ()
[𝑀𝑎𝑥] ' ' 𝑌!,# (1) 𝑋!,# = 0, ∀𝑖 ∈ 𝒫, ∀𝑡 ∈ [𝐾 − 𝑟! + 1, 𝐾] (2)

!*) #*)
𝑌!,# = 0, ∀𝑖 ∈ 𝒫, ∀𝑡 ∈ [𝐾 − 𝑝! + 1, 𝐾] 𝑍!,# = 0, ∀𝑖 ∈ 𝒫, ∀𝑡 ∈ [𝐾 − ℓ! + 1, 𝐾]
(3) (4)
660
-
-
' 𝑋!,# ≤ 1, ∀𝑖 ∈ 𝒫 (5) ' 𝑌!,# ≤ 1, ∀𝑖 ∈ 𝒫 (6)
#*)
#*)
-
' 𝑍!,# ≤ 1, ∀𝑖 ∈ 𝒫 (7) 𝑋!,# = 1, 𝑖 = 1, 𝑡=1 (8)

#*)
|𝒫| .&) .&)
'; ' 𝑌!,# + ' 𝑋!,#

!*) #*/01{.&'! ,3} #*/01{.&5! ,3}
|𝒫| .&) .&) .&)
'; ' 𝑌!,# + ' 𝑋!,# + ' 𝑍!,# <

!*) #*/01{.&'! ,3} #*/01{.&5! ,3} #*/01{.&ℓ! ,3}
.&) (9) |𝒫| .&) .&) (10)
+ ' 𝑍!,# < ≤ 1, 𝐾=1 ≥ '; ' 𝑌!,# + ' 𝑋!,#
#*/01{.&ℓ! ,3} !*) #*/01{.&'! ,3} #*/01{.&5! ,3}
.&)
+ ' 𝑍!,# < , ∀𝑘 ∈𝐾

#*/01{.&ℓ! ,3}
9
#
𝑋!(),# ≤ ' X 7,8 , ∀ i ∈ 𝒫 − 1, ∀ t ∈ K (11) 𝑍!,# ≤ ' 𝑋!,. , ∀ 𝑖 ∈ 𝒫, ∀𝑡 ∈ K (12)
8*)
.*)
𝑍!,# ≤ Y7,9(ℓ" , ∀ 𝑖 ∈ 𝒫, #
∀ 𝑡 ∈ K − ℓ7 (13) 1 − 𝑋!,# ≥ ' 𝑌!,. , ∀ 𝑖 ∈ 𝒫, ∀ 𝑡 ∈ K (14)

.*)
#
1 − 𝑋!,# ≥ ' 𝑋!(),. , ∀ 𝑖 ∈ 𝒫 − 1, ∀ 𝑡 -
.*) (15) ' D(𝑡 + 𝑟! ) ⋅ 𝑋!,# H ≤ 𝑑, ∀ 𝑖 ∈ 𝒫 (16)

∈ K #*)
-
-
' D(𝑡 + 𝑝! ) ⋅ 𝑌!,# H ≤ 𝑑, ∀𝑖 ∈ 𝒫 (16) ' D(𝑡 + ℓ! ) ⋅ 𝑍!,# H ≤ 𝑑, ∀𝑖 ∈ 𝒫 (18)
#*)
#*)
- - :
' 𝑡 ⋅ 𝑌!,# − ' D(𝑡 + 𝑟! ) ⋅ 𝑋!,# H ≤ 1 + M ⋅ K' Z7,9 M , ∀𝑖 ∈ 𝒫 (19)

#*) #*) 9*)
2.4. Implementation and computational experiments

The mathematical formulation described above was implemented computationally using
Python 3.9.16. The Pyomo package was used for implementation and the Gurobi solver 10.0
was used for solving. For the computational experiments were performed on a 3-question
test. To perform the analysis, the parameters were set to 𝑟! = ℓ! = 1. For each test, a grid of
values for the response times of each question will be made. This grid will have a lower limit
of 1 time slot and an upper limit of 6 time slots. The value of the response time for difficult
questions will be a function of the value of the response time for easy questions. e.g. If the
answer time for the easy question is 1, the answer time for the difficult questions will be 2,
3, 4, 5 and 6 time slots. In turn, the deadline will be varied, for each of the cases described
661
above, the deadline will be decreasing. In addition, the value 𝑑 will be generated as an upper
bound, this will have a value of 24 time slots, this will be calculated by using the following
expression: 𝑑 = ∑*!&' 𝑟! + 𝑝! + 𝑙! where all the questions will be difficult. Therefore the
number of instances to solve will be of 2* ∙ 5 ∙ 4 ∙ 3 ∙ 2 = 960.
3. Results
In the generation of instances for their respective analysis, all possible test sequences were
considered, that is, 2* possible combinations. These combinations are: {{h,h,e}; {e,e,e};
{h,e,e}; {h,h,e}; {e,e,h}; {e,h,h}; {e,h,e}; {h,e,h}}. Since it is assumed that the test-taker
will know the number of questions and the sequence of the test, 5 instances will be discarded,
since the RA strategy will dominate over RSA strategy regardless of the deadline. For
example, if the test sequence is {e,e,e} there will be no difficult question to skip so the RSA
strategy will not exist. On the other hand, if the sequence is {h,h,h} the use of the RSA
strategy would not be logical, since knowing the sequence of the test, the use of this strategy
would generate the additional cost of re-reading.
3.1. Sequential analysis {h,e,h}

It was found that the use of the strategies studied is indifferent if the deadline is less than
𝒎𝒊𝒏{𝒓 + 𝒑𝒉 , 𝟐𝒓 + 𝒑𝒆 } since the time to solve the test will not be enough to answer the first
question, so the use of one or the other strategy will lead to the same result, not answering
any question. On the other hand, we suppose the use of one strategy is indifferent to the other
when the time is more than enough to perform the test in its entirety so that if the deadline is
greater than 𝟒𝒓 + 𝒑𝒆 + 𝟐𝒑𝒉 no strategy will dominate over the other. Note that when facing
the sequence {h,e}, one strategy will dominate over the other, according to the values of 𝑝"
and 𝑝# , if the test deadline is between the values 𝒎𝒊𝒏{𝒓 + 𝒑𝒉 , 𝟐𝒓 + 𝒑𝒆 } and 𝒎𝒂𝒙{𝒓 +
𝒑𝒉 , 𝟐𝒓 + 𝒑𝒆 }. Thus, if 𝒑𝒉 > 𝒓 + 𝒑𝒆 the RSA strategy will dominate over RA strategy,
otherwise, if 𝒑𝒉 < 𝒓 + 𝒑𝒆 the RA strategy will dominate over RSA strategy. Finally, if the
test-taker is facing the last question of the test, and his deadline is between the values 𝟑 +
𝒑𝒆 + 𝟐𝒑𝒉 and 𝟒 + 𝒑𝒆 + 𝟐𝒑𝒉 , the dominant strategy will be the RA strategy.
3.2. Sequential analysis {e,h,e}

The result was that the use of the strategies studied is indifferent if the deadline is less than
𝒎𝒊𝒏{𝟐𝒓 + 𝒑𝒆 + 𝒑𝒉 , 𝟑𝒓 + 𝟐𝒑𝒆 } since the test resolution time will be sufficient to answer the
first question, that is, the easy question (facing an easy question does not require the use of a
strategy) but it will not be sufficient to answer the next question. On the other hand, the use
of one strategy will be indifferent to the other is when the time is more than enough to perform
the test in its entirety, so if the deadline is greater than 𝟒𝒓 + 𝟐𝒑𝒆 + 𝒑𝒉 no strategy will
662
dominate over the other. If the test deadline is between the values 𝒎𝒊𝒏 {𝟐𝒓 + 𝒑𝒆 + 𝒑𝒉 , 𝟑𝒓 +
𝟐𝒑𝒆 } and 𝒎𝒂𝒙 {𝟐𝒓 + 𝒑𝒆 + 𝒑𝒉 , 𝟑𝒓 + 𝟐𝒑𝒆 } one strategy will dominate over the other
according to the values of 𝒑𝒉 and 𝒑𝒆 . If 𝒑𝒉 > 𝒓 + 𝒑𝒆 the RSA strategy will dominate over
RA strategy, on the other hand if 𝒑𝒉 < 𝒓 + 𝒑𝒆 the RA strategy will dominate over RSA
strategy. Finally, if the test-taker is facing the last question of the test, and its deadline is
between the values 𝟑 + 𝟐𝒑𝒆 + 𝒑𝒉 and 𝟒 + 𝟐𝒑𝒆 + 𝒑𝒉 the dominant strategy will be the RA
strategy.
3.3. Sequential analysis {h,e,e}

The result was that the use of the strategies studied is indifferent if the deadline is less than
𝒎𝒊𝒏{𝒓 + 𝒑𝒉 , 𝟐𝒓 + 𝒑𝒆 } since the test resolution time will not be enough to answer the first
question, so that the use of one or the other strategy will lead to the same result, not answering
any question. In addtion, the use of one strategy will be indifferent to the other is when the
time is more than enough to perform the test in its entirety, so if the deadline is greater than
𝟒𝒓 + 𝟐𝒑𝒆 + 𝒑𝒉 no strategy will dominate over the other. If the test deadline is between the
values 𝒎𝒊𝒏{𝒓 + 𝒑𝒉 , 𝟐𝒓 + 𝒑𝒆 } and 𝒎𝒂𝒙{𝒓 + 𝒑𝒉 , 𝟐𝒓 + 𝒑𝒆 } one strategy will dominate over
the other according to the values of 𝒑𝒉 and 𝒑𝒆 . If 𝒑𝒉 > 𝒓 + 𝒑𝒆 the RSA strategy will
dominate over RA strategy, on the other hand if 𝒑𝒉 < 𝒓 + 𝒑𝒆 the RA strategy will dominate
over RSA strategy. Finally, if the test taker is facing the last question of the test, and its
deadline is between the values 𝟑 + 𝟐𝒑𝒆 + 𝒑𝒉 and 𝟒 + 𝟐𝒑𝒆 + 𝒑𝒉 the dominant strategy will
be the RA strategy. Note that in the tests performed similarities were found in the analysis,
these similarities were found when the sequence (S) of the test is {h,e}. Figure 3 shows a
graphical summary of what was described above.
bub
Figure 3. Summary table of dominance.
663
4. Conclusion and future works

Previous research has focused on practical use with the objective of finding strategies to
perform different types of tests, while in the area of psychology, various strategies have been
investigated with the objective of reducing stress and anxiety when facing a test. However,
these studies have not been approached from the algorithms, mathematics, and optimization
viewpoints. This study generated a mathematical scheduling model that generated a
standardized test with the objective of applying the strategies found in the literature in order
to find dominance properties. By performing several instances of a standardized three-
question test, it was concluded that one strategy would dominate over the other depending
on the parameters and design of the test. In particular, we observe that the use of one strategy
over the other will be indifferent if the maximum time to perform the test is not enough to
answer the first question or if the maximum time is more than enough to answer the whole
test. Finally, we show that the RA strategy will always dominate over the RSA strategy when
the last question of the test is asked. Studying these dominance properties can provide a guide
for test-takers to tackle standardized tests and obtain an expected result. For future works,
we leave open the problem with a sequence and number of unknown questions by the test-
taker, or where each question could have weights or a probability of error, increasing the
difficulty of the problem to be studied.
Acknowledgments
The authors are grateful for the support of the National Agency for Research and
Development (ANID), FONDECYT Project N°1211640.
References
DEMRE. (2023, ENERO). Retrieved from https://demre.cl/la-prueba/pruebas-y-
temarios/presentacion-pruebas-temarios-paes-invierno-p2024
Dufossé, F., Dürr, C., Nadal, N., Trystram, D., & Vásquez, Ó. C. (2022). Scheduling with a
processing time oracle. Applied Mathematical Modelling, 701-720.
Dürr, C., Erlebach, T., Megow, N., & Meißner, J. (2020). An Adversarial Model for
Scheduling with Testing. Algorithmica, 3630-3675.
Kesselman-Turkel, J., & Peterson, F. (1981). Test-Taking Strategies. Study smart series.
Retsef, L., Magnanti , T., & Shaposhnik, Y. (2018). Scheduling with Testing. Supplemental
Materials.
Theobald, M., Breitwieser, J., & Brod, G. (2022). Test Anxiety Does Not Predict Exam
Performance When Knowledge Is Controlled For: Strong Evidence Against the
Interference Hypothesis of Test Anxiety. Psycological Science, 10.
Viet, V. V. (2022). Learning strategy and its effect on academic achievement. Kasetsart
Journal of Social Sciences, 9.
Wang, T., & Zhang, J. (2006). Optimal partitioning of testing time: Theoretical properties
and practical implications. Psychometrika, 15.
664
Microcredentials: an opportunity towards the digital

transformation
Fernando J. P. Caetano1,2, Diogo G. Casanova1,3,4, Darlinda M. P. Moreira1,5
1
Universidade Aberta, Portugal, 2Centro de Química Estrutural, Institute of Molecular
Sciences, Instituto Superior Técnico, Universidade de Lisboa, Portugal, 3LE@D,
Universidade Aberta, Portugal, 4Centro de Investigação em Didática e Tecnologia na
Formação de Formadores, Universidade de Aveiro, Portugal, 5Centro de Estudos Globais,
Universidade Aberta, Portugal.
Abstract
This communication aims to present one of the online courses included in the
Microcredentials portfolio of the Universidade Aberta, in Portugal. We start
by presenting a framework of the European guidelines on Microcredentials,
defining its main characteristics, namely highlighting its advantages in the
context of reskilling or upskilling with a view to its applicability in the labor
sector. Following the guidelines of the Portuguese Ministry of Higher
Education, and specific EU funding under the Impulso Adultos programme, a
proposal for a Microcredential in Digital and Distance Learning was
conceptualized and developed for teachers´ training, with more than 1200
teachers already involved. An evaluation carried out through a survey
delivered to trainees shows very positive results in all items evaluated, with
strong emphasis on the transferability and applicability of the knowledge
acquired to work contexts.
These results show that this course, developed within the scope of
Microcredentials, broadly meets the needs of reskilling and upskilling, which
are increasingly important in the adaptability of adults to the new challenges
that digital transformation implies.
Keywords: Microcredentials; distance and digital education; digital
transformations; lifelong learning; higher education.

Microcredentials: an opportunity towards the digital transformation
1. Introduction
Working contexts and conditions are changing rapidly, and some studies indicate that over
the course of a year, between 20% to 70% of the population change jobs or even occupations
in response to the new challenges posed by technological and environmental transition
(Bakhshi et al. 2017; OECD, 2012). This panorama leads to the rapid mismatch between
initial formal training, defined on the basis of a set of pre-defined assumptions which are not
very flexible in nature and the skills required for the professional role. Therefore, there is a
pressing need for a constant renewal of existing training and the promotion of continuous
learning to acquire, adjust and update knowledge and skills.
It is in this context of rapid changes in professional skills and the urgent need for training
that Microcredentials have been mentioned as an educational strategy with potential to meet
the needs of professional qualification and requalification in the European and national space
(Kato, Galán-Muros & Weko, 2020; European Commission, 2020; MCTES, 2020).
The Recovery and Resilience Plan (PRR) in Portugal outlines post-covid objectives for the
training and requalification of the labor sector, with a focus on promoting digital transition
in companies and institutions and requalifying a significant portion of their workforce. Some
of the specific goals of this plan include:
• Providing digital skills training to 800,000 individuals, with personalized training

plans and online resources.
• Supporting the digitalization and requalification of public administration.
• Establishing programs, “schools”, and “alliances” for higher education training in
partnership with employers, including post-graduate diplomas and masters.
• Promote the digital transition in companies by retraining 36,000 workers.
By achieving these objectives, Portugal aims to enhance the competitiveness of its labor
sector, boost economic growth, and prepare its workforce for the digital age demands.
The purpose of this paper is to gather approaches and main characteristics of the so-called
Microcredentials and place them in the context of the training practices of Universidade
Aberta, the public distance learning university in Portugal, which follows an internationally
validated digital distance learning methodology supported by the UAb Virtual Pedagogical
Model (UAb-VPM) (Pereira et al. 2007; Mendes et al, 2019).
2. The main concept

A Microcredential is a certificate of short-term learning, formal or informal, which is
regarded as strategic by the European Commission to achieve its European Skills Agenda.
The European Commission (EC) has sought to discuss, within the scope of a number of
666
projects, the definition of Microcredentials and to identify the challenges and opportunities
in this area. According to the EC, Microcredentials are:
… the record of the learning outcomes that a learner has acquired following a small volume
of learning. These learning outcomes will have been assessed against transparent and clearly
defined criteria. Learning experiences leading to Microcredentials are designed to provide
the learner with specific knowledge, skills and competences that respond to societal,
personal, cultural or labor market needs. Microcredentials are owned by the learner, can be
shared and are portable. They may be stand-alone or combined into larger credentials. They
are underpinned by quality assurance following agreed standards in the relevant sector or
area of activity (European Council, 2022).
The European Union definition, largely shared by other authors (Brown, et a. 2021;
Wheelahan and Moodie, 2021) promotes the discussion of a tacit recognition of the
accomplishment of a learning outcome from a short learning experience developed in a
formal or informal learning context. This recognition usually comes in the aftermath of
responding to existing societal, personal, cultural, or labor market needs. According to the
EC, these needs typically originate from unexpected or conjunctural challenges projected by
technological or environmental transition. Microcredits can be offered on a stand-alone basis
or offered in conjunction with others (stackable) but must be guided by limited benchmarks
respected by the corresponding sector, be it the educational or business sector.
Regarding the context where the training takes place, although it was initially born in online
contexts, particularly through MOOCs and short courses held in distance learning platforms
such as Coursera or Futurelearn, the EC recommends that Microcredentials can be offered in
various educational environments such as face-to-face, online, hybrid or blended-learning
environments, or even in a work context. Regarding the entities that provide the training,
initially the need to adapt to the ECTS system led to an almost inevitable association with
higher education institutions. The last recommendation (European Council, 2022) opens the
possibility of Microcredentials having the involvement of other institutions such as
Vocational and Lifelong Learning Agencies or industry federations with training academies.
The need for the existence of a transversal unit of measurement such as the ECTS in order to
be used as a legal framework for evaluating the workload of the trainee/student during the
training period was made more flexible by no longer being included as mandatory by the
commission, and therefore, giving each member-state room to define its own legal
framework. Regardless of this European guiding document it is important to identify the
existence of Microcredentials in the Anglo-Saxon higher education space, in particular in the
United States, Australia, Canada, and more recently also in the United Kingdom.
The working group on Microcredentials, supported by the European Commission, published
a report in December 2020, addressing the need to define Microcredentials in a single format
667
and shared by its member states. Later, in 2022, the Council of the European Union adopted
the European Commission Recommendation on Microcredentials as a European strategy for
lifelong learning and employability. The Recommendation seeks to support the development,
implementation and recognition of Microcredentials across institutions, companies and
sectors. In this joint approach, member states are responsible for informing the European
Commission on appropriate measures to adopt the recommendations by December 2023. The
recommendations point to the need for member states to:
• Develop an ecosystem of microcredentials, including promoting the creation of new

microcredentials, encouraging the development of microcredentials in formal and
informal contexts, promoting the quality and transparency of Microcredentials and
fostering the development of partnerships between educational institutions and
social partners, employers and industry, innovation and research centers and local
and national authorities.
• Realize the potential of Microcredentials by integrating them into existing offers
and training, reaching a wider and more varied type of learners, including
disadvantaged and vulnerable groups, and integrate Microcredentials into
employment and labor market support policies.
This type of training can complement traditional learning and teaching by better enabling
students to acquire high-level skills. This process of Microcredentials, should also facilitate
access to a new type of learners (European Commission, 2020) or allow the return of former
students into higher education in a perspective of professional requalification. The strategy
foresees that by 2030, 60 % of all European adults undergo in training every year.
Microcredentials allow for opportunities to diversify learning and improve education through
shorter courses, extending the qualifications that each person already has.
In addition, Microcredentials can be grouped into subjects, being the proof of the learning
acquired and their certificates do not correspond to a full degree as, for example, the degree
that is obtained in a bachelor's, master's or doctorate.
For a good streamlining of the certificates awarded, it will be necessary to adopt a single,
transversal digital system that allows any university or employer to verify the information
elements of each certificate. This also means that secure access to the data must be ensured,
and they must also be verifiable free of charge. Transparency tools and processes are related
to the transparency of qualifications (the European Qualifications Framework); quality
assurance in higher education; credits for learning achieved; recognition of prior learning and
validation of non-formal and informal learning; lifelong learning and career management.
The European Council (2022) recommends Europass as the Digital Credentials Infrastructure
that could eventually facilitate the issuing, sharing and storage of all forms of learning in a
digital format, including Microcredentials.
668
3. Microcredentials at Univ. Aberta

Since 2009, Universidade Aberta (UAb) has been investing in Lifelong Learning (LLL) for
the personal and professional training of adult learners, offering a wide range of courses,
from short courses, with one or two ECTS, to post-graduation courses typically involving
from 30 to 60 ECTS. Since the development of this strategy, more than 13,000 people have
obtained certification in the various Lifelong Learning courses that have been offered at UAb.
For these participants, these courses have the advantage that they are usually more applicable
to their professional areas, less time-consuming and usually require less personal, financial
and professional investment.
The offer of LLL courses at UAb has become recognized within and outside its community,
as a natural and intrinsic situation to its own way of presenting itself. The model of online
distance learning, applied mostly in asynchronous format, has a wide capacity and
applicability among the target audience that is distributed across all regions of the national
territory and abroad with various schedule commitments, which means that this type of
education allows temporal and spatial flexibility suitable for students with personal and
professional limitations (in Portugal, distance learning is regulated by the decree-law
133/2019).
Lifelong Learning at UAb organizes the offer of a diverse set of courses in several areas.
These courses have been offered in a logic of providing their learners with a set of
professional and personal skills that allow them to quickly achieve their goals by linking
them with other formal and non-formal learning that they already have or will acquire.
Thus, it can be stated that, unlike other HEIs, inside and outside Portugal, that have never
had short courses on offer, the UAb pathway prepared the institution to start offering
microcredentials, both as a conceptual and a recognition point of view. For the UAb, the
concept linked to Microcredentials implementation corresponds to a natural and desired path
to follow, because the strategic importance that this educational model has for the adult
learners is recognized. As a natural result, the existing relations with the European
Association for Distance Teaching Universities two European funded projects related to
Microcredentials, and with other higher education institutions members of this association,
were stepping stones that lead to UAb being perceived by the sector as a national reference
entity for this area and as a "trusted provider" for the recognition of training offer and issuing
of Microcredentials. Furthermore, the Impulso Adultos (Adult Impulse) programme, boosted
the existing offer in Microcredentials leading UAb to set a target of training circa 6,000 adults
from 2022 to 2025.
669
3.1. Perspectives and Proposals

For the implementation of Microcredentials, some strategic areas of knowledge that are
already present in short courses at UAb and that have potential in various learning paths have
been established.
The Microcredentials were thus composed of training "packages" containing short-term
training courses (from 1 to 6 ECTS). At the UAb, Microcredentials are usually designed from
scratch or re-adapted from existing training courses. They are designed and developed in the
context of reskilling or upskilling having an applicable character in the workplace. Although
not mandatory, it is recommended to work with partner institutions in the development of the
training offers, particularly partners that represent the sector where learning outcomes
developed can be applied.
The partnership can be a joint scientific work (planning of activities, identification of content
and training objectives) or the identification of competences to be acquired (need analyses).
Regardless of the model chosen, the institution has not considered strategic the use of existing
courses, in particular, the use of course units from existing formal study programmes that
allow stackability (Bozkurt & Brown, 2021), i.e. the possibility of joining a set of
Microcredentials with the aim of forming a single piece that provides a degree or a
postgraduate diploma.
Regardless of the solution used, at UAb, Microcredentials training courses should always
include an element of assessment of the learning outcomes to prove that they are clearly met.
The learner can create their UAb Microcredentials portfolio whenever they finish the training
courses in which they are involved.
3.2. Distance and digital learning training modules - examples

UAb has already prepared about 20 training courses leading to Microcredentials which
include courses from 1 to 6 ECTS. For the sake of this communication, we will present the
Distance and Digital Education training course which is aimed at teachers of primary,
secondary and higher education in the education system in Portugal. To obtain a Microcredit
in Distance and Digital Education (DDE), participants need to undertake a 3 modules course,
with 4 ECTS and a duration of 4 months. This course enables teachers to design and monitor
a distance learning unit of a study programme/course and has been in offer since 2021,
leading to the certification of about 1200 higher education teachers. Its structure integrates
the following modules:
• E-activities in Course Design in DDE (1ECTS)

• Digital Teaching in Network (1ECTS)
• UC Project in Digital Environment (2ECTS)
670
The main objectives of this Microcredential, are:
• To reflect about pedagogical practice in Digital Distance Education contexts.

• Analyze communication and interaction processes in digital educational contexts.
• Mobilize relevant pedagogical strategies, namely at the level of activity design for
digital educational contexts.
• Conceive, design and develop a programme unit project for DDE.
• Design and develop e-activities for DDE environments.
• Select appropriate educational resources according to the context and the target
population.
• Select digital assessment instruments according to the nature and pedagogical
contexts.
This training provides informed answers to the problems teachers encounter in their teaching
practice in an online context and gives them the possibility to evaluate the effectiveness of
innovations in order to contribute to a successful digital education. According to UAb's
Virtual Pedagogical Model the asynchronous methodology allows flexibility in participation
and in the learning path, highly appreciated by students because they adapt learning to their
own pace. Another methodological aspect that we highlight is the assessment, where both
self-assessment and peer-assessment are strongly articulated with the course assessment.
4. Final remarks
At the end of each module all trainees are asked to respond to a survey of satisfaction with
questions about the structure and performance of the course, as well as aspects related to the
quality of the course, such as resources, activities, tools used, and the perceived self-
performance of each trainee.
The outcomes of the evaluation conducted to the participants have been overwhelmingly
positive, with notable success in the self-performance category. There was a unanimous
agreement among participants in their involvement with the proposed activities,
collaboration, and exchange of knowledge. Furthermore, the acquired knowledge and
collaborative processes were deemed easily transferable and applicable to each trainee's
respective work contexts. Although being an online course, the flexibility and the
organization of the course allowed the majority of the participants to have success in
achieving the learning outcomes. The course allowed three ending points – participants could
leave after each module being recognized with nano-credits related to the specific modules
they successfully completed. Notwithstanding, circa 50% of the participants finished all three
modules of the course and have been able to apply what they learnt in their own practice.
671
The results show that this short course developed at Universidade Aberta within the scope of
Microcredentials fits the concept and objectives for which it was created, i.e., it responds
positively to the immediate needs of reskilling and upskilling in a work context.
Acknowledgements
Project funded by the Recovery and Resilience Program (PRR), Portugal, and the European
Union - Next Generation, EU
References
Bakhshi, H., Downing, J. M., Osborne, M. A. & Schneider, P. (2017). The Future of Skills:
Employment in 2030. London: Pearson and Nesta.Brown, Mark & Mhichíl, Mairéad &
Beirne, Elaine & Mac Lochlainn, Conchúr. (2021). The Global Micro-credential
Landscape: Charting a New Credential Ecology for Lifelong Learning. Journal of
Learning for Development. 8. 228-254. 10.56059/jl4d.v8i2.525.
Bozkurt, A & Brown, M (2021) Microcredentials. Stackable, Combinable, or Transferable .
EdTechnica, https://edtechbooks.org/encyclopedia/microcredentials
DCU Connected, Micro-credentials, https://www.dcu.ie/connected/Microcredentials
DL 133/2019 - Decreto-lei 133/2019, de 3 de Setembro, Diário da República n.º 168/2019,
Série I de 2019-09-03.
EADTU, 2020. European Credit Clearinghouse for Opening up Education (ECCOE).
Available at: https://eccoe.eu/#/
European Commission (2020) “Achieving the European Education Area by 2025 and
resetting education and training for the digital age”, PRESS RELEASE, Publication 30
September 2020, https://ec.europa.eu/commission/presscorner/detail/en/ip_20_1743
European Commission (2020); A European approach to Microcredentials – Output of the
Microcredentials higher education consultation group - Final report, December 2020
https://op.europa.eu/en/publication-detail/-/publication/7a939850-6c18-11eb-aeb5-
01aa75ed71a1
European Council. (2022). Council recommendation on a European approach to
microcredentials for lifelong learning and employability. European Sources Online.
Official Journal of the European Union.
https://data.consilium.europa.eu/doc/document/ST-9237-2022-INIT/en/pdf
Kato, S., V. Galán-Muros and T. Weko (2020), "The emergence of alternative
credentials", OECD Education Working Papers, No. 216, OECD Publishing,
Paris, https://doi.org/10.1787/b741f39e-en.
MCTES (2020) Skills 4 pós-Covid – Competências para o futuro –
https://www.dges.gov.pt/pt/content/skills-4-pos-covid-competencias-para-o-futuro-0
Mendes, A. Q., Bastos, G., Amante, L., Aires, L & Cardoso, T. VIRTUAL PEDAGOGICAL
MODEL | Development Scenarios (2019), ISBN: 978-972-674-851-9
https://portal.uab.pt/wp-content/uploads/2019/09/MPV01_ENG.pdf
672
MicroBol (2020) - Microcredentials linked to the Bolonha Key commitments

https://microbol.microcredentials.eu/
OECD (2012). Better Skills, Better Jobs, Better Lives: A Strategic Approach to Skills
Policies, OECD Publishing. http://dx.doi.org/10.1787/9789264177338-en.
Pereira, A., Quintas Mendes, A., Morgado, L., Amante, L., & Bidarra, J. (2007) Modelo
Pedagógico Virtual da Universidade Aberta. Lisboa: Universidade Aberta.
http://hdl.handle.net/10400.2/1295
Wheelahan, L. & Moodie, G. (2021) Analysing micro-credentials in higher education: a
Bernsteinian analysis, Journal of Curriculum Studies, 53:2, 212-228, DOI:
10.1080/00220272.2021.1887358
673
Fostering cooperation between lecturers and TLC staff to improve

digital teaching. Experiences with a pool of discipline-specific
experts
Petra Panenka, Ann-Christine Hadam, Hans-Martin-Pohl
Department of teaching and studying, University of Applied Sciences Fulda, Germany.
Abstract
The necessity of discipline-specific teaching method skills of lecturers is
exacerbated by a partly insufficient cooperation between lecturers and media-
didactic staff in Teaching and Learning Centers (TLC). This paper aims to
illustrate how this lack of cooperation between lecturers and media-didactic
experts can be solved by a team of discipline-specific experts skilled in media-
didactic methods. Due to their knowledge of the discipline-specific
communication cultures these discipline-specific experts build a “bridge” in
cooperation between lecturers and media-didactic experts as well as technical
staff. Experiences gathered with the TLC of the University of Applied Sciences
Fulda show that the discipline-specific experts support lecturers by fostering
particularly informal cooperation between them and media-didactic experts or
technical staff. Therefore, the pool of discipline-specific experts is a new and
innovative way to support lecturers in acquiring discipline-specific digital
teaching skills, fostering discourse on teaching practices and further
developing digital teaching.
Keywords: Cooperation; digital teaching; discourse on teaching; digital skills.

Fostering cooperation between lecturers and TLC staff to improve digital teaching
1. We Need a Closer Cooperation Between Lecturers and Staff of TLCs!

During the COVID-19 pandemic, most university lecturers all over the world had to move
abruptly towards “online education”. The rapid start of such digital education brought not
only challenges for lecturers, students and staff of Teaching and Learning Centers (TLCs), it
also fostered digitalization in teaching and learning. Lecturers had to perform in online
classrooms by applying digital skills. Students had to adapt to digital, often unfamiliar
learning spaces and necessarily acquired new learning skills. Universities were forced to
advance their infrastructures and organization (Perez-Sanagustin et al., 2022). Particularly
TLCs had to develop support in online teaching methods for digital spaces.
Since March 2020, a transformation towards the digitization of the university system has
started, and the question of the sustainability of these changes has emerged. In this context,
staff of TLCs play a key role as they enable lecturers and students to adapt to continuous
changes in education by explaining digital tools and teaching skills to them. Within this
process, e-learning and media-didactic experts are particularly important as they can support
lecturers in developing courses, in trying out teaching methods or technologies (Karcher &
Nunn, 2021). In short: TLCs support lecturers in aquiring digital teaching skills.
Despite the urgent support needed in the past two years of the COVID-19 pandemic, still too
many lecturers do not see the necessity of cooperating with the TLCs staff or consider the
central services to be too little related to the teaching methods in their discipline. Even
lecturers who have been eager to aquire digital teaching skills are becoming less interested
in furthering their digital teaching in cooperation with media-didactic experts at TLCs. This
raises the question as to how to foster and/or maintain cooperation between lecturers and
media-didactic experts in TLCs for a sustainable development of teaching skills.
Cooperation is essential for the following illustrations. In essence cooperation (lat. co
together, opera tasks, work) between humans is the purposeful interaction of two or more
persons, groups or systems with a common interest. It is an intentional and planned way of
working together and by processes of mutual agreement on specific goals. It can be both
formal, in the case of existing structures or obligations, and informal, when there are no rules
in place (Pastoors & Ebert 2019). In terms of the “homo heterogeneous”, cooperation is
particularly understood here as a pursuing of longterm goals and acting sustainably (Rogall
& 2015). In line with Kubrick (2001), it is seen as an unfolding process of interaction which
can be better understood by interrelating the four perspectives “subject- or task-related”,
“social”, “organizing” and “communicative”. These perspectives illuminate the dynamics of
shared practices evolving into a shared collective culture with formal and informal
communication and collaboration.
This paper argues that a cooperation between lecturers, media-didactic and technical staff in
TLCs may be fostered by implementing a team of discipline-specific experts who know the
676
Petra Panenka, Ann-Christin Hadam, Hans-Martin-Pohl
department culture and discourse on teaching in their discipline. This sort of cooperation will
be illustrated by insights into the project Developing Online Formats Together - Innovative
and Sustainable (GO-IN), University of Applied Sciences Fulda (HFD), which aims to foster
such a cooperation by a pool of discipline-specific experts. First, there will be an illustration
of the necessity of enhanced cooperation between the TLC and the departments. (2) This is
followed by an overview of GO-IN (3). In a next step, examples of support by discipline-
specific experts are provided (4), the informal group discussions (4.1.) and the adaptation of
teaching material (4.2). Finally, the experiences are discussed (5).
2. The Necessity of Enhanced Cooperation between the TLC Dienstleistungen

Lehre und Studium and the Departments at the University of Applied Sciences
Fulda
At the University of Applied Sciences Fulda (HFD), there are currently around 730
employees and 9,700 students with diverse educational backgrounds, personal and
professional biographies. The HFD unites a variety of disciplines and has eight departments
with their own cultures of communication: Applied Computer Science (AI), Electrical
Engineering and Information Technology (ET), Food Technology (LT), Oecotrophology
(OE), Health Science (GW), Social and Cultural Sciences (SK), Social Work Science (PG),
Business Science (W). Students and lecturers are supported by the media-didactic experts in
the TLC of the HFD, the Dienstleistungen Lehre und Studium (DLS). This variety of
disciplines and diversity of students is a strength of the HFD, but it entails challenges. These
challenges in digital teaching were addressed by DLS. Since 2005, lecturers have been
supported in media-didactic teaching and technical support by a central e-learning service
and a cooperation project of many universities ("Digitally Supported Teaching and Learning
- digLL").
Despite HFD’s support services, teaching during the COVID-19 pandemic has brought
inconsistent development in digital teaching with regard to methods, tools and didactics
across the departments. The strengths and weaknesses of the HFD in digital teaching were
identified in the evaluation of the online semester 2020/21 (Kraft, Merkator & Pohl, 2021).
The evaluation included students (N=1.834) and lecturers (N=288). Data collection was
conducted via questionnaire survey and analysis by means of descriptive statistics. The
results revealed that there was good technical support, but partly problems in choosing
appropriate didactic methods among lecturers due to the overload of options. Therefore, the
identified key challenge was to address discipline-specific support and foster cooperation and
discourse about digital teaching methods between lecturers and DLS staff. The lack of formal
as well as informal cooperation between TLCs and departments in order to foster teaching is
not a new challenge. At the HFD, improving digital teaching through cooperation is the focus
of the GO-IN project, which is described in the following chapter.
677
3. The Project GO-IN and the Concept of the Pool of Discipline-specific experts
GO-IN is a project across disciplines; it is centrally managed by the Vice President for
Teaching and Learning and coordinated in DLS. It has the goal to identify existing digital
teaching practices which were developed during the COVID-19 pandemic. Lecturers are
particularly supported in adapting the digital teaching concepts identified and apply them in
a more developed way in their own teaching. The project has five pillars. First, a pool of
discipline-specific experts, with one expert from each discipline. Second, a professorial
advisory council, including one professor from each department. Third, discourse space(s),
which are spaces for discussions about digital teaching. Fourth, developement of teaching
and learning labs and fifth, producing teaching, learning and assessment materials. The
“heart” of the GO-IN-project is the pool of experts skilled in digital teaching methods and
tools who have a background in a certain discipline. The so called discipline-specific experts
of this pool foster particularly informal cooperation between lecturers and media-didactic
experts or DLS staff by building a “bridge” of communication. Once established, informal
cooperations sometimes transform into formal cooperation structures (see Figure 1).
Figure 1. Supported by the academic advisory council, eight discipline-specific experts build a bridge in
cooperation between the departments and DLS staff. Figure from: Christian Stickel (2023).
They support lecturers in developing a teaching concept, adapting certain teaching practices
or choosing an appropriate teaching tool. They are supported by media-didactic experts and
other technical staff in DLS as well as a professor from their discipline, who is a member of
the advisory council of the department, which forms the second pillar. The third pillar is the
discourse space(s) where the discipline-specific experts initiate communication about digital
teaching and learning by identifying discipline-specific needs in digital teaching methods.
They foster discourse about teaching and learning built on their knowledge about which
methods are preferred in the different disciplines. Furthermore, they bring together lecturers
within and across departments who are interested in the same teaching method(s) or digital
tool(s). With regard to the fourth pillar, development of teaching and learning labs, this
supports the departments in bringing together lecturers with the suitable person in DLS who
can support them in the design and application of innovative, digital-driven labs and the
678
related teaching methods in these new labs. They also establish cooperation between DLS
and the lecturers in regard to technical support for teaching settings. As part of the fifth pillar,
the discipline-specific experts support lecturers in producing teaching, learning and
assessment material by developing and creating these materials with media producers in
DLS.
4. Discipline-Specific Experts at Work: Two Examples

Based on the GO-IN project outline and the role of discipline-specific experts within the
project, the following examples illustrate informal (and formal) cooperation in practice. The
first example, the informal group discussions, addresses the question of how informal
cooperation with lecturers enhances a discourse about challenges in digital teaching and
enables tailor-made solutions. The second example, the adaptation of teaching material,
addresses how discipline-specific experts foster the adaptation of digital learning material
within and across departments by informal cooperation activities.
4.1. Informal Group Discussions

At the beginning of GO-IN, informal group discussions facilitated contacting lecturers and
establishing easy-going and non-obligatory regular meetings, the discourse spaces. The
informal group discussions enable the identification of good ongoing digital teaching
practices and prevailing challenges at the HFD. In five departments (AI, GW, LT, Oe, SK),
the discipline-specific experts fostered contact with nearly all lecturers in different ways,
depending on communication structure and culture in a department. In the departments AI,
LT, SK contact was established by the discipline-specific experts. In the department GW, the
discipline-specific expert began to participate in an existing formal discourse space. In the
department Oe, the best way to invite lecturers seemed to be an invitation by the member of
the advisory council. Despite the different ways of contacting lecturers for informal group
discussions, this led to two key developments: Firstly, they revealed good digital teaching
practices as well as challenges and, secondly, they initiated and established regular discourse
about teaching and learning between lecturers, discipline-specific experts and the DLS staff.
These discourse spaces are diverse and aligned to the communication culture in the discipline
and department.
In the department AI, the discourse space eLA (eLearning & Lehre-Austausch) was
implemented. There is an open discussion between professors and all other lecturers about
teaching digitization as well as applying digital methods and tools. The discipline-specific
expert coordinates and moderates the discourse space. Due to their sound knowledge in
teaching, they are able to pick up teaching examples that may be relevant for many lecturers,
applying KI in teaching.
679
The department GW already had the formal discourse space AG Qualität der Lehre for
discussion of teaching. Professors and lecturers participate here, treating varying topics
related to teaching and contributed to by participants. The discipline-specific expert gains
insight into good and challenging teaching examples and identifies particularly sustainable
structures of a discussion format which might not have been known before outside the
department.
Even though there is no specific title for discourse spaces in the department LT, the exchange
about teaching is conducted regularly in two kinds of discourse spaces, one for professors
and one for all other lecturers. Both discourse spaces are initiated by the discipline-specific
expert. At the beginning of some meetings, the discipline-specific expert, accompanied by a
lecturer, provides example(s) of a good digital teaching pattern (According to Kohls (2009)
patterns are schemes that document proven solutions in an easily accessible, practical form.).
This input is followed by discussions on the topic presented. The discipline-specific experts
deliver information and foster the discussion as they transmit their knowledge on teaching
methods in the communication culture of the department.
The department Oe also has no title for its discourse space, although such a space is initiated
regularly by the discipline-specific expert for interested lecturers. The participants are
professors and lecturers alike. The discipline-specific expert always presents patterns first in
a moodle course before an open exchange about the information presented starts. The
discourse is moderated by the discipline-specific expert. Hence, the discussions rely on their
knowledge about the teaching practice in the discipline.
Derived partly from the experience gained in the other departments and partly from extensive
informal group discussion, the department SK has just started to implement a discourse
space. Initiated by the discipline-specific expert, professors and all other lecturers will
participate in this discourse space together. This discourse space was already implemented
once by the department Oe and following meetings for discussions are planned. At this first
meeting, the discipline-specific expert presented a pattern which was discussed afterwards.
4.2. Adaptation of Teaching Material.

The adaptation of teaching material became particularly feasible after a well established
informal and partly developed formal cooperation between lecturers and discipline-specific
experts (as well as DLS staff) , which started to consolidate. The central goal of developing
digital teaching material is to provide templates which can be slightly modified and easily
applied by lecturers of all departments. Several templates for teaching, learning and exams
have already been developed in cooperation with lecturers, discipline-specific experts and
media producers.
680
A favourite template among lecturers is the digital learning template escape game. It was
developed to foster digitally assisted in-depth learning in class – virtual and in presence. The
idea for a learning unit with an escape game was proposed by a professor in the department
of LT. She herself did not have the time to compile such a template and was not sufficiently
skilled in media production. She asked the discipline-specific expert for assistance and the
latter in turn contacted one of the media producers in DLS. During the development of the
escape game by the media producers, the discipline-specific expert regularly presented the
evolving versions to the professor in order to tune them to the professor’s needs. Together
they designed two escape games. In a meeting of the pool of discipline-specific experts, the
participants discussed how escape games may be adapted for other courses. The solution was
to design an escape game template. Accordingly, the discipline-specific expert, the media
producer and five student assistants developed a template.
After the escape game template had been tested in a teaching course, the professor, supported
by the discipline-specific expert, shared her experiences with other professors in the LT
department. In particular, they presented it in the discourse space for professors described
above. The template was further presented in discourse spaces of four other departements
(Oe, AI, GW, SK). There, the discipline-specific expert from these respective departments
explained and presented the template, which led to new cooperations between lecturers and
media-producers. In the department Oe, the template has already been adapted to other in-
depth learning units in class. It has already been integrated into the course’s teaching concept.
In AI the implementation has started. The other two departments have declared a high interest
in using the escape game template.
5. Reflections on Experiences with the Pool of Discipline-Specific Experts

The development of digital teaching and learning in the last two years was shaped by
nessecities during the COVID-19 pandemic. Due to the support in digital teaching, the role
of TLCs changed and gained greater influence. TLC staff, whether media-didactic experts or
technical staff such as media producers, were particularly requested for solutions to technical
challenges, less for digital teaching methods and the development of digital teaching skills.
In this regard there is often still a lack of cooperation between lecturers looking for discipline-
specific teaching methods and media-didactic experts.
As the “heart” of the the GO-IN project of the HFD, the pool of discipline-specific experts
fosters digital teaching and learning by enhancing the cooperation between the departments
and the DLS media-didactic experts, media producers or other technical staff. The discipline-
specific experts particularly initiate informal cooperations which sometines segued into
formal cooperation structures. The experiences revealed that such a pool of discipline-
specific experts can build a ‘bridge’ between DLS staff by bringing in the knowledge about
681
the practiced communication culture of a department and the understanding of the relevant
teaching and learning methods in a specific discipline. However, the experiences in both
examples also demonstrate that formal cooperation by professorial support is highly
significant.
In a nutshell, the discipline-specific experts are aligned to the communication culture in the
department and as a pool of experts, they are therefore able to foster the cooperation between
lecturers of all departments and the DLS staff. Such an increase in personnel of the mid-level
staff of a university in discipline-specific consulting may therefore lead to sustainable
discourse and improvement in digital teaching and learning.
References
Karcher, S. & Nunn, C. (2021, November): Erfolgsfaktor „Kommunikation“ – Wie digitale
Lehre gelingen kann. Eine Reflexion über den Digitalen Lehrpreis in der Kategorie
„Engagierte Unterstützung der Lehrenden“. HINT, 2 (2021), 163-175.
Kraft, L., Merkator, N., & Pohl, H-M. (2021). Evaluationen zur Online-Lehre 2020/21.
Retrieved from University of Applied Sciences Fulda, Evaluation Web site:
https://express.adobe.com/page/dmobQEB2ysQop/?ref=https%3A%2F%2Fwww.hs-
fulda.de%2Funsere-hochschule%2Fa-z-alle-institutionen%2Fdienstleistungen-lehre-
und-studium%2Fevaluations-
news&embed_type=overlay&context=lightbox#evaluationen-zur-online-lehre-202021
Kohls, C. (2009). E-Learning-Patterns. Nutzen und Hürden des Entwurfsmuster-Ansatzes.
E-Learning 2009. Lernen im digitalen Zeitalter, Medien in der Wissenschaft.
https://doi.org/10.25656/01:3266
Kumbrick, C. (2001). Was ist Kooperation? Kooperation im Lichte der Tätigkeitstheorie.
Arbeit, 10(2), 149–166.
Pastoors, S. & Ebert, H. (2019). Bedeutung von Kooperation. In Psychologische Grundlagen
zwischenmenschlicher Kooperation (p. 1-5) Springer, Wiesbaden.
Pérez-Sanagustín, M., Iouri Kotorov, I., Teixeira, A., Mansilla, F., Broisin, J; Alario-Hoyos,
C., Jerez, Ó, do Carmo Teixeira Pinto, M., García, B; Kloos, C. D., Morales, M., Solarte,
M., Oliva-Córdova, L. M., & Gonzalez Lopez, A. H. (2022). Competency Framework for
Teaching and Learning Innovation Centers for the 21st Century: Anticipating the Post-
COVID-19 Age. Electronics, 11(3), 413.
Rogall, H. & Gapp-Schmeling, K. (2015). Homo heterogenus - das neue Menschenbild der
Ökonomie. In Wirtschaftsanthropologie. Nomos-Verl.-Ges., 2015.
682
(Digitally) transforming education in a large university

Iris Peeters1,4, Sylvia Grommen2,4, Hans Tubbax3,4
1
Group of Science, Engineering and Technology, KU Leuven, Belgium, 2Faculty of Science,
KU Leuven, Belgium, 3Faculty of Bioscience Engineering, KU Leuven, Belgium, 4Affiliated
with Leuven Engineering and Science Education Center (LESEC), KU Leuven, Belgium.
Abstract
The world is changing faster than ever and so is the context of higher
education. Students need to develop an agile mindset and corresponding
generic skills for the flexible workplace that awaits them and the many societal
challenges they face. Pedagogical and technological change can no longer be
postponed by higher education institutes. However, this (digital)
transformation is not an easy task.
This paper elaborates on the process of digital and educational transformation
at KU Leuven (Belgium), one of Europe’s oldest and largest research-based
universities, from a meta-perspective. Through several examples, challenges
and opportunities to trigger a shift in workforce, technology and culture are
explained. The applied network-based model of educational development,
pedagogical action research project methodology and informal initiatives
definitely play a crucial role.
Keywords: Digital transformation; higher education; management;
pedagogical action research; cultural change.

1. Introduction
The world is changing faster than ever and so is the context of higher education. Students of
today are not the same as students of yesterday or tomorrow, having different life, study and
career aspirations (Barhate, Dirani, 2022). At the same time, when manoeuvring through this
ever evolving society and work field, our ‘Generation Z’ students are expected to demonstrate
a fair amount of flexibility, with an attitude and generic skills set to match (Williams et al.,
2023). This was exaggerated and accelerated by the COVID-19 pandemic. Universities aim
to equip their students for this agile mindset with competences needed for a lifetime career
in this volatile, uncertain, complex, and ambiguous (VUCA) world, rather than traditional
content. They feel the urge for a learning evolution and “the development of updated and
agile curricula in […] higher education” as stated by the World Economic Forum (2016) and
become VUCA Universities as described by Korsakova (2020): validated, unique,
complementary, attractive. For this, a significant educational reform is necessary, which is
underpinned by several authors (e.g. Draskovicet al., 2020; Pup & Filep, 2021; Jackson &
Konczos Szombathelyi, 2022). However, this seems to be a very challenging ambition that
occupies higher education institutions worldwide. Moreover, where the rapid change in
educational technology offers tremendous opportunities to address the changing needs of
students, it also adds a complex digital component to educational transformation.
Change management and finding ways to consolidate results remain the most important
challenges in this process. Brooks and McCormack (2020) indeed state that full (digital)
transformation towards sustainable future-proof higher education requires shifts at different
levels: cultural change, workforce change and technology change.
Figure 1. Digital transformation in higher education related to Roger’s diffusion of innovation.

Sources: Brooks & McCormack (2020), Rogers (2003).
684
Figure 1 shows how this transformation is related to Roger’s diffusion of innovation (Rogers,
2003), evolving from a small group of innovators towards involving a majority of
stakeholders. This paper elaborates on the process of digital and educational transformation
at KU Leuven (Belgium), one of Europe’s oldest and largest research-based universities,
from a meta-perspective.
2. Methodology
2.1. Workforce change: network-based model of educational development and innovation

Peeters (2022) describes the dynamic model of discipline-specific educational development
at KU Leuven to implement centrally set policy priorities regarding educational innovation,
hereby ensuring alignment with bottom-up initiatives and translation to facultary contexts. In
this model, educational developers and an increasing number of educational technologists at
different levels are closely interrelated and all play an important role: at the faculty, group
(cluster of faculties) and university-wide (Figure 2). Peeters (2022) focuses on several group-
wide collaborative projects within the Group of Science, Engineering and Technology (SET
Group) on topics such as multicampus education, virtual mobility, MOOCs for Credit,
learning analytics, teaching assistant training and collaborative learning spaces. Additionally,
the launch of a SET Advisory Board Educational Innovation is mentioned as a way to allow
dialogue between vicedeans, other staff and students, increasing information exchange on
initiatives at different levels. In this context also communities of practice originate as typical
bottom-up approaches that are known to have a powerful effect on learning and teaching in
universities (Wenger-Traynor & Wenger-Traynor, 2015).
Figure 2. KU Leuven Learning Lab as a university-wide networking structure, involving educational developers
and technologists at different levels. The yellow dots represent project managers at the intermediate Group level,
other dots are staff members at the facultary and university level. The blue circle represents the Group of Science,
Engineering and Technology, in which LESEC operates as an additional network. Source:Peeters (2022).
685
The Leuven Engineering and Science Education Centre (LESEC), a research centre grouping
disciplinary teaching staff and educational developers in STEAM (Science, Technology,
Engineering, Architecture and Mathematics) with a focus on educational research and
practice, and KU Leuven Learning Lab (KULLL), a university-wide networking structure,
play a crucial rol herein.
2.2. Technology change: project-based approach including pedagogical action research

KU Leuven generally uses a project-based approach to allow defining clear goals that
translate the university’s vision on ‘Going Digital’ into concrete output, leading to
implementation and consolidation of results. Particular attention is given to the development
of transversal axes across study programmes related to expected (lifelong learning)
competences and international developments. All projects are intertwined with digital
transformation and many aim for a full implementation of technology. Principles of
pedagogical action research are applied in order to address real issues, modify practice as
well as contribute to theoretical knowledge. Hereby the cycle of observe - plan - act - reflect
is followed (Norton, 2009). Projects typically start small-scale, while tracking down and
involving the innovators. By making benefits visible through small-scale pilot studies the
group of early adopters is then expanded. Innovators may apply regularly for internal funding
to start such pilots, selection is done by a commission that links ideas to university wide
strategic aims. Consequently, the most successful projects in terms of positive evaluation by
involved students and teaching staff upscale towards being suitable for larger groups of
stakeholders, thus creating an early majority past the tipping point described by Rogers
(2003) (Figure 1). Principles of Sociocracy 3.0 (S3) are applied, which is an open source
framework of patterns and practices to collaborate more effectively in multidisciplinary non-
hierarchically structured teams (Priest & Bockelbrinck, 2017).
2.3. Cultural change: informal initiatives

Additionally to the more formal projects, several informal initiatives trigger cultural change
from the inside. For instance, members of the research centre LESEC have organized meet-
up events called ‘When ideas…’ since 2021 on topics such as learning analytics, learning
spaces, effectiveness in education, digital transformation, lifelong learning and flexible
curricula, assessment … Interested colleagues join these informal sessions to reflect and
discuss out-of-the-box, leaving comfort zones and sharing passion. Think-Pair-Share is used
as a cooperative discussion strategy, starting from a concrete question, as it is shown to be
effective to stimulate critical thinking and promote group participation (e.g. Deore & Arora,
2022). In some cases a ‘When ideas’ session leads to the reactivation of former projects,
concrete actions or publications in the LESEC news letter, hereby reaching more
stakeholders. Innovation and change is thus triggered bottom-up.
686

Initiatives and projects within the networking structure of educational innovation at KU
Leuven have shown to be effective in certain ways to facilitate aspects of (digital)
transformation. Especially workforce change and technology change are triggered, for
instance by breaking through hierarchical structures, involving different roles
collaboratively, hiring educational technologists, implementing blended learning including
adaptive learning paths, applying learning analytics … However, challenges become more
visible in this process and are mainly related to resistance to cultural change, expressed by
resistance at all levels to adapt traditional teaching and learning methods and organization.
In other words, resistance to cultural change acts as a catalyst of resistance to workforce and
technology change as well.
Based on some concrete exemplary cases and experiences this paper discusses both successes
and pitfalls related to the used methodologies.
3.1. Example: network-based LESEC team ‘Education Z’

LESEC organizes its research, development and innovation in several themes based on the
expertise, experience and interest of members. For each theme a team of LESEC members is
formed, whereby one member performs the role of coordinator and several subthemes are
defined in co-creation. The theme ‘Education Z’ focuses on creating learning experiences for
generation Z students that encourage them to become 'global citizens' with the necessary
knowledge, skills and motivation to thrive in their future career path and beyond. Global
competencies, active learning, collaborative learning spaces, online learning and assessment,
teams across borders and gerenation Z are current subthemes. In about four meetings a year
the team discusses insights from different perspectives about the ideal learning environments,
learning formats and assessment. This initiates more formal cooperation and actions e.g. by
setting up a master’s thesis or other project together, or by organizing quantitative and
qualitative data collection as a team.
This example shows that a network-based approach such as LESEC truly triggers exchange
and cooperation across boundaries. The fact that stakeholders with different backgrounds and
functions, but with common interests, are brought together induces initiatives that would
otherwise not originate or would be much smaller scale and less interdisciplinary. However,
the meso- and meta-level perspective also complicates certain processes. For instance, the
idea arose to start-up a co-creation project with students but it seemed unfeasible to do this
as a mutual initiative. Instead, several small-scale projects will run within different sub-
themes and Education Z will bring results together.
687
3.2. Example: towards implementing learning analytics with a collaborative project

One of the group-wide educational innovation projects of the SET Group concerns using
generated data by online learning material for increasing the efficiency and effectiveness of
learning (Tubbax & Peeters, 2023). In close collaboration with different stakeholders at all
levels of university, data are collected, analyzed and interpreted in order to give motivational
and action-oriented feedback to didactic teams. It soon became clear that the process of
making learning analytics scalable and sustainable is closely related to the digital
transformation of education at KU Leuven. The simultaneous migration towards a new
Learning Management System (LMS) made this symbiosis even more visible. However, it
seems very hard to convince a larger group than the innovators of the advantages of learning
analytics.
This example shows that a technology shift towards implementing learning analytics is
challenging but feasible, as long as all relevant stakeholders are involved. This is possible
within the framework of a project. However, expanding the group of actual users to an early
majority requires a slow and gentle process of repeated PDCA-cycles and action research
techniques. The tipping point in Figure 1 between early adopters and early majority, or
‘chasm’ as already described by Moore (1991), definitely is the biggest challenge. Generating
sufficient ‘proof’ for a ‘proof of concept’ is key to convince more stakeholders, while a clear
governance and policy is necessary to specify the framework. Tubbax & Peeters (2023)
further elaborate on this.
3.3. Example: learning spaces ‘ninja’ action to trigger cultural change

During a LESEC ‘When ideas…’ session on learning spaces in February 2022, the idea arose
to not wait for new high-level collaborative learning spaces to be developed, but to change a
traditional seminar room into a low-level active learning classroom simply by moving
furniture to a group-work arrangement. The ambition was to break the vicious circle
described by Peeters & Binnard (2018): the lack of use of innovative teaching practices
reduces the demand for new and flexible rooms, which reinforces the traditional habits of
teachers. This ‘ninja’ action bypassed KU Leuven procedures, but by organizing a close
follow-up of didactical teams through information charts and a local contact person, the plan
worked. Locally, lecturers are now asking for a second, similar active learning classroom.
Centrally, this is picked up by the involved stakeholders and triggers structural change in
available learning spaces.
This example shows that an informal brainstorm session can induce disruptive bottom-up
change, causing a complex interplay between different levels involved, leading to related top-
down actions. Informal actions can thus help to lower resistance to cultural change. However,
effects remain prone to the goodwill of a few innovators and need time.
688
4. Conclusions
Higher education can no longer avoid a disruptive innovation due to the emergence of online
learning as well as the evolved expectations of society concerning skills and attitude of
graduated students (Christensen & Eyring, 2011). However, this paper confirms that (digital)
transformation in a higher education institution is a challenging process and shares some
insights from KU Leuven (Belgium). Several examples show that a workforce and
technology change can be induced by a coordinated, project-based approach in which
different levels are involved and coordinated. Pedagogical action research is a valuable
method to feed practice as well as theory. However, a culture change is most difficult and
requires more than a project-based approach. After all, people, cooperation and willingness
to change became more important than following a plan. The combination of bottom-up and
top-down initiatives in one continuous PDCA-cycle is crucial, aiming for mutual goals. The
world is changing faster than ever and so should the context of higher education.
Acknowledgements
The authors would like to thank the Group of Science, Engineering and Technology and
LESEC for the networking opportunities and their financial support. The involved
stakeholders are thanked for their substantial contribution to educational innovation
initiatives that facilitate the digital transformation process at KU Leuven.
References
Barhate, B. and Dirani, K.M. (2022). Career aspirations of generation Z: a systematic
literature review. European Journal of Training and Development, 46(1/2), 139-157. doi:
10.1108/EJTD-07-2020-0124.
Brooks, C. D. and M. McCormack (2020). Driving Digital Transformation in Higher
Education. Louisville: CO, ECAR.
Christensen, C. M. & Eyring, H. J. (2011). The innovative university: Changing the DNA of
higher education from the inside out. San Francisco: Jossey-Bass.
Deore, M. & Arora, S. (2022). Effective Think-Pair-Share Pedagogical Strategy to Improve
Inferential Statistics Concept Understanding. Journal of Engineering Education
Transformations, 36(1), 25-30.
Draskovic, V., Jovovic, R., & Rychlik, J. (2020). Perceptions of the declining quality of
higher education in the selected SEE countries. Journal of International Studies, 13(4),
286-294. doi:10.14254/2071-8330.2020/13-4/20.
Jackson, K., & Konczos Szombathelyi, M. (2022). The influence of COVID-19 on sentiments
of higher education students - prospects for the spread of distance learning. Economics
and Sociology, 15(3), 226-256. doi:10.14254/2071789X.2022/15-3/13.
689
Korsakova, T. V. (2020). Higher education in VUCA world: New metaphor of university.

European Journal of Interdisciplinary Studies, 6(1), 93-100. doi: 10.26417/ejis-
2019.v5i2-282.
Moore, G. A. (1991). Crossing the Chasm: Marketing and Selling High-Tech Goods to
Mainstream Customers. New York: Harper Business.
Norton, L. (2009). Action research in teaching and learning: A practical guide to conducting
pedagogical research in universities. Routledge.
Peeters, I. (2022). Stimulating educational innovation in STEAM higher education: an
interdisciplinary collaborative true story. 14th International Conference on Education
and New Learning Technologies (EDULEARN'22), 4356-4361. doi:
10.21125/edulearn.2022.1040.
Peeters, I. & Binnard, E. (2018). Collaborative learning in space and time: bridging the gap
with ALINA. 10th International Conference on Education and New Learning
Technologies (EDULEARN'18), 7090-7099. doi: 10.21125/edulearn.2018.1674.
Priest, J. & Bockelbrink, B. (2017). Sociocracy 3.0. Retrieved from http://Sociocracy30.org.
Pup, Zs., & Filep, B. (2021). The impact of global socio-economic changes on the regional
role of universities. Economic Annals-XXI 190, 5-6(2), 33-47. doi:
https://doi.org/10.21003/ea.V169-04.
Rogers, E. M. (2003). Diffusion of Innovations: New York, Free Press.
Tubbax & Peeters, submitted. The symbiosis between learning analytics and digital
transformation in higher education. 9th International Conference on Higher Education
Advances (HEAd’23).
Wenger-Trayner, E. and Wenger-Trayner, B. (2015). An introduction to communities of
practice: a brief overview of the concept and its uses. Available from authors at
https://www.wenger-trayner.com/introduction-to-communities-of-practice. Online.
Williams, K.M., Wang, T., Holtzman, S., Leung, T.M., Cherfrere, G. & Ling, G. (2023).
Employer Expectations of 21st-Century High School Graduates: Analyzing Online Job
Advertisements. ETS Research Report Series. doi: 10.1002/ets2.12365.
World Economic Forum (2016). The future of jobs: employment, skills and workforce
strategy for the Fourth Industrial Revolution. World Economic Forum: Geneva.
http://www3.weforum.org/docs/WEF_Future_of_Jobs.pdf.
690
Enhancing online teaching: addressing the challenges faced by

early-career academics at Vietnam National University, Hanoi
Nghiem Xuan Huy1, Tran Thi Hoai1, Le Thi Thuong1,2, Nguyen Thai Ba1,2, Tuan Thu
Trang3
1
VNU Institute for Quality Assurance, Vietnam National University, Hanoi, 2PhD student
of Department of Sociology, VNU University of Social Sciences and Humanities, Vietnam
National University, Hanoi, 3VNU University of Education, Vietnam National University,
Hanoi., Hanoi
Abstract
The Covid-19 pandemic has posed significant challenges to teaching staff at
universities, particularly in the realm of online teaching. This study evaluates
the challenges faced by early-career academics at Vietnam National
University, Hanoi (VNU) in adapting to online teaching. A survey of 804
lecturers at VNU reveals that while they recognize the advantages of online
teaching and hold a favorable perspective towards it, they face challenges
related to proficiency in using new teaching technologies and learner
motivation. Based on the findings, this study proposes some recommendations
for early-career academics at VNU to address these challenges and enhance
their online teaching.
Keywords: Online teaching; online learning; educational technologies;
quality assurance; higher education; professional development.

Enhancing online teaching: addressing the challenges faced by early-career academics
1. Introduction
The advancement of science and technology has had a tremendously positive impact on
various aspects of society and holds a significant role in various fields with practical
applications. Technology has greatly benefited education by providing easy access to
academic resources, textbooks, and research materials online, eliminating the need for
physical libraries. Research indicates that online teaching and learning are increasingly
perceived positively (Thai & Tran, 2022). At the VNU, which encompasses 34 units
including 9 member universities, 3 affiliated schools, and 3 training centers, diversity is a
hallmark characteristic among its learners, who possess varying characteristics, needs,
perceptions, and concerns. Despite the growing use of technology and active teaching
methods at VNU, there is a lack of extensive research studies with a sufficient number of
subjects to accurately determine the impact of technology on active learning among learners.
Given the aforementioned reasons, this research aims at identifying the current status and
those challenges that early-career lecturers deal with in teaching online at VNU, and
proposing some solutions to help improve the quality of online instruction. The research
employed a quantitative data analysis approach, utilizing a sample of 804 responses from
VNU educators. The analysis took into account various factors including gender, teaching
field, teaching seniority, educational level, the country where the highest degree was
obtained, and the faculty unit at VNU.
The literature on online teaching suggests that incorporating technology into teaching
activities has various benefits, but limited in-depth investigations into the quality of online
teaching have been conducted. Online teaching involves instructional activities facilitated
through technology platforms in an online environment, differing from traditional, in-person
teaching in physical classrooms. Globally, research on active learning and the integration of
educational technology in teaching has been extensive, with a majority of studies being
empirical or qualitative in nature and focusing on specific training areas. Studies have been
conducted to enhance online teaching, with some utilizing analytical methods to clarify the
nature of online teaching activities in schools, while others have employed meta-analyses to
explore the impact of technology on students' learning in online classes. For instance,
research has shown that technology creates a more positive learning experience compared to
non-technology use (Le et al, 2022; Muhtadi, 2013; Tamim et al., 2011; Thai & Tran, 2022).
Studies indicate that technology has a generally positive impact on learning, with students
finding it appealing compared to traditional materials. However, selective and strategic use
of technology is necessary to avoid distractions and ensure effective collaborative learning
in online environments. Traditional classroom settings may be more effective for certain
692
Nghiem Xuan Huy, Tran Thi Hoai, Le Thi Thuong, Nguyen Thai Ba, Tuan Thu Trang
tasks compared to distance learning. Additionally, integrating technology effectively into the
learning process may require significant investments in infrastructure and resources.
Technology in online teaching can enhance student interaction during initial stages,
collaborative activities, and post-classroom learning. It can also improve teaching
effectiveness, increase access to global students, offer flexible scheduling, reduce costs, and
decrease environmental pollution. Studies have shown increased student scores in the first
year of implementation. However, limitations of technology in education include increased
opportunities for cheating, decreased attention span, imagination, and thinking ability among
students, as well as increased time, cost, and health-related burdens for lecturers. (McLaren
et al., 2022; Raja & Nagasubramani, 2018; Viorica-Torii & Carmen, 2013; Wang, 2020).
The enhancement of online education quality necessitates the implementation of robust
training programs and knowledge dissemination initiatives for instructors and students. This
entails upgrading IT infrastructure, improving students' IT competency, and providing
training for educational leaders and instructors on modern teaching techniques, scientific and
technical innovations, and digital technology in education (Le & Truong, 2020; Thai & Tran,
2022). Further research is warranted to gain a comprehensive understanding of the challenges
associated with integrating technology in online teaching and its impact on training quality.
Evidence-based solutions are required to enhance educators' skills, improve students' learning
outcomes, and address the dearth of studies in this domain. In this regard, VNU has
formulated guidelines to ensure high-quality online training that aligns with the quality
assurance requirements of the Government of Vietnam and VNU. These guidelines
encompass various aspects, including teaching content, teaching and learning activities,
testing and assessment, technical infrastructure, and learning resources, providing an
effective framework for the implementation of online teaching activities.
3. Research Methods
Data used in the study was collected through an online survey for new faculty members when
they participated in a training program on “New Teaching Methods and Application of
Information Technology in Teaching for Lecturers at VNU” from 2021 to 2022. The total
number of questionnaires collected was 903, after eliminating any disqualified responses, the
data collected from 804 faculty members were analyzed to address the research topics. The
survey data was processed and analyzed using Microsoft Excel and SPSS software.
In terms of the sample structure, the research team conducted a classification based on several
demographic variables including gender, field of teaching, teaching seniority, level of
education, country where the lecturer obtained their highest degree, and unit. Analysis of the
gender distribution in the sample revealed that the proportion of female participants (67.9%)
was more than twice that of male participants (32.1%). In relation to teaching seniority,
693
17.4% of the lecturers had 3-5 years of experience, 43.8% had 1-3 years of experience, 30%
had 6 months to 1 year of experience, and 8.8% had less than 6 months of experience. With
regard to education level, more than half of the lecturers held a Master's degree (50.6%),
while 38.3% held a Ph.D. degree, and the remaining 11.1% held a Bachelor's degree. In terms
of the country where the highest education degree was obtained, 62.6% of the lecturers
obtained their highest degree in Vietnam, 34.1% obtained it abroad, and the remaining 3.3%
provided invalid responses (see Figure 1 below).
12,70%
VNU University of Languages and International Studies
1,60% VNU University of Economics and Business
1,70% 28,40% VNU University of Engineering and Technology
1,90%
VNU University of Medicine and Pharmacy
2,00% VNU University of Science
4,40% VNU University of Education
VNU University of Social Sciences and Humanities
6,10% VNU School of Interdisciplinary Studies
VNU University of Law
8,30% VNU Center for National Defense and Security Education
15,70% VNU International School
8,60% Other units in VNU
8,70%
Figure 1: Description of the sample in units.
4. Online teaching at VNU: the current situation and challenges

The findings of this study reveal that although lecturers are aware of the advantages of online
teaching and express willingness to utilize this mode of instruction, they encounter challenges
related to their proficiency in using new teaching technologies and learner motivation. The
results of the research provide insights into how instructors utilize tools for online teaching,
their awareness of online teaching, and the challenges they encounter when working with
students in a virtual environment.
First of all, Table 1 shows how lecturers use teaching online tools in their work.
The research used Yes/No questions to gather information from instructors. The survey
results indicate that video conferencing, social media, online video, online quizzes, and
mobile applications are the most frequently utilized tools by lecturers, with over 60% of them
employing these tools. These tools were also frequently utilized during the COVID-19
pandemic when teaching and learning had to be entirely conducted online. However, some
other tools such as peer review tools and ePortfolios, are underutilized with less than 10% of
the surveyed lecturers using these tools in their classrooms. Another tool deemed necessary
for teaching, especially in online or blended teaching, is the learning management system
(LMS), which has not gained widespread popularity among lecturers at VNU. Only 41.7%
of lecturers reported using this tool during their teaching. This may be due to the recent
development of VNU's LMS system and its lack of synchronization with the information of
694
all training units, making it difficult for lecturers of some units to access and understand how
to use the system with their modules.
Table 1. Frequency of using tools/software in online teaching by lecturers.
Tool/software has used for Frequency Percent Tool/software has used for Frequency Percent
online teaching online teaching
Video conferencing 645 80.2% Online interactive activities 339 42.2%
Social media 583 72.5% LMS 335 41.7%
Online video 540 67.2% Wikis 121 15.0%
Online Quizzes 535 66.5% Blogs 103 12.8%
Mobile application 507 63.1% Courses from eBooks or 88 10.9%

iTunes
Online discussion forum 374 46.5% Peer review tools 58 7.2%
Online collaboration tools 357 44.4% ePortfolios 31 3.9%

Furthermore, the lecturers affiliated with VNU exhibit a clear understanding of the
advantages associated with delivering courses online. In addition to comprehending the
benefits offered by online teaching, the teaching staff demonstrates preparedness in terms of
their practical skills and teaching methods, as elucidated in the subsequent table:
Table 2: The awareness and practices in online teaching for lecturers.
Items Frequency Percent
I understand the benefits of using technology in teaching so I can make informed 777 96.7%
decisions about when and where to use technology in my teaching
I know how to evaluate the online teaching activities and tasks that I have/will 686 85.3%
be implementing
I've been exploring new technologies to integrate into my teaching and feel it's 639 79.5%
appropriate and doable
I am sure about which learning activity or task is best for online teaching 635 79.0%
I feel that learners are more interested in learning when technologies are 612 76.1%
integrated in my teaching activities, which encourages me to try applying new
technology to teaching activities
Online teaching is right for me 590 73.4%
I have a clear understanding of classroom assessment requirements when 573 71.3%

transitioning to an online teaching context
695
The results of the survey indicate that a high percentage of the 777 surveyed lecturers (96.7%)
expressed their understanding of the benefits of online teaching and their proficiency in
utilizing technology in their teaching activities. Likewise, 666 lecturers (85.3%) reported
having the ability to evaluate online teaching activities and tasks, while 639 (79.5%) claimed
to have successfully integrated new teaching tools into their instruction. Furthermore, 612
lecturers (76.1%) noted that the use of technology in teaching was more motivating for both
themselves and their learners, and 590 (73.4%) considered online teaching as a suitable
approach for themselves. These results indicate the willingness of lecturers to adopt new
technologies in their teaching, however, the successful implementation of such technologies
is still hindered by certain challenges. The survey findings highlight certain problems that
have impeded the lecturers in effectively integrating technology into their teaching activities.
Finally, many challenges in teaching online have been faced by teaching staff, as follows:
Table 3: Challenges for teachers in online teaching.
Challenges Frequency Percent
My ability to access and use online technologies is not good 447 55.6%
Learners lack interest and interest in my online lectures 390 48.5%
I'm not sure if the learning outcomes in online teaching are the same as those 362 45.0%
in face-to-face teaching
Lack of support on mechanisms and policies from VNU and training 344 42.8%
institutions
I don't know how to properly search for and use online content or resources 300 37.3%
I don't have a good understanding of effective pedagogical methods in online 282 35.1%
teaching
It took me too long to prepare for the online lesson 278 34.6%
The survey results indicate that a substantial proportion of faculty members encounter
difficulties when implementing online teaching methods. Specifically, 55.6% of the lecturers
reported challenges in accessing and utilizing online technologies. The attitude of learners
also posed a challenge for 48.5% of the lecturers in successfully implementing online
teaching methods. Additionally, a lack of experience in online teaching (45% respondents)
and insufficient support from VNU regarding policies and mechanisms (42.8% respondents)
are significant barriers to the implementation of online teaching methods by the faculty. It is
evident that while the lecturers at VNU have embraced the use of new technology, support
from the university and active participation from learners are critical factors for the successful
implementation of these teaching methods and technologies. The collaboration between
696
schools, lecturers, and learners will be instrumental in ensuring the success of teaching
innovations and improving the quality of learning experiences for learners.
5. Recommendations to improve the quality of online teaching

In light of the aforementioned findings, the research suggests three levels of implementation
to address the challenges encountered by early-career lecturers in online teaching, as follows:
At VNU level, it is recommended that guidelines, regulations, and mechanisms be developed
to facilitate the implementation of online and blended teaching practices across all member
institutions. This could include the development of comprehensive guidelines and regulations
that outline the best practices for online and blended instruction, as well as the establishment
of mechanisms to monitor and evaluate the effectiveness of these practices. Additionally, it
is suggested that regular seminars and professional development workshops be conducted to
support faculty in acquiring the necessary skills and knowledge to effectively utilize methods
and tools for online and blended instruction. These initiatives could provide faculty with the
necessary resources and support to enhance their competency in online teaching and ensure
the quality of instruction in the online and blended learning environments.
At VNU's member institution level, it is suggested to implement regulations and guidelines
specifically tailored for online and blended learning, with a particular focus on new faculty
members. These regulations and guidelines could provide a framework for new faculty
members to effectively engage in online and blended teaching practices, and could include
incentives to encourage their active participation in adopting innovative teaching methods
and educational technologies. Additionally, it is recommended that VNU's member
institutions establish mechanisms for monitoring and assessing the teaching activities of
early-career lecturers, with the aim of improving the quality of instruction. This could involve
regular evaluations of online teaching performance and feedback to help lecturers enhance
their teaching practices. Furthermore, it is crucial to establish a technical team with expertise
in educational technologies to provide support and guidance to new and novice instructors in
utilizing new teaching methods and technologies effectively. This could include offering
training programs, workshops, and ongoing technical assistance to ensure that faculty
members are equipped with the necessary skills and knowledge to effectively integrate
technology into their teaching practices.
At the individual level, it is encouraged for lecturers to actively engage in continuous
professional development opportunities to acquire and apply innovative teaching methods
and technologies in their instructional practices. This could include participation in training
sessions, seminars, and workshops that focus on the advancement of modern teaching
methods and technologies. Moreover, it is recommended that instructors introduce the
teaching methods and technologies they plan to utilize in advance to their students, and
697
encourage students to familiarize themselves with these tools. Gathering feedback from
students on the effectiveness of these methods and technologies can provide valuable insights
for instructors to refine their teaching practices. In addition, lecturers should engage in
regular self-reflection and evaluation of their teaching activities to promptly identify areas
for improvement and strive to enhance the overall quality of instruction. This could involve
critically examining instructional strategies, student engagement, and learning outcomes, and
making adjustments as needed to optimize the learning experience for students.
6. Conclusion
The integration of technology in online teaching has brought a multitude of benefits to early-
career academics in their pedagogical pursuits, and a majority of these lecturers are cognizant
of these advantages. Nonetheless, each lecturer still encounters at least one challenge (the
challenges of slacking proficiency in the utilization of new teaching technologies and the lack
of motivation among learners pose difficulties for online teaching) when delivering their
lessons online. To enhance the quality of online teaching for early-career academics at VNU,
it is imperative to have a coordinated effort from both the management and the educators
themselves. The management should establish appropriate policies for implementation,
support, and supervision to create an optimal environment for online teaching. Additionally,
the VNU Center for Teaching Excellence should play a proactive role in supporting and
encouraging early-career academics to actively explore and incorporate innovative teaching
methods into their online teaching practices.
References
Le, A. V., Dang, T. T. H., Bui, T. D., Vuong, Q. A., Phung, T. T. T, Do, D. L., Tran, T. T. B.
N. (2022). “Online teaching practice of Vietnamese high school teachers during the
COVID-19 pandemic: Analytical results from a large-scale survey”. Vietnam Journal of
Educational Sciences, 18(04), 1-11. doi: https://doi.org/10.15625/2615-8957/12210401.
Le, V. T., Truong, T. D. (2020). “Some solutions to improve the effectiveness of online training
in Higher Education in the context of the industrial revolution 4.0,” Vietnam Journal of
Educational Sciences, special(2), 33-36.
Muhtadi, M.A. (2013). The Impact of Technology on Education. AL-Nasser University
Journal. 1(1), 1-7. ISSN 2311-2360
McLaren, B. M., Richey, J. E., Nguyen, H., & Hou, X. (2022). How instructional context can
impact learning with educational technology: Lessons from a study with a digital learning
game. Computers & Education, 178, 104366.
Raja, R., & Nagasubramani, P. C. (2018). Impact of modern technology in education. Journal
of Applied and Advanced Research, 3(1), 33-35.
Statistics at Sciencedirect, Retrieved 23 August 2022, from https://www-sciencedirect-
com.db.lic.vnu.edu.vn/search?qs =active%20learning.
698
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What
forty years of research says about the impact of technology on learning: A second-order
meta-analysis and validation study. Review of Educational research, 81(1), 4-28.
Thai, L. H., Tran, A. T. (2022). “Improving the quality of e-learning at Thai Nguyen University
of education”, TNU Journal of Science and Technology, 227(06), 34–42.
Viorica-Torii, C., Carmen, A. (2013). The impact of educational technology on the learning
styles of students. Procedia-Social and Behavioral Sciences, 83, 851-855.
Wang, Y. H. (2020). Design-based research on integrating learning technology tools into higher
education classes to achieve active learning. Computers & Education, 156, 103935.
699
Steps towards enabling health professionals through future skills

Yvonne Sedelmaier1, Dieter Landes2
1
Vocational Education, SRH Wilhelm Loehe University of Applied Sciences Fuerth,
Germany, 2Department of Informatics, University of Applied Sciences Coburg, Germany.
Abstract
Education in healthcare must enable professionals to work in the health sector
for as long and as fulfilled as possible. Yet, new requirements are constantly
arising in the health sector, which continuously change the required
competences, e.g. due to new technological possibilities and increasing
interdisciplinarity. Several challenges arise: education in healthcare needs
awareness of required competences and their rapid change. At the same time,
addressing them in education presupposes an in-depth understanding of what
they actually are.
To tackle these issues, a teaching concept was developed that builds on self-
reflection of to-be professionals in healthcare. This concept includes
characterizing typical professional situations and deducing required (future)
competences for mastering these situations.
Beyond rising awareness to future skills, applying this teaching concept also
yields data that support a better understanding of required competences and
their importance across professions. A case study resulted in initial
competence profiles for several professions in healthcare.
Keywords: Future skills; teacher education; health; vocational education;
competence assessment; self reflexion.

Enabling health professionals through future skills
1. Introduction
Societies are getting older and health is a key aspect for maintaining quality of life. Education
in the health sector must enable health professionals to work as long and as fulfilled as
possible. In addition, requirements constantly evolve through megatrends such as digital
transformation, thus continuously changing the required competences of professionals, e.g.
due to new technological possibilities and increasing multidisciplinarity. So-called future
skills are gaining importance. Consequently, already the training of professionals and trainees
must pay attention to these future skills. In this context, student teachers play a central role,
as they must already address these future competences in the training of future professionals
for sustainably and permanently securing health care in an ageing society. To aggravate
things, the competences required in the future may not even be known yet. In particular, this
holds true for those of health professionals and their teachers.
The central research questions are derived from these considerations: What are relevant
future skills in health professions? How can student teachers be supported in addressing
future skills in their classes?
This paper contributes to answering these research questions through a teaching concept that
addresses both aspects jointly: student teachers are encouraged to increase their awareness of
future skills, while the future skills they identify shed a light on relevant competences for
health professions. The resulting documents will be analysed applying Grounded Theory
(Glaser & Strauss, 2009) to develop health competence profiles. As it turns out, competence
profiles have a high overlap across various professions in the health sector, yet emphasis on
specific competences differs largely between professions and several competences are
specific to particular professions.
The paper clarifies terminology and underlying concepts, before section 3 gives an overview
of the learning approach which serves two functions: collection of data on health future skills,
and making future skills explicit for future health teachers. Section 4 presents and discusses
some of the results of applying the concept. Section 5 provides a scientific underpinning of
the teaching concept. Section 6 summarizes the paper and gives an outlook to future work.
2. Clarification of terms: Competences and Future Skills

At present, there are hardly any systematic and methodologically sound studies on what
future competences do exactly mean, especially since they may differ significantly depending
on the field and region, e.g. between software professionals require and nurses or social
workers (Sedelmaier & Landes, 2015).
The concept of competence as such has been used in social and educational sciences for more
than 50 years, but is still very fuzzy and partly contradictory (Klieme & Hartig, 2007), thus
702
it is theory-relative (Erpenbeck & Rosenstiel, 2003). The starting point for discussions on the
concept of competence in educational science is the triad of social, self, and subject
competence introduced by Roth (1971). This triad is also reflected in Weinert's general,
cross-cutting concept of key competences (2001). Similarly, Erpenbeck (2003) distinguishes
four competence classes that can be broken down into a variety of sub-competences.
Various models agree that competences are a dispositional concept. They are tied to a person,
are based on personal characteristics (Erpenbeck & Rosenstiel, 2003), and aim at self-
organized action. Competences "are founded by knowledge, constituted by values and
attitudes, dispositional as abilities, consolidated by experience, and realized on the basis of
will or motives" (Maag Merki, 2009, p. 494). For Rhein and Kruse (2011, p. 80), the core
theoretical idea is that "it is the specific inter-play of knowledge, skills, abilities, personal
characteristics, experiences, and motivational structures {...} that constitute a competence,
without it being possible to reduce it to its individual components, although the description
of competences must always draw on these building blocks".
Competence is manifested in performance (Arnold, 2002, p. 31), which is interpreted
situationally or subject-specifically in almost all concepts of competence (Klieme & Hartig,
2007). This suggests that action competence is not only person-specific, but also situation-
specific and thus occupation-specific. Consequently, each occupational profile requires its
own competence profile, including general generic, occupational generic, and subject-
specific competences (cf. e.g. (Maag Merki, 2004)). Thus, competence profiles are not easily
transferable to other job profiles due to their subject-specificity.
The term “future competences” is difficult since competences as such are oriented towards
future challenges, without need for the word “future” to express this (Ehlers, 2020). Yet, all
definitions agree that competences are occupation-specific and continue to gain importance.
Future competences are often still unknown in detail, which is also true in healthcare.
3. Learning and Teaching Approach

Our learning concept for student teachers integrates identifying and understanding future
competences of health professionals in a self-reflexion task which aids both research
questions: qualitative data on required future competences of health professionals and
teachers are collected, while student teachers reflect on future skills and their integration in
their own classes later on. This is accomplished as outlined subsequently.
3.1. Intended learning outcomes

The learning approach focusses on three intended learning outcomes: First, student teachers
are aware of the importance of future skills. Second, student teachers are capable of recogni-
zing future skills and, third, of developing ideas how to address future skills in their classes.
703
Furthermore, we collect data for deriving a competence profile for health professionals
through qualitative reseach methods in three steps:
a. As a prerequisite, an understanding needs to be gained of how career reality in health
is changing and which competences will be required in future.
b. How can we gain awareness at future teachers about the competences and the
changing of the competences? How can we support student teachers to address these
potentially changing future competences in their classes?
c. In parallel, collected data are analysed by the instructor in order to derive a
competence profile for health professionals and their teachers.
3.2. Learning Approach

To that end, two exercises are used.
Exercise A: Characterize everyday professional situations

In order to introduce student teachers to the task, they are requested to describe everyday
professional situations and a typical working day including current and future tasks. These
descriptions should be profession-specific (e.g. for nursing, emergency medical services, or
surgical technology). Results are shared within the course for mutual inspiration.
Exercise B: Describe requried (future) competences

Based on these typical professional situations and tasks, students shall then derive competen-
ces that they currently need and will also need in the future for mastering the described
professional situations well. Attention should be paid to both professional and multidiscipli-
nary competences. It is important to develop “thick” descriptions and not just catchwords in
order to gain a detailed understanding of the competences. Furthermore, students should
focus on whether and how the required competences might change in the future. Students
work individually on these competence descriptions and hand their results in to the instructor.
3.3. Competence Descriptions for Systematic Self-Reflection

The instructor merges the resulting competence descriptions occupation-specifically using a
qualitative research approach that follows grounded theory (Glaser & Strauss, 2009). The
aim is to identify central competences, elaborate occupational specifics, and present them in
comprehensible, in-depth competence profiles. These competence profiles are the basis for
evaluating them in a competence assessment approach (CAT), which is intended to support
students' self-reflection in a goal-oriented way (Sedelmaier & Landes, 2014).
In CAT, the superordinate subject-specific and interdisciplinary competences are arranged
on three competence levels. Concrete partial competences are derived from each of these,
which form the basis for a systematic and repeatable self-assessment. Furthermore, CAT can
704
also be used to draw conclusions about the success of a particular teaching-learning concept
with regard to the achievement of the competence goals.
4. Results
This teaching approach has been implemented as a case study in an introductory course in
pedagogy of a part-time bachelor program of vocational education in healthcare at SRH
Wilhelm Loehe university. It trains future vocational teachers in healthcare. The case study
resulted in 21 student documents specifiying future skills in healthcare. 4 out of these 21
documents dealt with nursing-specific future skills, 4 with those for surgical assistants, 5
documents focused on the emergency medical services, and 8 took the perspective of
healthcare teachers. Documents were analyzed following Grounded Theory (Glaser
& Strauss, 2009) and coded with a total of 593 codes which were grouped into clusters (see
fig. 1 and 2).
4.1. Competence Profiles in Health Professions

In addition to technical and functional competencies (Rauner, 2011), empathy as well as a
structured approach and the ability to quickly grasp and link complex situations are of
particular importance in the healthcare sector. Another core competence is the ability to deal
with diverse stakeholders, such as multidisciplinary cooperation with other medical staff or
patients, relatives, or police and fire departments, etc. (see fig. 1, in german). Of particular
importance, especially for surgical assistants, is direct self-reflection and recognizing one's
own limits, while for emergency paramedics communicative aspects and for nurses empathy
are particular focal points.
Figure 1. Word Cloud of Future Skills for Health Professionals (in german).
705
4.2. Future Competences for Teachers in Health Education

Fig. 2 shows future competences which teachers in health education need. Though most of
these competences are also contained in fig. 1, different emphasis can be observed: Teachers
focus on methodological competences and the use of digital media in the learning process.
They see a special need for supporting pupils’ learning. Furthermore, they exhibit a higher
openness for learning. On the other hand, empathy has different detailed aspects, and
communication skills are less important for them.
Figure 2. Word Cloud of Future Competences of Teachers in Health Education.
5. Scientific Basis
In the learning context, competence profiles reflect the teaching-learning goals to be
achieved. In pedagogy, it is generally accepted that transparent teaching goals and the
negotiation of or agreement onTh teaching-learning goals significantly aids learning success
(Arnold & Erpenbeck, 2014). Numerous didactic concepts emphasise the importance of
competence profiles for planning and implementing learning processes (Arnold, 2015; Rhein
& Kruse, 2011). The necessity of formulating goals is also undisputed in pedagogy (Mager,
1992). For this purpose, we made our student teachers reflect on future competences of their
health profession and of their role as a teacher for these professions. This approach clarified
the intended learning outcomes for both teachers themselves and their subsequent pupils.
6. Conclusion and future work

Education in healthcare must enable professionals to work in the health sector for as long and
as fulfilled as possible. Yet, new requirements are constantly arising, which are continuously
changing the required competences of healthcare professionals. Consequently, education in
healthcare needs to be aware of required competences and their rapid change, while
addressing them in education presupposes an in-depth understanding of what they really are.
706
To that end, a teaching concept was developed that largely builds on self-reflection of to-be
teachers in healthcare. in particular by characterizing typical professional situations and
deducing required (actual or future) competences for mastering these situations. A first
implementation of the approach as a case study resulted in material from 21 students which
was analysed using Grounded Theory, resulting in 593 detailed codes. Clustering the data
from the 21 documents reveals competence profiles that have a large overlap in terms of
common competences, yet with different emphasis across professions. At the same time,
rising awareness with respect to future skills could be observed among student teachers.
By going through this learning concept, future health teachers get first ideas how to address
future skills in their classes later on. They could learn from models and get methodological
inspirations for their own classes. The learning approach reminded them of the importance
of learning goals, their explicit discussion with learners and the fuction and effectivity of
self-reflexion. These aspects are helpful for student teachers.
Nexts steps will include the collection of more data to consolidate the initial findings from
the case study. Results will be compared to further competences profiles, e.g. from technical
disciplines like Software Engineering. Some main differences in the competences profiles
will be highlighted in order to (further) develop domain-specific learning approaches and
subject matter didactics.
Then, the teaching cocepts will be elaborated further in order to enable future teachers to
recognise future competences and address them in their classroom. Here competence
assessment is an important aspect. A common assessment approach is self-reflection, which
is seen as a central aspect for learning (Henschel, 2019; Kauffeld, Reinecke, & Hennecke,
2009). Self-reflection is often implemented via learning diaries, but systematic, generalizable
evaluation is rare. This is where CAT (Sedelmaier & Landes, 2014) provides a connecting
bracket with its underlying competence profile. In addition, the detailed competence
descriptions are used as a basis for competence self-assessment with CAT. For this purpose,
the code categories are first used as concretizations in the competence model. From this,
statements can be derived from the individual statements of the interviewees in the research
documents, which enable a self-assessment of the students with regard to competence
development. These statements are collected in a questionnaire which can be mirrored to
intented learning outcomes of the course. The questionaire is handed over to students to
support self-reflexion with regard to specific health future skills.
References
Arnold, R. (2002). Von der Bildung zur Kompetenzentwicklung. Literatur- Und
Forschungsreport Weiterbildung. (49), 26–38.
707
Arnold, R. (2015). Bildung nach Bologna! Die Anregungen der europäischen

Hochschulreform. Wiesbaden: Springer. https://doi.org/10.1007/978-3-658-08978-8
Arnold, R., & Erpenbeck, J. (2014). Wissen ist keine Kompetenz: Dialoge zur
Kompetenzreifung. Baltmannsweiler: Schneider.
Erpenbeck, J., & Rosenstiel, L. v. (Eds.) (2003). Handbuch Kompetenzmessung: Erkennen,
verstehen und bewerten von Kompetenzen in der betrieblichen, pädagogischen und
psychologischen Praxis. Stuttgart: Schäffer-Poeschel.
Glaser, B. G., & Strauss, A. L. (2009). The Discovery of Grounded Theory: Strategies for
Qualitative Research. Chicago: Aldine Transaction.
Henschel, L. (2019). Competence Assessment and Competence Reflection in Software
Engineering Education. In 5th International Conference on Higher Education Advances
(HEAd'19). Valencia: Universitat Politècnica València.
Kauffeld, S., Reinecke, D., & Hennecke, M. (2009). Self-Assessment - ein Reflexionsansatz
in der Ausbildung? In S. Kauffeld, S. Grote, & E. Frieling (Eds.), Handbuch
Kompetenzentwicklung (pp. 388–408). Stuttgart: Schäffer-Poeschel.
Klieme, E., & Hartig, J. (2007). Kompetenzkonzepte in den Sozialwissenschaften und im
erziehungswissenschaftlichen Diskurs. Zeitschrift f. Erziehungswissenschaft (8), 11–29.
Maag Merki, K. (2004). Überfachliche Kompetenzen als Ziele beruflicher Bildung im
betrieblichen Alltag. Zeitschrift Für Pädagogik, 50(2), 202–222.
Maag Merki, K. (2009). Kompetenz. In S. Andresen (Ed.), Handwörterbuch
Erziehungswissenschaft (pp. 492–506). Weinheim: Beltz.
Mager, R. F. (1992). Preparing instructional objectives (2nd ed.). London: Kogan Page.
Rauner, F. (2011). Messen beruflicher Kompetenzen. Münster: Lit.
Rhein, R., & Kruse, T. (2011). Kompetenzorientierte Studiengangsentwicklung an der
Leibnitz Universität Hannover. In S. Nickel (Ed.), Der Bologna-Prozess aus Sicht der
Hochschulforschung (pp. 79–87). Gütersloh: CHE.
Roth, H. (1971). Entwicklung und Erziehung: Grundlagen einer Entwicklungspädagogik.
Pädagogische Anthropologie: Vol. 2. Hannover: Schroedel.
Sedelmaier, Y., & Landes, D. (2014). A Multi-Perspective Framework for Evaluating
Software Engineering Education by Assessing Students' Competencies. In 44th Frontiers
in Education (FIE), 2065–2072.
Sedelmaier, Y., & Landes, D. (2015). Swebos - The Software Engineering Body of Skills.
International Journal of Engineering Pedagogy, 5(1), 12–19.
Weinert, F. E. (2001). Vergleichende Leistungsmessung in Schulen - eine umstrittene
Selbstverständlichkeit. In F. E. Weinert (Ed.), Beltz-Pädagogik. Leistungsmessungen in
Schulen (pp. 17–31). Weinheim: Beltz.
708
“Map ourselves through Digital Storytelling”: pedagogical tool to

develop self-knowledge skills in advanced practice psychiatric-
mental health nursing
Carlos Laranjeira1,2, João Gomes2,3, Paula Carvalho2,4, Ana Querido1,2
1
Center for Innovative Care and Health Technology (ciTechcare), Polytechnic of Leiria,
Portugal, 2School of Health Sciences, Polytechnic of Leiria, Portugal, 3Hospital Center of
Leiria, Leiria, Portugal, 4Hospital Center of Médio Tejo, Tomar, Portugal.
Abstract
By creating digital stories people are encouraged to reminisce about events
with an impact on their life and personal and professional development. This
paper aims to determine nurses’ perspectives regarding a pedagogical tool –
Digital StoryTelling (DST). Our qualitative study design used video focus
groups involving 20 master’s students in mental health nursing during the
academic year 2020/2021. Students highlighted the creative content of digital
stories and their potential to develop self-awareness, socio-emotional
regulation and interpersonal relationships. The scarce digital literacy and the
difficulty in putting emotion in the content of the stories stood out as the main
barriers. Thus, a more individualized follow-up will be necessary to help
students. Further research may benefit from exploring how the DST tool can
assist a more diverse group of users.
Keywords: Digital stories; self-care; nurses; self-awareness; learning
strategy.

DST-Map “Map ourselves through Digital Storytelling”
1. Introduction
Telling stories is a way to disseminate information, and is a form of learning that calls for
sharing and reflection on what has been lived. Digital StoryTelling (DST) has been used as
a pedagogical strategy in different populations (children, elderly, students, and ill people),
educational contexts (primary, secondary, and higher education), and therapeutic settings
(Home Care for the elderly, Palliative Care, Mental Rehabilitation Units) (Mojtahedzadeh,
et al., 2021). A recent systematic review by Moreau et al. (2018) reaffirms the power of the
co-creation of digital stories in the training of health professionals, with a positive impact on
creative and reflective learning.
DST can broadly perpetuate the knowledge and sharing of the individual's lived experience,
either in archives for later viewing or through dissemination on digital platforms. The use of
DST combines first-person narrative with the use of multimedia (images, sounds, animation)
to produce a short video (3 to 5 minutes). In an era of people-centered care and increasing
pressure on care providers, particularly nurses, digital stories can have a positive effect in the
education and self-care behaviors of health professionals (Jáuregui-Lobera et al., 2020;
Laranjeira et al., 2021). According to Lambert and Hessler (2018), the development of DST
can be achieved through 7 stages, namely:
1. Start with an idea - Story cycle - writing the narrative. The cycle has a symbolic meaning
representing trust, integrity, and eternity. Participation in a group of students and teachers
helps to choose the theme and determine the story: What story do I want to tell? What do I
want the story to tell? What do I want to convey with the story? – The student’s experiential
content is shared in a group, replicating the traditional environment of “storytelling” and can
start from a theme of their own or one suggested by the teaching team based on real cases, or
common themes of the team (team-based learning);
2. Dramatic question - Select the emotions to share in the story - this step helps to identify
emotions in the narrative and to raise awareness of the emotions that you want to share and
include in the story;
3. “Finding the moment” and shaping the narrative – After identifying the theme and the
story, it is necessary to choose the scene and respective dramatic tension created with the
story. The moments of dramatic change and tension are the most impactful on the audience;
4. Visualizing the story – involves choosing the images that best convey the explicit and
implicit message of the story. It involves the student putting himself in the other's shoes and
perceiving how the scenes can have a more significant impact on the target audience;
5. Listening to the story – allows one to see the advantage of using the voice and the emotion
it conveys. The student chooses to use their voice or that of others to convey authenticity to
the narrated story;
710
6. Composition of the story using software – ensuring the combination of the various
elements of the story (voice, background sound, music, images) to maintain the sense of the
narrative;
7. Sharing the story – carried out with the group, the school community, and the target
audience for whom it is intended, on the internet. Sharing will be all the richer when it is
carried out and contextualized with the target audience.
During the story cycle, students are encouraged to reminisce about rescue stories with an
impact on their life and personal and professional development. The self-reflection and the
impact of their behavior on others will determine the assessment of how these situations can
contribute to the construction of desirable professional behavior. Reflecting upon the action
during the DST’s different stages can potentiate the development of metacognition skills.
1.1. Development of DST-Map approach

Regarding the innovation and creativity of training processes centered on students and with
a high potential for applicability in curricular flexibility, DST can be used as a pedagogical
strategy in the training of future health professionals and the advanced training of health
professionals, based on a low-complexity digital tool with high sustainability.
Its application can be enhanced with the re-creation of pedagogical environments, by
adapting spaces for the simulation of informal scenarios, and using video recording and high-
fidelity reproduction. In this sense, the communication laboratory was adapted with material
resources that allow the approximation of the current space to informal settings, with the
possibility of carrying out audiovisual recordings and autoscopy. Evidence shows that the
realism of high-fidelity scenarios promotes the development of clinical reasoning, clinical
competence, confidence, the integration of theory into clinical practice and the identification
of learning needs (Hanshaw & Dickerson, 2020). The use of video recording allows the
review of interactions during the story cycle, particularly the fleeting and non-repeatable
events that, in sharing, would very likely escape direct observation. Video recording
enhances learning by facilitating the emotional distance necessary for the reflective analysis
of the recorded material, which is very common in highly complex contexts.
This pedagogical tool has the following learning outcomes/educational objectives:
- Develop critical-reflexive self-analysis skills and potential for change;
- Develop narrative and reflective writing skills through the construction of the story script;
- Develop instrumental skills within the scope of digital literacy;
- Contribute to the consolidation of students' soft skills, particularly teamwork, emotional
competence and creativity.
711
1.2. Aim
This paper aims to determine nurses’ perspectives regarding a pedagogical tool – DST-Map
– which gives voice to those involved in the teaching-learning process in order to contribute
to self-knowledge and foster self-care behaviors.
2. Method
2.1. Study design

A descriptive qualitative study using online Focus Group Discussion (FGD) was performed
to gather information on students’ perceptions of DST. The Consolidated Criteria for
Reporting Qualitative Research (COREQ) checklist was used to report the study.
2.2. Participants
In the academic year (2020/2021), a purposive sampling strategy was used to enroll 20
master’s students in mental health nursing from the School of Health Sciences of Polytechnic
of Leiria (Portugal). Two FGDs were run (13 in the first group and 7 in the second group)
using Krueger and Casey’s (2015) methodological framework. Participants were selected in
accordance with the following inclusion criteria: (1) be a registered nurse enrolled in the
curricular unit "Personal Development and Therapeutic Communication", part of the Mental
Health Nursing master’s program; (2) can communicate in Portuguese; and (3) consent to
participate in an FGD. We excluded participants who had no access to any electronic device
to join the online interview.
2.3. Procedures
The online FGD sessions took place over two months (from February 2021 to March 2021)
using the videoconferencing application Zoom. The duration of FGDs lasted between 70 and
90 minutes. All FGDs were moderated by a clinical nurse and a nurse educator with a Ph.D.
degree and experience in conducting qualitative studies. Informed consent was obtained at
the start of each FGD. The focus group interview guide was based on the Gibbs reflexive
cycle (an integral part of reflexive analyses) (Li et al., 2020) and included three main
questions: a) What was your impression of the digital stories?; b) How has DST influenced
how you see yourself?; c) What did you feel/learn when you executed your DST? Probing
and supplementary questions were also asked.
The FG sessions were recorded with the participant’s permission and then transcribed for a
more efficient analysis of the discussion content and answers. Data analysis was performed
using the conventional content analysis method according to the procedure proposed by Elo
and Kyngas (2008). This method proposes three stages: open coding, creating categories, and
abstracting for data analysis. All the FG interviews were transcribed and imported into
712
WebQDA qualitative software program for data management. The findings were reviewed
by co-authors (researcher triangulation) to legitimately capture the essence of the findings.
3. Findings
The mean age of participants was 37 years (SD=9.2). Most of the focus group participants
were female. All participants had more than two years of nursing experience. The analysis of
the FGDs resulted in two main categories “Potentialities of DST” and “Barriers associated
with DST”.
Theme 1 - Potentialities of DST
The possibility of developing active strategies such as DST allowed for exploring the
creativity of the participants: “This strategy was very different (out of the box), I had never
imagined that the creative potential was so great” (FGD2). Additionally, there were numerous
reports highlighting the contributions of DST in the development of self-awareness, self-
reflection, and self-knowledge, essential components for the development of specialized
skills in Mental Health and Psychiatric Nursing. FGD1: “Understanding our emotions, we
can be more attentive to the emotions of others”; “I became more aware of my potential and
limitations”. FGD2: “The moments of self-analysis contributed to my personal
development”; “Looking back, I realize some of my reactions at the time I lived that story”.
At the same time, participants verbalized DST’s contribution to strengthening interpersonal
relationships with peers, and also to socio-emotional regulation. FGD1: “We can express
ourselves and express our feelings”; “Developing DST allowed us to explore the emotional
dimension of the facts, and that helps us achieve greater well-being. Sometimes, tackling
sensitive topics awakens experiences that I thought I had resolved.”. FGD2: “The cycle of
stories was useful to get to know my colleagues better and thus understand their attitudes and
behavior.”
The possibility of preserving history in time was mentioned by participants as a way of
ensuring its legacy for others: “That story was not only an academic work but also a kind of
legacy that I left for my children and for my family” (FGD1); “through my story, I was able
to honor a person who was very important in my life” (FGD2).
The potential of DST also makes it possible to develop skills that allow nurses to deal with
the therapeutic impasses that often arise in contact with people with mental illness: “I became
more sensitive to transferential impasses in my relationship with patients”, “I understand
better the difficulty of sharing what we feel and how much it sometimes affects therapeutic
relationships” (FGD1).
Theme 2 - Barriers associated with DST
713
During the FGD, several participants were unaware of the DST strategy, which caused them
anxiety in the initial phase of choosing the story: “The first impact was really the need to
expose something from the private sphere to the group of people who were strangers”
(FGD1); “as it was a pedagogical strategy that I didn't know about, it caused me fear and
anxiety” (FGD2). On the other hand, the participants highlighted the lack of digital skills in
the operationalization of the stories, which led to high consumption of time and resources for
their realization. FGD1: “The biggest challenge stemmed from my difficulty in using digital
tools” and “I'm not info-excluded, but having to use digital resources to build the story was
time-consuming, I even had to buy a program to help me with the DST”. FGD2: “I felt many
limitations in the construction of the final product, due to the lack of technical and digital
competence”, and “I realized that not everything that exists on the internet can be used”.
Several participants also mentioned the difficulty in selecting stories to share with their peers,
either because of the fear of exposure to others: “I felt uncomfortable and even
uncomfortable, knowing that I was going to share a story of my life” (FGD1); either because
of the difficulty of putting emotional content and affective tone in the stories “transmitting
emotion in the construction of my DST was the biggest challenge” (FGD1), “we are not used
to exploring emotions in an open way, and when we do that it creates fears”, “I had difficulty
identifying a significant experience that could be shared” (FGD2). This is because events that
arouse emotions tend to be remembered more easily than neutral events devoid of emotions.
4. Discussion
Given the rising complexity of the health care system in which nurses are needed to work,
educating future nurses capable of providing better quality care has become critical.
Assessment by competencies aims to verify the student's ability to manage concrete
situations. The focus is not merely on the specific task, but also on the strategies for solving
problems, mobilizing and articulating the resources that the student needs. These resources
are related to the domains of knowledge (knowledge of facts and concepts), knowing how to
do (mastery of skills and abilities), and knowing how to be (attitudinal posture and mastery
of situational relationships) (Panchenko, 2021).
Similarly to previous research, the DST tool was highly valued by the participants
(Henrickson et al., 2022). However, participants considered that more time was needed to
deepen and improve this methodology in the resource production phase, in order to fill small
gaps that were detected. Possibly, a more individualized follow-up would be necessary,
helping the students, case by case.
Based on our findings, education, and learning using DST can become extraordinarily
transformative and emancipating, through empathy, openness to experience, and a positive
orientation toward life (Laranjeira et al., 2021). Teaching the languages of images and
714
promoting critical readers of these images is an increasingly current challenge, as we move

towards a digital world, with audiovisual technology capable of creating virtual visual
scenarios, where the line that separates the mental image from the real one is very tenuous.
Learning is constructing meaning; meaning is what makes experience comprehensible and
coherent. Learning is a process through which meaning is attributed, starting from the
experiences and knowledge already acquired; it is a re-interpretation of an experience.
Making meaning is a fundamental element of adult learning (Panchenko, 2021).
DST contributes to understanding the learning process. Its purpose is to help professionals
become critically reflective and to participate more fully in discourse and action. Given the
effectiveness of digital storytelling, the educational application of this method is
recommended to improve social and emotional competencies among mental health nurses.
While the DST tool is a technique to enhance the learning processes in higher education, it
is also a strategy for enabling digital literacy. Finding audio or images with great symbolism,
especially in free services covered by Creative Commons licenses, requires time, patience,
and knowledge to refine searches (Hwang et al., 2023). This is a process that requires
dedication and persistence, in an attempt to find the ideal resource, which has the ability to
convey feelings, emotions, and the desired message. Finally, the almost generalized idea that
everything that exists on the Internet can be used is a frequent mistake. While it possible to
download content in the most diverse ways, the diversity of licenses means that not all content
is free to use.
5. Final remarks
DST is a robust and exciting methodology that allows for capturing, sharing, and preserving
stories. This path implies a break with some paradigms of traditional pedagogy, often too
closed in on itself. It is necessary to create openness and flexibility for informal learning,
which the vast majority of students today acquire through different technologies. The
recognition that these new technological and didactic skills are an essential ally to exploring
new ways of promoting the teaching/learning process and dynamizing new virtual learning
environments will have to be done naturally and not as an obligation enforced by
technological evolution. This study was performed only in a single higher education
institution and needs to be implemented on a broader scale. Further research may benefit
from exploring how the needs of more diverse groups of users could be met through the DST
tool.
References
Elo S, Kyngäs H. (2008). The qualitative content analysis process. J Adv Nurs. 62(1):107-
115. doi:10.1111/j.1365-2648.2007.04569.x
715
Hanshaw, S. L., Dickerson, S.S. (2020). High fidelity simulation evaluation studies in
nursing education: A review of the literature. Nurse Educ. Pract., 46, 102818.
doi.10.1016/j.nepr.2020.102818
Henrickson, L., Jephcote, W., & Comissiong, R. (2022). Soft skills, stories, and self-
reflection: Applied digital storytelling for self-branding. Convergence, 28(6), 1577–
1597. https://doi.org/10.1177/13548565221091517
Hwang, GJ., Zou, D., Wu, YX. (2023). Learning by storytelling and critiquing: a peer
assessment-enhanced digital storytelling approach to promoting young students’
information literacy, self-efficacy, and critical thinking awareness. Education Tech
Research Dev. [online first] doi.10.1007/s11423-022-10184-y
Jáuregui-Lobera I, Martínez-Gamarra M, Montes-Martínez M, Martínez-Quiñones JV.
(2020). Storytelling como instrumento de comunicación en contextos de salud. JONNPR,
5(8):863-90. doi:10.19230/jonnpr.3488
Krueger, R.A., Casey, M.A. (2015). Focus Groups: A Practical Guide for Applied Research,
5th ed.; SAGE Publications: Thousand Oaks, CA, USA.
Lambert, J., Hessler, B. (2018). Digital storytelling: Capturing lives, creating community
(5th ed.). Amsterdam: Routledge.
Laranjeira C, Carvalho P, Querido A. (2021). Digital Storytelling: Art-Based Pedagogy to
Improve Professional Development for Nurses. J Contin Educ Nurs., 52(10), 454-456.
doi: 10.3928/00220124-20210913-04.
Li, Y., Chen, W., Liu, C., Deng, M. (2020). Nurses’ Psychological Feelings About the
Application of Gibbs Reflective Cycle of Adverse Events. American Journal of Nursing
Science, 9(2), 74-78. doi: 10.11648/j.ajns.20200902.17
Mojtahedzadeh, R., Mohammadi, A., Hossein Emami, A., Zarei, A. (2021). How Digital
Storytelling Applied in Health Profession Education: A Systematized Review. J. Adv.
Med. Educ. Prof., 9(2), 63–78. doi.10.30476/jamp.2021.87856.1326
Moreau K, Eady K, Sikora L, Horsley T. (2018). Digital storytelling in health professions
education: a systematic review. BMC Medical Education. 10;18(1):208. doi:
10.1186/s12909-018-1320-1.
Panchenko, L. (2021). Digital storytelling in adult education: barriers and ways to overcome
them. Educational Technology Quarterly, 2021(4):673–688. doi.10.55056/etq.41
716
Competency-based education advances in Higher Education in

Health
Pau Martinez-Bueso, Olga Velasco-Roldán, Iosune Salinas-Bueno, Inmaculada
Riquelme-Agulló, Elisa Bosch-Donate
Departament Infermeria i Fisioterapia, Universitat Balearic Islands, Spain.
Abstract
INTRODUCTION. The incorporation of digital technology to rehabilitation
settings requires competency-based education advances in Higher Education
in Health or in life-long training. AIM. This work aimed at developing
knowledge in digital rehabilitation, exploring competences in Digital
Rehabilitation in higher education teachers, students, and professionals, and
creating pedagogical tools for them. A consortium of higher education
teachers from 5 European countries (DIRENE consortium) designed and
piloted a project in higher education studies from health and social care.
METHODS. This was a multicentre study that was performed through mixed
methodology. RESULTS. Digital competences in students, teachers, and
professionals were found. Digital content was created: a handbook and a
module on digital rehabilitation study. CONCLUSION. Assessing
Competency-based education advances in Higher Education in Health is an
essential need for the design and development of educational methods and for
enhancing the use of digital technologies in the rehabilitation practice.
Keywords: Digital rehabilitation; competences; higher education; teachers;
students; internationalization.

Competency-based education advances in Higher Education in Health
1. Introduction
Global need for rehabilitation is increasing due to population aging and increases in chronic
health conditions, among other causes (WHO, 2022). The World Health Organization
estimates that by 2050 over 2 billion people globally will need one or more assistive device
at some point in their lives (WHO, 2021). Therefore, the need for cost-effective services and
solutions for rehabilitation is evident. Digital transformation provides many opportunities,
however many rehabilitation service providers and healthcare professionals in Europe and
elsewhere, struggle using the new digital opportunities (Brunner et al, 2018). The
rehabilitation sector was strongly hit by the COVID-19 pandemic (Heiskanen et al, 2021).
Part of the challenges faced by the rehabilitation sector include the lack of technical
knowledge, capacities, and didactical competencies in the application of digital rehabilitation
(Longhini et al, 2022; Mikkonen et al, 2018). One of the main identified challenges in the
adaption of new digital opportunities, was the access to relevant and evidence-based
knowledge of which solutions to use to include digital rehabilitation into the scope of services
provided (Davies et al, 2022; Redecker et al, 2017). Another identified challenge was the
general competence and capability of rehabilitation workforce to design use cases, with the
users, for the existing digital solutions and for the different emerging technologies (Jarva et
al, 2022; Seeman et al, 2023).
Transnational cooperation in higher education offers an opportunity to create innovative
teaching and learning experiences, developing competences on scale to ensure access to
diverse digital knowledge in rehabilitation (Nezeha et al, 2020). That is the reason why we
created and finished the project, “Competences for the new era of user-driven digital
rehabilitation” (DIRENE): it addressed the emerging challenges brought by Covid-19 and
the need for the development of digital rehabilitation (DR) through higher education,
according to competence and practice to build the resilience of future rehabilitation systems.
2. Aims of the work

The overall objective of this project was to contribute towards the resilience of rehabilitation
systems through the development of DR competences of teachers, students and working life
professionals. The specific objectives of the project included: 1) developing knowledge in
DR for teachers, students, and professionals for competence development in DR and, 2)
increasing learning opportunities in DR through theory and evidence-based practice.
These objectives were achieved through active global cooperation and international
networking between universities from Finland, Spain, Austria, Germany, and Greece.
Different stakeholders were widely invited to contribute to the development work, including
partners outside Europe, who offered new expertise and reverse learning opportunities.
718
Pau Martinez-Bueso et al.
3. Methods
This was a multicenter European study that was applied across different DR disciplines. To
proportionate a multilevel perspective, a previous scoping review was done, and two different
studies were performed.
Firstly, a quantitative study was done to create a pilot questionnaire on competences for DR.
After the feedback, the questionnaire was refined and improved in writing and
comprehension. Definitive questionnaire was utterly completed by rehabilitation educators,
students, and professionals of the 5 countries.
Secondly, a qualitative, explorative, transversal research design was conducted. This
methodology was specifically used to have a basic descriptive qualitative approach about
learning needs related to DR, and barriers and facilitators to DR education. To do so, different
focus groups were conducted from different roles (healthcare professionals, health sciences
teachers and students)
4. Results
These objectives were achieved through developed Discussion papers, Framework, open
pedagogical Handbook and a study Module with open on-line learning materials for the use
of identified target groups in Europe and beyond.
The applied and empirical findings of interest to higher education professionals, students,
and professionals for competence development in digital rehabilitation were:
a. From the scoping review:
• Information collected on digital technologies for rehabilitation: data are in press

b. From the analysis through mixed methodology:
• Digital competence and training needs analysed through mixed methodology:

quantitative and qualitative research (see table 1 and 2)
• Sources of information developed: Open pedagogical Handbook for teachers,
trainers and the working life addressing the creation of on-line/virtual learning
material on DR based on mobile technologies, considering the variety of learners
needs in global contexts, considering clients, students and the current and future
working life professionals (“Handbook Mobile Education & Training of Digital
Rehabilitation Competencies”)
• An e-learning module on DR study module (10 ECTS cr) and open mobile learning
course (2 ECTS cr) on DR developed and implemented that is accessible across the
target groups and countries (“Introducción a la RHB Digital Rehabilitation”)
719
Table 1. Digital Rehabilitation competence levels (mean and standard deviation) in educators,
students ans rehabilitation professionals. *p<0.05.
Table 2. Learning needs on DR identified by the participants in the focus groups.
Learning needs Knowledge/skills

required
Digital eHealth literacy Searching and selecting reliable sources of

competencies information about health issues.
Communication Specific communication skills when using

skills technology-driven communication in
rehabilitation.
Technical skills Ability to use digital technologies in

rehabilitation.
Ethics and Confidentiality

regulation
Responsibilities
Legal framework Knowledge about the policies for the use of DR

for application of (data protection, coverage/ reimbursement of
digital technologies costs for the healthcare professional and for the
in rehabilitation user)
Specific DR Areas/fields of Knowing the field of applications of DR and

knowledge/skills application of DR recognizing the value for the user. This
includes being able to identify the benefits and
barriers of DR based on the needs of the user.
720
Pau Martinez-Bueso et al.
Findings and conclusions will be published in a format of discussion papers and distributed
widely through the networks of program partners and target groups. The findings are also
summarized into the following tables.
Results will be widely disseminated through websites, social media, multiplier events and
through new open learning opportunities. Directly, the project engages around 300
rehabilitation experts in Europe. Indirectly, through dissemination and data collection
activities, it is expected that over thousand people will be contributing to final outcomes.
4. Conclusions
We have produced various intangible and tangible results leading to the achievement of the
objectives. The transfer and impact of the project results influence higher education in
rehabilitation fields. As the demand and need for rehabilitation continues to increase during
a time of uncertainty, client-centered and cost-effective solutions are needed without
compromising clients rights and rehabilitation processes, while ensuring the access to skills
development for students, professionals and educators.
The participant higher education teachers and experts (in rehabilitation) will improve and
update their professional and pedagogical competencies. They will acquire new knowledge
and skills related to the digitalization of rehabilitation and especially in mobile solutions in
rehabilitation. Students (bachelor and master students as target groups and end users) will
develop their digital and professionals’ competencies, learn about the various opportunities
in rehabilitation and on the applicability of competences according to various client groups
and needs. Health professionals (as target groups and end users) will gain new knowledge on
the possibilities of digitalization in their everyday work. They will improve their digital and
professional competences and the ability to apply and implement new technologies to the
rehabilitation processes based on person-centred needs, specially with the digital content
created in this project (the handbook and the e-learning course on digital rehabilitation).
Partners in the project (EU and Africa) will enhance the quality and excellence of the
education. They also will be able to provide highly competitive education and to develop
rehabilitation curricula through digitalization utilizing the international network.
References
Brunner M, McGregor D, Keep M, Janssen A, Spallek H, Quinn D, Jones A, Tseris E, Yeung
W, Togher L, Solman A, Shaw T. (2018).An eHealth Capabilities Framework for
Graduates and Health Professionals: Mixed-Methods Study. J Med Internet Res
15;20(5):e10229. doi: 10.2196/10229.
Davies AC, Davies A, Abdulhussein H, Hooley F, Eleftheriou I, Hassan L, Bromiley PA,
Couch P, Wasiuk C, Brass A.(2022). Educating the Healthcare Workforce to Support
721
Digital Transformation. Stud Health Technol Inform 290:934-936. doi:

10.3233/SHTI220217.
Heiskanen T, Rinne H, Miettinen S, Salminen AL.(2021). Uptake of Tele-Rehabilitation in
Finland amongst Rehabilitation Professionals during the COVID-19 Pandemic. Int J
Environ Res Public Health, 18(8):4383. doi: 10.3390/ijerph18084383.
Jarva E, Oikarinen A, Andersson J, Tuomikoski AM, Kääriäinen M, Meriläinen M,
Mikkonen K.(2022). Healthcare professionals' perceptions of digital health competence:
A qualitative descriptive study. Nurs Open, 9(2):1379-1393. doi: 10.1002/nop2.1184.
Longhini J, Rossettini G, Palese A.(2022). Digital Health Competencies Among Health Care
Professionals: Systematic Review. J Med Internet Res, 24(8):e36414. doi:
10.2196/36414.
Nazeha N, Pavagadhi D, Kyaw BM, Car J, Jimenez G, Tudor Car L.(2020). A Digitally
Competent Health Workforce: Scoping Review of Educational Frameworks. J Med
Internet Res, 22(11):e22706. doi: 10.2196/22706.
McKinstry C, Iacono T, Kenny A, Hannon J, Knight K.(2020). Applying a digital literacy
framework and mapping tool to an occupational therapy curriculum. Aust Occup Ther J,
67(3):210-217. doi: 10.1111/1440-1630.12644.
Mikkonen K, Ojala T, Sjögren T, Piirainen A, Koskinen C, Koskinen M, Koivula M,
Sormunen M, Saaranen T, Salminen L, Koskimäki M, Ruotsalainen H, Lähteenmäki ML,
Wallin O, Mäki-Hakola H, Kääriäinen. (2018). Competence areas of health science
teachers - A systematic review of quantitative studies. M. Nurse Educ Today, 70:77-86.
doi: 10.1016/j.nedt.2018.08.017.
Redecker, C.(2017). European Framework for the Digital Competence of Educators:
DigCompEdu. Punie, Y. (ed). EUR 28775 EN. Publications Office of the European
Union, Luxembourg, ISBN 978-92-79-73494-6, doi:10.2760/159770, JRC107466
Seemann RJ, Mielke AM, Glauert DL, Gehlen T, Poncette AS, Mosch LK, Back DA. (2023).
Implementation of a digital health module for undergraduate medical students: A
comparative study on knowledge and attitudes. Technol Health Care, 31(1):157-164. doi:
10.3233/THC-220138.
World Health Organization WHO. (N.D.)(2022). mHealth programmes are sponsored by
government. Indicator metadata registry list. Retrieved June 2, 2022, from
https://www.who.int/data/gho/indicator-metadata-registry/imr-details/4774
World Health Organization. (2021). Rehabilitation. Key facts. Retrieved June 4th, 2022,
from: https://www.who.int/news-room/fact-sheets/detail/rehabilitation
722
Myth-busters at work: development of engineering identity and

employability through student research
Jeroen Lievens
Faculty of Engineering Technology, University of Leuven (Campus Diepenbeek), Belgium.
Abstract
For many (aspiring) engineering students, the prevalent image of engineering
identity is still unilaterally dominated by technical skills. This bias serves not
only to dissuade female prospective students but also to erode student
motivation for non-technical courses such as communication and other
employability skills. As an unfortunate result, also for the employer, many
starting engineers experience a gap between their own and the workplace’s
expectations. This paper reports on a didactic approach that has the students
actively research the workplace realities, and more particularly the
importance of communication skills as perceived by professional engineers.
On the basis of their own, partly self-directed research efforts, students
uniformly conclude that communication skills are in fact of great importance
for engineers. The indisputable scientific evidence, both quantitative and
qualitative, that they gather themselves turns out to be a powerful incentive for
students to recalibrate their understanding of engineering identity.
Keywords: Engineering education; professional identity; employability;
engineering curriculum; communication.

Development of engineering identity and employability through student research
1. Introduction
The field of engineering is notoriously diverse, allowing for myriad career options across
multiple domains and in several positions. Yet, at the core of what the engineering identity
is still all too often perceived to be is a surprisingly unilateral insistence on technical skills,
culminating in a stereotypical representation known as the engineering “nerd myth” (Beder,
1998; Brunhaver et al., 2018). This essentially masculine myth, disregarding the many social
dimensions integral to engineering, serves to discourage women from identifying with a
career in engineering (Faulkner, 2007). Another harmful consequence of this myth is that
engineering students tend to experience a lack of motivation for course topics that are non-
technical, such as communication and teamwork skills, the importance of which for
employability has nevertheless been underscored emphatically in, for instance, the Future of
Jobs Report by the World Economic Forum (2020). The gap between expectations and the
first real-life experiences in the engineering workplace, as described by researchers across
several economies (Dias, 2011; Kövesi & Kalman, 2020), has lead many young engineers to
switch jobs within the first six months of employment (KU Leuven, Antwerpen, et al., 2016),
leaving both the engineer and the employer, who has invested in recruitment and training,
with a sense of frustration and disappointment (Trevelyan, 2019).
The development of an engineering identity that is based on a sound and more nuanced
understanding of the realities of the workplace, is positive for both motivation (Frymier &
Shulman, 1995; Hock et al., 2006) as well as for employability and job satisfaction (De Coen
et al., 2018; Yorke, 2006). Increasingly, universities recognize the importance of bringing
engineering students into contact with the workplace; however, these interventions often
have an incidental character, functioning as curricular add-ons (e.g. a company visit or a one-
day shadowing internship) that fail to integrate meaningfully with the core curriculum.
Moreover, these interventions tend to position students in the passive role of observers. This
paper reports on an alternative approach, in which the students become active, in-depth
researchers of the workplace realities - with a focus on the importance of communication
skills, in this particular case - and report on their findings in a written paper as part of a
designated course in the curriculum. The proposition is that this approach helps students build
a well-informed understanding of workplace expectations and a more accurate sense of their
future professional selves, while simultaneously developing important engineering
competencies such as research and writing skills.
2. Methods
This section describes the setup of the second bachelor course “Statistics+”, which is part of
the curriculum of industrial engineering education of the joint program of KU Leuven and
Uhasselt at Campus Diepenbeek. As part of this course, students are asked to investigate
724
Jeroen Lievens
whether communication skills are important for professional engineers, and if so, which skills
are primarily important. Students are encouraged to explore several qualitative approaches
(e.g., interviews with engineers, focus groups, and literature reviews) facultatively in
conjunction with a mandatory quantitative approach in the form of a survey, which is uniform
for all students. The pivot of this survey is a questionnaire developed by the teaching staff
using Google Forms, which students are invited to send via e-mail to professional engineers
in their wider circle of acquaintances. Since 2012, when this approach was initiated, 2430
engineers have completed the questionnaire. Data are collected in Excel and consequently
analyzed, also using various statistical tools, by students. On the basis of the findings of their
qualitative and quantitative research endeavors, students write an academically formatted
paper answering the main research question – are communication skills important for
engineers? – and a self-composed additional research question focusing on specific
communication skills and/or correlations with the sector of employment, position, age,
gender, or size of the company. This additional research question allows students to invest
their personal interests and career plans in their research projects. Students interested in
construction engineering, for instance, could investigate which foreign languages are most
important in that sector. Throughout the process, students are supported in classes focusing
on, on the one hand, statistics, and on the other hand, research and writing skills.
This paper also reports on a small-scale survey, based on a questionnaire sent to 72 students
who took the course Statistics+ in the autumn of 2022. The questionnaire evaluated whether
the approach was successful in achieving its didactic goals. The questionnaire was completed
by 36 students.

The (unsurprising) answer to the main research question that students arrive at is that
professional engineers indeed perceive communication skills to be very important. With very
few exceptions, students report that their qualitative research efforts, typically interviews,
mainly serve to reemphasize the fundamental points that the questionnaire results make quite
definitively: that engineers, on average, spend a significant portion of their workday
communicating (Fig. 1) and that they perceive their communication skills to have a very
significant impact on their career path (Fig. 2).
Of course, it is not these conclusions in themselves that are the crux of the matter, but rather
the fact that the students themselves reach them on the basis of largely self-directed research.
In fact, many students end up arguing for the inclusion of more communication classes in the
curriculum based on this evidence. Alternatively, they may offer valid and useful suggestions
for new communication topics to be included in the curriculum or for fine-tuning specific
725
curricular pathways in function of domain-specific sectors. The two examples below are
included to demonstrate the potential of the student papers for curricular optimization.
a. In a 2021 student paper, students studied the recent evolution of the perceived
importance of virtual meetings (VMs) compared to face-to-face (F2F)
communication (Fig. 3)1. They concluded that the COVID-19 pandemic served to
accelerate an already incipient trend towards an increasing use of virtual meetings,
while this rise did not detract from the importance of face-to-face communications.2
These students remarked, and quite astutely so, that in our current curriculum,
classes on oral communication (e.g. meeting, negotiation, and presentation skills)
are restricted to face-to-face settings while such activities are likely to play out quite
differently in virtual settings (e.g. diminishing impact of body language). They
recommended that the communication classes should include a focus on virtual
environments in the future. Teaching staff is currently developing course materials
to follow up on that suggestion.
b. In another paper, students investigated the relationship between the perceived
necessity of mastering certain foreign languages and some common sectors of
employment for engineers (construction, electromechanical sector, chemistry, and
electronics-ICT). As Fig. 4 demonstrates, English is by far the most important
foreign language in all sectors3. French is generally speaking the second most
important language, with a marked outlier in construction engineering. In the
electromechanical engineering sector, however, German is the second most needed
foreign language. On the basis of these findings, the students recommended that
English should be a compulsory class for all specialisations in the engineering
program at our Faculty; that French should be compulsory for students in
construction engineering; that German should be compulsory for students in
electromechanical engineering; and that French, German and a proficiency training
course in English should be elective courses for all students in the master’s degree.
These findings were discussed with the curriculum designers of the Faculty, who
proceeded to change the curriculum accordingly.
1
The question relating to virtual meetings was added to the questionnaire in 2015.
2
In the updated graph shown here, including the 2022 results, the perceived importance of virtual meetings seems to be decreasing
again, very tentatively suggesting that the COVID-19 conditions may have been a stronger driver for the rise than other factors
(ecology, mobility, cost benefits…). Of course, this remains to be confirmed by data and research in the coming years.
3
Fig. 4 shows the graph updated on the basis of the most recent survey results.
726
Jeroen Lievens
Figure 1. Time spent on communication in an average work day (questionnaire results).
Figure 2. Perceived impact of communication skills on career path (questionnaire results).
Figure 3. Perceived importance of face-to-face communication and virtual meetings.
727
Figure 4. Perceived necessity of foreign languages for engineers in different sectors of employment.
In January 2023, upon completion of the course, the students were invited to complete a
questionnaire that aimed to evaluate the effectiveness of the described didactic approach. The
response rate was 50%, with 36 responding students out of 72. Figures 5, 6, 7 and 8 show
some of the results of the questionnaire, which used a five-point Likert scale for agreement.
Figure 5. “By doing this research, I have been able to establish ‘with my own eyes’ that communication skills are
important for engineers”.
728
Jeroen Lievens
Figure 6. “During this course, I have improved my research skills”.
Figure 7. “During this course, I have developed my writing skills”.
Figure 8. “During this course, I have improved my understanding of the engineering professional identity”.
729
From these figures, it seems fair to conclude that the approach was largely successful in
achieving its didactic targets.
4. Conclusion
To develop a strong sense of engineering identity, students need to have a sound
understanding of the lived realities of the workplace. Interventions based on observation are
certainly preferable to stereotypes and hearsay, but arguably, an even more potent way to
bust misleading and exclusionary myths about “the engineer” is to have students themselves
research the workplace and workplace expectations. Moreover, this approach allows for an
integration of this process of identity formation in the core curriculum as students can
simultaneously be taught vital engineering and employability skills such as research skills,
writing skills and even – in this particular case – statistics.
The approach described in this paper can be easily modified to fit the needs, concerns and
curricular goals of other fields of higher education, just as easily as it can be modified to
include other employability skills besides communication skills, such as teamwork skills,
leadership, life-long learning, creativity, problem-solving skills... A teacher design team
interested in this approach could ponder the following questions as a starting point: Which
misconceptions may exist among students about their future professional selves? Which
course in the curriculum could accommodate a student-driven research project in this aspect
of professional identity? Which didactic approach can maximize the interplay between
research project activities and other curricular goals? How can the approach allow the
personal interests of the students to co-define their research trajectory? Developing such a
course certainly requires significant effort, but it may well end up benefiting the students, the
teaching staff and the workplace all at once.
References
Beder, S. (1998). A bit of the Rain Man in every engineer? Engineers Australia, April 1998,
57.
Brunhaver, S., Korte, R. F., Barley, S., & Sheppard, S. (2018). Bridging the Gaps Between
Engineering Education and Practice. In R. B. Freeman & H. Salzman (Eds.), U.S.
Engineering in a Global Economy (pp. 129–163). NBER: University of Chicago Press.
De Coen, A., Goffin, K., Van Hoed, M., & Forrier, A. (2018). Techniek 10 jaar later:
Loopbaanpaden en – uitkomsten van STEM-studenten (Issue december).
Dias, D. (2011). Reasons and motivations for the option of an engineering career in Portugal.
European Journal of Engineering Education, 36(4), 367–376.
https://doi.org/10.1080/03043797.2011.593096
Faulkner, W. (2007). Nuts and Bolts and People’: Gender-troubled Engineering Identities.
Social Studies of Science, 37(3), 331–356. https://doi.org/10.1177/0306312706072175
730
Jeroen Lievens
Frymier, A. B., & Shulman, G. M. (1995). `What’s in it for me?’: Increasing content
relevance to enhance students’ motivation. Communication Education, 44(1), 40–51.
https://doi.org/10.1080/03634529509378996
Hock, L. F., Deshler, D. D., & Schumaker, J. B. (2006). Enhancing student motivation
through the pursuit of possible selves. In C. Dunkel & J. Kerpelman (Eds.), Possible
selves: Theory, research, and application (pp. 205–221). Nova Science Publishers.
Kövesi, K., & Kálmán, A. (2020). How to manage the study-to-work transition? A
comparative study of Hungarian and French graduate engineering students’ perception of
their employability. European Journal of Engineering Education, 45(4), 516–533.
https://doi.org/10.1080/03043797.2019.1622654
KU Leuven, U Antwerpen, UGent, UHasselt, & VUB. (2016). Eindrapport Studie Industrieel
Ingenieur 2020.
Trevelyan, J. (2019). Transitioning to engineering practice. European Journal of Engineering
Education, 44(6), 821–837. https://doi.org/10.1080/03043797.2019.1681631
Yorke, M. (2006). Employability in higher education: what it is – what it is not. Series one.
In M. Yorke (Ed.), Learning & Employability (Vol. 1). https://doi.org/10.1002/ir.162
World Economic Forum. (2020). The Future of Jobs Report 2020 (Issue October).
https://www3.weforum.org/docs/WEF_Future_of_Jobs_2020.pdf
731
Development and piloting of a micro-credential programme in

research ethics and integrity leadership – an example from
Estonia
Centre for Ethics, University of Tartu, Estonia.
Abstract
Micro-credentials are an emerging format of lifelong learning supporting
competence-development as well as deepening knowledge in the field. Still,
there is little research on implementing and sustaining micro-credentials in
higher education. The current paper aims to share experiences of development
and piloting of a micro-credential programme for a specific target group in
Estonia. The criteria for setting up a micro-credential programme as well as
initial lessons learned are outlined. Experience indicates that it takes some
time and sharing best practices to develop the programmes into a feasible form
of education. In case a programme is developed for a very specific target group
and no degree programme exists in the field, several tailor-made solutions
must be invented. By sharing best practices, developing and testing new
formats will make micro-credential programmes more effective.
Keywords: Micro-credentials; programme development; lifelong learning.

Development and piloting of a micro-credential programme
1. Introduction
Micro-credentials, or micro-degrees, are a new addition in the lifelong learning possibilities
and are avidly supported by the European Union (EU) (EU, 2021b). The new European Skill
Agenda (EU, 2021a) proposes using short-term training formats to provide swift and flexible
ways for people to deepen their knowledge and obtain new skills. A common EU approach
to micro-credentials was also established (EU, 2021b) to ensure alignment and transferability
among EU higher education (HE) institutions.
Similarly, Estonian Education Strategy 2021–2035 (MER, 2021) establishes that more
flexible and integrative options for continuing education should be provided, so all major HE
institutions offer various micro-credential programmes. In 2022–2023 academic year three
major universities in Estonia offer over 120 micro-credential programmes (Adamson, 2022)
(in addition to programmes offered by other HE institutions). Following European
Commission recommendations and national education strategies, micro-credentials are
accredited in credit points following ECTS (European Credit Transfer and Accumulation
System) which can then be used to complete modules in different degree levels and be
transferrable into different European universities (that follow the Bologna System) (EU,
2021b; McGreal & Olcott Jr, 2022). Universities state that micro-credentials provide an
option to integrate general skills as well as competencies of specific fields, the format is more
flexible and provides opportunities for requalification and upscaling (Adamson, 2022).
Generally, micro-credentials contribute to a more qualified workforce as well as a more
educated society.
Still, as micro-credentials are an emerging format of lifelong learning, there is little research
on implementing and sustaining micro-credentials in higher education (Selvaratnam &
Sankey, 2021). The current paper aims to share experiences of development and piloting of
a micro-credential programme for a specific target group in Estonia. First, the background of
the programme is introduced. Then, the description of the development process and the
piloting phase are provided. To conclude, lessons learned and recommendations are outlined
to support development and implementation of micro-credential programmes in Europe.
2. Need for research ethics and integrity leadership

HE institutions struggle with various issues pertaining to research ethics and integrity (REI),
e.g., increased competition, pressure to publish, technological advances that enable
plagiarism (Mustajoki & Mustajoki, 2017; Hyytinen & Löfström, 2017). This calls for a
systematic approach towards REI and its implementation (Bertram Gallant, 2011). HE
institutions have reacted to this need by setting up ethics committees, designating research
integrity officers, data management officers and delegating other individuals to create and
implement REI infrastructure (Wright & Schneider, 2010; Bertram Gallant, 2011). Usually,
734
the REI-related tasks are just added to the designated people with no additional training (or
receiving a few short training sessions), degrees in REI are not common. The question is:
how to prepare REI leaders who would “build the culture of integrity in HE institutions by
creating an environment where everyone can and will make ethical decisions”? (Tammeleht,
Löfström & Rodríguez-Triana, 2022).
In 2017 a new national code of conduct for research integrity was developed and the
document was approved in Estonia by all research institutions. The named document, in
addition to outlining the responsibilities of researchers, also stipulates the responsibilities of
research institutions in ensuring the REI infrastructure (Hea Teadustava, 2017). The need for
a more structured and systematic programme for educating REI leaders became urgent as
research institutions had limited levels of readiness for implementing the national code of
conduct. As a response, a micro-credential programme was set up by one university with
expertise in ethics.
2.1. Development of the micro-credential programme

As there was no degree programme to educate REI leaders, and a very specific target group
was kept in mind, a new programme was designed to fulfill the gap. The REI leadership
micro-credential programme was name Academic and Research Integrity: from theory to
practice. The aim of the programme is to prepare experts who could solve various REI issue
in their field and institution, and who would be ready to guide others.
The target group for the programme are people who have at least a bachelor’s degree and
who are already active in research and teaching in a research institution, especially on topics
connected with REI. Participants should be ready to apply the acquired knowledge in their
own institution by training others, advising on issues pertaining to REI and setting up REI
infrastructure.
The volume of micro-credential programme is 24 ECTS and lasts for two semesters. The
university provides micro-credential programmes to learners for a tuition fee. As for the
second semester tailored courses had to be developed for the target group (as there is no
existing degree programme about REI leadership), the price became slightly higher than the
average credit point price in the department. Still, the overall cost of the entire programme is
significantly lower than, for instance, in law, business or media.
Flexible timetables are also recommended by the university guidelines. The participants of
micro-credential programmes are expected to attend courses with degree students. Since the
participants of micro-credential programmes are usually individuals who work full time and
can devote a limited amount of time on travelling and attending lectures, the university
advises using session- or web-based learning formats.
735
2.2. Description of the learning process of the programme

The micro-credentials programme started with a full seminar day where we met the
participants face-to-face. At this first meeting, we got to know the learners and their
expectations for the programme. There was an introduction of contact persons, lecturers
responsible for the courses and learning aids (e.g., library, databases, ethics website, online
learning environment). Participants were also given a detailed overview of the autumn
semester schedule (see below) and the conditions for completing the courses.
Figure 1. Autumn semester schedule.
In the autumn semester, learners had to complete three courses (12 ECTS in total): “Critical
thinking and argumentation”, “Basics of ethics” and “Research integrity: framework,
requirements, values and principles of action” (see Figure 1).
All courses were structured in such a way that the final grade was formed continuously during
the semester, which means that learners had to constantly keep up with the course topics,
read the materials provided, participate in seminars, and solve homework assignments. The
final grade for all the courses was differentiated (from “A” to “F”).
In two of those courses (“Basics of ethics” and “Research integrity”) there were pre-recorded
video lectures in addition to online seminars that were compulsory to attend. But there was
also a possibility to compensate the absence from the seminar by submitting a written task.
In the course called “Critical thinking and argumentation” there were face-to-face seminars
but the attendance in these was not compulsory, since there were also recorded video
seminars from previous years.
Each of the course had a separate webpage where all important information about the course
was presented: completion conditions, schedule, instructions, additional materials, links to
736
join video seminars, communication forums and forums for posting independent works. In
addition, we created a separate general webpage for the micro-credentials programme. We
added communication forums there, through which we could share information with the
learners, and they could ask questions from us. There was also a place for learning diaries
where we posted a task for each week, to help the learners keep up with their courses and
combine the knowledge that they had gathered in different courses. Once a month, online
information sessions were also held. These were designed to convey information, listen to
learners' concerns, increase their motivation, and help them in the learning process.
In spring, learners must complete two courses (12 ECTS in total): “Ethics in an organization”
and “Workplace applications of practical ethics”. During spring semester, all seminars will
take place face-to-face, and for this we have planned one full day of seminars every month
(see Figure 2). On each seminar day (except for the last one), we will have three seminars
and lunch together, during which it is possible to discuss issues related to the micro-
credentials programme as well as to discuss ethical issues in a broader sense.
Figure 2. Spring semester schedule.
In the course called "Workplace applications of practical ethics", learners choose a supervisor
at the beginning of the semester and during the course every learner will develop one element
of the ethics infrastructure with the help of the supervisor.
In the subject "Ethics in an organization", seminars with various experts in the field take
place on every seminar day. Between the seminars, learners must independently read the
materials and make posts in the self-reflection diary. The final grade for both spring courses
is non-differentiating (“pass” or “fail”).
737
We have planned the very last seminar day of the programme as an outing to a country house
in a beautiful place of nature. There, learners are going to defend the last version of their
projects and will receive feedback from both supervisors and fellow learners. This will be
followed by a discussion round about the study-experience and how to implement and
disseminate knowledge obtained from the programme.
3. Initial lessons learned

The initial lessons learned were compiled when the first half of the micro-credential
programme had ended. The programme coordinator compiled feedback from learners (N=8),
lecturers (N=6), the programme head and other collaborators. The feedback was fed into the
development of the second half of the programme.
As the university guidelines encourage micro-credential learners to participate in the courses
already provided by degree programmes, this format was indeed followed. It was convenient
to incorporate additional learners into existing courses and they adapted relatively smoothly.
As the micro-credential learners had already at least graduate degrees (some even professors
in their field) and obtained quite significant work experience, they contributed significantly
to discussions during seminars and thus helped other learners gain valuable insights to
various ethical topics.
Nevertheless, there were also some drawbacks. Firstly, the set timetables were inflexible and
thus not convenient for micro-credential learners. They would have preferred fewer fixed
meetings but more intensive seminars when the meetings took place. Secondly, some learners
did not feel comfortable about learning together with undergraduate students and being
treated similarly to them. Namely, teaching undergraduate students need more structuring in
their studies, more fixed tasks and deadlines to keep up with their studies; their time-
management skills may still be limited. This format of teaching may not be suitable for
independent learners like the ones participating in micro-credential programmes. In addition,
already existing courses may have many participants and the individual needs of more mature
learners may remain unnoticed. Moreover, it may be difficult to adapt to the student’s role
and unfamiliar disciplines.
To alleviate the emerged issues, the lecturers of the existing courses were approached, and
agreements were made to support micro-credential learners. They would still cover the same
material but would get more flexibility in submitting their assignments. Organising regular
meetings to discuss learning and emotional issues were also perceived well by the learners.
Monthly online meetings gave them a chance to voice their concerns, share emotions and
support each other. Also, the programme team spent extensive hours in supporting the
programme participants. Many toyed with the idea of quitting the programme, the main
738
reason being the huge workload (studies and work-related responsibilities), but by the end of
the meeting everyone decided to continue with their studies.
The programme team saw another potential reason for not quitting – the programme is not
for free, and the studies were mostly financed by the participants’ employers. Thus, they had
taken the responsibility to finalise the programme. In addition to the certificate, the
participants would present a practical REI element to their institution/employer.
As a response to the lessons learned the courses for the second semester were designed from
the perspective of the micro-credential programme learners. They expressed a need to have
fewer meetings but then coming together for a full day. Thus, monthly seminar days were
organised. The time in-between the meetings would be devoted to individual reading and
compiling the practical final assignment of the programme (with more personal supervision).
Still, this kind of tailor-made approach is more time-consuming and expensive for the
organising institution. One solution to compensate for the expenses is to make the micro-
credential courses available for a wider public (e.g., through the Open Academy).
4. Conclusions and recommendations

All in all, micro-credentials programmes are an excellent addition to the lifelong learning
opportunities. They provide possibilities to deepen or enhance knowledge in different fields
while still being part of the labour market. Transferability of credit points ensures options to
pursue a degree in the field in case there is interest in it. Overall, micro-credentials contribute
to a more educated society.
Some recommendations for implementing a micro-credential programme:
• A preliminary information seminar could be organised to introduce the entire

programme. This will align participants’ expectations to the content offered.
• If possible, separate study groups should be created for learners of the micro-credential
programme, this way it is possible to better respond to learners’ needs and create a
better-suited schedule for them.
• Instead of a tight fixed schedule, fewer but more intensive seminar days should be
planned. Also, non-differentiated assessment should be used as this is more suitable for
independent/mature learners.
• The programme should have a designated contact person who is committed to regular
communication with the learners.
• Although online learning significantly adds flexibility, at least some face-to-face
meetings should be organised, because these remarkably increase learners’ motivation.
• To make a customized approach affordable for the organising institution, making micro-
credentials courses available for a wider public could be considered (so that, for
739
example, instead of taking part in the entire program, it would be possible to take only
one course).
Acknowledgements
Research is supported by the University of Tartu development fund for creating microdegrees
and the University of Tartu support for implementing research integrity at the university.
References
Adamson, M. (2022). Mikrokraadiprogrammide roll ja perspektiiv Eesti hariduses. Avatud
akadeemia blogi. Retrieved from:
https://www.tlu.ee/taxonomy/term/100/uudised/mikrokraadiprogrammide-roll-ja-
perspektiiv-eesti-hariduses (27.01.2023)
Bertram Gallant, T. (Ed). (2011). Creating the Ethical Academy: A Systems Approach to
Understanding Misconduct and Empowering Change. New York: Routledge.
EU. (2021a). Skills agenda for Europe. Retrieved from:
https://ec.europa.eu/social/main.jsp?catId=1223&langId=en&moreDocuments=yes
(27.01.2023)
EU. (2021b). Individual learning accounts and micro-credentials. Retrieved from:
https://education.ec.europa.eu/education-levels/higher-education/micro-credentials
(27.01.2023)
Hea Teadustava. (2017). Estonian Code of Conduct for Research Integrity. Tartu. Retrieved
from: https://www.eetika.ee/sites/default/files/www_ut/hea_teadustava_eng_trukis.pdf
(27.01.2023)
Hyytinen, H., & Löfström, E. (2017). Reactively, proactively, implicitly, explicitly?
Academics’ pedagogical conceptions of how to promote research ethics and integrity.
Journal of Academic Ethics, 15, 23-41.
McGreal, R., & Olcott Jr, D. (2022). A strategic reset: micro-credentials for higher education
leaders. Smart Learning Environments, 9(1), 9.
MER (Ministry of Education and Research). (2021). Education Strategy 2021–2035.
Retrieved from https://www.hm.ee/en/media/2919/download (25.01.2023)
Mustajoki, H., & Mustajoki, A. (2017). A new approach to research ethics: Using guided
dialogue to strengthen research communities. Taylor & Francis.
Selvaratnam, R. M., & Sankey, M. D. (2021). An integrative literature review of the
implementation of micro-credentials in higher education: Implications for practice in
Australasia. Journal of Teaching and Learning for Graduate Employability, 12(1), 1-17.
Tammeleht, A., Löfström, E. & Rodríguez-Triana, M. J. (2022). Facilitating development of
research ethics leadership competencies. International Journal of Educational Integrity.
https://doi.org/10.1007/s40979-022-00102-3
Wright, D. E., & Schneider, P. P. (2010). Training the Research Integrity Officers (RIO):
The federally funded. Journal of Research Administration, 41(3), 99-117.
740
Higher education student work placement and employability

Badroonesha Aumjaud1, Brinda Ramasawmy2, Brigitte Marie Francoise Driver2,
Deena Ramful-Baboolall1
1
Department of Agricultural & Food Science, Faculty of Agriculture, University of
Mauritius, 2Department of Agricultural Production & Systems, Faculty of Agriculture,
University of Mauritius, Réduit, Mauritius.
Abstract
This study aimed to evaluate student work placement experience with reference
to employability skills. An email/phone survey was conducted in 2016/2017
with on-site placement supervisors and alumni of the Faculty of Agriculture,
University of Mauritius, who had completed 6-month work placements.
Placement specifications and evaluation responses were retrieved from
records. 25 out of 28 (89.3%) food science and technology students had
secured employment less than one year after graduation, with 14.3% being
employed by organisations where they undertook the placement. 21 out of 25
(84%) alumni stated that the placement opportunity had enhanced their
employability. Placement providers commented positively on students’
attitude, progress and output. Student work placements may have contributed
to boost graduate employability. The Faculty of Agriculture 6-month work
experience programme was extended to other courses. From 2014 to 2023, it
has included 221 students who have been trained for jobs in the agricultural
and food sectors.
Keywords: Higher education; student work placement; food science;
agriculture; graduate youth unemployment; employability.

1. Introduction
Unemployment, working poverty, labour market inequalities and poor quality employment,
especially among the youth and women in developing countries, are concerns which need to
be addressed globally (ILO, 2019). The United Nations Sustainable Development Goal 8
(SDG 8) promotes inclusive and sustainable economic growth, employment and decent work
for all (UN, 2023). However, the International Labour Office reports major deficits in decent
work and warns that attaining SDG 8 seems unrealistic for many countries. Young people,
especially young women, are disproportionately affected (ILO, 2019). According to the
United Nations (2020), unemployment among youth is one of the greatest global issues.
OECD (2023) defines youth unemployment rate as the number of unemployed 15-24 year-
olds expressed as a percentage of the youth labour force. In 2016, it was estimated that over
40% of the world’s active youth was expected to be unemployed or have a job but live in
poverty (ILO, 2016). The COVID-19 pandemic exacerbated youth employment challenges
with anticipated scarring effects. The proportion of youth not in employment, education and
training increased to 23.3 % in 2020, an unprecedented level in at least 15 years (ILO, 2022).
Mauritius, a small island developing State (SIDS), located off the southeast coast of the
African continent in the Indian Ocean, has a population of 1.26 million (Statistics Mauritius,
2022a) and an adult unemployment rate estimated at 8.7% for the first quarter of 2022
(Statistics Mauritius, 2022b). Youth experience a higher-than-average unemployment rate
which was 25.3% in 2021 (World Bank, 2023a). Two main factors contribute to youth
unemployment: some sectors such as agriculture and export-oriented enterprises, no longer
attract low skilled youth due to working conditions and social status associated with these
jobs; youth competencies do not meet the demand of employers for proficiency in technical
skills (World Bank, 2019).
To tackle the youth employment crisis, the United Nations launched the Global Initiative on
Decent Jobs for Youth in 2016. The goal is to leverage knowledge, alliances and resources
to create action that leads to tangible results for young people (Decent Jobs for Youth
Organisation, 2023). The nexus between youth education and employment was recognised
by the United Nations declaration in 2015 with the following statement: “all countries stand
to benefit from having a healthy and well-educated workforce with the knowledge and skills
needed for productive and fulfilling work and full participation in society” (UN, 2018). Thus,
youth education plays a vital role in developing knowledge, values and competencies
required to secure decent jobs for a dignified life and sustainable growth.
Since independence in 1968, Mauritius has undergone economic transformation from a low-
income, agriculturally-based economy to a diversified, upper middle-income economy with
growing industrial, financial, and tourist sectors (Moody’s Analytics, 2023). Government
strategies have contributed to increase access to higher education and build the knowledge-
742
Badroonesha Aumjaud et al.
based economy towards the creation of an education hub (Knight and Motala-Timol, 2022).
The gross tertiary enrollment rate increased from 3 % in 1970 to 44 % in 2020 (World Bank,
2023b). However, graduate youth unemployment has emerged as a socioeconomic concern.
The proportion of unemployed youth who had attained tertiary education increased from
7.9% (2010) to 17.4% (2014) and 20.9% (2021) (Ministry of Labour, Industrial Relations,
Employment and Training, 2015; Statistics Mauritius, 2021). Skills mismatch has been
reported as a key determinant of jobless university graduates (Ministry of Labour, Industrial
Relations, Employment and Training, 2015; Ndyali, 2016). According to Hardin-Ramanan,
Ballasoupramanien, Gopee, Rowtho and Charoux (2017), corporates and educational
institutions in Mauritius identified lack of experience/expertise and no prior
preparation/training as important graduate work-readiness challenges.
To address youth unemployment and the graduate skills gap, the Government of Mauritius
has invested in initiatives such as the Youth Employment Programme and the Graduate
Training for Employment Scheme which engages employers in the skill formation process
(Ministry of Foreign Affairs, Regional Integration and International Trade, 2019). The
University of Mauritius has been supportive of the government’s vision to promote graduate
employability and initiated the Otto Essien Programme (United Nations Development
Programme/University of Mauritius) student placements project in 1994 and 1996-1998. The
Student Work Experience Programme (SWEP), was introduced in 1999 and was designed to
expose students to short work placements, real life tasks and activities. It was integrated with
the Work-based Learning (WBL) programme for a more structured, modular and progressive
approach, and offered to students as an optional experiential learning opportunity (UoM,
2023).
The Faculty of Agriculture of the University of Mauritius plays a leading role in national
capacity building for the agricultural, food and biotechnology sectors (UoM, 2023).
Interaction with stakeholders, alumni and students signalled the need to integrate student
work experience within the programmes of study. In 2002, the Faculty responded proactively
by including compulsory 6 to 8 weeks student placements in some undergraduate courses.
Student work placements in the field of study were organised by the Faculty of Agriculture
and undertaken in external organisations to create opportunities for application of acquired
theoretical knowledge. The Faculty experiential learning provision was embedded in other
undergraduate and postgraduate courses to benefit a larger number of students. In 2012, the
short placement was extended to 6-month sandwich work placement in one undergraduate
food science and technology course in light of students’ and employers’ feedback. The first
cohort of graduates who had completed a 6-month work placement entered the job market in
2016. In this context, the study intended to evaluate student work placement experience with
reference to employability skills to inform future practice and research.
743
2. Methodology
2.1. Placement students, organisations and academics

77 students undertook 6-month work placements in 2014 and 2016. Undergraduate students
belonged to the 18-24 age-group, with a high proportion of young women (81.2%). They
were from Mauritius and Rodrigues, a dependency of Mauritius and the smallest of the
Mascarene islands. Postgraduate students on the MSc Agribusiness Management, EU funded
intra ACP mobility programme, were from 4 African countries and Rodrigues. 31 public and
private organisations in the agricultural, food and hospitality sectors provided 6-month
placement opportunities. Each placement provider hosted 1 to 4 students who were mentored
by on-site supervisors. Placement providers designated one or more on-site supervisors
depending on the number of students being placed at the organisation. 21 academics of the
Faculty of Agriculture, University of Mauritius, were appointed as academic supervisors to
guide, monitor and assess students’ performance during the 6-month work placement, as per
guidelines of a Faculty work placement handbook.
2.2. Data collection participants, instruments, methods and analysis

On-site supervisors’ and students’ placement evaluation reports were maintained as records
by academic supervisors at the Faculty of Agriculture, University of Mauritius. Student work
placement reports were reviewed to retrieve objective evidence relating to learning outcomes.
The “supervisor’s final evaluation form” was completed by on-site supervisors. It consisted
of a quantitative assessment of student competencies and 1 open-ended question on student
performance. The “student work placement evaluation form” was filled in by students and
contained open as well as closed-ended questions on quality of experience at the organisation
concerned. An email survey was administered (December 2016-January 2017) with 18 on-
site supervisors from 14 host organisations (45% of placement providers in 2014 and 2016),
as well as 25 alumni of the Faculty of Agriculture, representing 32% of the student population
who had completed the 6-month work placement. Feedback questionnaires included open-
ended and closed-ended questions. The alumni feedback questionnaire defined the term
“employability” as “the ability to secure a relevant job and having the required knowledge,
skills and attitude to fulfil job responsibilities effectively”. A phone survey was conducted
to establish the employment status of 28 Faculty alumni who graduated with a degree in food
science and technology with 6-month work placement in April 2016. 5 academics and 1
administrative staff shared their reflections on the organisation, implementation and
assessment of student work placements. Their thoughts were critically analysed in light of
the internal context as well as the educational, economic and social landscape in Mauritius.
Quantitative data was processed by making use of descriptive statistics. Qualitative data was
categorised and themes were identified based on the frequency of reported issues to provide
insights into the outcomes of experiential learning.
744
3. Results & Discussion

Table 1. Alumni and placement providers’ feedback.
Alumni On-site supervisors
§ 25 out of 28 (89.3%) had secured § Positive comments on students’

employment attitude and output: skills
enhancement; progress in
§ 22 out of 28 (78.6%) had obtained a job in
learning and developing
the food sector
professional behaviour;
§ 4 out of 28 (14.3%) were employed by contribution to organisation’s
organisations where they undertook the activities
placement
§ All on-site supervisors
§ 6 out of 28 (21.4%) were recruited by a expressed their organisation’s
major group of food companies in Mauritius willingness to participate in the
student work placement
§ 21 out of 25 (84%) stated that the 6-month
programme in the future
placement had enhanced their employability
§ Suggestions for improvement:
§ Suggestions for improvement: exposure to
enhanced communication with
various departments of the organisation; on-
academic supervisor; more
site visit of academic supervisor; more
clarity in articulation of
interaction between on-site and academic
programme expectations; less
supervisors; payment of stipend; sandwich
paperwork; extension of
placement shifted to the end of the course to
placement activities to research
enhance chances of securing a job in the
undertakings
organisation
Table 2. Learners’ reported placement benefits and Faculty’s staff reflections.
Alumni & students’ reported benefits Academic & administrative staff reflections
§ 33 out of 40 (82.5%) rated placement § Academic supervisors: eye-opening

experience as good or excellent experience and connectivity with the
food system
§ 30 out of 38 (78.9%) considered that
there were job opportunities in § Administrative staff : constructive
placement organisations learning experience
§ 27 out of 38 (71.1%) were paid a § Challenges: securing high number of

stipend placements; ensuring student work
placement readiness; equivalence of
§ Best part of work placement and
work placement opportunities;
aspects relevant to the programme of
workplace language barrier for
study: development of knowledge, hard
regional students
and soft skills
745
89.3% of food science and technology students with 6-month work placement experience
secured employment in less than one year after graduation, with 78.6% obtaining a job in the
food sector and 14.3% being employed in host organisations. 84% of alumni surveyed stated
that the placement had enhanced their employability (Table 1). Furthermore, alumni and
students stated that they developed knowledge as well as hard and soft skills during the 6-
month work placement (Table 2). 82.5% rated their placement experience as good or
excellent and on-site supervisors commented positively on students’ attitude and output
(Table 1 & Table 2). According to a literature review on higher education work
placement/experience research in the United Kingdom, work placement benefits realised by
students include general skills development and greater ability to secure a job (Atfield, Hunt
and Luchinskaya, 2021). Other authors have also reported the positive effect of work
placement on employability skills in Italian and Malaysian contexts (Volpe, 2017; Abdul
Wahab, 2022). Teng, Ma, Pahlevansharif and Turner (2019) provided empirical evidence of
a significant relationship between soft skills development and job readiness from students’
perspectives in Malaysian and Chinese university settings. Two competing theoretical
mechanisms have been proposed to explain placement’s positive labour market outcomes:
the human capital effect which involves skills development; the signalling effect which
relates to the assumption that high-ability students opt for voluntary placements to get spotted
and not necessarily acquire skills (Atfield et al., 2021). Inceoglu, Selenko, McDowall and
Schlachter (2019) have postulated a theoretical model which depicts that intertwined learning
and re-structuring of identity processes in novel social environments influence career-related
outcomes of placement experiences. Thus, student’s receptiveness and ability might have
contributed to integrate learning cycles and identity change into meaningful competencies.
On-site supervisors valued students’ contribution and expressed willingnesss to maintain
their organisation’s participation as placement providers in the future (Table 1). Atfield et
al. (2021) reported that employers mentioned the following student work placement benefits:
inflow of novel ideas and cheaper human resource. Interestingly, in the present study, alumni
suggested payment of a stipend to students (Table 1) which could be important for financial
autonomy and self-esteem. Arguments for paid internships include motivation to deliver
quality work, job performance and efficiency (Belleenfant, 2022; Legal Service India, 2023).
Other suggestions for improvement related to communication/interaction between on-site
and academic supervisors (Table 1). Faculty’s staff reflected on their constructive experience
in arranging and supervising work placements. However, key challenges involved securing
a high number of equivalent placement opportunities and ensuring students were well
prepared for transition into real-world situations (Table 2). Divine, Linrud, Miller and
Wilson (2007) elaborated on HEI time and resource challenges in getting enough internships
offers in the US context. Moreover, previous studies reported that joint learning materials
developed between universities and employers, effective student support and integration of
reflexive activities were crucial for work placement success (Atfield et al., 2021).
746
4. Conclusions
From the findings of this study, it can be concluded that students who completed 6-month
work placements as a requirement of courses at the Faculty of Agriculture, University of
Mauritius, experienced positive outcomes in terms of skills development and employability.
Placement providers and Faculty’s staff demonstrated positive attitudes to the student work
placement programme. Since 2017, more students have benefited from 6-month work
experience. Some identified areas of improvement have been addressed to ensure quality of
experiential learning. For example, a professional development module was introduced to
enhance student preparedness for transition to employment. Language skills and intercultural
awareness is an emerging challenge in the context of higher education internationalisation.
Sustainability of workplace learning opportunities depends on continued engagement of
Faculty’s staff, students, alumni and stakeholders. It is important to strengthen external
organisation, alumni and student participation in design, planning and communication of
experiential learning activities for sustained career-related outcomes. More research is
needed on work placement/internship learning processes as well as benefits and costs to
students, HEIs and employers.
Acknowledgements
The authors wish to express their gratitude to: Associate Professor D Puchooa, Dean, Faculty
of Agriculture, University of Mauritius; Professor S Facknath; Mr K Boodhoo, Head of
Department of Agricultural Production & Systems; Dr S H Neetoo, Head of Department of
Agricultural & Food Science; Faculty’s staff, students, alumni and on-site supervisors.
References
Abdul Wahab, D. (2022). Who's to blame for graduate unemployment? Malaysia Now.
Atfield, G., Hunt, W., & Luchinskaya, D. (2021). Employability programmes and work
placements in UK higher education. Research report. UK: Department for Education.
Belleenfant, R. (2022). Top reasons to offer paid internships. Technology Advice.
Decent Jobs for Youth Organisation (2023, February 9). Retrieved from:
https://www.decentjobsforyouth.org/about
Divine, R.L., Linrud, J.K., Miller, R.H., & Wilson, J.H. (2007). Required internship
programs in marketing: benefits, challenges and determinants of fit. Marketing Education
Review, 17(2), 45-52. DOI: 10.1080/10528008.2007.11489003
Hardin-Ramanan, S., Ballasoupramanien, L., Gopee, S., Rowtho, V., & Charoux, O. (2017).
Towards graduate work readiness in Mauritius: a multi-stakeholder approach. Ebène,
Mauritius: MRC report.
Hardin-Ramanan, S., Gopee, S., Rowtho, V., & Charoux, O. (2020). Graduate work-
readiness in Mauritius: a multi-stakeholder approach. Journal of Teaching and Learning
for Graduate Employability, 11(1), 93-109.
747
ILO (2016). World employment social outlook. Trends for Youth 2016.
ILO (2019). Poor working conditions are main global employment challenge.
ILO (2022). Recovery in youth employment is still lagging, says ILO.
Inceoglu, I., Selenko, E., McDowall, A., & Schlachter, S. (2019). (How) Do work
placements work? Scrutinizing the quantitative evidence for a theory-driven future
research agenda. Journal of Vocational Behaviour, 110, 317-337.
doi.org/10.1016/j.jvb.2018.09.002
Knight, J., & Motala-Timol, S. (2022). Mauritius as a developing educational hub.
International Journal of Educational Development, 93, 102646.
doi.org/10.1016/j.ijedudev.2022.102646
Legal Service India (2023, February 8). Putting a full stop to unpaid internship. Retrieved
from: https://www.legalserviceindia.com/legal/article-6208-putting-a-full-stop-to-
unpaid-internship.html
Ministry of Foreign Affairs, Regional Integration & International Trade (2019). Voluntary
national review report of Mauritius. Port-Louis, Republic of Mauritius: MFARIIT.
Ministry of Labour, Industrial Relations, Employment & Training (2015). Enhancing the
employability of job seekers. Report of the Director of Audit. Mauritius.
Moody’s Analytics (2023). Mauritius economic indicators.
Ndyali, L. (2016). Higher education system and jobless graduates in Tanzania. Journal of
Education and Practice, 7(4), 116-121.
OECD (2023). Youth unemployment rate (indicator). doi: 10.1787/c3634df7-en
Statistics Mauritius (2021). Labour force, employment and unemployment.
Statistics Mauritius (2022a). Population and vital statistics, January-June 2022.
Statistics Mauritius (2022b). Labour force, employment and unemployment.
Teng, W., Ma, C., Pahlevansharif, S. & Turner, J.J. (2019). Graduate readiness for the
employment market of the 4th industrial revolution. The development of soft
employability skills. Education & Training, 61(5), 590-604. DOI 10.1108/ET-07-2018-
0154
UN (2018). World youth report. Youth and the 2030 agenda for sustainable development.
UN (2023, February 9). Making the SDGs a reality. Retrieved from: https://sdgs.un.org/
United Nations (2020). World youth report. Youth social entrepreneurship and the 2030
agenda.
UoM (2023, February 9). University of Mauritius web-site. Retrieved from:
https://www.uom.ac.mu/
Volpe, M. (2017). Assessment of internship effectiveness in South Italy universities.
Education & Training, 59 (7/8), 797-810. DOI 10.1108/ET-08-2016-0136
World Bank (2019). Mauritius: earnings mobility and inequality of opportunity in the labor
market in four charts.
World Bank (2023a). Unemployment, youth total (% of total labour force, ages 15-24)
(modeled ILO estimate) – Mauritius.
World Bank (2023b). School enrollment, tertiary (% gross) – Mauritius.
748
When intercultural education is problematic: the case of Russian

as a foreign language
Linda Torresin
Department of Linguistic and Literary Studies (DiSLL), University of Padua, Italy.
Abstract
This paper proposes a reflection on the issues posed by the intercultural
teaching of Russian as a foreign language (RFL) in the university context.
In particular, after defining the concepts of “interculturalism”/“intercultural
education” and outlining an updated picture of research on intercultural
education in RFL, two of the greatest problems of RFL intercultural education
are presented: on the one hand, the ambiguity of the theoretical treatment of
intercultural education (and, thus, of its practical applications) by RFL
scholars, and, on the other hand, the often essentialized, if not stereotypical,
portrayal of Russian culture in RFL textbooks, which in some cases distort
reality.
Finally, a possible solution is proposed to resolve these issues and make RFL
teaching more intercultural.
Keywords: Russian as a foreign language; intercultural education;
intercultural teaching; interculturalism; Russian culture; textbooks.

When Intercultural Education is Problematic: The Case of Russian as a Foreign Language
1. Introduction
When talking about the concept of “interculturalism” (Abdallah-Pretceille, 2006; Besley &
Peters, 2012; Cantle, 2012; Costa & Lacerda, 2007; Dervin, 2016; Kastoryano, 2018; Meer
et al., 2016; Penas Ibáñez & López Sáenz, 2006; Sarmento, 2014; Verkuyten et al., 2019;
Zapata-Barrero & Mansouri, 2022), we generally refer to a philosophy or viewpoint that
involves support for dialogue between cultures and challenges self-segregation of cultures,
ultimately leading to an “intercultural education” (IE), the foundations of which have been
laid by the policies of the European Union and UNESCO since the 1990s according to these
same assumptions (see, among others: Beacco, 2013; Byram, 2003, 2006, 2009; Byram &
Zarate, 1995; Byram et al., 1997; Byram et al., 2002, 2009; Coste et al., 2009; Deardorff,
2020; UNESCO, 2006, 2010, 2013).
In the research field of Russian as a foreign language (RFL), IE is commonly grounded in
two basic components: “intercultural communication” (mezhkul’turnaya kommunikatsiya)
and “intercultural communicative competence” (mezhkul’turnaya kommunikativnaya
kompetentsiya).
Scholars tend to qualify intercultural communication as “an adequate mutual understanding
of two participants of a communicative act who belong to different national cultures”
(Vereshchagin & Kostomarov, 1973, p. 43; hereafter, the translations are mine). The
precondition for and, at the same time, the guarantee of intercultural communication taking
place is intercultural communicative competence, which has been defined as “the individual’s
ability to exist in a multicultural society, to be successfully understood by representatives of
other cultures and by representatives of one’s own culture” (Azimov & Shchukin, 2009,
p. 134).
The topic of IE, which has become an object of study in the Russian context since the early
2000s with the publication of Gudkov’s (2000) and Ter-Minasova’s (2000) works devoted to
intercultural communication, has been extensively investigated in RFL from various
perspectives, with a focus on both theoretical and didactic-methodological aspects (see,
among others: Amelina, 2022; Antonova & Arsenyeva, 2019; Berdichevskiy, 2021;
Berdichevskiy et al., 2011, 2020; Chumak, 2009, pp. 195–199; Nemtchinova, 2020;
Petrikova et al., 2015; Tarchimaeva, 2017).
However, with a few exceptions (e.g., Torresin, 2022a), in RFL research, there is still a lack
of critical reflection on the concept of IE that may challenge certain established scientific
ideas or orientations and at the same time help to improve the intercultural teaching of RFL
in practice.
If IE in general, understood as above, alongside its undoubtedly positive sides, has over time
shown dark areas or critical aspects that are difficult to resolve to the point of being criticized
750
Linda Torresin
in its very nature and even being related to colonialism (see, e.g., the studies of Aman, 2015,
2017, 2019), IE in the RFL area, as we will see, does not escape this picture either.
This contribution aims to outline two of the greatest problems of RFL intercultural teaching
with a focus on the university context: on the one hand, the ambiguity of the theoretical
treatment of IE (and, thus, of its practical applications) by RFL scholars (§2.1), and, on the
other hand, the often essentialized, if not stereotypical, portrayal of Russian culture in RFL
textbooks, which in some cases distort reality (§2.2). Finally, a possible solution is offered
to solve the critical issues of IE in RFL and make RFL teaching more intercultural (§3).
2. Problems of RFL intercultural education

This paper discusses the following as the two main problems of RFL IE in the university
context: a) how RFL research approaches the concept of IE (§2.1) and b) how RFL textbooks
approach the concept of culture (§2.2).
2.1. Intercultural education and ambiguity in RFL studies

It is a well-established fact that scholars still do not agree on what should be meant by
“interculturalism” and the correlated concepts “intercultural communication,” “intercultural
communicative competence,” “intercultural dialogue,” and others, and that intercultural
processes may be looked at through different lenses, perspectives, and approaches (see., e.g.,
Delanoy, 2020; ten Thije 2020).
However, in the RFL area, the situation is even more complicated. In RFL research on IE, on
the one hand, a definition of IE—of its principles and characteristics referable to
interculturalism—is seldom provided (see Torresin, 2022a, 2022b), thus giving rise to a wide
variety of interpretations. On the other hand, the intercultural dimension, when defined and/or
interpreted, is represented in a way that often does not coincide with interculturalism (as we
outlined it) but is rather closer to “multiculturalism” (see Torresin, 2022b)—that is, a simple
juxtaposition of cultures with the aim of merely “decreasing inter-ethnic tensions,” in which,
rather than intercultural dialogue, cultures are invited to a more passive “education for
tolerance” (Azimov & Shchukin, 2009, p. 149).
Adding to the ambiguity of uses of the conception of IE in the RFL field is the essentialist
treatment of the concept of culture itself (which we will also return to in the next section
when discussing RFL textbooks).
Culture is generally conceived by RFL scholars in an exclusively national tone, that is, as
merely Russian (see Torresin, 2022a, 2022b), which testifies to a simplified and monolithic
view of Russian culture deviating greatly from the foundations and principles of IE. This
view completely excludes the transnational “Russophone” world (Caffee, 2013), which
751
includes Russian-speaking people who are not ethnic Russians, such as many contemporary,
internationally renowned Russian-language writers (see Torresin, 2023a).
To sum up, even the most recent RFL studies treat IE in an ambiguous way that does not
serve the development of IE itself, as the definitions of intercultural processes and related
concepts either lack or are vitiated by multiculturalism (i.e., the pursuit of tolerance rather
than dialogue between cultures) on the one hand and, on the other hand, by a simplified and
essentialized view of culture itself (understood as purely national, i.e., Russian, or, in other
words, in such a way as to exclude the equally important Russophone component).
2.2. RFL textbooks and the distortion of reality

If RFL research approaches the concept of IE in an ambiguous, simplistic, and essentialized
way that extends to the same characterization of the target culture, when considering concrete
teaching tools at the university level, such as RFL textbooks, the perception and
representation of Russian culture here is also rather questionable.
In fact, generally speaking, authors of RFL textbooks seem to rely on (or, if they are
academics, to belong to) the RFL didactic-methodological research illustrated above, with
which they share a simplified and essentialized view of Russian reality. This influence is
evident from the frequently biased, distorted, if not benevolent or even stereotyped portrayal
of the Russian world offered by many RFL textbooks (see Torresin, 2022c, 2023b).
In more detail, such a portrayal of the Russian world is conveyed through the omission of
some aspects (deemed secondary) over others for subjective and debatable reasons (see
Torresin, 2023b). For example, in the well-known RFL textbook Russkij yazyk: 5 elementov
(Esmantova, 2008–2011), which offers very few cultural topics, more space is reserved for
geography, whereas modern and contemporary Russian culture are completely ignored.
Among the deliberately omitted elements is also the Russophone aspect. In other words, the
only dimension represented in textbooks is generally that of national Russian culture. For
example, in Poyekhali (Chernyshov & Chernyshova, 2019–2022), the few cultural elements
given refer to the national dimension (Russian holidays, Russian recipes, etc.), especially the
classical (among the Russian writers mentioned, we may find Pushkin, Chekhov, Tolstoy,
and Dostoevskiy), while the transnational one (e.g., Russian-speaking world, contemporary
Russian-language writers, etc.) is not present. This reinforces the idea—already present in
RFL research, as we have seen—of Russian culture as purely national, monolithic, and static,
which is precisely the idea that passes, through the textbooks, to RFL learners.
Moreover, there are quite a few RFL textbooks that propose the myth of the “Russian soul”
(russkaya dusha) as a learning component (regarding the Italian educational context, see
Torresin, 2022c), thereby fostering the reinforcement of this monolithic and essentialized
idea of Russian culture. This occurs also in the very recent textbook Voyazh po-russki
752
Linda Torresin
(Moskalёva et al., 2020), where the Russian identity is depicted as unique and based on
certain constants (tension towards the vast spaces, tendency to sadness or anxiety, strong
sense of hospitality, etc.) attributable to the “Russian soul” (pp. 73, 250).
In summary, in many RFL textbooks, a distortion of reality takes place, resulting in the
partial, essentialized, and even stereotyped representation of Russian culture, interpreted as
national and endowed with fixed, unique, and unrepeatable traits (“Russian soul”).
3. Conclusions
As we have seen, the implementation of IE in the RFL context at the university level is
problematic for two reasons: on the one hand, because of the ambiguity of the theoretical
treatment of IE by RFL scholars, and, on the other hand, because of the biased and
essentialized—if not outright stereotypical—portrayal of Russian culture in RFL textbooks.
In my view, a solution to both problems would be improving the critical reflection on the
concept of IE. This could be done if RFL scholars and textbook authors were to dialogue
both with intercultural studies, particularly intercultural pedagogy, where “culture” is treated
as a complex and multifaceted object of study in an anthropological and sociological sense,
and with international intercultural studies, which recognize the complexity of culture.
Clearly, this requires a change that involves both RFL research and teaching practice and that
also goes through textbooks.
Funding
This study was conducted at Vilnius University with the financial support of the University
of Padua. Funding programme: “Seal of Excellence @UNIPD”. Project: “RETEACH”
(https://reteach.disll.unipd.it/). Project code: TORR_MSCASOE21_01.
References
Abdallah‐Pretceille, M. (2006). Interculturalism as a paradigm for thinking about diversity.
Intercultural Education, 17(5), 475-483. doi: 10.1080/14675980601065764.
Aman, R. (2015). In the name of interculturality: On colonial legacies in intercultural
education. British Educational Research Journal, 41(3), 520-534. doi:
10.1002/berj.3153.
Aman, R. (2017). Colonial differences in intercultural education: On interculturality in the
Andes and the decolonization of intercultural dialogue. Comparative Education Review,
61(S1). doi: 10.1086/690459.
Aman, R. (2019). Decolonising intercultural education: Colonial differences, the geopolitics
of knowledge, and inter-epistemic dialogue. London: Routledge.
753
Amelina, I. O. (2022). Organizatsiya mezhkul’turnogo dialoga v ramkakh

smeshannogoobucheniya russkomu yazyku kak inostrannomu. Russkyi yazyk za
rubezhom, 4, 33-40. doi: 10.37632/PI.2022.293.4.005.
Antonova, N., & Arsenyeva, I. (2019). Intercultural communication in teaching Russian as a
foreign language to Chinese and Vietnamese students. ICERI2019 Proceedings, 11558-
11564.
Azimov, E. G., & Shchukin, A. N. (2009). Novyy slovar’ metodicheskikh terminov i ponyatiy
(teoriya i praktika obucheniya yazykam). М.: Ikar.
Beacco, J.-C. (2013). Specifying languages’ contribution to intercultural education: Lessons
learned from the CEFR. Strasbourg: Council of Europe.
Berdichevskiy, A. L. (2021). Soderzhaniye mezhkul’turnogo inoyazychnogo obrazovaniya
vuze. Russkiy yazyk za rubezhom, 3, doi: 10.37632/PI.2021.286.3.001.
Berdichevskiy, A. L., Giniatullin, I. A., Lysakova, I. P., & Passov, E. I. (2011). Metodika
mezhkul’turnogo obrazovaniya sredstvami russkogo yazyka kak inostrannogo. Kniga
dlya prepodavatelya. M.: Russkiy yazyk. Kursy.
Berdichevskiy, A. L., Giniatullin, I. A., & Tareva, E. G. (2020). Metodika mezhkul’turnogo
inoyazychnogo obrazovaniya v vuze. Uchebnoye posobiye. M.: Flinta.
Besley, T., & Peters, M. A. (Eds.). (2012). Interculturalism, education and dialogue. New
York: Peter Lang.
Byram, M., & Zarate, G. (1995). Young people facing difference: Some proposals for
teachers. Strasbourg: Council of Europe.
Byram, M., Zarate, G., & Neuner, G. (1997). La compétence socioculturelle dans
l’apprentissage et l’enseignement des langues. Strasbourg: Conseil de l’Europe.
Byram, M., Gribkova, B., & Starkey, H. (2002). Developing the intercultural dimensionin
language teaching: A practical introduction for teachers. Strasbourg: Council of Europe.
Byram, M. (Ed.). (2003). Intercultural competence. Strasbourg: Council of Europe.
Byram, M. (2006). Languages and identities. Strasbourg: Council of Europe.
Byram, M. (2009). Multicultural societies, pluricultural people and the project of
intercultural education. Strasbourg: Council of Europe.
Byram, M., Barrett, M., Ipgrave, J., Jackson, R., & Méndez García, M. del C. (2009).
Autobiography of intercultural encounters. Strasbourg: Council of Europe.
Caffee, N. B. (2013). Russophonia: Towards a transnational conception of Russian-
language literature. PhD dissertation. University of California.
Cantle, T. (2012). Interculturalism: The new era of cohesion and diversity. London: Palgrave
Macmillan. Doi: 10.1057/9781137027474.
Chernyshov, S. I., & Chernyshova, A. V. (2019-2022). Poyekhali! SPb.: Zlatoust.
Chumak, L. N. (2009). Metodika prepodavaniya russkogo yazyka kak inostrannogo. Minsk:
BGU.
Costa, J. P., & Lacerda, T. (2007). A interculturalidade na expansão Portuguesa (Séculos
XV-XVIII). Lisbon: ACIME.
Coste, D., Moore, D., & Zarate, G. (2009). Plurilingual and pluricultural competence.
Strasbourg: Council of Europe.
754
Linda Torresin
Deardorff, D. K. (2020). Manual for developing intercultural competencies: Story circles.

Paris: UNESCO.
Delanoy, W. (2020). What is culture? In G. Rings & S. Rasinger (Eds.), The Cambridge
handbook of intercultural communication (pp. 17-34). Cambridge: Cambridge University
Press. doi:10.1017/9781108555067.003.
Dervin, F. (2016). Interculturality in education: A theoretical and methodological toolbox.
London: Palgrave Pivot. doi: 10.1057/978-1-137-54544-2.
Esmantova, T. (2008-2011). Russkiy yazyk: 5 elementov. SPb.: Zlatoust.
Gudkov, D. B. (2000). Mezhkul’turnaya kommunikatsiya: problemy obucheniya. M.: Izd-vo
Moskovskogo universiteta.
Kastoryano, R. (2018). Multiculturalism and interculturalism: Redefining nationhood and
solidarity. Comparative Migration Studies, 6(17). doi: 10.1186/s40878-018-0082-6.
Meer, N., Modood, T., & Zapata-Barrero, R. (2016). Multiculturalism and interculturalism:
Debating the dividing lines. Edinburgh: Edinburgh University Press. doi:
10.1515/9781474407106.
Moskalёva, L., Shakhmatova, T., & Murtazina, D. (2020). Russian Journey. SPb.: Zlatoust.
Nemtchinova, E. (2020). Developing intercultural competence in a Russian language class.
In E. Dengub, I. Dubinina, & J. Merrill (Eds.), The art of teaching Russian (pp. 333-358).
Washington (DC): Georgetown University Press.
Penas Ibáñez, B., & López Sáenz, M. C. (Eds.). (2006). Interculturalism: Between identity
and diversity. New York: Peter Lang.
Petrikova, A., Kuprina, T., & Gallo, Ya. (2015). Osnovy mezhkul’turnoy didaktiki. M.:
Russkiy yazyk. Kursy.
Sarmento, C. (2014). Interculturalism, multiculturalism, and intercultural studies:
Questioning definitions and repositioning strategies. Intercultural Pragmatics, 11, 603-
618. doi: 10.1515/ip-2014-0026.
Tarchimaeva, L. C. (2017). Pedagogicheskaya strategiya mezhkul’turnogo obrazovaniya
vusloviyakh obucheniya russkomu yazyku za rubezhom. Pedagogicheskiy IMIDZH,
4(37), 111-119.
Ten Thije, J. D. (2020). What is intercultural communication? In G. Rings & S. Rasinger
(Eds.), The Cambridge handbook of intercultural communication (pp. 35-55).
Cambridge: Cambridge University Press. doi:10.1017/9781108555067.004.
Ter-Minasova, S. G. (2000). Yazyk i mezhkul’turnaya kommunikatsiya. M.: Slovo.
Torresin, L. (2022a). La “competenza comunicativa interculturale” nell’insegnamento del
russo come LS (RKI). Teorie e pratiche didattiche, problemi e criticità. Scuola e Lingue
Moderne, 1-3, 22-28.
Torresin, L. (2022b). Vuzovskoye prepodavaniye russkoy literatury na urokakh RKI v
mezhkul’turnoy perspektive. In R. A. Ogorodnikova & A. A. Emel’yanova (pod red.),
Rossiya i mir: transnatsional’nye kommunikatsii i vzaimoproniknoveniye kul’tur: Sbornik
statey mezhdunarodnoy mezhdisciplinarnoy nauchnoy konferentsii (Moskva, 22 aprelya
2022 g.) (pp. 356-365). M.: Knigodel.
755
Torresin, L. (2022c). “Russkaya dusha” v prepodavanii RKI: kriticheskiy analiz

stereotipnykh diskursov v sovremennoy didakticheskoy praktike. JIKUVIP, X-3.
Novi Sad: University of Novi Sad, 451-461.
Torresin, L. (2023a). K voprosu o rabote s khudozhestvennym tekstom na urokakh RKI
vuzakh: transnatsional’nyi podkhod. IV mezhdunarodnaya konferentsiya
“Natsional’naya identichnost’ skvoz’ prizmu dialoga kul’tur: Rossiya i ibero-
amerikanskii mir”. Rostov-na-Donu [in print].
Torresin, L. (2023b). Kakaya kul’tura? Problema kul’turnogo elementa v uchebnikakh po
RKI. Aktual’nye problemy prepodavaniya russkogo yazyka kak nerodnogo/inostrannogo:
rossiyskie i zarubezhnye praktiki. Stavropol’: FGAOU VO “Severo-Kavkazskiy
federal’nyi universitet” [in print].
UNESCO. (2006). UNESCO guidelines on intercultural education. Paris: UNESCO.
https://unesdoc.unesco.org/ark:/48223/pf0000147878 (last accessed: 20/01/2023).
UNESCO. (2010). Education for intercultural understanding. Paris: UNESCO.
https://unesdoc.unesco.org/ark:/48223/pf0000189051 (last accessed: 20/01/2023).
UNESCO. (2013). Intercultural competences: Conceptual and operational framework.
Paris: UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000219768 (last accessed:
20/01/2023).
Vereshchagin, E. M., & Kostomarov V. G. (1973). Yazyk i kul’tura. Lingvostranovedeniye v
prepodavanii russkogo yazyka kak inostrannogo. M.: Izd-vo MGU.
Verkuyten, M., Yogeeswaran, K., Mepham, K., & Sprong, S. (2019). Interculturalism: A new
diversity ideology with interrelated components of dialogue, unity, and identity
flexibility. European Journal of Social Psychology, 50(3), 505-519. Doi:
10.1002/ejsp.2628.
Zapata-Barrero, R., & Mansouri, F. (2022) A multi-scale approach to interculturalism: From
globalised politics to localised policy and practice. Journal of International Migration
and Integration, 23, 775-795. doi: 10.1007/s12134-021-00846-w.
756
Digital pedagogy for the present: an artificial intelligence

methodology for curriculum development
Michelle Rufrano1, Jean-Ezra Yeung2
1
Department of Sociology & Anthropology, Fordham University, U.S.A., 2Columbia Startup
Lab, Columbia University, U.S.A.
Abstract
In an age of information, course creation at the university-level poses new
challenges in that the constant influx of new data makes it difficult for
instructors to update courses with the latest research. Using Artificial
Intelligence (AI), we created a curriculum that matches the contemporary
experiences of students. This approach was used to help design a course on
Aging and Adulthood using the life course model. We were able to create a
topological map of topics and subtopics that provided the instructor with the
ability to quickly understand the shape of the data, thematic connections, and
data voids. Choosing from a myriad of research articles from a derived
skeleton of topics, the instructor was then able to design a unique course with
the most relevant research and supplement the syllabus with seminal works in
the field.
Keywords: Artificial intelligence; topological data analysis; digital pedagogy;
curriculum development; life course; epigenetics.

Digital pedagogy for the present: an AI methodology for curriculum development
1. Introduction
The first time I traveled to Spain I was 20 years old and when I left I cried all the way to the
airport. My señora was my best Spanish teacher. We would pantomime and dance to make
our meanings known. We needed no words when we parted, on her face was the sadness that
can only follow the great joy of being forever changed by someone significant—linked lives.
Linked lives is one of the five characteristics of the life course model. A framework that
articulates the cumulative effects of our life experiences over time (Elder Jr., 1998, p. 4). A
tool that can help us identify vulnerable periods within the context of historical events and
create opportunities for informed interventions (Elder Jr., 1998, p. 5). Inspired by my own
life experiences, I endeavored to teach a course entitled Coming of Age: Adulthood grounded
in the life course model. However, the life course model is a novel approach that has only
become part of public health curricula as of the last decade (Begg et al., 2015, p. 1).
Therefore, a meaningful course would not only need to cite seminal works in the field, and
discuss the key innovations in epigenetics, but also be relevant to students. I have now
successfully taught Coming of Age: Adulthood twice due to the application of artificial
intelligence in outlining aging.
Using Artificial Intelligence (AI), the curriculum was up-to-date even given the COVID-19
pandemic and the recent explosion of research in the fields of the life course model,
epigenetics, and exposome. Almost half (60,000+) of the articles (130,000+) containing the
search term “epigenetics” were published in PubMed during the last five years. The
preparation for this course was cut down to a few hours, only requiring a quick 15-minute
update from semester to semester.
2. Methodology
Using PubMed, a biomedical literature database accessed at
https://pubmed.ncbi.nlm.nih.gov/, the researchers extracted articles over 365 days ending on
December 8th, 2021, using search terms from a life course glossary curated by the
researchers. They used their domain expertise to determine the set of terms and boolean
operators (and/or). Data were extracted using the Biopython API (Cock et al., 2009) and
transformed data was loaded into a custom-built interactive Python Dash web app (Hossain,
2019) that leveraged topological data analysis. This included generating a topological
network (Singh et al., 2007), computing a persistence diagram (Zomorodian & Carlsson,
2004), and identifying the representative cycle (Tralie et al., 2018). Additionally, a
knowledge graph was constructed to supplement topological understanding by seeing the
conceptual relationships involved in different areas of the topological network.
758
Michelle Rufrano, Jean-Ezra Yeung
Figure 1. Topological network of Pubmed articles on the life course model
The procedure used to extract topics and sub-topics was a multi-step process. First, the
knowledge graph was used to identify the key concepts and their relationships. Those
concepts were mapped to the topological network’s clusters, which provided a rough
segmentation of articles into topic or sub-topic groups (Figure 1). In the topological network,
lines between nodes represent connectivity and similarity, and each node represents a set of
articles. The node color represents its similarity index, which is any function that summarizes
its relative distances from other data points into a single relative distance value. To determine
whether a cluster is a strong signal, the persistence diagram visualizes the overall distribution
of hierarchical clustering, so that we can determine how many topics are likely and when a
topic becomes saturated upon initial or repeated identification (Figure 2). In computing the
persistence diagram, we also extracted the representative cycle to determine any persistent
topological signal, as it highlights the most salient relationships between concepts across
articles. Upon discussion and review of an exported comma separated file (.csv) or Excel file
(.xlsx) containing article metadata and their cluster assignments, we determined topics and
selected works to include in the curriculum. Topics were labeled, created, modified and/or
collapsed to simplify the organization. Altogether, our AI tool extracts the shape of the
research topics without having to manually sort through article titles and abstracts. Thus, this
framework readily provides a skeleton of topics, subtopics, and their relationships.
759
Figure 2. Persistent Diagram of PubMed articles
3. Findings
In the first semester teaching the course, there were 11 topics and some selected subtopics
(Figure 3). As a semester is around 14 weeks at Fordham University, roughly one topic can
be used for each week and larger or more complex topics are split over two weeks (e.g.
Mental Health).
Figure 3. List of Topics and Subtopics
760
With the topic list set based on the topology of the life course, the instructor was then able
to select readings from each topic, resulting in a comprehensive, up-to-date syllabus. The
instructor supplemented a few seminal works to help guide the discussion of the significance
of this research from a social context. Overall, the total time for the course creation was
roughly two days.
The topological tools also allow researchers to recognize topic connectivity and circularity
(identified as loops) as well as research voids. The instructor can use these patterns to discuss
the relationships and limitations of data sources with students. The topological network of
aging and the life course model clearly demonstrated that understudied groups such as
women, racial and ethnic minorities, people who are differently abled or sexual minorities,
etc. are often left out of the dominant discourse of research—a significant research void.
4. Discussion
The utilization of AI expedited the process of course development with the most updated
research/ readings and positively impacted students. The individual articles covered a myriad
of topics relevant to the contemporary experience of students. So much so that discussion
easily flowed from week to week as we contended with studies on climate change, early
onset ADHD, longitudinal effects of early life adversities (ELA), epigenetic processes, and
maternal health from all over the world (Vergunst & Berry, 2022; Nigg et al., 2020; Suglia
et al., 2021; Simons et al., 2021; Vedam et al., 2019).
Students’ reactions were overwhelmingly positive, even saving articles to help them discuss
the realities of mental health with parents or the longitudinal effects of pre-term births with
personal doctors (Brenner & Bhugra 2020; Heikkilä et al., 2021).
Every semester students evaluate their courses and below are two anonymous statements
from Fordham University Student Course Evaluations (2022) that specifically spoke to the
course content:
I really appreciated the way the course was setup–up– with us reading
current studies that then relate to the following lecture. The studies having
been applicable to current events made the class extremely engaging and
interesting.
I really liked the presentations, they were informative and opened my mind
up to topics and issues I hadn't really learned about before.
Here are two direct emails from students after the course (2022):
Thank you for such an incredible semester of thought-provoking
discussions and readings. I truly learned so much
761
Taking your class was one of my most fulfilling experiences at Fordham

because the discussions and discourse we were able to have amongst peers
was a chance to explore deeper thoughts about what shapes us as human
beings and what effects are present amongst us in school communities,
relationships, work environments, and the world in general.
As an instructor, I seek to teach the courses that inspire this kind of passion in my students,
and while I may be an expert in my respective fields of sociology and public health—I do
not know everything. However, with the help of this AI-powered methodology, I can enter
every semester confident that the research studied in my courses can better prepare my
students for life.
Novel data techniques and tools are no longer required just for data scientists but for higher
education instructors seeking to maintain relevance and clarity in a rapidly evolving
information landscape. Big data is ubiquitous—its overflow causes information fatigue
syndrome, a weakening of our ability to think and discern (Han, 2017, p. 60). IBM defines
characteristics of big data as having high levels of volume, velocity, and variety (“Big Data
Analytics”, 2023). The course taught required deconstructing text data from article titles and
abstracts. Typically it would take months to manually extract topics and relationships with
moderate success in identifying complex patterns. Furthermore, findings could become
outdated during the months of preparation.
To maintain relevance in the present, the COVID-19 pandemic prompted us to integrate other
sources of unstructured data (e.g. social media and government sources). Therefore, the
methodology is adaptable to any text data source and can be used in other contexts requiring
timely and significant research updates, such as the escalating impact of climate change,
addressing evolving social crises in psychotherapy, and identifying misinformation in new
media.
The limitation of this methodology is that the results are not repeatable as they require an
individual to interpret the topological diagrams and knowledge graph to generate findings.
However, the expectation is that the user is knowledgeable in the field and that they bring
their expertise and lens to unveil an original research narrative.
References
Begg, M.D., Fried, L.P., Glover, J.W., Delva, M., Wiggin, M., Hooper, L., Saxena, R., De
Pinho, H., Slomin, E., Walker, J.R. and Galea, S. (2015). Columbia public health core
curriculum: short-term impact. American journal of public health, 105(12), e7-e13.
https://doi.org/10.2105/AJPH.2015.302879.
Big Data Analytics | IBM. (n.d.). Retrieved from https://www.ibm.com/analytics/big-data-
analytics.
762
Brenner, M. H., & Bhugra, D. (2020). Acceleration of anxiety, depression, and suicide:
secondary effects of economic disruption related to COVID-19. Frontiers in
psychiatry, 11, 1-21. https://doi.org/10.3389/fpsyt.2020.592467
Cock, P. J Antao, T., Chang, J. T., Chapman, B. A., Cox, C. J., Dalke, A., Friedberg, I.,
Hamelryck, T., Kauff, F., Wilczynski, B., & de Hoon, M. J. L. (2009).
Biopython: freely available Python tools for computational molecular biology and
bioinformatics. Bioinformatics, 25(11), 1422-1423.
https://doi.org/10.1093/bioinformatics/btp163
Elder Jr, G. H. (1998). The life course as developmental theory. Child development, 69(1),
1-12. https://doi.org/10.1111/j.1467-8624.1998.tb06128.x
Han, B. C. (2017). In the swarm: digital prospects (Vol. 3). MIT Press.
https://doi.org/10.7551/mitpress/11055.001.0001
Heikkilä, K., Pulakka, A., Metsälä, J., Alenius, S., Hovi, P., Gissler, M., Sandin, S. &
Kajantie, E. (2021). Preterm birth and the risk of chronic disease multimorbidity in
adolescence and early adulthood: A population-based cohort study. Plos one, 16(12), 1-
15. https://doi.org/10.1371/journal.pone.0261952
Hossain, S. (2019). Visualization of bioinformatics data with dash bio. C. Calloway, D.
Lippa, D. Niederhut, & D. Shupe (Eds.), In Proceedings of the 18th Python in Science
Conference. Austin, Texas, 126–133. https://doi.org/10.25080/Majora-7ddc1dd1-012
Nigg, J. T., Sibley, M. H., Thapar, A., & Karalunas, S. L. (2020). Development of ADHD:
Etiology, heterogeneity, and early life course. Annual review of developmental
psychology, 2, 559-583. https://doi.org/10.1146/annurev-devpsych-060320-093413
Simons, R. L., Lei, M. K., Klopach, E., Berg, M., Zhang, Y., & Beach, S. S. (2021). (Re)
setting epigenetic clocks: An important avenue whereby social conditions become
biologically embedded across the life course. Journal of Health and Social
Behavior, 62(3), 436-453. https://doi.org/10.1177/00221465211009309
Singh, G., Mémoli, F., & Carlsson, G. E. (2007). Topological methods for the analysis of
high dimensional data sets and 3d object recognition. PBG@ Eurographics, 2, 091-100.
http://dx.doi.org/10.2312/SPBG/SPBG07/091-100
Suglia, S. F., Appleton, A. A., Bleil, M. E., Campo, R. A., Dube, S. R., Fagundes, C. P.,
Heard-Garris N.J., Johnson, S. B., Slopen, N., Stoney, C. M., & Watamura, S. E. (2021).
Timing, duration, and differential susceptibility to early life adversities and
cardiovascular disease risk across the lifespan: Implications for future
research. Preventive medicine, 153, 1-8. https://doi.org/10.1016/j.ypmed.2021.106736
Tralie, C., Saul, N., & Bar-On, R. (2018). Ripser.py: A lean persistent homology library for
python. Journal of Open Source Software, 3(29), 925.
https://doi.org/10.21105/joss.00925
Vedam, S., Stoll, K., Taiwo, T. K., Rubashkin, N., Cheyney, M., Strauss, N., McLemore,
M., Cadena, M., Nethery, E., Rushton, E. Schummers, L., Declercq, E., & the GVtM-US
Steering Council (2019). The Giving Voice to Mothers study: inequity and mistreatment
during pregnancy and childbirth in the United States. Reproductive health, 16(1), 1-18.
https://doi.org/10.1186/s12978-019-0729-2
763
Vergunst, F., & Berry, H. L. (2022). Climate change and children’s mental health: a
developmental perspective. Clinical Psychological Science, 10(4), 767-785.
https://doi.org/10.1177/21677026211040787
Zomorodian, A., & Carlsson, G. (2004, June). Computing persistent homology. In
Proceedings of the twentieth annual symposium on Computational geometry (347-
356). https://doi.org/10.1145/997817.997870
764
New model to evaluate values, beliefs and assumptions in the

recognition of prior learning
Phil O'Leary
Recognition of Prior Learning Extended Campus Munster Technological University, Cork,
Ireland.
Abstract
This research will present the model created for a study on the Recognition of
Prior Learning (RPL) in Ireland. This study aimed to identify the values,
beliefs, and assumptions operating in RPL between the Assessor, Mentor and
Candidate in higher education (HE).
A critical constructivist grounded theory; this paper is focused on the
conceptual framework used for the research which adapted Van Kleef’s (2007)
model of RPL with Schein’s (2004) model of organizational culture. Schwartz
(2012) theory of values also helped with analysis of any values in the data.
Findings show that similar value systems operate in RPL, with honesty as the
primary value, and fairness, openness, and equity also present. The remaining
findings and conclusions show that RPL is a challenging field of practice and
that resources and training are essential. The assumptions show RPL requires
the standards to be upheld.
Keywords: Values; beliefs; assumptions; recognition of prior learning.

New model to evaluate values, beliefs, and assumptions in RPL
1. Introduction
Recognition of Prior Learning (RPL) is an important concept in lifelong learning and it refers
to an assessment process where value is given for all forms of prior learning; formal, non
formal and informal (CEDEFOP, 2018; Werquin, 2010). RPL provides for access to
education and for the award of credit. RPL is part of lifelong learning policy development,
but there has been little emphasis on understanding the viewpoints of the actors within RPL
or the philosophical basis of its practice (Hamer, 2016, Travers, 2017). This research took a
critical constructivist grounded theory approach to examine the values, beliefs, and
assumptions operating in RPL, aspects of which will influence interactions.
This paper will describe how Van Kleef’s (2007) model of RPL was adapted to provide the
theoretical framework required to examine the values, beliefs and assumptions present.
2. Methodology
Grounded theory was chosen as this research focused on the social processes in RPL and the
values, beliefs, and assumptions that might be considered significant. This is a relatively
understudied aspect of RPL and grounded theory provided the means to report the emerging
themes in the data.

This research followed Charmaz (2006) and employed a critical constructivist grounded
theory methodology to examine what values, beliefs, and assumptions were present in RPL
in a higher education setting.
Semi-structured interviews provided the initial data and the questions were focused on ‘what’
values, beliefs, and assumptions were important in RPL, and ‘why,’ this was so. The initial
purposive sampling developed into theoretical sampling, and grounded theory techniques
supported the analysis of the 82 interviews.
Three rounds of coding proceeded from initial open codes (573), to focused codes (5), and
theoretical codes (3). The second round of coding provided the conceptual categories and the
primary findings. The more abstract third round of coding provided framework for
discussion.
2.2. Theoretical framework created a new model of RPL

The theoretical framework selected Van Kleef’s (2007) model of RPL as the most suitable
for this research, as her model combined aspects of adult learning theory and practice and
suitably represented RPL (see Figure 1). Van Kleef (2007) also included values and beliefs
in her model while other RPL models in the literature did not (Harris, 1999; Osman, 2004).
766
Phil O'Leary
Figure 1. Van Kleef’s model of RPL. Source:Van Kleef (2007).
To locate values, beliefs, and assumptions within the RPL process, Scheins (2004) theory of
organizational culture was employed. Schein’s (2004) theory was suitable as the constructs
within his model were relevant to the focus of this research (see Figure 2).
Figure 2. Schein’s theory of organizational culture. Source: Schein (2004).
Integrating Van Kleef (2007) and Schein’s (2004) models provided the means to locate the
values, belief’s and assumptions operating in RPL in a meaningful way. Figure 3 illustrates
this adapted model.
767
Figure 3. Adapted model of RPL. Source: Schein (2004), Van Kleef (2007).
On the adapted RPL model (Figure 3) each of the elements therein interact to bring about the
RPL assessment. And although they are represented here simply, it is a more reciprocical
relationship as described by Schein (2004) that occurs in practice. Values, beliefs and
assumptions drive the process now and artefacts are represented through the ‘professional
practice’ and ‘assessment methods’ of the higher education setting.
In addition to this model (Figure 3), this research employed one more element, Schwartz’s
(2012) theory of values which supported the identification of any values arising in the data
and the literature. Schwartz’s (2012) model provided a consistent frame of reference and
thereby supported the analysis.
3. Primary findings
Due to space limitations this paper only briefly reports the primary findings which were
available in the second stage of coding. Here the focused codes were categorized into themes
to yield five conceptual categories; the primary findings;
1. The values in RPL; honesty, fairness and openness are to the fore
2. Beliefs supporting lifelong learning and RPL
3. Assumptions; grouped into what RPL provides and upholding the standards
4. Challenges posed by RPL; it is a challenging field of practice
5. Supports for RPL; resources and training are essential
768
Phil O'Leary
Figure 4 shows how these five conceptual categories were combined into three overarching
abstract theoretical codes, providing the most abstract level of reference for the data.
Figure 4. The emergence of the theoretical codes.
Figure 5 locates the primary findings within the adapted model of RPL. A more detailed
reporting of the primary findings was not possible within the space constraints of this paper.
Figure 5. Primary findings in the adapted model.
This paper is focused on Van Kleef’s (2007) adapted model which was employed to
successfully locate values, beliefs, and assumptions within RPL. The discussion will consider
this model.
769
4. Discussion
The primary findings provided empirical evidence of the values, beliefs, and assumptions
present according to the Assessor, Mentor and Candidates in HE in Ireland. While space did
not allow for a detailed presentation of these findings, Figure 5 provided a top-level summary.
This discussion will focus on the adapted RPL model that provided the conceptual framework
for this research.
This research furthered Van Kleef’s (2007) model of RPL by incorporating Schein (2004).
Van Kleef’s (2007) original model combined aspects of adult learning theory and RPL
practice. She drew from the humanist and critical traditions (Dewey (1938), Friere (1972),
Knowles (1970), Mezirow (1978), Kolb (1981), Jarvis (1987), and Schön (1983). Van Kleef
incorporated the knowledge and skills of the RPL practitioner, the assessment, and the social
context itself in her model. Significantly, she positioned ‘basic beliefs and values
underpinning RPL’ (p. 13) as key driving forces. She then positioned the remaining elements
as interacting with each other.
However, there were limitations with Van Kleef’s (2007) model, the first of which is that she
did not position ‘assessment methods’ as directly influencing ‘basic beliefs and values
underpinning RPL’ (p. 13). It might have been appropriate to do so. The second limitation is
that she did not explicitly name the values underpinning RPL, although she did identify her
beliefs about RPL in the paper. In this Van kleef (2007) stated her belief is that learning does
occur outside of HE, and that this learning can be assessed without compromising the
standards. A final limitation is that Van Kleef (2007) did not include the influence of the
Mentor or the Candidate in her model specifically, two important additional actors in RPL.
Notwithstanding these limitations, this researcher considered Van Kleef’s (2007) model a
very good starring point for this study.
In addition to Van Kleef (2007), this researcher considered Schein’s (2004) model of
organizational culture useful. It was possible to bring Schein’s (2004) insight about the levels
of organizational culture into Van Kleef’s (2007) model, thereby locating values, beliefs and
assumptions therein.
The grounded theory methodology provided the means to analyse the 82 interviews and
Figure 4 illustrates where this analysis culminated. The values, beliefs and assumptions in
the data are seen as driving the RPL process. Although space is limited here, this research
finds that honesty was the primary value required in RPL according to the participants. There
were beliefs in the data in support of RPL and lifelong learning similar to Van Kleef’s (2007).
The assumptions show that what RPL can provide in terms of access and credits is dependant
on the maintenance of the standards of HE. The challenging nature of RPL provision and the
requirements for supports were also foregrounded in the data.
770
Phil O'Leary
5. Conclusion
Further developing Van Kleef’s (2007) model provided the means to locate values, beliefs,
and assumptions effectively in an RPL process. The grounded theory methodology identified
these values, beliefs, and assumptions in an Irish HE context. The findings have implications
for future policy development including the requirement to re-frame the policy discourse to
reflect the challenging nature of RPL provision.
References
CEDEFOP (2018). European Inventory on Validation of Non-Formal and Informal
Learning; Country report Ireland. Retrieved from
https://www.cedefop.europa.eu/en/publications-and-resources/country-
reports/european-inventory-validation-non-formal-and-informal-learning-2018-update-
ireland.
Charmaz, K. (2006). Constructing Grounded Theory; a practical guide through qualitative
analysis. London: Sage.
Dewey, J. (1938). Experience and Education, New York: Simon & Schuster.
Friere, P. (1972). Pedagogy of the Oppressed, Harmondsworth: Penguin Books.
Hamer, J. (2016). Assessment Philosophy: A Critically Conscious Tool for Ethical Skills
Recognition. PLA Inside Out: An International Journal on Theory, Research and
Practice in Prior Learning Assessment. Retrieved from:
https://plainsideout.org/index.php/home/article/viewFile/100/157
Harris, J. (1999). ‘Ways of seeing the recognition of prior learning (RPL): What contribution
can such practices make to social inclusion?’ Studies in the Education of Adults, 31, 124-
139.
Jarvis, P. (1987). Adult learning in the social context, London: Croom Helm.
Knowles, M. S. (1970). The modern practice of adult education: Andraogy versus pedagogy,
New York: Association Press.
Kolb, D. (1981). Learning styles and disciplinary differences, in Chickering, A. W., ed., The
modern American college, San Francisco: Jossey-Bass, pp. 232-255.
Mezirow, J. (1978). Perspective Transformation, Adult Education, 28, pp. 100-110.
Osman, R. (2004). ‘Access, equity and justice: Three perspectives on Recognition of Prior
Learning (RPL) in Higher Education,’ Perspectives in Education, 22, 139-145.
Schein, E. (2004). Organizational Culture and Leadership, San Francisco: Jossey-Bass.
Schön, D. (1983). The Reflective Practitioner: How Professionals Think in Action, New
York: Basic Books.
Schwartz, S. (2012). An overview of the Schwartz theory of basic values, Online readings in
Psychology and Culture, 2, 11.
Travers, N. (2017). ‘Inherent tensions within the practices of prior learning assessment at
SUNY Empire State College.’ in: Jelly, K. & Mandell, A., eds., Principles, Practices
771
and Creative Tensions in Progressive Higher Education. Rotterdam, Boston, Tapei:

Sense Publishers, pp. 215-241.
Van Kleef, J. (2007). Strengthening PLAR: Integrating theory and practice in post-secondary
education. Journal of Applied Research on Learning, 1, pp. 1-22.
Werquin, P. (2010). Recognising Non-formal and Informal Learning; outcomes policies and
practices. Paris: OECD.
772
Academic rankings as a source of metrics and benchmark tools

for continuous improvement at Técnico Lisboa
Carlos Carvalho, João Fernandes
Studies and Planning Office, Instituto Superior Técnico, University of Lisbon, Portugal.
Abstract
Academic rankings ot only provide an opportunity to increase visibility and
reputation of institutions but also can be used as benchmark tools for quality
improvement, despite much criticism of biased coverage and
flawed methodologies. As part of the University of Lisbon, Técnico Lisboa – a
leading engineering and technology school in Portugal – has monitored and
examined rankings through its Rankings Observatory in the context of
strategic options, but institutional transformations have been an obstacle to
exactly measure Técnico’s worth and weight in the context of the University of
Lisbon. Those transformations have not contributed to leverage the positioning
of the University of Lisbon in major rankings either. This paper discusses the
impact of institutional transformations on the University of Lisbon and Técnico
in rankings, describes Técnico’s attempts to overcome these constraints and
gives examples of how it applies rankings as internal benchmark tools towards
continuous improvement.
Keywords: Academic rankings; benchmark tools; quality improvement.

1. Introduction
The University of Lisbon (ULisboa) acquired its current status in 2013, after the merger of
Universidade Técnica de Lisboa (UTL) and Universidade de Lisboa (UL). Among 18
institutions, Técnico Lisboa (Técnico), which was UTL’s engineering and technology school,
has maintained its name and role in the current ULisboa. This merger has allowed ULisboa
to become a ‘world-class university’, in a trend of recent years in Europe (Docampo et al,
2015). However, it also brought major constraints. Técnico has undergone a decrease in
autonomy and diminished visibility and ultimately it has affected its brand identity. It became
difficult to exactly measure its performance or establish its ‘worth’ and reputation, given that
the rankings list ULisboa and not Técnico.
In its recently adopted Strategic Plan 2020-2030 (2022), Técnico highlights its ambition ‘to
become a Europe’s leading (top 20) Engineering, Science & Architecture school’ by 2030,
among other major targets in terms of governance and positioning in the national and
European contexts. There has also been a wide internal debate on possibly changing its
statutes and adopting a foundational framework, with resulting changes in governance, with
more autonomy, more flexilibility financially, among other structural transformations.
Despite widespread critique, rankings have been increasingly used by institutions to be part
of decision-making strategies or as a benchmarking of quality assurance (Hazelkorn, 2013).
This criticism ranges from indicators adopted to the weight some have in scores (Fauzi et al.,
2020). Through its Rankings Observatory (Observatory), Técnico has focused on specialised
rankings in Engineering and Technology, in its fields and subjects of intervention. It operates
as an intelligence unit on these matters, which has sought to overcome existing limitations
by using ranking metrics to quantify its contribution to ULisboa, analyse its performance
through indicator analysis, and compare Técnico’s performance against its peer institutions.
Other limitations still endure. An example of this is the reputational component of rankings,
given that some ranking sources have a strong focus on reputation. Reputation surveys
(academic, employer) refer to ULisboa and not to Técnico, which makes it impossible to
accurately determine Técnico’s value and weight, as reputation has a substantial impact on
ranking scores, as in QS and THE (Vidal & Filliatreau, 2014).
It is important to note that although the Observatory monitors different rankings, analyses
and benchmark exercises focus fundamentally on QS and THE ranking sources.
2. The challenge to overcome the absence of Técnico in rankings

As previously mentioned, one of the major challenges faced by Técnico is the fact that it is
not listed in any ranking, and its performance is diluted in the participation of ULisboa. As
an added challenge, the merger of UL and UTL led to the situation in which some of the
774
scientific fields of Técnico are not exclusive. This is particulary noticeable in areas such as
chemistry, biology, physics, maths and even in some engineering fields. The attempt to
overcome this barrier is to quantify, as precisely as possible, how much Técnico weighs in
terms of percentage on ULisboa and consequently to understand how much it contributes to
its performance in rankings by subject, in subject areas taught at Técnico.
For this purpose, we chose to look at two dimensions: scientific output (bibliometric data)
and volume of students. The former is measured through the volume of publications indexed
in SCOPUS whereas the volume of students corresponds to the current number of students
enrolled. The subject areas where these dimensions come closest to 100% are those in which
Técnico’s weight is greater or almost exclusive in the context of ULisboa.
Based on the organization of broad fields in SCOPUS, table 1 provides key outcomes
concerning the number of publications of Técnico and ULisboa, i.e. Técnico’s percentage of
contribution to ULisboa in each subject area.
Table 1. The weight of Técnico Lisboa in the context of ULisboa as per SCOPUS fields.
No. of SCOPUS publications (2015-

2019)1 Weight %
SCOPUS Broad field
(IST/UL)
Técnico ULisboa
Chemical engineering 899 1356 66.3%

Chemistry 1459 2415 60.4%
Computer Science 1180 1650 71.5%
Earth & Planetary Sciences 529 1918 27.6%

Energy 1025 1305 78.5%
Engineering 3502 4106 85.3%
Environmental Sciences 1241 3246 38.2%
Materials Science 1844 2216 83.2%
Mathematics 1035 1180 87.7%
Physics & Astronomy 2735 3871 71.1%
1
Extractions via Affiliation ID: Técnico (60004956); ULisboa (60106051);
775
According to the above data, three broad fields taught at Técnico are the greatest contributors
to the ULisboa’s performance and outcomes: Mathematics (87.7%), Engineering (85.3%)
and Materials Science (83.2%). These are followed by Energy (78.5%), Computer Science
(71.5%) and Physics and Astronomy (71.1%). The remaining fields account for less than
70%, which suggests that it is reasonable to consider that other ULisboa’s institutions, in
particular the Faculty of Sciences, significantly contribute to its performance and results.
As regards the volume of students, table 2 provides the volume of students enrolled in
Técnico vs ULisboa in the fields taught at Técnico and its weight in the framework of
ULisboa, according to the data retrieved from the Portuguese Directorate-General of
Statistics of Education and Science (DGEEC).
Table 2. Técnico’s weight in the context of ULisboa as per the number of students enrolled in
three subject areas, according to the DGEEC classification2.
Natural sciences, Engineering, Information and

Subject areas maths & manufacturing and Communication
statistics construction industries Technologies (ICTs)
Técnico 1130 9624 130

ULisboa 4802 13080 326
% Técnico
within 23.5% 73.6% 39.9%
ULisboa
The data shows that the number of students enrolled in Engineering, manufacturing and
construction industries at Técnico accounts for 73.6% of the ULisboa student population,
followed by Information and Communication Technologies (ICTs), with 39.9%.
As for Natural Sciences, Mathematics and Statistics, it is also reasonable to consider that
other schools of ULisboa have a relevant contribution and therefore it is more difficult to
exactly determine Técnico’s weight and standing in ranking sources in these subject areas.
This therefore justifies the option for the Observatory to focus on Engineering and
Technology sector rankings.
In this regard, it is possible to understand how much Técnico weighs percentage-wise in the
context of ULisboa, in subject areas and student population, but these outcomes only reflect
an approximate idea of that influence. Finally, it can be said that this is an exercise of trying
2
Source: DGEEC database: Students enrolled in 2020/21; Figures relative to 1st and 2nd cycle students
776
to measure the ‘weight’ of a merged institution in the context of its parent, which could be
applied to other examples.
3. Applicability and instrumentalizaiton of rankings

This section disccuses and explores some examples of how Técnico’s Observatory applies
rankings, namely in the analysis of ranking indicators and scores, and benchmark excercises,
which stimulates a quality culture within the institution in its areas of activity, as argued by
Berbegal-Mirabent & Ribeiro-Soriano (2015).
3.1. Analysis of indicators and scores

Because the key challenge is to tackle the "non-presence" of Técnico in the rankings, the
strategy involves improving the positioning of ULisboa, preferably in collaboration with
other schools. In this regard, we would need to be able to accurately understand the impact
of Técnico on ULisboa in all indicators, for which access to data (ie. student population,
financial data, etc.) from other schools should be needed.
Ranking indicators provide a good comparison tool because they are a common benchmark
applied equally to all institutions. Reputation indicators (academic, research and employers)
depend on surveys that are conducted among stakeholders. At this level, Técnico’s partners
perceive their partnership with Técnico and not with ULisboa. Nevertheless, these surveys
are conducted among parent institutions, in this case ULisboa, because Técnico is not eligible
for that purpose and its reputation may not contribute significantly to the reputation of
ULisboa. A number of indicators should also be defined to exactly measure the weight of
Técnico in the positioning of Ulisboa and use them as benchmarks for improvement, given
that the the analysis of indicator scores allows us to identify, for example, underperforming
indicators. The examples that follow draw on the latest edition of the THE ranking by subject
(2022) in Engineering and Technology. The figure below shows how each indicator evolved
in the period 2020-2022.
Citations
60
Industry
Income
Intl outlook
40
Research
20
Teaching
Global
0 score
2020 2021 2022 2023
Figure 1. ULisboa performance per indicator and global score. Source:THE (2020-2023).
777
From the above, it can be said that these observations are critical to identify underperforming
indicators or even those which can perform even better. The figure shows that the Citations
indicator, which has been constantly decreasing across the 2020-2023 period, should object
of concern as it weighs 30% according to the THE WUR by subject methodology.
Table 3 reveals the outcome of an hypothetical exercise of applying a growth percentage to
the score of an indicator and, thorugh that, calculate a precise rank.
Table 3. An hypothetical approach by applying a growth % to the results of ULisboa.
Global Ind Intl Inhouse Inhouse

Name Cits Res Teach
Score Income Outlook Score Rank
University of
36.1 52.8 37.6 53.5 26.3 23.6 36.1 141
Lisbon
A (5% growth) - 55.4 39.5 56.2 27.6 24.8 37.2 132
B (10% growth) - 58.1 41.4 58.9 28.9 26.0 38.9 120
C (10% growth
in Research & - 52.8 37.6 53.5 28.9 26.0 36.9 134
Teaching)
These results are merely hypothetical and rely on the assumption that the next edition of the
ranking would remain unchanged. However, it gives an idea of how the global score and
standing would be if we apply a growth percentage in some or all indicators. The inhouse
score and rank were calculated by the Observatory, because the THE ranking classifies
institutions in bands from the 201st place onwards (201-250…).
3.2. National benchmarking

Benchmark exercises include comparisons with peer universities, either in the national or
international contexts. Figure 2 below allows us to observe how the ULisboa globally
performed against its peers in Portugal, Universaity of Porto (UPorto) and Universidade da
Beira Interior (UBI), in a defined time period: 2020-2023. It reflects the evolution of the
global score in the ranking, and it is aimed chiefly at marketing purposes.
778
40 37,6 37,1 37,1

35,7 36,1
34,5 35,1 34,6 35,1 35,1
34,1 34,1
35
30
25
20
2020 2021 2022 2023
Universidade de Lisboa Universidade do Porto Universidade da Beira Interior
Figure 2. Evolution of global score for the Portuguese TOP 3 HEIs. Sources: THE (2020-2023).
The table below provides the annual mean variation between 2019 and 2023 in each indicator,
for ULisboa and Univ. Porto. Based on the percentage rate of growth (or decrease) in that
period, we can observe an estimated position for 2024. This can be a useful way of looking
at the evolution of the institution during the period in question and take action in advance if
a downward trend in the performance of ranking indicators is observed.
Table 4. 2019-2023 variation for ULisboa and UPorto and potential 2024 scenario.
Annual Mean Variation 2019 - 2023 Potential scenario 2024*

Dimension
UL UP UL UP
Citations -3.1% 2.1% 51.2 59.2
Industry Income 2.7% 1.3% 38.6 39.6
International Outlook 3.7% 2.5% 55.5 51.6
Research 8.2% 10.9% 28.5 27.5
Teaching 2.6% 4.5% 24.2 27.8
Global Score 1.7% 4.1% 36.0 38.7
The potential scenario for 2024 allows the governing bodies of the institution to anticipate a
likely future estimate if performance keeps the pattern of recent years. It also also helps
identify priority areas of intervention, according to ranking methodologies.
3.3. International benchmarking

In the pursuit of quality, international benchmarking has also been very important to observe
ULisboa’s performance in some international networks. Table 4 shows the performance of
ULisboa in a network of institutions of Engineering and Technology, CLUSTER, of which
ULisboa is part. Among other things, we can track how the object of becoming one of the
20th leading Engineering and Technology institutions in Europe can be achieved or not, and
779
how these targets may be re-defined. Considering the the scores in each dimension we may
clearly underline that reaching the top 20 in 2030 seems to be unrealistic.
Table 4. Performance of CLUSTER institutions.
Industr
Rank Rank Intl Global
Institution Cits y Res Teach
(Europe) (world) outlook score
income
11 50 KULeuven 72.4 96.9 73.7 67 57.7 67.7
12 53 KTH 67.3 64.9 47.2 74.1 72.3 66.2
15 64 KIT 62.6 64.7 81.2 68 65.1 53.7
16 74 Polito 66.7 57.3 55 57.5 59.1 60.3
28 108 Aalto Univ 77.4 53.1 74.9 37.7 45.4 54.5
TU
46 162 48.3 89.7 54.6 47.9 41.8 48.8
Darmstadt
Trinity Col
70 216 74.4 36 86.2 28.4 24.4 44.6
Dublin
87 261 UCLouvain 54.1 58.6 75.5 28.6 33.7 42.2
Grenoble
90 266 46.7 39.9 66 36.3 38 42.1
Inst. Tech.
141 452 ULisboa 52.8 37.6 53.5 28.6 23.6. 36.1
172 541 UPC 40.1 39 52.9 26.3 31 34.1
4. Conclusions
This paper briefly analyses what a subsidiary institution can do to measure its weight within
the context of its parent for ranking purposes, as a result of a merger. This need comes from
the fact that a potential Técnico position in university rankings cannot be done alone by
looking at how the parent university is performing ‘by subject’ because we have to consider
the potential input of other schools, specially those who share the same subjects with Técnico,
such as as physical sciences, biology, math or chemistry.
Due to these constraints the analysis of rankings undertaken by Técnico, for purposes of its
own autonomous strategies and policies, should mainly focus on Engineering and
Technology field rankings to determine, as closely as possible, a potential Técnico position
on university rankings. The analysis of ranking indicators and scores proves to be relevant to
stimulate a quality culture through comparisons with peer institutions, nationally and
internationally, in a number of activities, according to ranking methodologies.
As it is a recent activity at Técnico, there is still no visible impact so far in terms of objective
results and improvements, however the rankings theme has gained significant awareness
among the management board and it has made its way into strategic planning for the next 10
years and has a significant role in quality monitoring by providing key indicators for the
780
yearly activity plan. Técnico is currently better aware of its performance and position among
Portuguese universities.
References
Berbegal-Mirabent, J., Ribeiro-Soriano, D., (2013). Behind league tables and ranking
systems. Journal of Serivce Theory and Practice, 25(3), 242-266, doi 10.1108/JSTP-04-
2013-0059.
Docampo, D., Egret, D., Cram, L. (2015). The effect of university mergers on the Shanghai
Rnaking. Scientometrics. Published online. doi: 10.1007/s11192-015-1587-5.
Fauzi., M. A., Tan, C. N-L., Daud, M. & Awalludin, M. M. N. (2020). University rankings:
a review of methodological flaws. Issues in Higher Education, 30(1),79-96, 2020.
Hazelkorn, E. (2013). Rankings and Implications for Quality Assurance in Higher Education,
Exploring Quality Assurance Through the Africa-EU Partnership Policy Workshop EU-
Africa Joint Strategy, Gabon, Africa, May, 2013.
Strategic Plan 2020-2030 (2022). Instituto Superior Técnico. Retrieved January 20, 2022
from (https://tecnico.ulisboa.pt/en/tag/tecnico-strategic-plan/).
Times Higher Education (THE) (2022). World university rankings. Retrived in January 2022
from (https://www.timeshighereducation.com/world-university-rankings).
Vidal., P., & Filliatreau, G. (2104). Geographical Comparison of World University Rankings.
Higher Education Evaluation and Development, 8(1), 1-14. doi
10.6197/HEED.2014.0801.01.
781
Undergraduates as researchers in humanities and social sciences

courses: articulating assessment by means of micro and macro
cooperative and integrated tasks
Mireia Trenchs Parera, Andreana Pastena
Department of Humanities, Universitat Pompeu Fabra, Barcelona, Spain.
Abstract
The proposal presents an experience of formative-summative and research-
based assessment method developed for Humanities and Social Sciences
undegraduates. Such assessment experience is organized in micro-tasks that
guide the students towards the development of an original and cooperative
piece of research, or macro-task. Students are invited to conduct a study with
first-hand data and think about the transferability of its results to society and
a potential professional future. Here, we describe each assessment procedure
and task as well as the tools developed to support the students in the learning
process. We also provide examples of the macro-tasks developed by the
students, and draw conclusions as regards the effectiveness of such assessment
experience. Following a socioconstructivist approach to the teaching-learning
process, the method aims at fostering meaningful and contextualized learning
and assessment at university, as well as undergraduates’ individual autonomy
and cooperative skills.
Keywords: Competences; continuous assessment; cooperative learning;
research-based assessment; task-based learning.

Undergraduates as researchers in humanities and social sciences courses
1. Introduction
In the context of today’s globalized societies, students are required to acquire multiple
competences and skills that allow them to function in the professional world. For that
purpose, we designed an assessment experience for Humanities and Social Science students
that requires conducting a piece of research with first-hand data and thinking about the
transferability of its findings to society. The result has been formative-summative and
research-based assessment organized in micro-tasks guiding students towards the
development of an original piece of research, i.e. the macro-task. The experience has
undergone continuous revision year after year since 2018-2019.
2. A research-based assessment experience

The experience is the result of two teaching innovation projects implemented during
academic years 2018-2019 (36 PlaCLIK 2018-2019 2) and 2022-2023 (PlaCLIK,
E2022014355) and funded by Universitat Pompeu Fabra. The objective was the design of a
research-based assessment method that would foster meaningful learning. Our aim was to
give students the opportunity to put into practice what they learn by developing an original
research project of their choice and linking its results to a profesional area of their interest.
2.1. Pedagogical approach

The proposed evaluation method is based on a socio-constructivist approach to the teaching-
learning process (Vygotsky, 1978). As such, the presentation of contents and the
demonstration of methods and analyses by the instructor are combined with the leadership
and autonomy of the student both inside and outside the classroom. The idea behind our
assessment method is that students become self-directed learners, getting involved in the
learning process and developing more general skills and competencies than those strictly
related to the learning of contents, such as time management, goal setting, self- and peer-
assessment, data collection, and resource use, among others (Grow, 1991). A formative-
summative type of evaluation and a task-based approach highlight the role of the learner
(Skehan 1998; Ellis, 2003; Nunan, 2006). In addition, the eminently practical and
collaborative approach to the tasks means that, in this context, the role of the instructor
acquires the value of a guide and an adviser in learning. The main aim of such an approach
is that students develop those competences that make them autonomous in the analysis of –
in our case– intercultural spaces and discourses from a critical, interdisciplinary perspective
and, thus, become capable of conducting their own applied research.
784
2.2. Context and participants

The experience has been implemented in two English-Medium Instruction undergraduate
courses – Discourse Analysis and Intercultural Spaces, Languages and Identities – offered
by the Universitat Pompeu Fabra, during several consecutive academic years.
Discourse Analysis is a course shared by third- and fourth-year students from the BA degrees
in Humanities, Global Studies, and Political and Administration Sciences. It is conceived as
an introduction to concepts and methods for the analysis of oral and written discourses in a
variety of disciplines and professional areas such as politics, history, literary criticism,
journalism, and advertising, among others. The course aims at the development of students’
autonomy in the use of discourse analysis to investigate the relationship between language,
author’s intention, and social context with a critical perspective.
Intercultural Spaces, Languages and Identities is a course shared by third- and fourth-year
students from the BA degrees in Humanities and Global Studies. The objective is to
understand socialization spaces that are shared by interlocutors from different cultural and
linguistic backgrounds, by exploring such intercultural spaces from a transdisciplinary
perspective that includes concepts and methods from critical discourse analysis, educational
sociolinguistics, applied linguistics, linguistic anthropology, and social psychology.
Since they are offered in English-Medium Instruction, both courses usually receive a high
percentage of international students on either short-term or long-term mobility.
2.3. The structure: Micro- and macro- tasks

As mentioned, course assessment is organized around a macro-task, that is, an applied
research project the results of which have to be related to a professional field of students’
interest such as public administration, journalism, education, social mediation, advertising,
and cultural institutions, among others. To encourage collaborative work, projects have to be
carried out in pairs. Students choose their own project partners but choosing a partner from
a different cultural and linguistic background is recommended as we wish to promote
meaningful intercultural interactions and the integration of international students.
In order to develop this piece of research, students are required to carry out several micro-
tasks – both in class and out of class – that will guide them towards the completion of the
Final Project Report (i.e. the macro-task). Also, in line with cooperative learning principles,
micro-tasks may be done either individually or in small groups, fostering negotiation and co-
construction of meanings. Specifically, students are required to do the following micro-tasks:
1. MODULE ACTIVITIES: These are tasks done by the students in-class individually,
in pairs or in small groups. They include practice tasks related to the theoretical
concepts studied in class and linked to each course module, for instance, analyzing
interview excerpts in order to detect language ideologies, or thinking about
785
examples of everyday linguistic profiling. After the actitity, results are shared with
the group and the instructor, and discussed. When the task cannot be finished in
class, students have the possibility to complete it shortly after the session. The
Module Activities are compulsory for formative assessment, yet no specific grade
is assigned; only the number of completed activities contribute to the final
summative grade, accounting for continuous participation. The aim is that students
familiarize themselves with terms and concepts specific to the field and learn how
to apply them. For instance, two examples of Module Activities are:
a. Students individually complete a questionnaire taken from the literature
and used to collect research data on intercultural sensitivity; then, they
critically discuss the clarity and appropriateness of questionnaire items
with classmates.
b. In small groups, students explore the university’s linguistic landscape by
taking photos from signs and posters on-site and surrounding areas; then,
they share the images in a collaborative online space (i.e. UPF’s Aula
Global), and conduct a semiotic and discourse analysis that is shared with
the whole group.
2. PROJECT PROPOSAL: In pairs, students develop a proposal of the research project
they wish to carry out. They choose the topic, the research questions and objectives,
the kind of data they plan to collect, and the concepts they will use for the analysis.
In the proposal they have to include at least one of the course compulsory readings
and one of the optional ones as useful theoretical or methodological references for
the project. They also have to include, at least, a new reference as a result of an
initial documentary research on the topic of choice. The aim is that students develop
strategies to plan tasks and to work autonomously with academic rigor. The proposal
does not receive a specific grade, but it must be approved by the instructor before
the start of the project.
3. PEER REVIEWS: Each student individually has to conduct a critical review of the
Project Proposal of another group. Peer Reviews include several related tasks: (a)
during a one-hour Seminar session, each student discusses the proposal with its
authors and, as a peer reviewer, provides oral feedback; the feedback is reciprocal
since this activity is done in groups of two pairs each; (b) peer reviewers prepare
individually a one-page-long written peer review of the assigned proposal upon the
basis of what has been discussed in class; and (c) reviewers provide oral feedback
during the Oral Presentations. The aims of this task are that students become more
aware of the elements of a research project, put into practice their critical skills in a
typically academic activity, and engage in peer collaboration. Peer Reviews are
evaluated by the instructor, but they are made available to the proposal’s authors so
that they may also be used to improve the project.
786
4. ORAL PRESENTATION: Students in pairs present their project in an advanced

state of completion in a conference format and with visual support (i.e. PowerPoint
and other materials like audios, videos or images, if necessary). Usually, Oral
Presentations are done in-class at the end of the course; however, depending on the
number of students enrolled, students may also be asked to record a video of their
presentation and share it in the virtual classroom to receive feedback. The aim is
that students, in pairs, practice those academic and professional competences linked
to public speaking such as, for instance, time management and information synthesis
and selection. Students are given suggestions by their classmates, especially by
those assigned as peer reviewers. They also receive feedback from the instructor
who, afterwards, evaluates them formally.
Consequently, all the assessment activities are interrelated and contribute to the final
summative grade that students receive. As such, each micro-task works as a milestone, being
functional and essential to reaching the final goal (see Figure 1). Indeed, each assessment
milestone, as well as the critical feedback provided by the instructor and the classmates, guide
the student towards the development of the final macro-task, that is, the FINAL PROJECT
REPORT. For this research-based task, students in pairs have to investigate a real
intercultural space of their choice and interest with a focus on language, and collect their own
data. The analysis may include such methods and instruments as interviews, focus groups,
(participant) observation, questionnaires, discourse analysis of relevant documents, and
visual data on linguistic landscape. In doing so, it is compulsory to use theoretical concepts
studied in class, as well as those included in the recommended readings for each module.
The research is not to be conducted from an academic point of view but putting oneself in
the shoes of a professional such as a journalist, publicist, NGO employee, cultural mediator,
or instructor. This professional role is chosen by the students themselves. The project will
result in a final written report that, in the future, students may include in a portfolio for
potential employers. For this purpose, the report has to include a recommendations section,
in which students provide suggestions for those stakeholders that may be interested in their
results. Students hand in the Final Project Report after the course is finished in order to have
the opportunity to integrate the feedback and suggestions received by the instructor and the
peer reviewer(s), as well as all the relevant theoretical concepts tackled during the course.
787
Figure 1. Design of the assessment method. Source: Own elaboration.
2.4. Guiding the students: Student Guide, seminars, and rubrics

In order to guide students towards the successful completition of each assessment task, we
developed a comprehensive Student Guide, providing detailed instructions and
recommendations on the content and form of each activity. Such Guide may be found in the
virtual classroom since day one, along with a detailed calendar of the sessions and deadlines,
and examples of studies conducted in intercultural spaces that support the students in
becoming acquainted with both useful theoretical concepts and research methods.
Also, two one-hour Seminar sessions are explicitely devoted to the development of the micro-
tasks (i.e. Methodology Seminar in Figure 1), providing examples and guidelines for the
elaboration of the Project Proposal, Peer Reviews, and Oral Presentation. During these
sessions, students have the opportunity to ask questions and clarifications, as well as share
their research topic with their classmates. Anyhow, after each task, the instructor provides
individualized feedback to the students, either in class or in individual or pair tutorials.
Furthermore, the Final Project Report is evaluated following an assessment rubric that is
shared with the students before the submission of the final product, and that may serve as a
self-assessment tool. Indeed, in our model, self-assessment functions as a counterpart to
formative and summative instructor assessment and an essential activity of the learning
process. Similarly, the course includes a rubric for the oral presentation that both the
instructor and the peer reviewers use for assessment and peer feedback.
3. Examples of students’ research projects

In order to illustrate the variety of projects carried out by the students, we present here a
selection of five of the most successful Final Project Reports. For each of them, we report
the title, the professional role undertaken by the students, and their chosen stakeholders:
788
• The exchange student experience at Universitat Pompeu Fabra: Students assumed the
role of Program Consultants expert in academic mobility and, after their research,
provided suggestions for the Office of International Affairs at the university.
• Attitudes and acceptance towards the South Korean community and ‘Hallyu Wave’ within
Catalan media and multicultural society: Students assumed the role of journalists and
cultural consultants, and the recommendations were directed to South Korean
entertainment companies, cultural entities interested in South Koreans’ integration in
Catalonia, and the Catalan media.
• Linguistic identities: The use of Spanish in informal intercultural spaces in Barcelona:
Students acted as reporters and cultural mediators, and offered suggestions to universities
and their language programs as well as the Barcelona’s city council .
• Students’ language attitudes towards ordering at the UPF cafeteria: Students acted as
researchers specializing in interculturality, and recommedations were offered to the
Mobility Office at that university, the administration, and the cafeteria staff.
• Political discourse and linguistic landscape of graffiti in the city of Barcelona: Students
took the professional role of anthropologists at a government institution like the town
hall. Stakeholders profiting from the outcome of the study would potentially be local
authorities and politicians who can use the information “to apply measures that may
improve the quality of life of the citizens or generate a positive social change”.
4. Conclusion
The experience presented here is the outcome of four years of teaching to three different
cohorts of students of Discourse Analysis and four of Intercultural Spaces, Languages and
Identities. At the each term, the instructor modified their methodology and evaluated its
impact, as regards the learning process and the students’ workload1. An in-depth analysis of
this constant revision process and the degree of satisfaction of the instructor and the students
–who assessed the instructor and the sessions every year– deserves another communication
or article. However, although the experience has entailed a high degree of dedication and
work by the instructor, we consider the proposal highly satisfactory for several reasons.
First, such research-based assessment method provides students with the opportunity to
reflect on how the knowledge acquired in the university context and as a result of academic
research can be useful in the real and professional world. Also, it allows them to put into
practice theoretical concepts and developing skills and competences related to academic
1
Though students’ evaluations were always extremely positive in all survey items, the survey item
“The workload required corresponds to the credits of the subject” always received a lower grade, i.e.
8.7 on a ten-point scale in 2019-20, 7.9 in 2021-22, and 8.9 in 2022-23.
789
research, such as time management, respecting deadlines, data collection and structuring and
carrying out an autonomous research project.
Second, the method puts the students at the center of their autonomous learning process. Also
in content lectures and methodological seminar sessions, students have a focal role, since
they have to acknowledge the relevance of such content for successfully carrying out each
task. As such, the process highly satisfies the students, as shown by the average degree of
satisfaction expressed in their teaching evaluations2.
Third, it encourages contact, interaction, and collaboration among students from different
degrees and diverse academic and professional interests, by promoting co-construction of
meaning and knowledge. Also, in the case of the Intercultural Spaces, Languages and
Identities course, the assessment method allows for the interaction between students from
diverse linguistic and cultural backgrounds and, thus, the development of plurilingual and
pluricultural competences which, in turn, constitute part of the course contents.
Fourth, since the whole process requires more individualized instructor-student interactions
in class and out-of-class tutorials for guidance, it allows for a better understanding of
students’ interests and needs. As a consequence, the instructor may develop content and
activities responding better to those needs, while creating a relationship of mutual trust.
Finally, this assessment experience may be tailored to other teaching contexts, as
demonstrated by the rapid adaptation to online teaching during the mandatory lockdown
established due to the COVID19 pandemic between March and June 2020, fully coinciding
with one of the courses and resulting in extremely positive evaluations by the students.
Lastly, we wish to conclude by quoting Claudia who, when evaluating the Discourse Analysis
course, wrote: "Overall, I've appreciated the general organization, content and activities of
the course. In particular, I found useful and engaging the peer-assessment system. I also
found refreshing the clear and ordered presentation of the content seen in class, the
assessment deadlines and the evaluation criteria."
Acknowledgements
The authors wish to thank UPF for the funding provided by UPF’s Pla d’Ajuts a la Qualitat
i a la innovació en l’Aprenentatge (36 PlaCLIK 2018-2019 2 and, specially, E2022014355
2022-23), as well as GREILI and ALLENCAM research groups for their support.
2
In 2019-20, the mean grade given by the students was 9.23 for the survey item “In general terms I
am satisfied with this subject”; in 2021-22, it was 9 and, when assessing the instructor, 9.62; in 2022-
23, the mean grade for overall satisfaction was 9.6.
790
References
Ellis, R. (2003). Task-based Language Learning and Teaching. Oxford University Press.
Grow, G. O. (1991). Teaching Learners to be Self-Directed. Adult Education Quarterly,
41(3), 125–149.
Nunan, D. (2006). Task-based Language Teaching. Ernst Klett Sprachen.
Skehan, P. (1998). A Cognitive Approach to Language Learning. Oxford: Oxford University
Press.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes.
Cambridge, MA: Harvard University Press.
791
Threshold tests as a way to encourage long-term, self-regulated

learners in engineering
Danica Solina1, Chris Wong2, Kate Crawford3, Elaine Huber4
1
School of Mathematical and Physical Sciences, University of Technology Sydney,
Australia, 2Faculty of Engineering and IT, University of Technology Sydney, Australia,
3
Eviva Pty Ltd, Australia, 4The University of Sydney Business School, University of Sydney,
Australia.
Abstract
Traditionally, educators evaluate the effectiveness of a new assessment in a
pre-requisite early subject via pass rates, grades and student satisfaction. It
may be more appropriate to measure the impact on student results in later
stage subjects. In this work we report on the impact of changing assessment,
over 10 years, at a metropolitan Australian university for an initial calculus
subject (Mathematics 1) on two follow-on subjects: Mathematics 2 and
Fundamentals of Mechanical Engineering. Earlier research found that Online
Mastery Tests can harm later learning outcomes even though failure rates drop
within the pre-requisite subject. Here we show that Paper Threshold Tests,
requiring greater engagement, metacognitive strategies result in fewer passes
in the pre-requisite subject however, they were also resulting in a major
inversion of the grade distribution toward higher grades in the follow on
subjects where 60% of students now obtain marks > 75%.
Keywords: Assessment; mastery learning; progression analysis.

Threshold tests as a way to encourage long-term, self-regulated learners in the engineering
1. Introduction
A university education serves multiple purposes including providing a strong foundation in
the field of choice, transferable practical skills, critical thinking skills, and networking
opportunities. The underlying objective is the creation of lifelong learners who have the
capability to keep up with advancements in their field and to adapt to changes in their careers
(Nilson, 2013).
However, many students commence university only having mastered surface learning
techniques, resulting in an erosion of learning and decay of knowledge/ learning loss between
semesters (Dills et al., 2016). To combat this, assessments should be designed to encourage
students to develop deep learning approaches to maximise knowledge retention for
application in the subsequent subjects in the next semesters.
As university subjects often have large enrolments (n >500), it is not feasible to personalise
subjects for each student. Instead, assessments should be structured to allow students to
develop the learning skills to progress through Bloom’s taxonomy (Bloom et al., 2001)
moving from recall, to understanding, application, evaluation and creation.
This work examines the impact of the type of assessment used in a pre-requisite mathematics
subject at an Australian university on students’ long-term retention of concepts and their
ability to transfer their learning to subsequent subjects. The focus will be on the comparison
of Online Mastery Tests (OMTs) and Paper Threshold Tests (PTTs) in the first-year calculus
subject, Mathematics 1 (Math1) and their impact on the grade distribution in the two
following subjects, Mathematics 2 (Math2) and Fundamentals of Mechanical Engineering
(FME). The investigation covers a ten-year period, with the latter subject (FME) having
remained unchanged in delivery or assessment. This work had human ethics clearance
(HREC 17-1158).
2. Assessment structures
2.1. Math1 assessment and within subject performance

Math1 is structured to run over 12 weeks where there are 11 weeks of lecture material and
11 weeks of associated tutorials, and in some years, computer labs. Math1 is offered in all
three study sessions. Session 1(S1) has the largest intake and consists primarily of students
attempting the subject the first time. Session 2 (S2) includes repeat students and students who
had completed Foundation Mathematics (FM) before attempting Math1. Session 3 (S3) was
introduced in the summer of 2017 to give students a chance to catch up on Math1 if they had
either failed or completed FM in S2.
794
Between 2008 and 2020, assessment in Math1 was altered in an effort to improve learning
outcomes and pass rates. The first three iterations included a 40% Compulsory Final Exam
(CFE), one (later two) Class Tests (CT), weekly review questions called Routine and Review
Sheets (RRS) and Mathematica Computer Lab Participation marks (CLP) .
The fourth iteration of Math1 assessment was the most significant change, which introduced
Online Mastery Tests (OMTs) through the Webassign the online platform. These tests were
a combination of Mastery learning (Block & Burns, 1976) and Keller Plan/Personalised
system of instruction (Rae, 2011). The concept behind Mastery Tests is that if students have
a strong grasp of fundamentals, then they can tackle more complex concepts. These tests can
be repeated (within reason) to give the student the time to attain proficiency.
It was required that students obtain 80% in each of four OMTs. Students were given three
opportunities to attain the 80% requirement in each of the four OMTs. The first OMT was
on assumed knowledge and was worth a nominal 5%. The following three tests covered 2 to
3 weeks of content over the first 8 weeks. It was possible to attain a pass in Math1 by attaining
the minimum requirement in each of the OMTs and an Optional Final Exam (OFE). OMTs
only required a final answer to be submitted whilst OFEs required fully worked solutions.
In S2 2017 the OMTs were replaced with the paper equivalent, Paper Threshold Tests (PTTs).
The Math1 assessments and weighting are listed in Table 1.
Table 1. Assessment iteration in Math1, session and years used and assessment breakdown.
Iteration, Session & Assessment Breakdown and percentage weight

Year
Quiz/Assess Mastery test Final Exam
5% RRS, 25% CT, 10% - 60% CFE

1. S1 and S2 2008
Assignment
2. S1 2009 10% RRS, 25% CT, 5% CLP - 60% CFE
3. S2 2009 – S1 2014 10% RRS, 25% 2×CT, 5% CLP - 60% CFE
4. S2 2014 – S2 2016 62.5% OMT 37.5% OFE
5. S1 2017 48% OMT 52% CFE
6. S2 2017 – S3 2020 50% PTT* 50% CFE*

Note: * Note weightings changed to 50/50 Autumn2018).
The grade distribution in Math1 for each assessment iteration is given in figure 1 where
F:Fail (<50%), P: Pass(50–64%), C:Credit (65–74%), D:Distinction (75–84%) and H: High
Distinction (85–100%).
795
Figure 1. Grade distribution in Math1 with assessment iteration (as outlined in Table 1) and session
The introduction of OMTs in iteration 4 (S2 2014) resulted in a significant drop in fails.
Although this decrease seemed positive, tutors began to report that students were just pattern
matching instead of fully understanding the material. This minimal approach carried over
into Math 2 as well, according to a progression analysis by (Coupland et al., 2017).
Considering these findings, Math 1 assessment underwent a 5th iteration, in which a
minimum requirement was introduced for the Compulsory Final Exam (CFE) - now requiring
fully worked solutions.
Figure 1 shows the reintroduction of a final exam (iteration 5-S1) resulted in a significant
increase in fails. The graphed data is the result after a supplementary exam was given. Pre-
supplementary exam the failure rate was 46%. It was concluded that OMTs were not
supporting learning of the subject material. Discussions with 80% (n=100) students who
attempted the supplementary exam showed that OMTs did not align well with final exam
expectations. Students believed they were doing well due to high OMT scores (80%+) but
did not invest extra effort on the final exam. Furthermore, OMT only required final answers
whilst the final exam required fully worked solutions.
In iteration 6 (S2 2017), OMTs were replaced with PTTs. It should be noted the only other
subjects taken in common during their starting semester, Chemistry and Physics, had little
change in delivery at the time PTTs were introduced.
As with the first OMT, the first PTT also examined assumed knowledge. The first test was
administered in weeks 1 and 3 to give students a chance to either revise or withdraw before
796
census date in week 4. A score of 80% was required to pass. The questions were structured
to reflect the sequence in which the material would be covered in the subject. Throughout the
semester, the test was referred to as a reminder of the assumed knowledge.
The last three PTTs examined 2 to 3 weeks of content at a time, covering the first 8 weeks.
Tests had similar questions to the OMTs, however students were required to show complete
working/reasoning where a question was given more than 1 mark.
These tests examined general procedural knowledge and application. Application was
included to place the mathematics in context and to break the habit of using only “x, y, z” as
variables. These tests had a threshold requirement of 70%. As with the OMTs, students were
allowed to sit the tests three times in order to achieve the required mark. Tests were
administered in the second hour of a two hour tutorial, with feedback given a week after the
first attempt of each test.
From figure 1, it is clear that the introduction of the PTTs resulted in a significant increase
in failure rate. However, grade distribution within a subject and low failure rate may not
accurately measure the quality of the learning outcomes.
2.2. Progression Analysis

An alternative approach is to examine the impact of Math 1 performance on the subsequent
subjects. A typical mechanical engineering student math program is as follows:
Students without pre -calculus Founda'on Math
Students with pre -calculus Math1 Math2
FME
Math2 has undergone a number of assessment iterations. For clarity, we have labelled these
iterations A, B and C. These are listed in Table 2 and Figure 2.
Table 2. Assessment iteration in Math2, session and years used and assessment breakdown.
Iteration, session, years Assessment Breakdown and percentage weight
30% CTs,
A. S12008 – S3 2013 60% CFE
10% Assignments
B. S1 2014 – S1 2017 62.5% OMTs 37.5% OPE
C. S2 2017 – S3 2020 52% (50%*) OMTs 48% (50%*) CFE

Note: *weight change in S2 2017
797
The grade distribution data in Math1 from 2008 to 2018 is presented in figure 2. The data is
grouped based on differences in assessment and iteration. Note that fails in Math2 were high
before the inclusion of OMTs (iteration 3B).
The introduction of a compulsory final exam in Math1 (iteration 5C) shows a shift to higher
grades (D/H) in Math2. It is unclear however whether this shift occurs because of the
introduction of the compulsory final exam in Math 1 (iteration 5C) or the introduction of the
compulsory final exam in Math 2 (iteration 5C/6C).
MM1/MM2 Assessment Regime

F P C D H
100% 4 5
8 7
13 12
9
90% 15 13
13
32
80% 18 18
26
70% 19 22
30
60% 21
Propor%on
31
50% 37
32 33
40%
49
30% 34
41
21
20% 15
33
28 4
10% 14 13 12
8 9
0%
1A 2A 3A 3B 4B 5C 6C
Figure 2. Grade distribution in Math2 with both Math1 assessment (numbers) and Math2 assessment (letters)
FME/MM1 Assessment Regime
F P C D H
100%
7 9 9
10 11
90%
7 12
30
17
15
80% 21
13
70%
26
23 20
60%
26
Propor%on
33
50% 41
40% 31
35
36
24
30%
30
20% 23
36
18
16 15
10%
4
0% 1
1 2 3 4 5 6
Figure 3. Grade distribution in FME with Math1 assessment iterations
798
A better indicator is FME, where the subject assessment has not changed appreciably during
the time of analysis. The inclusion of the final exam in Math1 (iteration 5) resulted in a
dramatic reduction in FME fails and the introduction PTTs in Math 1 (iteration 6) resulted in
significant increase in higher grades (D/H) as per Figure 3.
The introduction of PTT’s in Math1 lead to a significant improvement in student performance
in both Math2 and FME, as shown by the grade distribution in Figures 2 and 3. This resulted
in over 60% of students in Math 2 and 55% in FME achieving a Distinction or higher and
demonstrates successful transfer of knowledge applied to subsequent subjects.
Furthermore, the delivery of Math 2 was almost identical for 5C and 6C highlighting the
impact of the effect of Math1 assessment on Math2. Moreover, the results in FME indicate
that students were competently applying the mathematics that they learned in Math1 to a
subject in the Engineering Discipline.
This grade inversion was found to be robust for students coming from Math1 from S2 2017
to S12019 into Math2 and FME from S1 2018 to S2 2019 indicating the learning outcomes
were not a one-off phenomenon and skills were transferable to follow-on subjects.
The most important aspect of PTTs is that they were only used in a first semester subject, yet
the learning outcomes and skills were carried into follow on subjects without PTTs. This
suggests that the assessment methods used in the first semester play a crucial role in
determining a student's approach to learning in subsequent semesters.
PPTs effectively promote deep learning by encouraging the use of metacognitive strategies
instead of surface rote learning (McGuire, 2018). Use of application problems require
students to identify important information, use mathematical concepts, and demonstrate their
thought process through clear worked solutions and evaluation of the results.
Furthermore, the timing of the three tests follow Nilson’s (2013) recommendation for the
development of self-regulated learners. The first test provides an initial evaluation of the
student’s knowledge and effectiveness of their learning approach, with feedback in the
following week to encourage an early start. The second test two weeks later allows the student
to reflect on the change in their learning approach. The final test at the end of semester offers
a chance to refine their learning approach, helping them to build and retain their mathematical
skills.
3. Concluding remarks
We demonstrated the effectiveness of PTTs in a first-year calculus subject by analysing
progression data to improve learning outcomes and ensuring long-term retention of
knowledge. The results showed the importance of implementing PTTs in early semester
subjects which can have a positive impact on subsequent subjects.
799
However, there are drawbacks to using PTTs. Creating 12 distinct PTTs, solution sets, and
marking criteria is a time-consuming process, and more versions are required as class size
grows to maintain academic integrity. This requires more institutional support for this form
of assessment such as appropriate workload allocation and additional resources.
PTTs were difficult to administer during COVID (2020) where S1 had 800+ & S2 300+
students. Due to cost of invigilation for multiple tests, a PDF download/upload approach
was taken with reliance on student honesty. Unfortunately, some students cheated, leading to
changes in assessment for 2021, and the impact is yet to be analysed.
Learning conditions and quality at many universities have declined due to cost constraints.
From this research, investing more resources in first year can have a significant impact on
student success in later years by creating independent learners with better long-term
outcomes.
References
Block, J. H., & Burns, R. B. (1976). Mastery Learning. Review of Research in Education, 4,
3–49. https://doi.org/10.2307/1167112
Bloom, B., Airasian, P.W., Cruikshank K. A., Mayer, R. E., Pintrich, P.R., Raths, J. &
Wittrock, M. C. (2001). A taxonomy of learning, teaching and assessing: A revision of
Bloom’s taxonomy of educational objectives. New York, NY: Longman.
Coupland, M., Solina, D., & Cave, G. E. (2017). Mastery Learning: Improving the Model. In
A. Downtown, S. Livy, & J Halls (Eds), 40 years on: We are still learning! Proceedings
of the 40th Annual conference of the Mathematics Education Research Group of
Australasia (pp. 189-196). Melbourne: MERGA. https://eric.ed.gov/?id=ED589527
Dills, A., Hernández-Julián, R., & Rotthoff, K. W. (2016). Knowledge decay between
semesters. Economics of Education Review, 50, 63–74.
https://doi.org/10.1016/j.econedurev.2015.12.002
Nilson, L. B. (2013). Creating Self-Regulated Learners: Strategies to Strengthen Students’
Self-Awareness and Learning Skills. Stylus Publishing, LLC.
McGuire, S. Y. (2018). Teach Yourself How to Learn: Strategies You Can Use to Ace Any
Course at Any Level. Stylus Publishing, LLC.
Rae, A. (1993). Self-paced learning with video for undergraduates: A multimedia Keller Plan.
British Journal of Educational Technology, 24(1), 43–51. https://doi.org/10.1111/j.1467-
8535.1993.tb00640.x
800
The PASSt project: predictive analytics and simulation of studies

aimed at quality management and curriculum planning
Gabriel Wurzer1, Shabnam Tauböck1, Markus Reismann1, Christian Marschnigg1, 3,
Sukrit Sharma2, Karl Ledermüller3, Julia Spörk3, Maria Kokovsky3
1
Zentrum für strategische Lehrentwicklung, TU Wien, Austria, 2TU.it, TU Wien, Austria,
3
Evaluierung & Qualitätsentwicklung, WU Wien, Austria.
Abstract
Quality management has become a crucial factor for improving student
success, with reporting being widely used to scrutinize curricula for possible
bottlenecks and resource deficiencies. Predictive capabilities in that context
have, however, been often limited to simple regression models acting on
historical data, which might not always be available when curricula change
often; furthermore, work in curricular planning often demands “what if”-
scenarios that are beyond extrapolation, such as determining the influence of
changes in procedure on student success, which in itself is based on a multitude
of intertwined factors such as social background and individual performance.
In the PASSt project, we have been using Machine Learning and Agent-Based
Simulation for Predictive Analytics in that sense. As a result, we have been
developing an extensive toolset for curriculum planning which we want to
outline in this paper, together with some lessons learned in that process. Our
work will help practitioners in higher education quality management
implement similar methods at their institutions, with all said benefits.
Keywords: Data modeling; machine learning; agent-based simulation;
predictive analytics; quality management; curriculum planning.

The PASSt project: predictive analytics & sim of studies aimed at QM and curriculum planning
1. Introduction
Reporting is nowadays commonplace in higher education quality assessment: On the one
hand, this allows for a guidance of the curricular planning process, on the other hand such
assessments are also prescribed by governmental regulation (e.g. the EU “Standards and
Guidelines for Quality Assurance in the European Higher Education Area (ESG)”, see ENQA
2015). Digitalization has been long lagging behind requirements in that area, with data
repositories primarily serving study administration but not analytics; as a consequence policy
makers have explicitly made Digital Transformation in that area a priority (e.g. the EU
“Digital Education Action Plan” [EC 2018], especially “Priority 3: Improving education
through better data analysis and foresight” [ibid.]).
In the course of the PASSt project, we have been extending the scope of study analytics at
the partner universities to prediction using Machine Learning and Simulation while at the
same time improving the data basis. This paper reports on the toolset we developed in that
course for curricular planning and quality assurance, and gives some lessons learned for
people seeking to implement similar measures in their institution. In more detail, we:
• Initially define activity dimensions within curricular planning and quality

assessment, both acting as our requirements and context (Section 2, “Background”).
• We then review work that is similar in aim or outcome (Section 3, “Related Work”).
• Next, we describe the data basis that is an outcome of a lengthy discussion process
between project partners and a wider field of universities (Section 4, “Base Data”).
• The core of our contribution herein lies in the description of our curriculum planning
toolset (Section 5, “Developed Toolset”) on the curriculum and aggregate level.
• Before concluding, we also discuss some lessons learned (Section 6, “Discussion”).
2. Background
Our work is aimed at supporting curricular planning in study committees typically headed by
the provost/dean of studies, where work typically happens on the following two design levels:
Curriculum: View of a whole study with strong focus on amount of students (input/output,
dropout rates), sequences of courses in different phases (e.g. introductory phase, intermediate
phase, finishing phase) and semester-wise view of throughput through this system of these
lectures implicitly or explicitly connected by their preconditions.
Lecture: View of an individual course in terms of its location within a curriculum (prescribed
vs. actually taken semester), capacity (number of examinations per semester), type (lecture,
lab, seminar, etc.; mandatory, elective or optional), success rate (as a function of numbers of
times already taken; or curriculum if the course is shared between multiple studies), credits
granted for successful completion, periodicity (one-time, every semester, every year, every
802
Gabriel Wurzer et al.
two years in seldom cases), temporal view of course utilization by students, rules for
accreditation, preconditions for inscribing to a course, and so forth.
Even though many of these factors are beyond planning (e.g. number of students starting
their studies), many can be influenced implicitly using the following action dimensions which
are, quite interestingly, situated only at the lecture level:
Location within the curriculum and preconditions for lectures. An attribution of study
phase (beginning, intermediate, finishing phase) ensured by preconditions is selective with
regards to students opting in (or out, e.g. by not continuing a study; while this may seem cruel
many of the efforts go towards self-awareness over whether a study is suitable for a student
or turns out to be the wrong choice). Another aspect is the semester in which students actually
take a lecture (vs. the prescribed semester), since this has to do with the perception of students
and the success rate (more in due course for the latter).
Capacity. This dimension is occupied with the capacity of lectures in terms of examinations
per semester as well as the capacity for students seeking to enroll; many social and effort-
wise questions range into that field and are enforced by specific rules (e.g. not being able to
take two large lectures that have project character in one term). Where ressources are scarce
capacity may also be changed by changing periodicity.
Success rate. Students are sometimes impeded by lack of previous knowledge, in which case
additional lectures on basics (“introduction to …”) or shifting lectures from one semester to
the other can be used to raise success rates; on the other hand, there might be reasonable
doubts on fair assessment by lecturers, which must be further addressed in the process of
quality management.
Clearly there is also a social dimension in all of this; for example, taking one’s mouth too
full will result in a danger of dropping out, and therefore the aim must also be on ensuring
that a curriculum is adequate in the light of different target groups which must be empowered
to study at their own pace. Even though the project is not focused on individual feedback, we
have the option to simulate individual study performance [in credits] based on different (also:
social) characteristics, which gives a more realistic model of maximal effort invested.
3. Related Work
As said, our work is targeted at curricular planning rather than student self assessment or
individual feedback which a lot of approaches seek to facilitate. Some of the examples of the
latter include the Ingram (2020) agent-based classroom lessons model, which is an example
of Agent-Based Simulation on the individual level, but does not offer general insight for
further study analytics. In fact, we are convinced our Agent-Based Simulation is i the first of
its kind, from having researched the literature to the best of our knowledge. Research
803
analytics with the intended goal of curriculum planning is also novel; there are a handful of
approaches which try to understand curricula in higher education – e.g. De Silva et al. 2021
– however there is lack of a tool approach as novel methods proposed but not implemented.
4. Base Data
Universities are very different in terms of what they assess in their administrative processes
on which study analytics builds; therefore it was necessary to negotiate, in view of this
project’s goals, what data should be included in a “common data structure” serving as base
data for futher analysis among all project partners. The outcome (also compare with Figure
1) is structured as follows:
Base Entities. Students (sid) and Lectures (lid, title, name, type, credits) with their metadata.
Several Curriculums (cid, … ) are also present, which are to be outlined in due course.
Linking Relations. Lectures are assigned to a Curriculum by the presence of a Curriculum
Assignment; this is, however, a theoretical assignment; the actual realization of that relation
is the Study, which links a Student to a Curriculum. In this course a student takes an
Examination for a Lecture. The outcomes and semester of this examination are further taken
by us to infer additional data (semester actually taken; distribution of semesters taken;
success rate; periodicity and base semester [winter or summer term]; capacity of a lecture).
Figure 1. Base data structure for the PASSt project.
804
5. Developed Toolset
Based on the Examinations within the base data, we can infer the actual semester distribution
of Lectures within a Curriculum. The most frequent semester (see red bar in the left part of
Figure 2) is then used to build up replica of a curriculum (lectures in semesters, see
background table in the left part of Figure 2) that can be simulated by use of Agent-Based
Simulation, where each agent represents an individual student. The student numbers are
inferred from Study within the base data, and can be altered to simulate “what-if” scenarios
(see right part of Figure 2). Likewise, capacities of lectures (examinations/semester), type
(mandatory, elective or optional), success rates (likelihood of students successfully taking a
lecture), periodicity (one-time, every semester, yearly, every two years) and anchor semester
(winter or summer) are inferred from Examination and Study. Student performance (maximal
effort in credits/semester) can either be inferred from the mean credits of all students per
semester (see again right part of Figure 2), or using a Machine Learning prediction of credits
based on individual factors such as (ordered by importance) credits in previous semesters,
education background, existence of multiple studies, gender and citizenship (more details cf.
Spörk et al. 2021 [in German]).
Figure 2. Simulation Input. (left) Curriculum with one lecture being currently edited, (right) number of students in
winter and summer cohort, number of credits taken per term
Given their maximal performance, students seek to enroll in lectures and take examinations
(subject to maximal capacities = lectures as servers with queues; examinations governed by
the aforementioned success rate). As outcome, we can observe the semester-wise utilization
of a curriculum where most prominent lectures are highlighted in tabular form (left in Figure
3) or visualized as a sim “semester tunnel” (right in Figure 3).
In addition, we have been developing a portal that lists aggregate measures for a whole
curriculum (see Figure 4): There we depict student beginner numbers, performance in credits,
and status (active, completed and aborted per semester). Furthermore, we have the regulatory
measure of “examination-active” students (i.e. more credits than a certain credit threshold).
805
Figure 3. Simulation results. (left) Tabular depiction of utilization, (right) visualization as “semester tunnel”
Figure 4. Portal depicting aggregated view per curriculum.
6. Discussion
The developed curriculum simulation is a queueing system with servers = lectures and clients
= students with individual amounts of credits (mean credits or determined by Machine
Learning). The question is whether that is “realistic” or not. Analysis of actual examinations
show that there is a Poisson-curve-like utilization of lectures, where students take less than
their maximal amount of credits in the first semester followed by more than the maximum
before the performance decreases based on study success and remaining credits to be made.
This behavior can be modelled in the simulation, however not as emergent outcome but as a
prescribed utilization curve; the cause for this is still an open question, or is this effect because
of examinations being taken one semester too late (and thus it would appear that students
806
have more effort in the second rather than in the first semester). Regardless, one sure take-
away is that it would make sense to reward studying uniformly, since no resources are wasted.
It would seem that our approach results in a “recommendation engine” that automatically
proposes the best way to increase student success and studyability; however, this is not the
case; the most important contribution of our work lies in bringing the numbers to the table; a
discussion is always needed since each study commission knows its “problematic” areas, and
solutions are based on many factors (but we are contributing some ground truth, of course).
In this context, we also wish to emphasize the role of the aggregate views available in the
portal (see again Figure 4), which add an overview of the base data per curriculum (filterable
also per semester) as well as a prediction component (e.g. predicted ECTS/semester).
In practical terms, our toolset offers an unprecedented data density for evidence-based
curriculum planning, as we would call it: We have reporting based historical data, prognosis
based on Machine Learning and/or extrapolation of the former (e.g. linear regression); we
also support “what-if” type of capacity planning using our simulation. The intended audience
of such tools ranges from study commissions over provosts/deans to the university
management, at which level benchmarking and goal-setting is important not only because
one wants to improve studyability but also since may be connected to financing (e.g. via
objective agreements set by the state).
“The ultimate goal is to help HEIs quality management processes and procedures”
summarizes what we aim for best (thank you to reviewer 3 for that perfect one-liner).
However, as also noted by the same reviewer, care must be taken as to not employ these
methods in order to single out lectures with adverse intents, or (as we would also add) to give
a negative prognosis on an individual level. Therefore, an integral part of the PASSt project
is to also look at the ethical and legal implications when applying that technology, resulting
in a legal guide (e.g. pseudonymization as requirement) as well as a Code of Practice
document which states what computations may or may not be performed on the base data.
7. Conclusion
We presented a curriculum planning tool based on the individual lecture level that is
simulated in an agent-based manner, plus an analysis tool that shows aggegated student data
for students of a curriculum; both approaches are supplemented by a machine-learning
prediction that can predict student performance based on individual factors. The outcome is
that many levels of inquiry are needed (and: possible, with our approaches) in order to
increase student performance; however most analysis results demand a post-hoc discussion
with the stakeholders.
807
Acknowledgments
The PASSt project is funded by the Austrian Federal Ministry of Education, Science and
Research under the call “Digital and Social Transformation”. This paper builds on two
previous articles in the Journal for Higher Education Development [Zeitschrift für
Hochschulentwicklung], Spörk et al. 2021 as well as Tauböck et al. 2023 [in press] as well
as one conference paper in MathMod 2022 [Wurzer et al. 2022]. The paper has been written
from scratch with the focus on tool support for curriculum planning and QM, leaving out
legal and ethical aspects which form yet another basis (kudos go to Johannes Kepler
University Linz - especially to Anna Schöfecker, Gerhard Mühlbacher, Günther Gruber,
Florian Gundl and Stefan Pittner).
References
De Silva, L.C., Rodríguez-Triana, M., Chounta, I.-A., Pishtari, G.. (2021). An overview of
analytics for curriculum understanding and optimization in Higher Education.
Proceedings 11th International Conference on Learning Analytics & Knowledge
(LAK21), 270-285.
EC (2018). Commission Staff Working Document on the Digital Education Action Plan.
EUR-LEX Document 52018SC0012, Retrieved from: https://eur-lex.europa.eu/legal-
content/EN/TXT/?uri=CELEX%3A52018SC0012&qid=1647943853396 (accessed Jan
19, 2023).
ENQA (2015). Standards and Guidelines for Quality Assurance in the European Higher
Education Area (ESG). Brussels, Belgium. ISBN 978-9-08-168672-3. Available from
https://www.enqa.eu/wp-content/uploads/2015/11/ESG_2015.pdf (accessed Jan 19,
2023).
Ingram, F. (2020). An Agent-based Classroom Lessons Model and Simulation. Doctoral
Thesis, Lancaster University. Available from: https://www.researchgate.net/
publication/354625019_An_Agent-based_Classroom_Lessons_Model_and_Simulation
(retrieved 16th Jan 2023).
Spörk, J., Tauböck, S., Ledermüller, K., Krikawa, R. and Wurzer, G. (2021). Analysis of
studyability by means of prediction and simulation models, in Journal for Higher
Education Development [Zeitschrift für Hochschulentwicklung], 16(4), 163-182.
Tauböck, S., Schöfecker, A., Ledermüller, K., Krakovsky, M., Sharma, S., Reismann, M.,
Marschnigg, C.G., Mühlbacher, G., Spörk, J., Schadler and Wurzer G. (2023 [in press]).
PASSt - Predictive Analytics Services für Studienerfolgsmanagement. Journal for Higher
Education Development [Zeitschrift für Hochschulentwicklung], Special Issue
“Sonderausgabe Hochschullehre - Digitalisierung der Hochschullehre – Projekte
österreichischer Hochschulen 2020–2024”.
Wurzer, G., Reismann, M., Marschnigg, C., Dorfmeister, A., Tauböck, S., Ledermüller, K.
and Spörk, J. (2022). PASSt-A: Agent-based student analytics aimed at improved
feasibility and study success. Proceedings of MathMod 2022, IFAC PapersOnLine, 55
(20), 361-366.
808
An innovative, technology-enhanced instructional approach to

address the diverse competencies of STEM students in math
classes
Corinna Modiz, Andreas Körner
Institute of Analysis and Scientific Computing, TU Wien, Austria, Strategic Education
Development Center, TU Wien, Austria.
Abstract
The transition from secondary to higher education in mathematics raises
several challenges for students of STEM (Science, Technology, Engineering,
Mathematics) studies. These include amongst others different emphases and
requirements and the prohibition of computer algebra systems in basic math
education at universities, which leads to a high dropout rate in the first year
of study. To address these challenges, a so-called three-level system was
introduced at TU Wien (Vienna University of Technology) in first-year
mathematics courses to support and encourage students with different levels
of high school knowledge. This approach entails several requirements such as
a higher correction effort which is countered by the use of an e-learning system
called Möbius. It not only provides automatic grading of exercises but also
individual and immediate feedback for the students. The analysis of the grade
distribution of two years indicates a positive influence of the introduced three-
level system on first-year STEM students.
Keywords: Innovative teaching; evaluation and assessment; e-learning
strategies; new teaching models; STEM education; blended learning;
technology enhanced teaching.

An instructional approach to address the diverse competencies of STEM students in math classes
1. Introduction
During the transition from secondary to university mathematics education, students
experience various difficulties, which makes teaching methods at university an emerging
research field, see Voskoglou (2019) and Körner et al. (2014). According to Pinto & Koichu
(2022), there are several approaches in order to help students to overcome those challenges.
All of which address different origins, including the discontinuitiy of mathematics education
from secondary to university level. This describes the ability to perform algorithms to solve
problems taught in secondary mathematics educations in contrast to deeper mathematical
understanding required in university math courses of STEM studies.
In line with the results of Rylands & Coady (2009), students' mathematical competences and
their ability to pass first-year STEM courses depend strongly on their mathematical
background, i.e. the type of school attended and the quality of mathematics teaching. This
results in a diverse group of students, some of whom do not have basic knowledge of calculus
and cannot follow the constructive material, while others feel underchallenged if the level of
mathematics taught is lowered.
Furthermore, the intensive use of computer algebra systems (CAS) in secondary mathematics
education has a negative impact on students' understanding of how to manually do math and
what it should look like, as pointed out in a Danish study by Jankvist & Misfeldt (2021). As
the use of CAS is not allowed in university STEM courses of bachelor level, these deficits in
general mathematical skills and understanding are reflected in students' results, not only in
pure mathematics courses.
These challenges students are facing lead to a high dropout rate among first-year students in
STEM studies, see Geisler, Rolka & Rach (2023), and therefore need to be addressed. The
following section presents a general concept of the so-called three-level system that addresses
these challenges in mathematics education. In addition, its application in various
mathematical subjects at the beginning of the Bachelor's programme is briefly presented,
including the use of an e-learning system called Möbius, formerly Maple T.A., which is
explained in more detail in Winkler, Körner & Urbonaite (2012).
2. A three-level system to address different mathematical competencies
2.1. Structure of the three-level system

In order to meet the needs in third level mathematics education of a heterogenous group with
different backgrounds and skills, a three-level system is designed. The system is established
in the context of constructive alignment (objectives, assessment, education and teaching
activities), explained further in Biggs (1996) and Wang et al. (2013). The proposed approach
states what set of skills is expected from students as a learning outcome of the course
810
(objectives). The three-level system focuses on providing feedback on students' current

knowledge and skills so that they can continuously adapt their learning (assessment), whether
they start with a low level of knowledge or already have more developed skills. Figure 1
illustrates the basic idea of the three learning levels of the proposed system (education and
teaching activities).
Figure 1. Overview of the key aspects of the three-level system.
The basic level (Level 1), clarifies the basic requirements for each chapter of the course so
that students can enter with the necessary mathematical tools, also outlined in Dlouhá, D.,
Pospíšil & Dlouhá, K. (2022). This level is essentially based on secondary school
mathematics and therefore focusses on algorithmic math knowledge, but is adapted to the
topics of the respective course as it progresses. In this context, a set of exercises is provided
at the beginning of a chapter and has to be solved within a deadline of one to two weeks. The
completion is mandatory, but does not influence the final grade. Furthermore, students have
an infinite number of attempts to complete this assignment. An example of this level
presenting important concepts that the students should be familiar with in order to follow a
course on real functions is illustrated in Figure 2.
Figure 2. Basic concepts necessary for the topic of real functions.
The focus level (Level 2), addresses the learning outcomes to be achieved in the course.
These are mostly the examples that were offered in the traditional form of the course solely.
This level focusses on the application of results thought in level 1 to answer further advanced
problems. For example, the basic knowledge about an exponential function is used to find
range in which such a function must be bijective.The examples to be worked on in the
framework of this level are discussed weekly in exercise groups where, depending on the
811
respective course, the exercises will be either presented by a tutor or the students themselves.
A certain percentage, e.g. at least 50%, of exercises has to be solved by the end of the course.
The linking level (Level 3), facilitates the linking of knowledge and skills of the different
chapters. This level is oriented towards the final exam of the lecture, where the chapters are
not asked piece by piece, but an example on two or more aspects is covered. Students who
have not been exposed to this kind of examples usually have difficulties in the final exam,
even if they can easily solve tasks on a single topic. In addition, the deepening of knowledge
is promoted. To practise combining different topics independently, a typical example would
ask students to give a function that satisfies the properties known from level 2, such as
bijectivity and continuity. Of course, linking is less present at the beginning than in the later
chapters of the course. Generally, exercises of this level are not mandatory, but give bonus
points based on the quality of discussion of the student’s solution with a tutor.
2.2. Application Scenarios

As of today, the three-level system was applied to two math courses for first-year students at
TU Wien. It was first developed for the course Mathematics 1 for Electrical Engineering,
Technical Physics, Geodesy and Geoinformation, and later on also applied to Practical
Mathematics 1 for Technical Physics.
Mathematics 1 consists of a lecture and an exercise course, which are graded separately. The
grade of the exercise course is composed of the percentage of second level examples solved,
the quality of presentations, the sum of points of the three exams during the semester and
possibly bonus points. The grade for the lecture course is based on the points achieved in the
final exam that takes place at the end of the semester.
The three-level system was originally established for the course Mathematics 1, which
consists of a lecture and an exercise. Therefore, it is directly applicable to this course in all
aspects described and was later adapted to Practical Mathematics 1 while maintaining the
basic idea.
3. Implications of the three-level-system
3.1. Emerging challenges

The following are the main challenges selected based on student feedback and lecturer
experience. An obvious challenge is the increased workload for students and teachers in a
course implementing the three-level system. Compared to traditional educational systems in
this type of first-year mathematics course that implemented only the secondary level, those
involved face an increased catalogue of demands during the semester. Several teachers and
students complain about such systems - students about the increased workload, teachers about
812
the increased supervision and correction effort. This system is only accepted if it tends to
have a positive effect on students during the semester (preparation of midterm tests, weekly
homework, short quick tests, etc.) and leads to better results in the final exams.
Furthermore, the three-level system addresses another aspect of individual feedback. First
semester students tend to avoid self-assessment tests in the first weeks of the semester to
avoid a negative feedback. The first compulsory test of the first semester is usually the first
serious feedback students receive, and this feedback is usually rather negative. There are
several reasons why the first test is negative, one important reason being the lack of training
of the examples in different versions.
3.2. Solution approaches

To address these challenges, several aspects are considered in the three-level system. The use
of an e-learning system, which provides a set of examples for students to practice with, allows
for individualised, low-threshold feedback for students. This feedback is only available to
the students and not to the teacher. Such examples appear partially in weekly homework and
midterm tests, so that students see the benefit of practicing these examples during the
semester. The first and second stages of the three-level system help to achieve this effect on
students. The use of the e-learning system Möbius makes it possible to provide randomised
examples and to assess students' results immediately after completion. This means immediate
feedback for the students and no correction effort for the lecturer. In addition, the examples
provided are randomised by algorithms, i.e. students are given the same question style, but
certain parameters (e.g. numbers, functions, vectors, matrices, etc.) are varied. Such a
situation forces students to practice their arithmetic skills more than with the traditional,
unmodified examples. In the traditional form, students tend to read only the solution of
already solved examples and in the exam, the resulting lack of skills to solve the example
causes problems. The three-level system avoids such effects from the beginning since
students cannot copy solutions in level 2 but need to solve their individual examples.
Tomilenko & Lazareva (2020), who use a comparable CAS, have shown that this has a
positive effect on students' test scores.
3.3. Additional advantages

Students have constant access to the pool of examples provided via the e-learning system. In
addition to the weekly homework, the examples are used to assess the basic knowledge
(Level 1) and to prepare for the mid-term tests during the semester. This continuous training
of the focused topics provided in the assignment definition of the examples prepares students
for the final test of the course at the end of the semester. In the traditional framework,
exercises during the semester and final exam were considered as two different parts. In the
813
three-level system, the skills required for linking learning outcomes (Level 3) contribute
greatly to the skills required for the final exam.
In addition to the advantages in the learning process of the students, the lecturers also benefit
from this system. They receive constant feedback on the students' current level of knowledge
and can react accordingly. If repetitions or question rounds in the lecture or tutorial seem
helpful, they can offer such measures in the course.
4. First Analysis of the results of three-level system courses

This section presents and compares the results from exercise courses between 2019 (113
participants) and 2022 (91 participants) to show an initial trend of the impact of the
introduced three-level system. These years were chosen since the three-level system was
introduced after 2019 and 2022 provides the most up-to-date results after three years of
running the three-level system. The data was collected and obtained from the student
administration system of TU Wien.
Figure 3. Grade distribution (best “1” to worst “5” grade) of the exercise courses of 2019 (left) and 2022 (right).
Figure 3 shows the grade distribution of the exercises course of Mathematics 1 in the summer
term 2019 and 2022 respectively. In both diagrams, “1” corresponds to the best possible
grade, while “5” means that the course could not be completed successfully. While in the
summer term of 2019 only 50% of the participant of the exercise course were able to pass,
this percentage increased to 61% in 2022.
A first analysis of the grade distribution of the final lecture exam also shows a positive trend
in the results between the years without and with the introduced three-level system. This
tendency with regard to the influence of the three-level system is visible in the overall results,
but is not easily verifiable statistically, since students in Austria do not have to take the final
lecture exam immediately after attending the exercise course.
814
5. Discussion
The three-level system is introduced as an approach to address students' challenges due to
mathematics education deficits in the first semester of STEM studies, in accordance with the
observations of Pinto & Koichu (2022). However, challenges in the transition from secondary
to tertiary mathematics are not only related to a lack of skills. They also concern the
interaction and communication between higher education teachers and students, as well as
the different experiences of students during the transition phase due to different
characteristics such as race, gender, sexual orientation or cultural background, see Nasir et
al. (2012), Rodd & Bartholomew (2006). Therefore, further development and investigation
of the three-level system and continuing teaching methods with respect to these aspects is
required.
As mentioned in section 3, automatic grading of randomised exercises with the e-learning
system Möbius counteracts the increased workload for the teacherNevertheless, the
development of examples, including the initial idea, the layout, the algorithm behind the
actual exercise, the meaningful randomisation and the grading code is a time-consuming
process. Introducing an e-learning tool therefore takes time at first for the initial development,
but offers many advantages for teachers and students, e.g. individual, immediate feedback,
the constant possibility to access the pool of examples and to observe the learning progress
of the students.
To get a first impression of the impact of the three-level system on student results, the grade
distribution of the 2019 and 2022 exercise course is presented in section 4. It shows a positive
trend in the development of student results based on the increased number of participants
who passed the course after the introduction of the three-level system. As the comparison is
limited to two years, the positive influence needs to be further substantiated by examining
the results of other semesters using more informative statistical methods. Furthermore, the
exact analysis of the impact of the three-level system on the final lecture examination needs
to be further investigated beyond the positive trend. When merging the results of the exercise
and lecture, cohort statistics are required in order to include students who have taken
examinations beyond their year group.
It is planned to introduce the presented three-level system in other courses at TU Wien, e.g.
in courses of Linear Algebra for Technical Physics. To this end, the approach must be adapted
to the corresponding topics, e.g. include basic matrices and vector arithmetic in the basic
level. In addition, this subject raises requirements for the automatic evaluation of examples,
e.g. ambiguous answers such as the evaluation of an orthonormal basis of a vector space,
which can consist of vectors of different order or linear combinations. Therefore, the example
pool and the corresponding grading library need to be further developed.
815
References
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education 32,
347–364. doi: 10.1007/BF00138871.
Dlouhá, D., Pospíšil, L. & Dlouhá, K. (2022). High school mathematics knowledge level of
technical university students. 8th International Conference on Higher Education
Advances (HEAd’22). doi: 10.4995/HEAd22.2022.14268.
Geisler, S., Rolka, K. & Rach, S. (2023). Development of affect at the transition to university
mathematics and its relation to dropout — identifying related learning situations and
deriving possible support measures. Educational Studies in Mathematics. doi:
10.1007/s10649-022-10200-1.
Jankvist, U.T. & Misfeldt, M. (2021). Old Frameworks - New Technologies. Canadian
Journal of Science, Mathematics and Technnology Education. 21, 441–455. doi:
10.1007/s42330-021-00164-4.
Körner, A., Winkler, S., Pöll, C. & Breitenecker F. (2014). Erlangen von Verständnis und
Erlernen von Fertigkeiten in der Mathematik einmal anders. Zeitschrift für
Hochschulentwicklung, 9(4), 101-115. doi: 10.3217/zfhe-9-04/07.
Nasir, N. S., Snyder, C. R., Shah, N. & Ross, K. M. (2012). Racial Storylines and
Implications for Learning. Human Development, 55(5-6), 285-301. doi:
10.1159/000345318.
Pinto, A. & Koichu, B. (2022). Diverse perspectives and experiences of university
mathematics teachers on improving the secondary-tertiary transition. Educational Studies
in Mathematics. doi: 10.1007/s10649-022-10196-8.
Rodd, M. & Bartholomew H. (2006). Invisible and special: young women’s experiences as
undergraduate mathematics students. Gender and Education, 18:1, 35-50, doi:
10.1080/09540250500195093.
Rylands, L.J. & Coady, C. (2009). Performance of students with weak mathematics in first-
year mathematics and science. International Journal of Mathematical Education in
Science and Technology, 40(6), 741-753. doi: 10.1080/00207390902914130.
Tomilenko, V. A., Lazareva, E. G. (2020). Effective Use of a Computer Grading System for
Teaching Mathematics at a University. ITM Web Conference, 35, 03017. doi:
10.1051/itmconf/20203503017
Voskoglou, M. (2019). Comparing Teaching Methods of Mathematics at University
Level. Education Sciences, 9(3), 204. doi: 10.3390/educsci9030204.
Wang, X., Su, Y., Cheung S., Wong, E. & Kwong, T. (2013). An exploration of Biggs’
constructive alignment in course design and its impact on students’ learning approaches.
Assessment & Evaluation in Higher Education, 38(4), 477-491, doi:
10.1080/02602938.2012.658018
Winkler, S., Körner, A. & Urbonaite B. (2012). Maple T.A. in Engineering Educations. IFAC
Proceedings Volumes, 45(2), 906-911. doi: 10.3182/20120215-3-AT-3016.00160.
816
Development of scientific skills in Higher Education with a flipped

classroom-contest approach
Miguel Garcia-Bosque1, Carlos Sánchez-Azqueta2, Concepción Aldea1, Esther
Cascarosa3, Santiago Celma2
1
Department of Electronics Engineering and Communications, University of Zaragoza,
2
Department of Applied Physics, University of Zaragoza, 3Department of Specific Didactics,
University of Zaragoza.
Abstract
In this work, the flipped classroom methodology has been applied to the
laboratory sessions in the subject Physical Techniques I of the Degree in
Physics at the University of Zaragoza, Spain. The proposed sessions have been
distributed in two main parts. The first part consists on flipped laboratory
sessions in which during the before-class sessions, the students must
understand, design and customize the designs that will have to characterize
experimentally in the laboratory sessions. The second part consists of a student
contest activity where the students compete against their pairs while improving
their learning about the topics presented in the flipped classes. The proposed
approach could increase the depth of the acquisition of experimental skills,
helping students to acquire a better understanding of the concepts under study
in laboratory sessions.
Keywords: Active learning; collaborative work; flipped classroom; laboratory
skills; model-based learning.

Development of scientific skills in Higher Education with a flipped classroom-contest approach
1. Introduction
Model-based learning is the construction of mental models of phenomena by integrating
information about their structure, function and causal mechanism, and mapping that
information into analogous systems (Gobert and Buckley, 2000). A model can be seen as a
representation of a target and serves as a “bridge” connecting a theory and a phenomenon
(On and Oh, 2011). If we consider the classical approach of Gilbert (1991), models are built
by an induction process based on simplified representations of systems. Teaching of physics
(and engineering) has traditionally focused on the development of a conceptual
understanding of the relations between the contents, laws and principles of the matter.
Physics education research has uncovered serious difficulties in learning (Duit, 2009)
because of the lack of models that emphasize the construction of knowledge (Ceberio et al.,
2014). Nowadays, model-based learning is becoming an essential part in the evolution of the
strategies in science teaching. Despite the difficulty of knowing the nature and content of the
mental models of students, instructors can design different strategies to address and evaluate
them. In this paper we introduce the flipped classroom as a relevant instructional strategy.
The flipped classroom is a pedagogical method that transfers certain learning processes
outside the classroom and uses class time to facilitate and enhance other processes of
acquisition and practice of knowledge (Wang et al., 2018; González-Gómez, 2017). It
promotes autonomous learning and also frees classroom time to carry out active activities in
which participation and the exchange of ideas are promoted. From a model-building point of
view, students are encouraged to become more personally involved in their construction,
while at the same time the necessary conditions for these models to be contrasted and
discussed with their peers are promoted. As a consequence, a flipped classroom has to be a
comprehensive approach that combines direct instruction with constructivist methods with
the main objective of increasing student engagement and involvement in the course content
to improve their conceptual understanding.
In this work, we propose to apply the flipped class methodology to the experimental part of
the compulsory subject Physical Techniques I of the Degree in Physics at the University of
Zaragoza, Spain. Adding to the advantages of active learning, the flipped classroom allows
to dedicate more laboratory time to apply knowledge acquired using active learning
strategies, and therefore the learning process can be better followed by the teacher.
2. Proposed Teaching Methodology

The classical interpretation of model-based learning engages students in an iterative process
oriented to build a mental model of a given phenomenon. There are many ways to achieve it
such as inductive processes, simplifications, comparisons, integration of information, etc.
(Cascarosa, et al., 2020). Although model-based learning needs to be carried out by the
818
Miguel Garcia-Bosque et al.
students, the instructors can adopt strategies to facilitate the acquisition of the models, and to
assess that they have been built correctly. It is in this scenario where the flipped classroom
methodology is applied, allowing to create an environment of collaborative learning in the
classroom, and involving the students from the beginning in their own learning process. By
the application of a methodology based on the flipped classroom, the authors expect to
contribute to a correct development of mental models by the students.
2.1. Participants and scope

Participants are the students enrolled in the course Physical Techniques I, which aims to
provide basic skills in metrology and electronic instrumentation for the measurement of
physical magnitudes. In Physical Techniques I, students must be able to analyze basic
electronic circuits, determine specifications and tolerances for a measurement, and design the
full setup to carry out a measurement, calculating the contribution of every stage to the
uncertainty. The theoretical contents of this course have a weight of 4 European Credit
Transfer System (ECTS) credits while the experimental part has a weight of 2 ECTS credits.
The experimental part of the course is divided in four laboratory sessions: the first session is
introductory and intended for students to become familiar with the measuring instruments;
the second session teaches students to analyze the stationary and transient behavior of
resistor-capacitor (RC) and resistor-inductor-capacitor (RLC) circuits; the third session
studies experimentally the behavior of several circuits such as a non-inverting amplifier and
a comparator, and the last session deals with filtering and signal conditioning. The basic
concepts and lab procedures are previously explained during the theory blocks, which take
place before the laboratory sessions.
In this course, students face the basic concepts of Electronics and, in particular, Circuit
Theory for the first time. Furthermore, the laboratory sessions require the use of several
instruments that students are not used to manipulate. As a consequence, students often need
guidance by the instructors, which leaves less space to implement active learning activities.
2.2. Example of application

Traditionally, before the realization of a laboratory session, students receive a theoretical-
practical explanation about the goals and the procedures to be followed. Subsequently, during
the session, they have access to specific supporting materials (instrumentation user manuals,
component data sheets, etc.). Also, the instructors, present during the sessions to help in the
realization of the different tasks, have to deal with several groups simultaneously, making it
difficult to provide an immediate and comprehensive answer to all questions raised by the
students. The proposed flipped laboratory sessions are distributed in 2 main parts.
1. Flipped Laboratory. Before the laboratory sessions, the students are given the designs and
their intended application. With the help of electronic simulation software, they must
819
understand the designs, choose some parameters to make them work as intended and, finally,
verify the correct operation of the final designs. During the laboratory sessions, the students
have to implement the designs with commercial components and characterize them
experimentally, comparing the results with those obtained by simulation. The main idea of
this part, which will be more deeply explained afterwards, is to introduce students to the
standard design-simulation-characterization process typically used by electronic designers.
2. Student contest. This practice is planned as a very dynamic activity where the students
compete against their pairs while improving their learning about the topics presented in the
flipped classes. This activity is divided in two sessions: an initial one based on solving a real
problem using Toolkit for Interactive Network Analysis (TINA) simulations, and a second
one consisting on a quiz. Unlike the flipped laboratory, which is an experience that has
already been carried out, the student contest is an activity that we have newly introduced in
the current course, where we have applied it to the second laboratory session to obtain
preliminary results. If these results are satisfactory as it is expected, the approach will be
extended to all laboratory sessions in future courses.
3. Proposed experience
During the before-class activities, the students, divided in groups of 2 or 3 depending on the
total number of students enrolled, have access to different materials and references related
with the topics of study as well as to an explanation of the main concepts covered. They are
encouraged to refer to other subjects such as electromagnetism to ensure a good link with the
previously study topics. Also, they have some references to look for the main figures of merit
and performance criteria of the system under study. Although some main references are
provided by the instructors, the students are encouraged to look for related information and
references. The students have to carry out TINA simulations of some performances. For this
purpose, they have access to different tutorials, explicative videos and manuals of how to use
this software. TINA is a SPICE-based electronic design and simulation software. In the
laboratory session, the students will use the free version TINA-TI offered by Texas
Instruments. The groups have access to the material two weeks before the laboratory session.
But the instructor is going to acquire the role of mentor supervising and assessing in the
preparation of the flipped class. Therefore, as a minimum, one week before the laboratory
session, a meeting between the teacher and the groups is planned so that students can explain
the development of the work they are doing as well as to ask for help with the organization
of the flipped class and the bibliographic search. During the laboratory sessions, the students
must experimentally characterize the circuits that they have previously studied and simulated.
The circuit that they have to characterize experimentally, the RLC circuit in the session 2, is
the one that combines the two concepts: a circuit in which two energy storing elements
820
exchange energy periodically, and also a circuit in which an energy dissipating element (the
resistor) gives out a part of the total energy in every oscillation cycle. Also, the operation of
this circuit can be linked to that of mechanical damped oscillations, so that they can relate
with all aspects covered in courses on Mechanics about the actual damped oscillating
frequency versus the undamped frequency (that of the LC circuit) and the ratio of energy lost
per cycle, which is given by the quality factor Q.
To conclude the flipped class activities and also as a way to promote the interest of the
students in the presentations of their colleagues, we propose to perform a student contest.
This contest consists of two classroom sessions in an interactive classroom where each
student can win extra points (half of them in each student contest session) for the total grade
of the course. This special classroom contains some television screens with their respective
cooperative tables and a main bigger screen with a projector. Making use of the MirrorOp
Sender application, the students can connect their devices to the TV screens to benefit the
collective work and the share of information and in this way, everyone in the group can easily
see what the others want to show. The teacher has access to all the screens of the room so
that he/she can show in the main big screen some interesting information of some groups that
can be useful for the rest.
For the first session of the student contest, the students have to solve a case study using all
the knowledge they obtained in the flipped classes. In particular, it was found convenient that
students made use of TINA simulations to solve the case study. They have one hour to read
and solve the case with the help of all the tools and information they have available in their
computers, the material of the course uploaded in the Moodle Learning Platform and the
Internet. As a deliverable, at the end of the session, the students have to send to the teacher a
slide with a summary of their solution of the problem. In the last minutes of the class each
group has to present and defend their work. The teacher is in charge of deciding which work
was the winner to give the group the extra points. In one case study, the students are given a
design which consists of a temperature detector connected to an amplifying stage which
output will be proportional to the temperature. The student must propose a solution to
mitigate the output error by high interferences in the wire connecting the sensor with the
amplifier. Several possible signal conditioning circuits are given to the students and they have
to choose the appropriate circuit. In other, the students are given a radiation detector contains
a Geiger-Muller tube, which produces an instantaneous current pulse when a particle passes
through it. This current decays rapidly in the absence of new particles. The students must
include an additional circuit that manages to make the presence of the particle visible by
prolonging the duration of the voltage peak supplied by the detector. In a similar way as in
the previous case of study, the students are given several possible circuits and they must
select the appropriate one and integrate everything in a final design (Figure 1).
821
(a) (b)
Figure 1. Scheme of systems where the students must include a circuit in (a) temperature (b) radiation detector.
The second session of the student contest consists on a quiz that contained different questions
related to the flipped classes. To solve the questions, the students have free access to all the
material of the course. To guarantee the collaborative objective of the session, the students
only have access to one question of the quiz at a time and they cannot come back in the quiz.
At the end of the quiz, the students know their punctuation but not the specific questions that
they answered right or wrong. In this way, they have to decide whether or not they want to
try to solve the quiz again as the winner is the group with more right questions when the time
ends. For this second session the students have one half hour to solve the quiz. The rest of
the class is left for the teacher to explain the main solutions to the questions of the quiz and
debate with the students their solutions.
4. Assessment
To evaluate how the flipped classroom influences the learning outcomes and the construction
of the knowledge model, the assessment of the students is divided in several actions along
the course. Among the evaluation actions designed, we can cite specific quizzes, laboratory
papers or talks containing information on the studied systems, their operating principles and
related relevant information. Working through these resources, student must be able to
construct his model and to check and assess his knowledge model.
The assessment of the activity has been focused on two main points. The evaluation of the
quality of the resources created by the students and their performance in the flipped class as
well as their performance in the student contests. And personal motivation and satisfaction
of the students. To investigate how effective the tandem learning outcomes-flipped classroom
is, a series of specific evaluation activities have been programmed both in the pre-class
activities and in the face-to-face (F2F). A battery of quizzes associated with the use of
simulation tools and applets in pre-class activities has been designed, which not only reveals
the degree of understanding of the topics covered but also how they have been carried out,
one of the bases of the flipped. In addition, both the learning and follow-up resources
generated can be downloaded to any device, making it possible to do them anywhere and
allowing a flexible learning environment. Another series of issues associated with F2F
activities are associated with verifying both the associated specific learning outcomes and
the effectiveness of these self-study sessions.
822
The evaluation of the laboratory sessions is done attending to skills that students are expected
to develop along the course. The skills encompass technical aspects such as the
implementation of the experimental set-up, the measurement process, and also behavioral
aspects such as student autonomy and attitude, time spent, or the elaboration of the final
report. The assessment of the skills related to the assembly of the instrumentation is carried
out by a questionnaire before the laboratory session, together with viewings of videos of
handling the devices. Finally, the assessment of the skills related to the communication of
the results is carried out by a report that students have to hand out within the following days
after the realization of the lab session (Sánchez-Azqueta et al., 2019). To evaluate the
experience and motivation of the students, a specific survey has been created since evidence
suggests that there exists a strong correlation between student motivation and satisfaction,
and the success of virtual learning activities. To validate the effect that the inclusion of the
flipped classroom has on the results of the students enrolled in the course under study, a
comparison should be carried out with the grades obtained by students during the academic
years when it was not already implanted, but the strategy presented in this work has been
performed in a current academic year.
5. Expected results and Conclusions

This work presents the use of flipped classrooms combined with the student contest as a
pedagogical tool to implement model-based learning in higher education. In spite not having
final results yet, some advantages over the traditional approach are expected using this
methodology, such as increased student motivation and autonomy, or a deeper understanding
of difficult concepts (Gilboy et al., 2014). In addition, it encourages the use of specific
recourses to support the study of the topics and to acquire a deeper understanding of these
processes, improving student motivation (Chen et al., 2014). The flipped classroom allows
more time for active learning to happen in the classroom, further they develop a collaborative
work in a realistic environment, which fosters their learning autonomy and active,
participative teaching, promoting the conditions leading to the establishment of peer learning.
In summary, the methodology proposed in this work allows students to map the different
pieces of information into a general system to develop their own mental model.
In this respect, it can be considered that while non-interactive resources are more convenient
to present the topics and main ideas, interactive resources are more powerful to establish
links between them to build the models (Sánchez-Azqueta et al., 2016), and to check whether
the conclusions drawn from the models are true (O´Flaherty and Phillips, 2015). A process
is created so that students engage in the development of the mental models of the topics
covered in class with tools aimed for facilitating their construction and also assessing their
correctness. The implication of every student is required for this process and therefore fosters
their learning autonomy, but also the exchange and social cooperation with their peers to
823
analyze their own knowledge models, which entails sharing hypotheses, amending their
thoughts and working with their cognitive disagreements.
References
Cascarosa, E., Sánchez-Azqueta, C., Gimeno, C., & Aldea, C. (2020). Model-based teaching
of physics in higher education: a review of educational strategies and cognitive
improvements. Journal of applied research in higher education, 13(1), 33-47.
Ceberio, C., Almudí, J. & Zubimendi, J. (2014). Analysis of the arguments created by
students in introductory courses of university physics when tackling problematic
situations. Enseñanza de las Ciencias, 32(3), 71–88. doi:10.5565/rev/ensciencias.1112
Chen, Y., Wang, Y., Kinshuk, & Chen, N.S. (2014). Is Flip enough? Or should we use the
flipped model instead?. Computers & Education, 79, 16-27.
Duit, R. (2009). Bibliography-stcse: Students’ and teachers’ conceptions and science
education. Kiel: University of Kiel.
Gilbert, S.W. (1991). Model building and a definition of science. Journal of Research in
Science Teaching, 28(1), 73–79.
Gilboy, M.B., Heinerichs, S., & Pazzaglia, G. (2015). Enhancing student engagement using
the flipped classroom. Journal of nutrition education and behaviour, 47(1) 109-114.
Gobert, J.D. & Buckley, B.C. (2000). Introduction to model-based teaching and learning in
science education. International Journal of Science Education, 22, 891-894.
González-Gómez, D., Jeong, J.S., Cañada Cañada, F., & Gallego Picó, A. (2017). La
enseñanza de contenidos científicos a través de un modelo Flipped: propuesta de
instrucción para estudiantes del Grado de Educación Primaria Enseñanza de las Ciencias,
35(2), 71-87. doi:10.5565/rev/ensciencias.2233
O'Flaherty, J. & Phillips, C. (2015). The use of flipped classrooms in higher education: A
scoping review. The Internet and Higher Education, 25, 85-95. doi:
10.1016/j.iheduc.2015.02.002
Oh, P.S. & Oh, S.J. (2011). What teachers of science need to know about models: An
overview. International Journal of Science Education, 33(8), 1109-1130.
Regueras, L., Verdú, E., Muñoz, F., Pérez, M., de Castro, J. & Verdú, M. (2009). Effects of
competitive e-learning tools on higher education students: A case study. IEEE
Transactions on Education 52(2), 279–285.
Sánchez-Azqueta, C., Gimeno, C., Celma, S., & Aldea, C. (2016). Using the wiimote to learn
mems in a physics degree program. IEEE Transactions on Education, 59(3), 169–174.
Sánchez-Azqueta, C., Celma, S., Cascarosa, E., Aldea, C., & Gimeno, C. (2019). Application
of a flipped classroom for model-based learning in electronics. International Journal of
Engineering Education, 35(3), 938-946.
Wang, J., Jou, M., Lv, Y., & Huang, C.C. (2018). An investigation on teaching performances
of model based flipping classroom for physics supported by modern teaching
technologies. Computers in Human Behavior, 4, 36-48.
824
The essentials of science communication in a course engaging both

for students and professionals
Maurizio Dabbicco1, Franco Liuzzi2, Sandra Lucente1, Massimo Trotta3
1
Department of Physics, University of Bari, Italy, 2The Factory srl, Bari, Italy, 3Istituto per i
Processi Chimico-Fisici, Consiglio Nazionale delle Ricerche, Italy.
Abstract
On the one hand, science is increasingly entering the governance of modern
society. On the other hand, society seems to be lagging behind scientific
achievements, increasingly unable even to understand its language. The
language gap is based on the well-known intimidating appeal of STEM
programs and the unreasonable timidity of scientific degree courses to include
communication skills as mandatory. An open course was experimented to
provide students with the necessary tools to decode and encode the essential
elements of science communication, regardless of disciplinary content. The
course, which was designed following the 5W+H "golden rule" of non-
academic science communication, has been offered for three consecutive years
to nearly one hundred students. The students' perspective is analyzed in terms
of their perception of the study effort, the achievement of their own learning
expectations and the level of general satisfaction. The main result is that it is
possible to involve undergraduates, graduates and professionals in rethinking
their approach to communication of scientific content to the general public,
with rewarding results for all players, once the focus is on the greatest common
divisor instead of the lowest common multiple.
Keywords: Science communication; soft skills; open course.

The essentials of science communication in a course engaging both for students and professionals
1. Introduction
On November 27th, 2017, in the Sala Capitolare of the Senate of the Italian Republic, Pietro
Greco gave a speech on the correlation between science and democracy, asserting that
“scientific knowledge is no longer one of the many elements that nourish culture and reshape
the individual and collective life of women and men. But it has been, for some time now, the
main engine of the production system and of the social dynamics of the planet itself.”, (Greco,
2018). The guiding idea of this founder of the science communication in Italy is that a society
in which science is the primus movens becomes a knowledge society and the rights of
citizenship must include the right to scientific citizenship.
Shortly afterwards, the SARS-Cov-2 epidemics demonstrated the importance of scientists
having the necessary skills to communicate clearly with colleagues outside their discipline,
with experts in nonscientific disciplines, as well as with the general public (Sandal, 2022).
However, scientists traditionally have not been provided communication training in their
education and the effective communication beyond their discipline tends to be a rare attribute
(Stevens 2019). Science communication courses in Italy are mostly one- or two-year master's
courses, now offered by many universities and scientific institutions; or they are short
curricular courses in many degrees in "Communication Sciences"; or are offered as short
courses accessible to PhD and graduate students (Trench, 2021). Despite the calls for science
communication to be included in STEM education (Baram-Sabari, 2015; Dahm, 2019), and
although training activity in various aspects of science communication have been published
(Mannino, 2021), a discipline-free approach to developing an essential communication skill
toolbox, like those elaborated by the QUEST EU funded project (QUEST, 2021), has not
been integrated into most STEM curricula in Italian universities (Davies, 2021). Moving from
these considerations, the authors devised a path to provide students with the necessary tools
to decode and encode the essential elements of science communication, regardless of the
disciplinary contents.
The title of the award-winning novel by Paolo Cognetti, The eight mountains, (Cognetti,
2018) originates from an ancient Nepalese myth telling that at the center of the world there
is a very high mountain, Mount Sumeru, and around it there are eight seas and eight
mountains. The meaning of the legend is in the question: who will have learned more, those
who have gone around the eight mountains, or those who have reached the top of Mount
Sumeru?
In the world of disciplinary teachings increasingly isolated from each other; in the world of
countless sources of "free" (dis)information and global interconnections, a similar question
arises when thinking of a valuable modality to introduce students to science communication.
Without pretending to answer the question of the Nepalese legend, leveraging on their
professional experiences, the authors devised a course in Communication of Science,
826
informed around few general principles and that also provides the essential toolbox for social
media communication. The course has been running online synchronous since 2020 and has
been delivered until now to about one hundred students and professionals with very different
background knowledge and communication skills. The content and format of the course has
been consistently refined based on participant feedback gathered at the end of each of the
three editions proposed so far and on the assessment of the communication skills acquired.
In this paper, the content of the course in its latest version is illustrated in detail, also
providing the reasons for the didactic choices. The effectiveness of the proposal is analyzed
using a few parameters that consider the point of view of the attending students: Perceived
Study Effort - PSE, Achievement of Learning Expectations - ALE and Level of General
Satisfaction - LGS. The cross-reference of these parameters with the gender, knowledge base
and motivation of the participants allowed to draw a synthetic and yet detailed picture of the
success of the methodology adopted and of the proposed teaching contents. This ‘subjective’
students’ perspective complements the more ‘objective’ assessment of the learning outcomes
by homework assignments, individual and team presentations and the final production of
social media content for the event LumineScienza, yearly registered in the program of the
International Day of Light®, promoted by the United Nations Educational, Scientific and
Cultural Organization - UNESCO (Lucente, 2023).
2. The Narrative Structure of Communication of Science Course

Quite common in the communication realm is the 5W + H rule, and science communication
directed at the general public is no different. The course content was designed in modules
according to this general rule and the following definitions:
• How module: do it and let them know, communication strategies, technical skills of
image creation and video making, the unspoken words.
• What module: identify the core message, be sure to know it and take your time to
study it. Put in evidence, be concise, be responsible, fact checking, sniffing fakes.
• When module: it is not that any time is the right time, planning and delivering
strategies, one more event?
• Where module: different public <-> different media <-> different tools, identify the
best medium for delivering your message, basics on social media platforms.
• Who module: a different public needs a different language, follow the audience
while shaping it, going out while looking in.
• Why module: think before say. Another drop in the ocean, is there any reason for
your message to be relevant, good reasons and bad results.
827
The subject of the course is organized in six modules, each consisting of four hours of lectures
and four hours of individual or teamwork introduced by a renowned testimonial, as a 'living
example' of how to do well. After a few permutations, the modules are now ordered as follow:
Why – Who – What – Where – When – How, as detailed by the course curriculum in Table
1.
The format balances one-to-many lectures and Q&A session with testimonials, individual
and team homework, intermixed with presentations and open discussions about the shaping
of LumineScienza. A detailed discussion about choosing that specific ordering can be found
in (Lucente, 2023); to the purpose of this communication is it worth to note that the first three
modules set the grounds for the following three and help in building a common language and
a shared horizon for the quite heterogenous group of participants, whose distribution in terms
of gender, basic knowledge and motivation of the enrolled students in the three editions of
the course, is given in Table 2. Enrolment is open to students from the host university as well
as to people from other academic institutions and to anyone with a high school qualification.
Approximately 42% of participants were not students at the host university and most of them
declared professional interest in choosing the course.
On the one hand, the distribution by gender closely matches that of the student population of
the hosting university, that represents 58% of the enrolled students to Communication of
Science. On the other hand, the very same title of the course preferably attracted students
from the PE group of disciplines (70%) with respect to the SH group (11.7%), in contrast to
student population of the hosting institution, about 60% / 20% in the SH / PE degree courses.
3. The Students’ Perspectives

During enrolment, students were asked for the main motivation for choosing the course. Table
2 shows that their response is evenly distributed among the three options, with about one third
motivated by practical reasons, for example to complete the number of eligible credits needed
in their degree course; personal and professional interest equally share the remaining part.
At the end of the teaching activity, students were asked to fill a feedback form consisting of
three sections. The first section is meant to assess their appreciation of the course content. LS
students appreciated the What, Whe’s modules more than average. SH students found less than
average interest in the How module while appreciated most the Why and Whe’s. PE students
stay on average for all modules. However, all differences are around one standard deviation,
indicating that students from all background knowledge find stimulating activity that help them
to decode their "natural communicative inclination" within a methodological framework and to
appropriately codify the less mature aspects of their respective communication strategies. The
only exception is for those who declared Professional motivation in attending the course, whose
828
average appreciation of all modules is one standard deviation below average. Professionals may
have found too basic some content, especially of the Why and Whe’s modules.
Table 1. Course curriculum of Communication of Science.
Topics Actions Outcomes
Get to know Self-introduction of teachers and students; description Respect of the time
2h of methodology and evaluation criteria; outline of the assigned - Technical
course content; graphical illustration of the class issues sorted out – First
composition; Q&A. time on video
Why module Spoken and unspoken communication; basic principles Elaboration of the
8h of communication strategies; visual identity; coordinate personal pay-off – Tips
image; comunication campaign; examples; case history; for reading behind the
creativity is not without boundaries; listening of surface – Main mistakes
professional communicators. to be avoided
Who module In the land of lies, truth is a disease (G. Rodari); know Short sensory experience
8h the subject, the context, the public, the goal you want to on perceptions – team
achieve; know yourself; the narrative techniques: working – brain storming
objects, slides, posters; verbal, paraverbal and non- – focus groups.
verbal language; two RSVP working methods: Respect,
Stimulate, Valuate and Present.
What module What is scientific?; the reliable and unreliable source; Serching the databases –
8h read and write a scientific text; the jargon; what is the Google trends – cut a
red thread with society; tell the uncertainties of science; long story short – the
errors, inaccuracies, oversights; the inner balancing of 5W+H golden rule
scientific method and also the dark side of science.
Where module Attention is a rare commodity; content marketing Main features of popular
4h strategies; language experimentations; the publics of social media – content
social media. creations for Insta and fb
When module Social media manager: basic principles, tips and tricks; Schedule delivering
4h the editorial plan; copy it; stay away from; monitoring media content –
in real time; post analysis. benchmark analysis
How module Framing; basic principle and social media adaptation; Personal video: 3’ –
8h objects, people, background; screenplay; timing; editing. video editing software
Am I sure? The only certainty of science is uncertainty; the rigour Question and Answer
2h of the method instead of the method of rigour; one game – graphical
question has many answers; unbias yourself. abstracts – hidden CV
Let’s face it What do we think about the course; what do you think; Self-evalaution – identify
2h feedback priorities
829
Table 2. Distribution of the total population of students over three years.
Gender Basic Knowledge Motivation

Life Physical & Social &
Male Female Practical Personal Professional
Science Engineering Humanities
36.7% 63.3% 18.3% 70% 11.7% 36.7% 31.6% 31.7%
100% 100% 100%

Basic Knowledge is classified according to the European Research Council index. Practical motivation refers to
those whose attendance was instrumental to completing their eligible ETCS; Personal motivation refers to those
who chose to attend because of personal interest; idem for Professional. 100% = 60 compiled feedback forms.
The second section of the feedback module is meant to assess to what extent students achieved
their own learning expectations (ALE), their level of general satisfaction (LGS) regarding the
whole experience and the perceived study effort (PSE), compared to their initial outlooks. The
responses were collected over a five-grade scale and translated in integers from –2 to +2: +
indicating fulfillment and satisfaction, – indicating disappointment and discontent. The
averaged response is reported in Table 3 together with the distribution across the gender,
knowledge and motivation categories. Almost all students are aligned in deeming the learning
objectives substantially achieved, with the SH group being more satisfied than the average,
positively correlated to a PSE significantly higher than the average. The reported LGS is
positive for all, with the notable exception of the LS group, probably because all teachers have
PE or SH background and are somewhat biased in the examples and analogies proposed. An
essential and specific feature of Communication of Science is the direct and regular
involvement of students in the production of media content (video, poster, text, tweet,
slideshow) working in small groups, along with the creation and management of the web
event LumineScienza at the end of the course (on fb and Insta). The content realized and
posted for LumineScienza (graphics, videos, texts, ads) is evaluated by the teachers and
contributes 50% to the final grade. The groups are formed by teachers to be as much
heterogeneous as possible. A great deal of time is devoted to team building: at least two
sessions each year are dedicated to introducing the methodologies of brainstorming and focus
group. Students are constantly invited to step out the comfort zone and face new challenges.
The third section of the final feedback form is meant to assess how students would qualify their
experience in the acquisition of new technical skills, teamwork, work planning and in
identifying their own biases and preconceived opinions. Instead of using a numerical index,
options were proposed using an "emotional" scale from disappointing to exciting. At first
glance, Figure 1 shows an overall "positive" evaluation highlighted by the dominance of the
bluish area in all rows, corresponding to a Surprising and Exciting evaluation of the activity.
Τhe categorized analysis is then focused on the other side of the colored spectrum (green, red
and orange), considering only values exceeding the corresponding Global averages by more
than one standard deviation. Gender category does not show significant difference, except for
830
Males that found Team Work slightly more Tiring than average (25% against 20.7(2.6)%). 20%
of the SH group found Team Work Pointless and Work Planning Tiring or even Disappointing,
as compared to averages around 3.5(5)%. This is consistent with the below-average level of
LGS (see Table 3) expressed by the same group and compares negatively with the above-
average values of PSE and ALE. The apparent inconsistency finds a possible explanation in the
fact that teamwork and the use of planning tools are widespread practices in the disciplines
belonging to the SH group. A similar picture emerges for the Professionally motivated category,
that also generally appreciated less than average the content of some modules. Work planning
was Tiring for 35(8.2)% of all students, distributed in all categories, with peaks of 45% for
Practically motivated, a category representing students mainly interested to securing the ECTS
in their career. It is worth pointing out that Team Work, which is based on comparison and
collaboration among different points of view, allows everyone to "look in the mirror", in front
of their own preconceptions (Bias Identification), and this experience was unanimously
reported either as Surprising or Exciting. It is always difficult to recognize and accept your own
cognitive biases, so it is encouraging to note that this was the most valued skill across all
categories. Professionals, PE and SH students stand out in well above average surprise and
excitement, surprise and hype shared by the authors as well.
Table 3. Student perspective of the learning outcome.
Student Gender Basic Knowledge Motivation Average

perception M F LS PE SH PRAC PERS PROF (St.Dev.)
ALE 0.9 1.0 0.9 1.0 1.4 1.1 0.8 1.0 1.0 (0.2)
LGS 1.2 1.3 0.7 1.5 0.9 1.1 1.3 1.4 1.3 (0.3)
PSE -0.3 -0.4 0 -0.5 1.5 0 -0.4 -0.5 -0.3 (0.7)
ALE: Achievement of Learning Expectation; LGS: Level of General Satisfaction; PSE: Perceived Study Effort.
LS: Life Science; PE: Physics & Engineering; SH: Social & Humanities. PRAC: Practical; PERS: Personal;
PROF: Professional motivations, as defined in the caption of Table 2.
831
Figure 1. Synthetic graphical representation of the students’ appreciation of being engaged in laboratory activity.
Options are expressed on the ‘emotional’ scale illustrated by the color code. Relative percentage is proportional
to colored area. The area of each colored square corresponds to 100%.
4. Conclusions
Climbing Mount Sumeru on the straight track, ascending it along gentle spiraling paths or wandering
the eight mountains is a matter of personal choice. However, scientific literacy cannot be guaranteed
until all scientists have learned the basic skills necessary to communicate with clarity and empathy.
Communication of Science claims to provide such basic skills in a format compatible with any
academic degree program. The syllabus focuses on common principles with respect to specialized
languages and communication strategies. The greatest common divisor of the communication of
scientific contents of any discipline to a non-specialized public on social media is preferred to the
least common multiple of STEM languages and disciplinary communication.
Communication of Science is funded by UniBa, call Competenze Trasversali. This work is
supported by PANDORA (Horizon Europe Seed - UniBa) to foster scientific citizenship.
References
Baram‐Tsabari, A., Osborne, J. (2015). Bridging science education and science
communication research. J. Res. Sci. Teach., 52, 135–144. doi: 10.1002/tea.21202.
Cognetti, P. (2018). The eight mountains. London: Vintage.
Dahm, R., Byrne, J. &Wride, M.A. (2019). Interdisciplinary communication needs to become a core
scientific skill. BioEssays News Rev.Mol.Cell.Dev.Biol.,41:e1900101. doi: 10.1002/bies.201900101.
832
Davies, S.R., Franks, S., Roche, J., Schmidt, A. L., Wells, R. & Zollo, F. (2021). The landscape
of European science communication. JSCOM, 20 (03), A01. doi: 10.22323/2.20030201.
Greco, P. (2018). Rivista online della Biblioteca "Giovanni Spadolini". Speciale: Scienza e
umanesimo. Intervento di Pietro Greco. Minervaweb n. 44 (Nuova Serie), aprile 2018.
Verified 24.4.23: https://www.senato.it/4800?newsletter_item=1933&newsletter_numero=186.
Lucente, S., Liuzzi, F., Trotta, M. & Dabbicco, M. (2023). Do 5W+H commute? Examples in
Calculus and Communication of Science. 4th Higher Education Learning Methodologies
and Technologies Online. HELMeTO 2022. Communications in Computer and Information
Science, vol ##. Springer, Cham. doi: 10.1007/978-3-031-29800-4_42.
Mannino, I. et al. (2021). Supporting quality in science communication: insights from the
QUEST project. J. Science Commun. 20(3), 1-22, A07. doi: 10.22323/2.20030207.
QUEST (2021). QUality and Effectiveness in Science and Technology communication.
Verified 24.4. 23: https://questproject.eu/toolkits/
Sandal, M. (2022). Come sta la comunicazione della scienza in Italia?. Il Tascabile. Verified
24.4.23: https://www.iltascabile.com/scienze/comunicazione-scienza-italia-pandemia/
Stevens, S., Mills, R. & Kuchel, L. (2019). Teaching communication in general science
degrees: highly valued but missing the mark. Assessment & Evaluation in Higher
Education, 44(8), 1163-1176. doi: 10.1080/02602938.2019.1578861.
Trench, B. & Bucchi, M. (2021). Global spread of science communication: institutions and
practices across continents. In Routledge Handbook of Public Communication of Science
and Technology. Routeledge. doi:10.4324/9781003039242-6.
833
Teachers of natural sciences: some challenges and perspectives in

Brazil
Cristina Leite
University of São Paulo, Brazil.
Abstract
The teaching of Natural Sciences has a fundamental role in contemporary
education. However, the traditional emphasis on specific content unrelated to
the context of the student has made it difficult to understand and act in an
increasingly complex and interconnected world. It is fundamental and urgent
to incorporate an interdisciplinary and integrated perspective in teacher
education. This work brings to light the challenge of the integration of areas
in the natural sciences from a contemporary perspective. In addition, it
presents some elements results of the construction of a proposal for teacher
education for Brazilian basic education.
Keywords: Teaching of natural sciences; interdisciplinarity; teacher
education.

Teachers of natural sciences: some challenges and perspectives in Brazil
1. Introduction
The role and importance played by science and technology in the contemporary world is
unquestionable. They are found in practically everything around us and have a strong
influence on the way we think, act and live. To deny students basic knowledge such as:
telecommunications, astronomical discoveries, energy, genome, transport systems,
properties and transformations of materials, climate change, transgenics, among others, is to
deprive this group of a sense of belonging to a society that is highly dependent on this
knowledge.
However, science has for many years been traditionally taught by emphasising memorisation
and the application of formulas that mean little to basic education students. A strong
emphasis on long lists of specific content, without links to the student's context, often
prevents the necessary depth of understanding and acting in a world that is complex, in which
the variables are many and interconnected. The lack of meaning in the learning of these
immense lists of contents, whose only goal is the achievement of tests and exams, has been
alerted by different teachers, managers and researchers in the area of teaching of natural
sciences.
It is essential to give the teaching of Natural Sciences a meaning more in line with the
potential that this area has in human development throughout history. To this end, it is
essential to incorporate a change of focus: the specific contents are important and essential,
but they need to be at the service of something more comprehensive. The contents, therefore,
should be seen as means and not as ends of the teaching of Natural Sciences.
Several researchers and educators have been warning for some time about a crisis of meaning
in the teaching of natural sciences. If, in the past, the inclusion of scientific themes in basic
education had as assumption the learning only of scientific contents distant from the students'
reality, today this perspective does not make sense anymore, especially after the invention
and dissemination of the internet in which specific knowledge is available at any time
(FOUREZ, 2003).
2. Urgent action on Brazil

In Brazil, the debate has been urgent due to a major reform in basic education (BRASIL,
2017) and the implementation of a Common National Curricular Base (BRASIL, 2018) in
which the knowledge of the natural sciences is guided by interdisciplinarity and the
integration of areas. The primacy of interdisciplinarity and its correlates is already found in
the original Parecer CNE-CEB 7, 2010, p.29: “In the organization and management of the
curriculum, the disciplinary, multidisciplinary, interdisciplinary and transdisciplinary
approaches require the careful attention of the school institution, because they reveal the
836
Cristina Leite
worldview that guides the pedagogical practices of educators and organize the student's
work.”
The area of knowledge Nature Sciences lists skills and competences to be developed
throughout Basic Education in the thematic axes: Matter and Energy, Life and Evolution and
Earth and Universe. Thus, contents that were previously seen in a fragmented way now need
a more integrated look, so that the conceptual and methodological tools of the curricular
components enable students to investigate, analyse and discuss problem situations that
emerge from different socio-cultural contexts, as well as understand and interpret laws,
theories and models, applying them to solve various problems of individual, social and
environmental nature.
Faced with the challenge of teachers education for a world in deep transformation, the
University of São Paulo, especially the Chair of Basic Education, linked to the Institute of
Advanced Studies (CEB-IEA/USP) and in partnership with some university teachers, has led
the debate and preparation of a project for an Interdisciplinary Science Degree, whose aim
is, among others, "[. ...] to educate teachers to work in basic school, implying a creative,
systematic and responsible inter-transdisciplinary approach to school contents, providing a
real transversality in their education". (PPC, 2022, p.20). The participants are divided into
students in the initial education process and teachers working in basic education,
concomitantly and shared.
The idea of Inter-Transdisciplinarity can be understood in three different aspects: as an
approach or methodology that facilitates the access to complex objects; as an action that
overcomes or transits between distinct disciplinary fields; and as a education for or in
teaching-learning processes between distinct disciplinary fields (PPC, 2022).
In order to structure the learning spaces, the degree is divided into 4 large educations blocks:
General Transdisciplinary Education; Transversal Pedagogical Education; Autonomous
Project-based Education; Specific Interdisciplinary Education. The first three are common to
other qualifications and not exclusive to the area of natural sciences, thus allowing for great
integration in the education of teachers from different fields of school knowledge.
3. The sensitive case of natural science

The proposal for integrated teaching of Physics, Chemistry and Biology requires greater
mastery of the three areas of knowledge by the teacher, as well as greater in-depth knowledge
of astronomy and geosciences. This fact represents a major change in terms of science
education in Brazil, leading to innovations also in the teachers’ education for basic
education.’
837
The curriculum integration proposed for the Nature Sciences in the BNCC seems to relate to
the searches for function and meaning of school knowledge highlighted by Lenoir (2006)
when analyzing the differences between visions of interdisciplinarity of different approaches
and regions of the planet. The competences indicated in the document for nature sciences
present concerns both in epistemological and social plans, as well as focus on the meaning
of school knowledge, in the more practical and operational plan, by focusing its propositions
in empirical social and environmental issues, and in processes and practices of scientific
research, thus presenting concerns both in the function and in the meaning of school
knowledge.
The integration of the areas of natural sciences faces a number of challenges ranging from
methodological differences and language barriers to conflicts of interest, the resistance of
teachers to work collaboratively, the lack of time and resources for integrated projects, the
need for new forms of assessment and structural difficulties.
One of the main obstacles to the implementation of proposals that integrate knowledge and
the development of interdisciplinary proposals in basic education is the initial teachers’
education, usually based on a fragmented view of knowledge and the absence of'
interdisciplinary teaching materials (SILVA; MAGALHÃES, 2016).
Given so many obstacles involved both in the integration of the areas of natural sciences in
basic education, as in the development of integrative proposals, the interest of this research
focuses on the impact on the curricular organization of teacher education brought by the
changes made in the area of Natural Sciences and their Technologies, by its unprecedented
character, requiring a differentiated profile of the teacher of biology, physics, and chemistry,
in accordance with an integrative and interdisciplinary discussion.
Undergraduate courses of interdisciplinary teacher education, although necessary, are still
experienced in the early stages of construction and implementation in Brazil (GATTI et al,
2022). It is important to highlight that many of the existing degrees in Nature Sciences in the
country, due to the tradition of fragmented disciplines, have difficulties in overcoming the
content-based and disciplinary model. They end up reinforcing in their curricula the junction,
the sum of physical, chemical and biological knowledge and not a real articulation among
them.
One of the main criticisms that is made to the teaching of natural sciences is its
compartmentalised, watertight and poorly articulated teaching. The lack of articulation in the
undergraduate courses makes the challenge for natural sciences teachers almost
insurmountable. Although the municipal and state education departments try to overcome the
problem by offering some continuing education courses, the degree courses are still far from
interdisciplinarity.
838
Cristina Leite
3.1. Building an undergraduate course focused on teacher education in an

interdisciplinary perspective in Science
In an attempt to promote an expansion of the relations between the scientific knowledge
mobilized, the proposal of the new Interdisciplinary Bachelor's Degree in Sciences (USP)
carries out some movements: at the same time that it develops a panoramic vision of the
scientific culture about our knowledge of the Universe, for example, it delves into its various
phases and areas, with the objective of articulating the whole and the part. The introductory
curricular component, to be taught by teachers with knowledge in different fields of
knowledge (astronomy, physics, chemistry, biology, geoscience, education, philosophy and
engineering) has the purpose of providing an overview of this knowledge that will be more
and better deepened throughout the course. The idea is to investigate and debate on the
knowledge of the Universe produced by the natural sciences over time and the main
technologies and tools important to this construction.
Figure 1. Science, Technology and the construction of knowledge.
The education in natural sciences aims to, while working on the fundamental concepts
proposed by the BNCC (BRASIL, 2018): Matter, Energy; Life and Evolution; and Earth and
Universe, structure and organize them from major themes that interconnect these topics, as
well as by fundamental aspects and themes in science, such as: Patterns and transformations
in nature; Observations, modeling and simulations in the construction of knowledge in
Natural Sciences; Energy and the challenges of the 21st century; Matter that forms us and
surrounds us and its impacts on modern life; Life, health and environment; Light, radiations,
their impacts and applications.
Furthermore, the purpose of this course is to situate this knowledge in time and space,
providing reflections and debates involving the nature of scientific knowledge, its
839
investigative processes and practices, its relationship with technology, with history and with
culture and its relationship with educational spaces, especially the school.
In order to give life to innovative proposals in basic education, it is also part of the objectives
of the new Interdisciplinary Bachelor's Degree in Sciences (USP) to structure supervised
internships through the development of inter/transdisciplinary projects in partner public
schools.
The proposal of the new Interdisciplinary Bachelor's Degree in Sciences (USP) dialogues
with the concept of transversality introduced in the Brazilian educational regulatory
framework (CNE/CEB, 2010, p. 24): "transversality guides to the need to institute, in
educational practice, an analogy between learning theoretically systematized knowledge
(learning about reality) and real life issues (learning in reality and from reality)." And has
as reference the idea that to teach in an interdisciplinary way, the mere juxtaposition of
content is insufficient. The necessary change consists of a change in the understanding and
attitude of the future teacher (LENOIR, 2006; FAZENDA, 1995), and will only be possible
with the overcoming, at least in part, of an exclusively disciplinary curriculum structure.
The work described here fits into the research model called Experiences Report (RE) and
tries to seek to "reach subjects, events and temporalities, the Report imbues other processes
and knowledge, linked to modalities of scientific constructions more apt to recognize the
importance of the use of narrative skills" (DALTO and FARIA, 2019).
4. Some considerations
It is essential to highlight that we are immersed in a very dynamic world full of concrete
challenges that urge us at all times. Living responsibly and actively requires a certain degree
of scientific literacy and, therefore, the mastery of a series of competences and skills in which
inter-transdisciplinarity is very present. Real-life problem situations are not, in general,
specific to one component. From this perspective, the Natural Sciences proposed for the new
Interdisciplinary Bachelor's Degree in Sciences (USP) intends to educate teachers who can
meet the current educational demands, starting from real problem situations in which the
knowledge of natural sciences is mobilized, in order to build a more just and equitable
society.
Acknowledgments
The Chair of Basic Education, for the opportunity of collaborative work.
To the Working Group involved in the construction of the Nature Science Proposal for
Interdisciplinary Bachelor's Degree in Sciences (USP).
840
Cristina Leite
To Fapesp, for the financial support.
References
Brasil, 2017. Ministry of Education. Law No. 13.415/2017.
Brasil, 2018. Ministry of Education. Base Nacional Comum Curricular.
Dalto, M.R. Faria, A.A., 2019. Experience report: A scientific narrative in the post-
modernity. Estudos e Pesquisas em Psicologia. State University of Rio de Janeiro.
Fourez, 2003. Crisis in Science Teaching? Investigations in Science Teaching - V8(2), pp.
109-123.
Gatti, B. Guimaraes, L. Puig, D. 2022. A cartography in teacher education for basic
education: innovative practices and solutions in curriculum proposals. São Paulo:
CEB/IEA. University of São Paulo.
PPC, 2022. Pedagogical Course Project: Interdisciplinary Science Degree. São Paulo:
CEB/IEA. Institute of Advanced Studies, University of São Paulo.
Silva, Magalhaes, 2016. Importancia da interdisciplinaridade na área de ciencias da natureza
no ensino médio. Revasf, 2016.
Lenoir, Y. de. Three interpretations of the interdisciplinary perspective in education
according to three distinct cultural traditions (2006).
Fazenda, I. C. A. Critical-historical review of interdisciplinary studies in Brazil. Association
for Integrative Studies Newsletter, v. 17, no 1, 1995, p. 1-9.
841
Designing co-curriculum experiential learning practice in

permaculture for studying science classics
Ming Li
Office of University General Education, The Chinese University of Hong Kong, Hong Kong.
Abstract
General education (GE) plays a critical role in university education. It
embraces a broad intellectual pursuit through the study of and reflection on
liberal arts, science knowledge, and real-world situations. The inspiring yet
challenging common-core GE foundation course UGFN1000 In Dialogue with
Nature, which requires students to study science classics, was launched at
CUHK in 2012. To enhance students’ learning of the selected science classics,
such as Origin of Species and Silent Spring, co-curriculum experiential
learning practice in permaculture was designed and employed. This paper
illustrates the design of the pre-session preparation, practice sessions, and
post-session reflection. It provides a practical and insightful reference to
educators interested in developing experiential learning through permaculture
in GE and beyond.
Keywords: Experiential learning; general education; in dialogue with nature;
science classics; permaculture.

Designing co-curriculum experiential learning practice in permaculture for studying science classics
1. Introduction
General education (GE) has been a key component of higher education in the 21st Century. It
plays a critical role in nurturing students to meet the challenges of the ever-changing world
and preparing them to be engaged citizens with global awareness (Office of University
General Education, 2013). Taking the opportunity of the educational structure reform in
Hong Kong in 2012 (Wang & Gano-Phillips, 2017), a common-core GE Foundation
Programme was developed and integrated into the new four-year undergraduate curriculum
at The Chinese University of Hong Kong (CUHK). Each year, the science classic-reading
course UGFN1000 In Dialogue with Nature in the GE Foundation Programme requires 3,600
junior undergraduates to reflect on the intellectual pursuit of nature and science through the
study and discussion of selected excerpts of science classics, such as Charles Darwin’s Origin
of Species, James Watson’s DNA: The Secret of Life, Rachel Carson’s Silent Spring, and
Joseph Needham’s The Shorter Science and Civilization in China.
Previous findings suggested that studying science classics as a common-core requirement is
very challenging for students (Hoi, Wong & Pang, 2017; Kiang, Ng & Cheung, 2015; Lau,
Li & Liao, 2018). Furthermore, without concrete experience of the contexts of the science
classics and direct exposure to the related real-life situations, students may find it difficult to
fully appreciate the ideas, problems, and dilemmas in the classics and, therefore, their
relevance to the students’ lives and contemporary situations. To enhance students’ learning
in this science classics-reading GE course, experiential learning has been employed to bridge
the book knowledge and real-life situations and to foster reflection. (Li, 2018) Experiential
learning is a transforming process whereby knowledge is created from experience (Kolb,
1984). In Kolb’s experiential learning theory, an experiential-learning cycle divides the
learning process into four stages, including concrete experience, reflective observation,
abstract conceptualization, and active experimentation (Kolb, Boyatzis, & Mainemelis,
2001). Most students agreed that experiential learning through farming helped them
understand the course materials and fostered reflection in the GE course (Li, 2018). The
practice can foster a sense of relevance and immediacy, thereby enhancing understanding of
dilemmas discussed in the classics and fostering deep reflection on the challenges and ethics
of applying science in real-life situations.
While it is largely considered that farming is the foundation of sedentary civilizations, it has
brought global concerns regarding food safety, environmental pollution, biotechnology
ethics, and sustainability (Food and Agriculture Organization, 2017). The development of
sustainable agriculture is of paramount importance to alleviate the pressing of these global
challenges. Among various agriculture models, permaculture is a philosophy of living with
nature and a practical design to maintain an agriculturally productive ecosystem that
preserves biodiversity, stability, flexibility, and sustainability (Holmgren, 2002; Mollison &
Holmgren, 1978). It has three tenets (care for the Earth, care for people, and fair share), which
844
Ming Li
can be achieved by actualizing eleven design principles (Mollison, 2013). Since the
philosophy, principles, and practical designs of permaculture echoes with many core issues
stressed in the science classics, permaculture is deemed suitable to become a core theme in
the development of the co-curriculum experiential learning practice to enhance students’
learning this science classic-reading GE course.
For instance, permaculture’s tenet “care for the earth” aligns with Carson’s viewpoint on the
human-nature relationship, in particular, the use of pesticides. Back in the 1960s, Carson had
a detailed discussion in Silent Spring that the “web of life” is highly complicated. There are
times that we inevitably intervene in the relationship, but we should be keenly aware that our
acts will bring consequences to other places in the world and in the distant future (Carson,
1962). Today, the global use of insecticides has increased nearly one-fold from 2.3 million
tons in 1990 to more than 4.1 million tons in 2019 (Food and Agriculture Organization 2022).
The problem of excessive use of insecticides and herbicides weakens the insect biodiversity
worldwide and results in an increase of super-pests. Aligning with Carson’s desire for
biological control, permaculture aims at reducing or eliminating the external input for
maintaining or restoring the natural systems. The permaculture principle of “using biological
resources” guides farmers to adopt biological insect control instead of pesticide (Mollison,
2013). In our experiential learning practice, we did not apply any pesticides. Instead, we
planned tropical milkweed (Asclepias curassavica) to attract ladybugs to control the
population of aphids in the rooftop garden. Students can therefore have a better understanding
of the ideas in Silent Spring and foster reflection by practicing this permaculture principle.
More examples include the connection between the permaculture principle of “accelerating
succession and evolution” and natural selection and man’s selection described in Charles
Darwin’s Origin of Specie, as well as the permaculture principle of “relative position” and
the natural pattern of yin-yang theory in Joseph Needham’s The Shorter Science and
Civilization in China.
2. Objective
Given that permaculture emphasizes the coherent understanding of the natural systems and
values the harmonious coexistence of human society and the land community, which is highly
relevant to the broad coverage of the science classics in the GE course, it was deemed an
appropriate subject for the experiential learning practices in the science classics-reading GE
course. Therefore, co-curriculum practices in permaculture have been designed and
employed to enhance the understanding of the selected science classics. This paper aims to
illustrate the designs of experiential learning sessions for educators interested in developing
similar experiential learning practices.
845
3. Design of Practice Sessions

The purpose of the practice sessions is to enhance students’ learning in the science classic-
reading GE course through the experience of permaculture practice in real-life situations.
Students could choose this experiential learning practice as an optional assessment
component which contributed to a total of 3% of students’ course grades. A rooftop garden
at the Hui Yeung Shing Building at CUHK had been established as the site of practice. The
experiential learning practice was designed with three parts: pre-session preparation,
practical sessions, and post-session reflections (Figure 1).
Figure 1 Schematic diagram of the design of the practice sessions.
3.1. Pre-session Preparation

Engaging students before the experiential learning sessions was essential for effective
learning. The pre-session preparation aimed at equipping students with pre-requisite
knowledge and skills, fostering commitment, and building communication among students
in groups before the practical sessions. Blended learning was employed here using two e-
learning instruments: (i) the “Experiencing Permaculture” micro-module and (ii) the tailor-
designed farming companion mobile App “Growing Edibles 101” (GE101).
3.1.1. Micro-module “Experiencing Permaculture”

The permaculture micro-module was comprised of three short introductory videos about
sustainable agriculture, permaculture principles, and designs (Experiencing Classics
Website, 2022). The first episode “Agriculture and Sustainable Development” introduced the
development of agriculture in Hong Kong and the world, the relationship between
agriculture, classic texts, and social issues, and the definition of permaculture. The second
episode “Permaculture Principles” and third episode “Permaculture Design” explained the
permaculture principles and practice, respectively, which helped students glimpse the
philosophy of permaculture. By watching these micro-module videos, students could gain
pre-requisite knowledge about sustainable agriculture and permaculture before attending the
practice sessions. Students were required to complete a short quiz on the micro-module
before registration for the sessions.
3.1.2. Mobile App “Growing Edibles 101” (GE101)

From our experience, students generally did not understand the relationship between nature
and farming. Insufficient knowledge of farming techniques also presented an obstacle to their
experiential learning. A considerable amount of time had to be spent on explaining basic
846
Ming Li
knowledge and skills in each practice session. Meanwhile, inadequate attention or mere
forgetfulness on the part of the students could lead to poor growth of the crops, resulting in
a disappointing learning experience. Therefore, the mobile App GE101 was developed for
blended learning (Figure 2). The App contained the following features which helped students
learn prior to and during the practice sessions:
• connecting the knowledge, attitude, and value taught in the GE course with farming
practices by asking reflective questions and displaying inspiring famous quotes;
• equipping students with essential knowledge and basic skills through tailor-made
content, illustrations, and demonstration videos;
• facilitating the selection and planting of seasonal crops;
• issuing reminders and confirmations of daily farming tasks for student groups;
• serving as a platform for communication among students in groups and between
teachers and students.
Figure 2 Screen captions of the mobile App GE101. The panels from left to right are the pages on farming
techniques, crop information, group work reminder, and the message board for communication.
3.2. Practice Sessions

The practice sessions were a series of two-hour biweekly or triweekly hands-on practice and
discussion in groups at the rooftop garden over ten weeks. The sessions aimed to provide
hands-on experience in practicing permaculture, revisiting concepts, and fostering discussion
of selected science classics. Each session included an introduction, demonstration of farming
tasks, hands-on practices, group discussion of classics, and the learning experience (Table 1).
Farming practices included soil-mixing, seeding, watering, weeding, pest control,
composting, applying fertilizers, pruning, and harvesting. Apart from participating in the four
practice sessions, students were responsible for taking care of their own crops between the
sessions. It was crucial to carefully select the seasonal crops with appropriate growth periods
(70 days in our case) so that students could harvest and enjoy their crops at the end of the
sessions. If the growth was unsuccessful, it was also an excellent opportunity to reflect on
the farming process and learn from the mistakes.
847
Table 1 Session plan of each practice session.
Session 1 (Day 0)
Introduction • Introducing the aims of the practice sessions, the rationale behind experiential
learning, and the connection between studying science classics and permaculture
• Introducing the management and safety issues at the rooftop garden
Practice • Demonstrating and practicing soil-mixing, seeding, irrigation
Discussion • Discussing the hands-on experience and the value of good farming soil
• Reminding students to bring food leftover for composting in the next session
Session 2 (Day 21)
Introduction • Reading the excerpt from the classics Silent Spring
• Briefing the relationship of the practice to Silent Spring
Practice • Demonstrating and practicing weeding and composting
Discussion • Sharing thoughts or questions on the value of weed
Session 3 (Day 49)
Introduction • Reading excerpts of the classics Silent Spring and Origin of Species
• Briefing the relationship of the practice to Silent Spring and Origin of Species
Practice • Demonstrating and practicing weeding, pest controlling, pruning, and fertilizing
Discussion • Sharing thoughts or questions on the meaning of pest and their relationships with
other plants and animals in the rooftop garden
Session 4 (Day 70)
Introduction • Reading excerpts of the classics DNA: The Secret of Life and The Shorter
Science and Civilization in China.
• Accessing the yield and quality of students’ crops
Practice • Demonstrating and practicing harvesting and seed saving
Discussion • Sharing thoughts or questions on the meaning of seeds and farming cycles
according to patterns in nature.
• Reviewing the whole process of permaculture farming over the ten weeks, such
as the lessons learned, the most impressive moments, and their feelings at
different stages.
• Encouraging students to continue rooftop farming after the practice sessions
3.3. Post-session reflection

Students were required to submit a short post-session reflection essay to consolidate their
reflection after the practice sessions. Shortlisted students’ reflections can be viewed on the
Experiencing Classic Website (2022). A year one medical student has agreed to share her
experience and reflection in this paper:
“The activity was truly one full of insights, in terms of inducing philosophical reflection
on the science classics and our daily lives, especially when the world is in such an
abnormal state. To start with, the activity not only granted me the opportunity to have
hands-on experience in farming but included some sort of reflecting sessions. This linkage
between theory and practice allowed me to have a more in-depth understanding of the
view that Carson had: nature had her way of maintaining a balanced ecology, and it is
more sophisticated that may be beneficial even for humans. Examples would be weeds,
which happen to be ideal indicators for the soil’s condition. Moreover, seeing bees while
848
Ming Li
watering plants was really something special for me as it shows that humans must rely
on nature to yield fruits. Apart from the aforementioned, planting also brought me some
insights into my daily life. Participating in the production of fertilizer from food waste,
eating a freshly harvested leave, putting my fingers deep in the soil, and hugging bushes
are all not part of my daily life, especially the latter ones when the environment is
somehow considered contaminated that from childhood I have been taught not to touch
things when it is not necessary because they are ‘dirty’. It was quite an experience that
now I feel like having broken through a bit compared to myself in the first planting session
– it feels good to be in direct contact with nature, and things that seem dirty, like soil,
may sometimes be even cleaner than ourselves, who are contaminated with artifacts like
plastic as so on. Additionally, eating the juicy radish I harvested really gave me
satisfaction: the love of freshness felt like something I genetically possessed, and the effort
of planting really paid off. My dream to do farming in my old years has been reassured!”
Students were invited to fill in a simple questionnaire and join focus group interviews to give
feedback on the practice sessions. From the questionnaire, about 95% of the students agreed
at different degrees that the practice sessions were helpful in learning the science classics in
various aspects (Table 2).
Table 2 Students’ feedback on the practice sessions of experiential learning through

permaculture (n=44). The figure represents the percentage of each score.
Score 1 2 3 4 5 6
a) Deepen understanding of the science classics 0.0 2.3 2.3 31.8 43.2 20.5
b) Extend the learning beyond the courses 0.0 2.3 0.0 15.9 61.4 20.5
c) Relate science classics with daily lives 0.0 2.3 2.3 20.5 56.8 18.2
d) Enhance motivation for learning 0.0 0.0 0.0 22.7 54.5 22.7
(Six-point Likert scale: 1 strongly disagreed, 6 strongly agreed).
4. Conclusion
Experiential learning through co-curriculum practice sessions in permaculture was designed
and employed to enhance the understanding of science classics in a common-core university
GE course. This paper has illustrated the design of the experiential learning practice, which
consisted of pre-session preparation, practice sessions, and post-session reflection. The
design and practice shared in this paper are helpful for educators interested in developing
similar experiential learning practices in GE and beyond.
Acknowledgments
Gratitude is given to HUANG Xueqing Shirley for her contribution to this paper. This project
was supported by the Teaching Development and Language Enhancement Grant.
849
References
Carson, R. (1962) Silent Spring. Boston: Houghton Mifflin.
Experiencing Classics Website. (2022). Experiencing Permaculture. Retrieved Feb 10, 2023,
from https://expclassics.oge.cuhk.edu.hk/experiencing-permaculture/
Food and Agriculture Organization. (2017). The Future of Food and Agriculture: Trends and
Challenges. Retrieve Feb 10, 2023, from https://www.fao.org/3/i6583e/i6583e.pdf
Food and Agriculture Organization. (2022). FAOSTAT: Pesticides Use. Retrieved Feb 10,
2023, from https://www.fao.org/faostat/en/#data/RP/visualize
Hoi, W. H. S., Wong, W. H., & Pang, K. M. (2017) Confronting students’ science anxiety
through ‘in dialogue with nature’, in The European Conference on Education, California,
CA. https://papers.iafor.org/submission36285/
Holmgren, D. (2002). Permaculture: Principles & Pathways beyond Sustainability. Hepburn,
Vic.: Holmgren Design Services.
Kiang, K., Ng, A., & Cheung. D. (2015) Teaching science to non-science students with
science classics. American Journal of Educational Research, 3(10), 1291-1297. doi:
10.12691/education-3-10-13.
Kolb, D. A. (1984) Experience as the Source of Learning and Development. Upper Saddle
River, NJ: Prentice Hall.
Kolb, D. A., Boyatzis, R., & Mainemelis, C. (2001). Experiential learning theory: previous
research and new directions, in Perspectives on Thinking, Learning, and Cognitive Styles
(pp. 227-248). Routledge.
Lau, K. C., Li, M., & Liao L. (2018) Assessment of students’ views of nature of science in
the context of the discovery of DNA structure, in Proceedings of Multidisciplinary
Academic Conference (Pp.190-200). Czech, Prague.
Li, M. (2018) In dialogue with nature: experiential learning through farming in a university
common core science general education course. Journal of Applied and Physical Sciences.
4(3):101-111. http://www.tafpublications.com/platform/Articles/full-japs4.3.4.php
Mollison, B. C. & Holmgren, D. (1978). Permaculture One: a Perennial Agricultural System
for Human Settlements. London: Transworld Publishers.
Mollison, B., & Slay, R. (2013). Introduction to Permaculture (2nd Edition). Tasmania,
Australia: Tagari Publications.
Office of University General Education. (2013). General Education Foundation Programme.
Retrieved Feb 10, 2023, from
https://www.oge.cuhk.edu.hk/index.php/en/programme/university-general-
education/general-education-foundation
Office of University General Education. (2013). Our Mission. Retrieved Feb 10, 2023, from
https://www.oge.cuhk.edu.hk/index.php/en/overview/mission
Wang, X., & Gano-Phillips, S. (2017). Implementing general education in Hong Kong:
Government policies, institutional responses, opportunities and challenges. Frontiers of
Education in China, 7(2), 253-269.
850
Relationship between time management and class attendance in

university students: clustering techniques for detection of profiles
Santiago Porras Alfonso1, Julio César Puche Regaliza1, Silvia Casado Yusta1, Athénaïs
Sauvée2, Paula Antón Maraña1, Joaquín Antonio Pacheco Bonrostro1
1
Department of Applied Economy, University of Burgos, Spain, 2Department of Private Law,
University of Burgos, Spain.
Abstract
Low class attendance by university students is one of the factors that may be
related to abandonment ratios, which constitute a serious socio-economic
issue. The aim of this work is to confront the influence of the time management
capacity of first-year students with their class attendance. With a factorial
analysis of a survey carried out, four factors emerged: the students' perception
of how they manage their time, the time they spend on less productive tasks,
the ability to finish tasks on time and the use of time management tools.
Moreover, with classification trees it was seen that students who are able to
finish the tasks on time, have a greater capacity for concentration and spend
less time on trivial tasks, have higher class attendance. With these profiles
identified, it is expected to guide them to improve their time management,
increase their class attendance and, as a consequence, decrease dropout rates.
Keywords: Time management; class attendance; factorial analysis;
classification trees.

Relationship between time management and class attendance in university students
1. Introduction
High university dropout rates are a prominent social as well as economic problem. On the
one hand, students change their life goals and on the other hand, the investment made, and
the opportunity cost have a strong impact on the organizations and governments in charge of
university educational management. In Spain, 33% of students do not finish the degree in
which they began their higher education stage, of which 21% drop out and the remaining
12% change their degree, which implies an annual loss of €974 million (Pérez & Aldás,
2019). Undoubtedly, figures are high enough to try to reduce the dropout rate of students.
The reasons for school dropout can be approached from three perspectives (González
Tirados, 1985): (1) from the perspective of the student's characteristics (aptitudes, abilities,
motivation, etc.) (Esteban et al., 2016), (2) from the perspective of the institution (complexity
of the studies, student-teacher relationship, educational methodologies, etc.) (Bartual
Figueras & Poblet Farrés, 2009), and (3) from the perspective of society (family, cultural,
socioeconomic, etc.) (Cervero-Fernández et al., 2017).
Among all these reasons, we pay special attention to those related to the student himself.
Concretely, to the way in which these students manage their time. Other authors have studied
time management by students. For example, Adams & Blair (2019) found a significant
correlation between time management and grades obtained, although there was no difference
according to gender, age or entry qualification. In contrast, Oreopoulos et al., (2022) suggest
that low-touch programs that offer scheduling assistance, encouragement, and reminders for
studying lack the required scope to significantly affect academic outcomes. On the other
hand, Murray et al., (2022) provide methods, resources, and strategies to help students better
manage their time, offering their combined experience in mentoring and training STEMM
students and their time management skills, and Khiat (2022) uses an automated adaptative
time management enabling system to guide students in managing their time more efficiently,
thus impacting in their performance.
There are also studies that analyze the determining factors regarding the degree of attendance
to classes by university students, such as the one carried out by Moores et al. (2019), and
even Sloan et al. (2020) indicate that there are studies in which poor time management is
pointed out, among other factors, as a cause of school absenteeism. Nevertheless, we believe
that the relationship between time management and student attendance in face-to-face classes
has not been sufficiently analyzed due to its economic and social importance as mentioned
above.
So, to deepen this gap, the objective of the work proposes to analyze the relationship between
the management that students make of their time and class attendance in university studies.
The identification of student profiles will make it possible to offer proposals for improvement
in time management and, consequently, we hope that there will be an increase in class
852
Santiago Porras Alfonso et al.
attendance by students. We also hope that this increase in class attendance will have a direct
impact on the reduction of university dropout rates.
To achieve this objective, the rest of the work is divided into 3 further sections. In section 2,
we show the methodology used to develop the study. Section 3 shows the results obtained
and finally, section 4 presents a series of conclusions and possible future lines of research.
2. Methodology
In this section is explained the dataset used, the factorial analysis and classification trees
employed in the study.
2.1 Data Collection

To collect the answers from the students, a survey about time management, based on the
publication by Neill (1996) has been carried out. The questionnaire consists of 28 questions
which answers are represented by a scale that goes from 1 to 8; 1 being the statement “doesn't
describe me at all; it isn't like me” and 8: “this statement describes me very well; it is very
much like me”.
The sample is made up of 135 first-year students of Business Administration and
Management, and Finance and Accounting degrees. This data has been cross-referenced with
face-to-face attendance at the classes of the subject Applied Computer for Business, common
to both degrees in three levels, low for students who attended less than one third of the classes,
medium for the students who attended from one to two thirds, and high for those who attended
more than two thirds. Personal data such as gender or age has not been included in the study
because of privacy limitations.
2.2 Factorial analysis

A factorial analysis of the survey data has been carried out, with a double objective. Firstly,
to identify representative factors of student time management and secondly, to reduce the
dimensions of the data set.
2.3. Cluster analysis

With the factors obtained and normalized, the objectives are, on the one hand, to classify the
students in different groups according to their attendance and, on the other hand, to see how
time management factors influence this classification.
To do so, among the range of existing classification techniques, it has been decided to use
recursive partitioning for classification trees Breiman (2017). The reason is to obtain an easily
interpretable taxonomy by levels that allows dividing the students according to the rules of
the tree based on the values of the factors which allows to see their importance and influence
853
in the classification. This gives the possibility of identifying in which aspects of time
management each student should be guided individually.
3. Results
The results have been obtained using the programming language for statistical computing R,
highlighting the nFactors library to calculate the number of factors for the factor analysis,
stats library for factor analysis and rpart2 library for classification trees.
3.1 Factorial analysis.

Initially, a correlation matrix of the responses to the survey is calculated. In Figure 1 it can
be observed a slight correlation between the survey answers.
Figure 1 Correlation Matrix.
In order to ensure that the factor analysis can be carried out with guarantees, the KMO values
have been calculated with a value of 0.84 greater than 0.5 and Barlett's sphericity test, which
offers a K-squared value of 204.72 and a p-value of < 2.2e-16. This indicates that the factory
analysis can be carried out with guarantees.
Four metrics have been tested to select the number of factors according to Raîche (2013):
Eigenvalues: Eigenvalues with a value higher than one. The acceleration factor indicates
where the elbow appears. The optimal coordinates (OC) correspond to an extrapolation of
854
the preceding eigenvalue by a regression line between the eigenvalue coordinates and the last
eigenvalue coordinates. The parallel analysis criterion compares the eigenvalues of the
sample correlation matrix with the eigenvalues obtained from a random correlation matrix
for which no factors are assumed. According to the results obtained, the number of factors to
be retained is 4.
Analyzing the results of performing a factorial analysis with varimax rotation shown in Table
1, Factor 1 represents the student's perception of his good time management. Factor 2
explains the waste of time. Factor 3 indicates the student's ability to complete tasks on time,
and Factor 4 explains whether the student relies on any tool to keep track of his time.
Regarding to Factor 4 it is true to that only question 12 has the highest value, but we decide
to keep it because other questions has similar values and influence in Factor 4.
3.2 Cluster analysis.

The survey data has been divided randomly in train (110 students) and validation (25
students). The process to train the classification tree is a cross-fold validation, with a fold of
5 and 10 repetitions. The maximum depths tested are 1, 3, 5, 7 and 9. The metric to choose
the best configuration is the accuracy of classification. The results obtained indicate that the
maximum depth obtained is 5 and an accuracy of 36.8% which is low. However, with the
validation set results shown in Table 2, the accuracy is 40%. Observing that it's difficult
distinguishing between students with a high or medium assistance.
The final classification tree can be observed in Figure 2, this taxonomy allows to classify the
students according to their answers to the survey.
The values shown in the leaves of the tree correspond to the probability of each class, high
(left), low (center) and medium (right) value. Analyzing the taxonomy, it can be concluded
that Factor 3 is the most important. That is, whether or not they complete their tasks on time.
Those who do not complete their tasks tend to have a low-medium attendance and those who
do comply to a medium-high attendance.
855
Table 1 Components matrix result of the factorial analysis.
Question Factor 1 Factor 2 Factor 3 Factor 4

01 My life is very well organised. 0.592 -0.171 0.265
02 I manage the way I use my time really well. 0.710 -0.232 0.201 0.103
03 I make effective plans for getting things done. 0.505 -0.150 0.310 0.365
04 I am good at breaking complex tasks down into
0.513 -0.221 0.231
achievable chunks.
05 I use my time effectively. 0.712 -0.287
06 I procrastinate over doing difficult tasks. 0.476 -0.109
07 I accurately predict how long tasks will take. 0.531 0.128 0.185
08 I waste a lot of time. -0.244 0.589 -0.190
09 I am on top of my important tasks at the
0.185 0.376 0.353
moment.
10 I accomplish what needs to be done each day. 0.217 0.509 0.268
11 I do the most important tasks during my most
0.254 0.183 0.237
energetic periods of the day.
12 I prepare a daily or weekly "to do" list. 0.346 -0.125 0.174 0.370
13 I spend a lot of time mucking around. 0.433 -0.333
14 I meet deadlines on time 0.753 0.112
15 I easily get distracted from important tasks -0.117 0.632
16 I get important tasks done on time. 0.202 0.810
17 I find myself procrastinating over tasks that need
-0.163 0.584 -0.423
to be done.
18 I have a weekly schedule on which I record fixed
0.378 -0.107 0.269 0.286
commitments.
19 I spend too much time on trivial matters 0.568 -0.124
20 I always complete tasks before they are due 0.194 0.642
21 Despite interruptions, I get important tasks done. 0.234 0.676
22 I am in control of how my time is spent. 0.617 -0.189 0.178
23 I am satisfied with the way I use my time. 0.708 -0.241
24 I find distractions to be very tempting. -0.112 0.605 0.152
25 I monitor progress towards my goals. 0.582 0.315 0.171
26 I have a hard time concentrating. -0.253 0.656 0.351
27 I am hopeless at time management. -0.382 0.623 0.385
28 I balance work, rest, and play. 0.629 -0.118 0.146 0.136
Table 2 Confusion matrix for validation data.
High Low Medium

High 4 4 3
Low 1 1 2
Medium 4 1 5
856
Figure 2 Final classification tree.
4. Conclusions
Poor class attendance is a problem that affects the university community. With the survey,
four factors that define the temporary management of students have been identified. The
elaboration of the taxonomy yields weak results in terms of precision but has allowed us to
see the relationship between time management factors. It can be concluded from these weak
results that there are other factors that affect attendance, and this cannot be fully explained
by time management. Although, the tree taxonomy allows us to identify student profiles to
orienteering them in order to improve their time management, increase their class attendance
and decrease dropout rates. The study is limited by the protection of personal data of the
students.
As future lines of work, it is intended to expand the survey to analyze other causes such as
repetition of the course, the family environment, the perception of difficulty of the subject,
or if the student does other activities that affect attendance. It is also intended to expand the
sample with different subjects and degrees to enrich the results and make them more global.
Establishing a relationship between time management and dropout rates, this presents a
challenging problem due to the difficulty involved in contacting students who have dropped
out and having their willingness to complete the surveys.
857
Methodologically, several models will be compared, such as Ordinary Least Squares or

Spline Regression.
Acknowledgements
This work was supported by research grants provided by the University of Burgos and IFIE
in the project “Análisis de la gestión del tiempo en estudiantes universitarios: mejora en
asistencia a clase y calificaciones a través de machine learning y gamificación.” under the
convocatory:“Convocatoria de Ayudas a Grupos de Innovación Docente reconocidos para la
elaboración de materiales docentes para los años 2023 y 2024”
References
Adams, R. V. & Blair, E. (2019). Impact of time management behaviors on undergraduate
engineering students’ performance. Sage Open, 9(1), 2158244018824506.
Bartual Figueras, T. & Poblet Farrés, M. C. (2009). Determinantes del rendimiento
académico en estudiantes universitarios de primer año de Economía. Revista de
Formación e Innovación Educativa Universitaria, 2(3), 172-181.
Breiman, L. (2017). Classification and regression trees. Routledge.
Cervero-Fernández, A., Bernardo, A., Esteban, M., Tuero, E., Carbajal, R. & Núñez, J. C.
(2017). Influencia en el abandono universitario de variables relacionales y sociales.
Revista de Estudios e Investigación en Psicología y Educación, 12(núm. Extr.), 46-49.
Esteban, M., Bernardo, A. & Rodríguez-Muñiz, L. J. (2016). Permanencia en la universidad:
la importancia de un buen comienzo. Aula Abierta, 44(1), 1-6. doi:
10.1016/j.aula.2015.04.001.
González Tirados, R. M. (1985). Análisis de las causas del fracaso escolar en la Universidad
Politécnica de Madrid. Madrid: Centro de Investigación y Documentación Educativa.
Khiat, H. (2022). Using automated time management enablers to improve self-regulated
learning. Active Learning in Higher Education, 23(1), 3-15.
Moores, E., Birdi, G. K., & Higson, H. E. (2019). Determinants of university students’
attendance. Educational Research, 61(4), 371-387.
Murray, S. A., Davis, J., Shuler, H. D., Spencer, E. C., & Hinton, A. (2022). Time
management for STEMM students during the continuing pandemic. Trends in
biochemical sciences.
Neill, J. T. (2016). The University Student Satisfaction and Time Management
Questionnairev9. Retrieved from
https://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Assessmen
t/Survey/TUSSTMQ9 (Accessed: December 01,2022).
Oreopoulos, P., Patterson, R. W., Petronijevic, U., & Pope, N. G. (2022). Low-touch attempts
to improve time management among traditional and online college students. Journal of
Human Resources, 57(1), 1-43.
858
Pérez, F. & Aldás, J. (2019). Indicadores sintéticos de las universidades españolas, Valencia:
Fundación BBVA-Instituto Valenciano de Investigaciones Económicas. Available
at: https://www.fbbva.es/wp-content/uploads/2019/04/Informe-U-Ranking-FBBVA-
Ivie-2019.pdf (Accessed: December 14, 2022).
Raîche, G., Walls, T. A., Magis, D., Riopel, M., & Blais, J. G. (2013). Non-graphical
solutions for Cattell’s scree test. Methodology: European Journal of Research Methods
for the Behavioral and Social Sciences, 9(1), 23.
Sloan, D., Manns, H., Mellor, A., & Jeffries, M. (2020). Factors influencing student non-
attendance at formal teaching sessions. Studies in Higher Education, 45(11), 2203-2216.
859
The engagement in university students: preliminary psychometric

analyses of the Spanish version of the engagement vs. disaffection
with learning scale
Alba Aza1, Georgina Guilera2, Chuen Ann Chai2, Juana Gómez-Benito2, Estefania
Guerrero2, Maite Barrios2
1
Department of Personality, Evaluation and Psychological Treatments, Universidad de
Salamanca, Spain, 2Department of Social Psychology and Quantitative Psychology,
Universitat de Barcelona, Spain.
Abstract
The level of active participation of students in a learning activity is referred to
as student engagement. Research indicates that engagement and disaffection
are two complementary factors with two domains: behavioral and emotional.
Unfortunately, there are no multidimensional instruments available in
Spanish. Therefore, a study was conducted to analyze the psychometric
properties of the Spanish version of Engagement vs. Disaffection with
Learning (EDL). A sample of 194 participants was analyzed in terms of
distribution of item responses, factor structure, internal consistency, and the
correlation with other measures of engagement and personality traits. Results
confirmed the scale's reliability in terms of internal consistency. However,
some problematic items were identified. The Spanish version of the EDL
appears to be a promising scale for assessing engagement among Spanish-
speaking students, with the need to review some of its previous items. The EDL
could be a useful tool for educators and researchers in the field of education.
Keywords: Disaffection; education; engagement; motivation; university.

The engagement in university students: preliminary analyses of a scale
1. Introduction
Initially, engagement is a phenomenon linked to the labour world with its construct of
burnout (Maslach & Leiter, 1997). Although, due to its implications, it quickly spread to the
education sector, especially in higher education institutions. The concept definition of student
engagement is the degree of active student participation in a learning activity (Skinner et al.,
2008).
It is often conceived as a multidimensional phenomenon involving academic and personal
factors related to the student, as well as factors related to the context in which the learning
process takes place (e.g., campus facilities, social environment, services, etc.) (Sinclair et al.,
2003), constituting itself as a subjective construct (Moreira et al., 2020). Likewise, emotional
support from teachers (Mazer, 2017) and greater communicative clarity (Titswort et al.,
2010) have been linked to higher levels of student engagement. Traditionally, three are three
identified domains in the engagement phenomenon: affective, cognitive and behavioral,
which are all highly correlated (Bond & Bedenlier, 2019; Eccles, 2016). However, other
authors consider it more appropriate to differentiate the negative aspects (i.e., disaffection)
from the positive ones (i.e., engagement), arguing that it is more disaffection than a lack of
engagement (Skinner et al. 2008, 2009).
Research in this area has shown that student engagement predicts learning, motivation,
performance, and academic progress (Lin et al., 2019; Wang, 2017; Zamarripa et al., 2022).
Students who exhibit high levels of engagement are more prone to preparing for classes by
engaging in activities such as reading, studying extensively for exams, or completing
homework assignments. They could think about the applicability of course content to their
lives, how they can implement practical knowledge and skills, and how the subject can
benefit their future careers (Mazer, 2017).
To promote engagement in our students. Currently, the international scientific literature
offers numerous instruments to measure engagement or components of it. However, it can be
challenging to find instruments that effectively capture cognitive, emotional, and behavioral
factors all at once (Moreira et al. 2020). Additionally, many of these instruments have not
been validated for use in the Spanish language. Therefore, the objective of this research is:
1) to validate the Engagement vs. Disaffection with Learning scale in a sample of Spanish
university students and 2) to investigate the factors and characteristics that are related to high
levels of engagement.
862
Alba Aza et al.
2. Method
2.1. Participants
The inclusion criteria were (1) being a university student at the time of responding the
questionnaire, and (2) having a proficient level of Spanish. The study sample comprised 194
students, with 154 women (79.4%) and 38 men (19.6%), and 2 non-binary individuals
(1.0%). The ages of the participants ranged from 17 to 48 years, with a mean of 20.6 (SD =
3.83). The participants’ educational background was diverse, with 95.4% pursuing higher
education degrees or certificates, 3.6% enrolled in master's or postgraduate programs, and
1.0% in doctorate programs. Most of the participants (76.1%) reported living with their
family of origin, while 9.1% lived alone, and 7.6% cohabited with their own partner or
family.
2.2. Measures
The EDL is a 27-item instrument that assesses students’ engagement versus disaffection in
the classroom by measuring behavioral and emotional participation or rejection in classroom
learning activities (Skinner et al., 1990, 1998; Wellborn, 1991). All items are scored on a
four-point Likert scale, which ranges from 1 = does not describe me at all to 4 = describes
me totally. This scale aims to measure four domains of engagement: Behavioral engagement
(5 items), Behavioral disaffection (5 items), Emotional engagement (5 items) and Emotional
disaffection (12 items). Nevertheless, the items in the Emotional disaffection scale can be
consolidated into five, as some items are variations of different negative emotions, such as
frustration, boredom, or concern. The scores of these four factors are obtained by adding the
scores of each of the items that are part of it. However, other research has found the better
functioning of the two-dimensional structure: commitment vs. discontent or emotional vs.
behavioral (Skinner et al. 2009).
The original English-language version of the Agentic Engagement Scale (AES; Reeve, 2013)
includes five items measuring students’ dialectical and transactional participation in class
(e.g., “I defend my opinions even if they are not in line with those of my classmates”). Item
responses are presented on a 7-point Likert scale, ranging from 1 (completely disagree) to 7
(completely agree). Items were translated and adapted into Spanish through a parallel
translation and reconciliation process.
The Spanish version of the Mini International Personality Item Pool–Five-Factor Model–
Positively Worded (Mini-IPIP-PW; Martínez-Molina & Arias, 2018; original version by
Donnellan et al., 2006) was administered. This instrument assesses the Big Five personality
domains of extraversion, agreeableness, emotional stability, and openness to experience
through 20 items. Items are scored on a 5-point Likert scale (1 = not at all - 5 = completely)
to assess each personality trait. The Spanish Mini-IPIP-PW has demonstrated adequate
863
validity and reliability (α ≥ .90), and a positive relationship with engagement (Qureshi et al.,
2016).
The Utrecht Work Engagement Scale for Students (UWES–9S; Schaufeli & Bakker, 2004)
is a nine-item scale for assessing work engagement, characterized by three domains: vigor,
dedication, and absorption (e.g., “My studies inspire me with new things”). Items are scored
on a 7-point Likert-type scale, ranging from 0 (never) to 6 (always). The three-factorial
structure of the UWES-9S has been confirmed, showing a high internal consistency (α = .84).
An ad-hoc questionnaire was also administered to gather sociodemographic data, including
participants' gender, age, current field of study, and admission GPA.
2.3. Procedure
The data collection for this study took place between October and November 2022 and
utilized a convenience sample method. The invitation to participate was sent to undergraduate
university students via their virtual campus. Participants were directed to an online
questionnaire hosted on the Qualtrics platform (https://www.qualtrics.com), which included
the previously mentioned instruments. Participants were encouraged to share the link with
their social media contacts (e.g., Twitter, Instagram). Before starting the questionnaire, all
participants were informed about voluntary participation, anonymity, and confidentiality in
the study, and gave an online informed consent. The study was conducted in accordance with
the Declaration of Helsinki (World Medical Association, 2001) and received approval from
the University of Barcelona's bioethics commission.
2.4. Statistical analysis

Descriptive statistics of sociodemographic variables and items of the EDL scale were
analyzed, as well as data on skewness, kurtosis, and response endorsement to evaluate the
floor and ceiling effects. The acceptable value of skewness and kurtosis are those in the range
[–1, + 1] (Ferrando and Anguiano-Carrasco, 2010). Statistical analyses were performed using
the Jamovi program version 2.3.21.
To assess internal consistency, Cronbach's alpha values and item-total correlations were
calculated. A minimum alpha value of 0.70, as recommended by Nunnally and Berstein
(1994), was used as the threshold for a reliable measure. Pearson's correlation coefficient was
used to compute the correlations between the domains of the EDL.
The convergent and discriminant validity were assessed through correlation analysis between
the scale domains of the EDL and the AES, UWES-9S, and Mini-IPIP-PW. Pearson's
correlation coefficients were computed and interpreted according to Cohen’s criteria (1988).
864
Alba Aza et al.
3. Results
3.1. Item-level descriptive analysis

Table 1 includes the distribution of the scores of the items on the EDL scale. The items with
the lowest scores were items 24, “When I'm in class, I feel bad” (M = 1.52, SD = .76) and
25, “I get cranky when I'm doing activities in class” (M = 1. 41, SD = . 64) and those with
the highest scores are items 1 “I strive to do well in college” (M = 3.29, SD = .68) and 9 “I
enjoy learning new things in class” (M = 3.17, SD = .80).
The skewness and kurtosis indices show an adjustment to normality in most items, although
there is a slight positive asymmetry. In this way, there is an observatiton that some items
concentrate with the largest percentage of answers in the answer option 1 = does not describe
me at all, showing a floor effect of that item. Also, some items show a ceiling effect by
concentrating the highest percentage of responses in option 4 = describes me totally.
3.2. Internal consistency

According to Cronbach's alpha, internal consistency levels are satisfactory for the four
domains. Table 2 shows the levels of both Cronbach's α and the item-test correlation for items
structured in the four domains. Items with critical item-domain correlations (< .40) are bold
in this table.
3.3. Convengent and discriminant evidence

There was a large positive correlation between the domain of behavioral and emotional
engagement (r = .63) and a large negative correlation found between this domain and
behavioral disaffection (r = -.73). Similarly, a large correlation (r = .70) was observed
between the emotional engagement domain and the UWES-9S engagement scale. The
UWES-9S demostrated moderate correlations with the remaining domains of the the EDL.
865
Table 1. Descriptive statistics of the EDL items.
Domain/ M SD Skewness Kurtosis Percentage of Item

Item Endorsement
1 2 3 4
Behavioral
engagement
Item 1 3.29 .68 -.55 -.33 .05 11.2 46.5 41.7
Item 2 2.74 .82 -.24 -.44 7.0 29.4 46.5 17.1
Item 3 1.95 .97 .67 -.62 40.6 32.1 18.7 8.6
Item 4 3.02 .76 -.46 -.01 3.2 18.2 52.4 26.2
Item 5 2.96 .80 -.30 -.56 3.2 24.6 45.5 26.7
Emotional
engagement
Item 6 2.67 .85 -.16 -.58 8.6 32.1 42.8 16.6
Item 7 2.96 .79 -.52 .03 4.8 18.7 52.4 24.1
Item 8 2.19 .82 .28 -.42 19.8 47.1 27.3 5.9
Item 9 3.17 .80 -.77 .18 3.7 13.4 44.9 38.0
Item 10 3.06 .79 -.45 -.40 2.7 19.8 46.0 31.6
Behavioral
disaffection
Item 11 1.79 .92 .97 .04 47.6 32.3 12.8 7.0
Item 12 1.57 .75 1.21 .95 56.7 32.1 9.1 2.1
Item 13 1.71 .87 1.00 .07 52.4 28.9 14.4 4.3
Item 14 2.60 .89 .12 -.81 8.6 41.2 31.6 18.7
Item 15 2.62 .92 .06 -.89 9.6 39.0 31.0 20.3
Emotional
disaffection
Item 16 2.01 .78 .54 .08 25.7 52.4 17.6 4.3
Item 17 1.89 .79 .53 -.34 34.8 44.4 18.2 2.7
Item 18 1.79 .83 .76 -.26 44.4 35.8 16.6 3.2
Item 19 2.02 .96 .56 -.69 35.8 34.8 20.9 8.6
Ítem 20 1.81 .94 .86 -.32 48.1 28.9 16.6 6.4
Ítem 21 3.09 .87 -.71 -.16 5.9 16.0 41.7 36.4
Ítem 22 1.83 .82 .84 -.12 43.3 35.8 15.0 5.9
Ítem 23 2.24 .98 .42 -.80 24.6 41.2 19.8 14.4
Ítem 24 1.52 .76 1.51 1.86 61.5 28.3 7.0 3.2
Ítem 25 1.41 .64 1.58 2.27 66.8 26.7 5.3 1.1
Ítem 26 2.91 1.00 .44 -.96 10.2 24.1 29.9 35.8
Ítem 27 2.96 .97 -.53 -.75 9.1 21.4 34.2 35.3
866
Alba Aza et al.
Table 2. Cronbach’s α and item-test correlations levels for the EDL scale.
Domain/ Cronbach’s α Item- Domain/ Cronbach’s α Item-

Item domain Item domain
correlation correlation
Behavioral engagement 0.70 Item 4 .69
Ítem 1 .40 Item 5 .67
ítem 2 .61 Emotional disaffection .77
Item 3 .13 Item 1a .54
Item 4 .61 Item 1b .51
Item 5 .65 Item 1c .42
Emotional engagement .84 Item 2a .57
Item 1 .63 Ítem 2b .53
Item 2 .71 Ítem 2c .42
Item 3 .62 Ítem 3 .59
Item 4 .63 Ítem 4 .44
Item 5 .59 Ítem 5a .62
Behavioral disaffection .81 Ítem 5b .59
Item 1 .48 Ítem 5c .40
Item 2 .54 Ítem 5d 34
Item 3 .65
Note. Items with item-domain correlation < .40 are in bold
Table 3. Pearson correlation between EDL scale with other variables.
Scales Behavioral Emotional Behavioral Emotional

engagement engagement disaffection disaffection
EDL
Behavioral engagement -
Emotional engagement .63** -
Behavioral disaffection -.73** -.55** -
Emotional disaffection -.48** -.64** .61** -
Mini-IPIP-PW
Extraversion .65 .16* -.01 -.14
Agreebleness .17* .14* -.09 .01
Consientiousness .24** .16* -.36** -.25**
Emotional stability .07 .20* -.01 -.18*
Openness to experience .14 .17* .04 -.02
AES .36** .43** -.19* -.32**
UWES-9S .53** .70** -.49** -.59**
Admission GPA .19* .12 -.19* -.07
Note. *p<.005, **p<.001
4. Discussion
The results of this study show that despite finding satisfactory levels of internal consistency
according to the four-factor model proposed by Skinner (2008), there are some items (p.e.,
item 3 “When I'm in class, I participate in discussions”) that show a low discriminative
capacity to differentiate between high and low values of engagement. This lack of
867
differentiation between behavioral engagement and disaffection has been previously

observed in other studies (Immekus & Ingle, 2019).
Our findings support the correlation between the EDL and other forms of engagement, as
work engagement, in line with the study of Skinner (2009). On the other hand, we did not
find the expected results according to academic achievement (Wang, 2017). However, we
observe relations between behavioral engagement/disaffection and conscientiousness and
between emotional disaffection and Emotional stability (Muenks et al., 2017).
Despite the novelty and significance of the research topic, this study is subject to limitations,
such as the use of a convenience sampling method for participant selection. Moreover, it is
essential to employ a larger sample size to conduct a confirmatory factor analysis of the scale
and assess its internal structure.
5. Conclusion
The EDL scale is a promising instrument for studying engagement in the Spanish population,
with satisfactory internal consistency levels and evidence of convergent validity with other
engagement measures and personality variables. However, it is necessary to remove or
review in depth the functioning of some items to ensure the adequate performance of the
EDL scale.
References
Bond, M., & Bedenlier, S. (2019). Facilitating student engagement through educational
technology: Towards a Conceptual Framework. Journal of Interactive Media in
Education, 2019(1). doi:10.5334/jime.528
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd Ed). New York:
Routledge.
Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The mini-IPIP scales:
tiny-yet-effective measures of the Big Five factors of personality. Psychological
Assessment, 18(2), 192-203. doi: 10.1037/1040-3590.18.2.192
Eccles, J. S. (2016). Engagement: Where to next? Learning and Instruction, 43, 71–75. doi:
10.1016/j.learninstruc.2016.02.003
Ferrando, P. J., & Anguiano-Carrasco, C. (2010). El análisis factorial como técnica de
investigación en psicología. Papeles del Psicólogo, 31, 18-33.
Immekus, J. C., & Ingle, W. K. (2019). A test and cross-validation of the factor structure of
the engagement versus disaffection with learning instrument among middle school
students. Journal of Psychoeducational Assessment, 37(4), 504-511. doi:
10.1177/0734282917742569
Lin, L. C., Hung, I. C., Kinshuk, & Chen, N. S (2019). The impact of student engagement on
learning outcomes in a cyber-flipped course. Educational Technology Research and
Development, 67(6), 1573–1591. doi: 10.1007/s11423-019-09698-9
868
Alba Aza et al.
Maslach, C., & Leiter, M. P. (1997). The truth about burnout: How organizations cause
personal stress and what to do about it. San Francisco: John Wiley & Sons.
Mazer, J. P. (2017). Associations among classroom emotional processes, student interest, and
engagement: A convergent validity test. Communication Education, 66(3), 350-360. doi:
10.1080/03634523.2016.1265134
Moreira, P., Cunha, D., & Inman, R. A. (2020). An integration of multiple student
engagement dimensions into a single measure and validity-based studies. Journal of
Psychoeducational Assessment, 38(5), 564-580. doi: 10.1177/0734282919870973
Muenks, K., Wigfield, A., Yang, J. S., & O'Neal, C. R. (2017). How true is grit? Assessing
its relations to high school and college students’ personality characteristics,
selfregulation, engagement, and achievement. Journal of Educational Psychology,
109(5), 599–620. doi: 10.1037/edu0000153
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd Ed.). New York:
McGraw-Hill.
Reeve, J. (2013). How students create motivationally supportive learning environments for
themselves: The concept of agentic engagement. Journal of Educational Psychology,
105, 579-595. doi:10.1037/a0032690
Schaufeli, W. B., Bakker, A. B., & Salanova, M. (2006). Utrecht Work Engagement Scale-9
(UWES-9) [Database record]. APA PsycTests.
Sinclair, M. F., Christenson, S. L., Lehr, C. A., & Anderson, A. R. (2003). Facilitating student
engagement: Lessons learned from Check & Connect longitudinal studies. California
School Psychologist, 8, 29–42. doi: 10.1007/BF03340894
Skinner, E., Furrer, C., Marchand, G., & Kindermann, T. (2008). Engagement and
disaffection in the classroom: Part of a larger motivational dynamic? Journal of
Educational Psychology, 100(4), 765–781. doi:1037/a0012840
Skinner, E. A., Kindermann, T. A., & Furrer, C. J. (2009). A motivational perspective on
engagement and disaffection: Conceptualization and assessment of children's behavioral
and emotional participation in academic activities in the classroom. Educational and
Psychological Measurement, 69(3), 493–525. doi: 10.1177/0013164408323233
Skinner, E. A., Wellborn, J. G., & Connell, J. P. (1990). What it takes to do well in school
and whether I’ve got it: The role of perceived control in children’s engagement and school
achievement. Journal of Educational Psychology, 82, 22–32. doi: 10.1037/0022-
0663.82.1.22
Skinner, E. A., Zimmer-Gembeck, M. J., & Connell, J. P. (1998). Individual differences and
the development of perceived control. Monographs of the Society for Research in Child
Development, 63(2–3), v–220. doi: 10.2307/1166220
Titsworth, S., Quinlan, M. M., & Mazer, J. P. (2010). Emotion in teaching and learning:
Development and validation of the classroom emotions scale. Communication
Education, 59(4), 431-452. doi: 10.1080/03634521003746156
Wang, F. H. (2017). An exploration of online behaviour engagement and achievement in
flipped classroom supported by learning management system. Computers & Education,
114, 79–91.doi: 10.1016/j.compedu.2017.06.012
869
Wellborn, J. G. (1991). Engaged and disaffected action: The conceptualization and

measurement of motivation in the academic domain. Unpublished doctoral dissertation,
University of Rochester.
World Medical Association. (2001). Declaration of Helsinki ethical principles for medical
research involving human subjects. Bulletin of the World Health Organization, 79(4),
373–374. doi: 10.1515/9783110208856.233
Zamarripa, J., Rodríguez-Medellín, R., & Otero-Saborido, F. (2022). Basic psychological
needs, motivation, engagement, and disaffection in Mexican students during physical
education classes. Journal of Teaching in Physical Education, 41(3), 436-445. doi:
10.1123/jtpe.2021-0006
870
Increasing student engagement with COIL Padlet

Barbara Therese Miller1, Susan Göldi1, Meng Hsien (Jenny) Lin2
1
School of Business, University of Applied Sciences Northwestern Switzerland FHNW,
Olten, Switzerland, 2California State University Monterey Bay, Seaside, CA, United States.
Abstract
Student engagement undoubtedly deserves attention and time of lecturers and
researchers. This paper aims to showcase how to increase the engagement of
international marketing students with a simple, technology-enhanced course
addition based on an online bulletin board (Padlet) collaboration between two
universities in different continents.
This conceptual paper provides (a) for other educators a case study, detailing
the efficient collaboration of two lecturers in different universities for a joint
course design; and (b) for researchers some initial data of increased student
engagement.
Keywords: Student engagement; COIL; international marketing; course
design; experiential learning.

1. Introduction
Engaging students into the learning process is essential for deep learning experiences. In
recent decades education thankfully has added new forms to traditional classroom lecturing,
where the professors recited theoretical input, explained, and narrated as the central figure in
the educational setting. A plethora of interactive methods and technologies, including case-
study based teaching, integrating online exercises and games into lectures provide
opportunities for student engagement. These innovative forms require certain investments in
training and time from the lecturers which can be a limiting factor of a wider deployment.
However, due to simple technology-enhanced tools, stimulations of student engagement can
be nowadays achieved, even for challenging international education topics.
In this paper we present a course design how to increase the engagement of international
marketing students with a simple addition based on an online bulletin board (Padlet) shared
between two universities in different continents. A description of the lecturers’ collaboration,
which was initiated via a Collaborative Online International Learning (COIL) between the
two universities and involved 40 students, is included. Their joint development of a learning
cycle on the topic of cultural difference in the context of marketing is explained in detail.
Several observed indicators of increased student engagement are also summarised.
This paper addresses several themes addressed in the conference, namely (a) Globalisation
& International, (b) e-learning strategy, and (c) Innovative teaching and learning experiences.
2. Theoretical background and method

In the research community, there is agreement that “student engagement” is a complex
phenomenon, with one consensus being on the core dimensions (cognitive, behavioural, and
affective/ emotional). (Kuh, 2003, Kahu, 2013, Bond et al., 2020). Even though the exact
definition of student engagement and its measurement remains controversial, researchers
observed positive impact on students’ learning (Fredrick & McColskey, 2012; Kahu &
Nelson, 2018; Bowden et al., 2021), with Bond et al. (2020) providing an overview of most
used indicators of student engagement per dimensions based on 243 studies. These indicators
range from “learning from peers” and “deep learning” for cognitive student engagement, to
“participation/ involvement” and “achievement” for behavioural student engagement and
“positive interaction with teachers and peers”, “enjoyment” and “interest” for affective
student engagement; however to fully understand and measure student engagement, a
quantitative analysis of easily measurables indicators may not be enough, and should be
accompanied by observations of the learning environment and a better understanding of
students’ reflections and perceptions (Bond et al., 2020).
872
The experiential learning theory as introduced by Kolb (1984) stipulates learning as a cycle
which creates knowledge through the transformation of students’ experience with reflective
practice. In a simplified way, the students’ learning cycle can be described by the phases of
concrete experience, reflective observation, abstract conceptualisation and active
implementation, and deep learning is defined as moving from isolated to integrated
knowledge. Providing the opportunity to students of strong experiences and stimulation of
reflections with links to theoretical related material should achieve the highest level of
learning (Kolb & Kolb, 2011). This approach also allows the lecturers to better understand
students’ learning by observing the students and by reviewing their reflections. The obvious
relationship between experiential learning activities and student engagement has been
reported by several studies (Winsett, 2016, Li et al., 2019).
International marketing education has a long tradition of using technology to enrich students’
learning (Bell et al., 2001), with tools having drastically improved over the years. Hence,
sophisticated technology-enhanced international teaching approaches such as COIL have
been possible since over a decade, with the aim to bring lecturers and students from remote
distances together (Jie & Pearlman, 2018). The integration of experiential learning theory
with COIL has successfully been applied to internationalise an entire business school’s
curriculum and to develop intercultural skills among students (Nava-Aguirre et al., 2019).
This conceptual paper proposes that combining experiential learning theory with the
opportunities provided by technology-enhanced online collaboration can efficiently
contribute to a positive and interactive environment for students and increased student
engagement. Next, the paper provides a detailed course design for International Marketing,
enriched by a simple added course activity, namely COIL Padlet.
3. International Marketing course design

The eight public Swiss Universities of Applied Sciences and Arts are closely linked to the
economy, culture and society, with their educational concept focusing on providing highly
practice-relevant degrees. At the University of Applied Sciences Northwestern Switzerland
(FHNW), lecturers experiment on providing students in addition to theoretical knowledge
with applicable skills for future jobs. The course design of International Marketing in the BA
programme of International Management was up to 2021 based on interactive and theoretical
study units. Additionally, due to the COVID travel restrictions students’ mobility such as
international study abroad was limited. Hence the lecturer in the spring of 2021 explored
opportunities to enrich this 12-weeks autumn semester syllabus for students with
international experience; the overall aim was to allow students to learn about cultural
difference in international advertisements in line with the course objective. The International
Office at FHNW provided the contact to a professor at a partner university in the US. The
873
two lecturers agreed on a lean collaboration for an international exchange for the students,
with the additional aim of improving student engagement. The learning experience took place
on a shared online space on Padlet which as a website application is an online bulletin board
where students wrote their posts and provided comments and reactions to posts of students
abroad (called COIL Padlet for this project). The discussions were structured along four tasks
and called COIL #1 to #4 (named COIL(s) subsequently). Students were able to receive
bonus points (5% of total grade) on providing a reflection on their learning.
Table 1. Summary of adapted International Marketing course design.
Original course design Course addition (COIL Padlet)

Lecturer’s Prepare students’ Collaboration between two universities; plan 12-week
preparation reading list, prepare semester syllabus with every 3 weeks a COIL, specify
lecture content, case task descriptions and two-week time windows for
studies and exercises activity; create and deploy Padlet before semester start
Students’ Prepare reading on Prepare for COIL # 3 (selection of a still or video

preparation international marketing advertisement with a cultural aspect)
Study units Discussion of students’ Topics on COIL Padlet: COIL #1: Students’ interests;
reading; theoretical COIL #2: Food; COIL #3: “Cultural” advertisement;
input from lecturer; COIL #4: Religious holidays
case-study group work
Reminder to comment on Padlet during two-week time
window; time in classroom to allow discussion on COIL
Padlet; educational dialogue on students’ observations
(retrospect); task descriptions on Padlet; reminder for
COIL #3 (finding a “cultural “advertisement)
Assessment Exam Exam; reflection paper (bonus points)
The asynchronous discussions on the COIL Padlet were planned across the semester (Table
1), involving topics that built on each other. The COIL Padlet website was made available
from the beginning of the semester and two-week time windows for working on each active
COIL were defined. The time windows were helpful as students did not share the teaching
day nor was the curriculum at the two universities fully aligned. Students were able to work
in the classroom on the Padlet during the four “COIL” study units (for 20 minutes). Simple
tasks, except for COIL #3, were intentionally designed to easily engage students (excerpts
from COIL Padlet: “What are your interests?”; “What is a typical food in your culture?”).
In line with the experiential learning theory, the lecturer asked the students to reflect after
COIL #1 in groups whether they observed more differences or commonalities across the two
sets of students (see Table 2). The lecturer synthesised their findings in an educational
dialogue with them. For the more involved COIL #3 students had to select a still or video
874
advertisement with a cultural aspect which they also had to describe with a marketing theory.
Two steps (in italics in Table 2), preparation of learning cycle and explicit abstract
conceptualisation, were added to the experiential learning cycle by Kolb (1984) to better
describe the learning experience of the students and the efforts involved.
Table 2. Mapping of COIL Padlet on experiential learning cycle including effort.
Experiential COIL Padlet Effort for students Effort for lecturer

learning cycle Implementation
Preparation of Set up joint Padlet across None Moderate, one day for
learning cycle two universities, design four collaboration (email,
common COILs for students calls) of two lecturers
Concrete In-class COIL #1, #2 and #4 Little, as time was Little, encouraging
experience were built on simple topics allocated to post/ and reminding
on cultural difference comment students to post and
comment
Reflective Time allocated in class to Little, additionally Little, facilitating an

observation reflect in groups on implicitly educational dialogue
commonalities/ differences to synthesise students’
across COIL #1 discussions observations
Abstract Students select a suitable Moderate, students Little, reminding

conceptualisation “cultural” advertisement in to match their students to find
(implicitly) preparation for COIL #3 learning to COIL #3 advertisement
Active COIL #3 on a “cultural” Little, as time was None

experimentation advertisement with a allocated to post/
marketing theory comment
Abstract Written reflection (one page) Moderate, 1-2h to Moderate, half-day

conceptualisation a few weeks after COIL #3 summarise their grading/ providing
(explicitly) reflection feedback to students
To encourage students’ abstract conceptualisation, the lecturer created a reflection for

students. The prompts (Table 3) were crafted with the aim of encouraging critical reflection,
while at the same time allowing students to make connections to marketing theory.
Prompts were structured to help gain insight into the cognitive (Q1, Q4) and affective student
engagement (Q2, Q3) dimension, and were based on research related to assessment rubrics
from Brock University (Brock University, n.d.), University of Edinburgh (The University of
Edinburgh, n.d.) and ongoing work of one of the authors (Miller & Staley, 2023).
875
Note that the reflection reports are discussed here as a mean for both students (to explicitly
conceptualise) and lecturers (to learn about students’ development implicitly) and the results
are not the central focus of this conceptual paper.
Table 3. Prompts provided to students for their written reflection on COIL #3
Reflection on Your experience with the COIL #3:

existing marketing Q1. What did you know about cultural aspects in promotion before COIL #3?
knowledge and Q2. How did you feel during the task, and what about the interaction with
emotions. other students?
Connection to Your take-aways, learnings from COIL #3

academic Q3. Have you learnt anything which you will use for future international
marketing concept marketing tasks?
& evidence of Q4. Did your understanding of culture change? Does a marketing theory from
development your reading/ class discussion make more sense now?
4. Observations and learnings

At the end of the semester, the collaborating lecturers reviewed their individual class
observations, including the reflections provided from the students. Overall, they considered
the COIL Padlet implementation, following an adapted experiential learning cycle, a success
in terms of effort-benefit considerations. As summarised in Table 2, the perceived effort for
the added tasks was little, however the benefits were significant. The lecturers were pleased
by the positive learning environment created in class due to the COIL Padlet and the level of
keenness of students to contribute. They also observed that students actively engaged with
each other outside the allocated time in class. Students mentioned that they are interested in
more interaction with partner students via synchronous lectures which the lecturers are
investigating for future collaborations. They have already shared at their respective
universities the observed positive behaviour of the students and the overall course design.
Whereas the behavioural dimension of an increased student engagement was captured
directly by the lecturers’ observations, they used the reflection reports to gain insight into the
cognitive and affective dimensions. Selected quotes from these reports (as shown in Table 4)
further support the claim that students were engaged in the COIL Padlet (structure and
possible indicators, in brackets, as suggested by Bond et al. (2020)).
876
Table 4. Selected quotes from students’ reflection reports.
Cognitive • “Personally, it always helps me a lot when I get practical examples of theoretic
student content. Through the intercultural exchange, I now have examples in mind that
help me to internalize the theoretical knowledge.”
engagement • “From the lectures we learnt about high- & low context cultures but from COIL
(Learning #3 I got to see practical examples of that and how different the communication
is which made it easier to understand.”
from peers,
• “Personally, I perceived the interactions with students of other nationalities
deep during COIL #3 as very insightful. What especially developed during this COIL,
learning) was my perspective on the cultural iceberg model, as …”
• So, thanks to this exchange, I will stop assuming that even if the cultures are
similar, there are tools that cannot be used in all countries and that is why I must
inform myself first.”
• Also, in terms of academic knowledge COIL has also helped me to deeply
understand different ads and how stereotypes and subliminal messages are used.
• “… During the last three years of studies, we had a look at Hofstede’s
dimensions... But it was the first time that I have really adapted my thinking and
tried to implement the theoretical knowledge when talking to US students.”
Affective • “The interaction with the international students is very fascinating and
informative. Many students are highly involved into the discussion and give their
student
own experience or expertise through their own cultural lens.”
engagement • “Overall, these COIL tasks have been very fun and insightful to a completely
(Positive different culture than ours. In combination with our marketing classes, it opens
many views on how to reach the target audience ...”
interaction,
• “… Secondly, this task was fun. The students posted so many amusing ads.
enjoyment, Studying interactively like this and keeping in touch with each other helped to
interest) eases the difficulty of the pandemic.”
• Overall, I am grateful for having been able to be part of this experience.”
5. Conclusions
We believe the presented case study should allow international educators to explore similar
course designs without much effort and contribute to a wider deployment of modern and
technology-enhanced forms of teaching. Further research on measuring student engagement
can use the reported lecturers’ observations and students’ quotes as starting points.
Limitations of this paper are the small number of students involved and the explorative
approach of this new course design and could be addressed systematically in future projects.
References
Bell, J., Deans, K., Ibbotson, P., & Sinkovics, R. (2001). Towards the “internetalization” of
international marketing education. Marketing Education Review, 11(3), 69-79.
877
Bond, M., Buntins, K., Bedenlier, S., Zawacki-Richter, O., Kerres, M. (2020) Mapping
research in student engagement and educational technology in higher education: a
systematic evidence map. Int J Educ Technol High Educ 17, 2
Brock University. (n.d.). Role of Reflection. https://brocku.ca/pedagogical-
innovation/resources/experiential-education/role-of-reflection/
Bowden, J. L., Tickle, L., & Naumann, K. (2021) The four pillars of tertiary student
engagement and success: a holistic measurement approach, Studies in Higher Education,
46:6, 1207-1224
Fredricks, J. A., & McColskey W. H. (2012). The Measurement of Student Engagement: A
Comparative Analysis of Various Methods and Student Self-Report Instruments. In
Handbook of Research on Student Engagement, 763–782. Boston, MA: Springer
Jie, Z., & Pearlman, A. M. G. (2018). Expanding access to international education through
technology enhanced collaborative online international learning (COIL)
courses. International Journal of Technology in Teaching and Learning, 14(1), 1-11.
Kahu, E. R. (2013). Framing student engagement in higher education. Studies in
HigherEducation, 38, 758 - 773.
Kahu, E. R., & Nelson K. (2018) Student engagement in the educational interface:
understanding the mechanisms of student success. Higher Education Research &
Development, 37:1, 58-71
Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and
development. Englewood Cliffs, NJ: Prentice-Hall
Kolb, A.Y., & Kolb, D. A. (2011). Experiential Learning Theory: A Dynamic, Holistic
Approach to Management Learning, Education and Development. In Armstrong, S. J. &
Fukami, C. (Eds.) Handbook of management learning, education and development.
Kuh, G. D. (2003) What We're Learning About Student Engagement From NSSE:
Benchmarks for Effective Educational Practices, Change: The Magazine of Higher
Learning, 35:2, 24-32
Li, H., Öchsner, A., & Hall, W. (2019) Application of experiential learning to improve
student engagement and experience in a mechanical engineering course. European
Journal of Engineering Education, 44:3, 283-293,
Miller, B. T., & Staley, L. (2023). Project report on “Building competencies through students’
self-reflection”. [Unpublished manuscript]. University of Applied Sciences Northwestern
Switzerland FHNW.
Nava-Aguirre, K. M., Garcia-Portillo, B. I., & Lopez-Morales, J. S. (2019). Collaborative
online international learning (COIL): An innovative strategy for experiential learning and
internationalization at home. The Palgrave handbook of learning and teaching
international business and management, 721-746.
The University of Edinburgh. (n.d.). Reflection Toolkit. Assessing Reflection.
https://www.ed.ac.uk/reflection/facilitators-toolkit/assessment
878
Positive effects of mindfulness practices on academic performance

and well-being
Katia G. Karadjova-Kozhuharova1, Ruth L. Baker2
1
University Libraries, Georgia Southern University, USA, 2University Libraries, Georgia
Southern University, USA.
Abstract
This qualitative study explored potential positive effects of mindfulness
practices among university students. The mindfulness practices were provided
at an experiential space, the Brain Booth at the university library, meant to
learn about the mind-body connection, reduce stress, and optimize learning.
The Brain Booth activities were available to students at any time during the
library working hours. The findings showed that the Brain Booth was
experienced as helping students to 1) Relax and Destress; 2) Calm down; 3)
Foster focus and clarity; 4) Actively engage and reenergize. A strong
preference was outlined for mindfulness activities without a digital component
or screen time. The study showed that the Brain Booth is helping students, at
their point of need, with some of the major factors that affect their academic
performance the most, as well as with their overall well-being.
Keywords: Mindfulness; well-being; university library; academic
performance; students; Brain Booth.

Positive effects of mindfulness practices on academic performance and well-being
1. Introduction
One of the major trends in higher education lately has been that student needs are increasingly
front and center with a focus on supporting students holistically. Nowadays, most of
university students are Generation Z, the most ethnically diverse generation yet walking on
university campuses. Also, students have to navigate in a very fast-paced societal
environment, which very often leads to a lack of concentration known as mind wandering.
Studies show that Generation Z multitasks daily at least across five screens and that two of
the factors affecting their academic performance the most are stress and anxiety (Seemiller
& Grace, 2016). The Brain Booth is an initiative in a higher education setting addressing
these student challenges through mindfulness and contemplative pedagogy.
The Brain Booth at the university library is an experiential space to learn about the mind-
body connection, reduce stress, and optimize learning. Mindfulness, as defined by Kabat-
Zinn (1994), “means paying attention in a particular way: on purpose, in the present moment,
and nonjudgmentally.” Zajonc (2013) outlines that “contemplative pedagogy offers
educational methods that support the development of student attention, emotional balance,
empathetic connection, compassion, and altruistic behavior, while also providing new
pedagogical techniques that support creativity and the learning of course content.” The Brain
Booth initiative introduces mindfulness and contemplative pedagogy to the campus
community and offers mindfulness practices through intentional brain breaks and activities
that support emotional self-regulation and foster singular thoughtful focus. The Brain Booth
activities include: biofeedback, coloring, curated game station, gratitude-express, light and
sound therapy, massage chair, meditation, origami, virtual reality-immerse, and more.
Students showed great interest in the initiative. It is known that different activities do not
work the same way for everyone, hence the Brain Booth has been offering a variety of such
to accommodate and be able to provide support at the point of need. These activities are
available to students at any time during the university library work hours.
The following major trends emerged from the scholarly publications in the last five years
(2017-2022), after a systematic literature review on positive effects of mindfulness practices
and contemplative pedagogy in academic settings (We are unable to include here the
complete list of identified and reviewed publications. A full bibliography is available upon
request.):
• Use of physical spaces devoted to mindfulness practice. There is evidence for an

increasing use of physical spaces devoted to mindfulness practices on campus
premises, especially in academic libraries (e.g., Chun et al., 2021; Duffy et al., 2021;
Gibson & Regan; Hartel et al., 2017; Karadjova, 2018, 2019; Karadjova-
Kozhuharova & Baker, 2022; Ruhlmann, 2017).
880
• Incorporating mindfulness practices in university/college curricula. Some scholars

and practitioners have designed and conducted ongoing mindfulness training in the
classroom across curricula to support an attentive, present centered, and non-
reactive mental mode (e.g., Bartel et al., 2018; Colaianne et al., 2020; Fung et al.,
2019, Hartel et al., 2017; Ramasubramanian, 2017; Reeve et al., 2021; Schwind et
al., 2017; Warren & Deckert, 2019) or in specific curricula such as music (Bartos et
al., 2022) and chemistry (Vitha, 2022), as well as focused on the role of the
Instructor (Placito de Rango, 2018; Schwind et al., 2017).
• Most of the studies have focused on using breathing techniques, meditation and
journaling as mindfulness practice activities in their studies (e.g., Azevedo &
Menezes, 2020; Egan et al., 2022; Ramasubramanian, 2017; Schwind et al., 2017).
Most of the studies concluded with a recurring call for more research and rigorous
testing of the effectiveness of mindfulness practices on academic performance (e.g.,
Bamber & Scheider, 2022; Duffy et al., 2021; Finkelstein-Fox, et al., 2018; Schwind
et al., 2017). It was also noted that brief mindfulness activities might have positive
behavioral outcomes without the need for full mindfulness courses.
Considering all of the above, conducting a study on the potential positive effects of the
mindfulness practices in the Brain Booth, which offers a variety of mindfulness activities,
was a natural next step for the researchers involved with the initiative.
2. Methodology
The researchers employed a qualitative inquiry to explore potential positive effects of
mindfulness practices among university students. The research question was: How do
university students experience the mindfulness activities with the Brain Booth? A
questionnaire with two open-ended questions through an anonymous Google form was made
available to the Brain Booth visitors for the duration of the Fall 2022 semester. The consent
form preceding the questionnaire required the acknowledgement of a student status to be able
to complete it. The two questions were: 1) Which activities did you find helpful and why?
and 2) How would you describe your overall experience with the Brain Booth activities? The
collected narratives from 95 respondents were analyzed using the In Vivo Coding method to
formulate initial codes from the texts. Afterwards these were grouped in emerging themes
and patterns to outline the findings of the study. In general, relying on individuals’ self-
reported effects might be acknowledged as a limitation of a study, but due to the nature of
this study, it should not be considered a disadvantage. For example, if a student reports that
a mindfulness activity has helped them to relax or to destress, they are actually the reliable
source of such a report.
881
3. Findings
The overall results of the study were very positive. 100% of the respondents found their
experience with the Brain Booth helpful and pointed out at least one mindfulness activity as
having a positive effect on their academic performance or well-being. The major findings on
how university students experience the Brain Booth activities were that they help them to: 1)
Relax and destress; 2) Calm down; 3) Foster focus and clarity; 4) Actively engage and
reenergize. Specific activities mentioned the most were coloring, origami, and the puzzle
station, followed closely by the massage chair, light & sound therapy, the tabletop gaming
station, and the pedal study desks.
3.1. Relax and Destress

The majority of the respondents, 72%, experienced the activities as helping them to relax and
destress. Here are some specific examples from the narratives:
“A convenient unwind from the heavy load that college can press on a person.”
“It helped me destress.”
“I had a great experience. I love how there’s different stimuli for relaxation.”
“It gives me a way to distract myself and relief myself of stress.”
“It really relaxed my body and mind.”
3.2. Calm down

Many of the students, 27%, described that the activities were helpful with calming them down
and helping with anxiety. Below are some specific examples from the narratives:
“…very helpful because it helped calm me down much faster…”
“They were very fun and helped calm me down.”
“…were relaxing and help calm any nerves that I had for my exam.”
“I loved it and it helped bring my anxiety down.”
3.3. Foster Focus and Clarity

A significant number of students, 25%, found the activities to be helpful with fostering focus
and clarity, which is extremely important in addressing the challenge of the well-known
‘mind wandering,’ namely, the lack of concentration issue. Here are specific examples from
the narratives:
“It made my mind feel clear.”
882
“…helpful before my calculus exam.”

“It really helped my brain kick-start and focus easier on my studies.”
“…it gets my brain going so I can do my homework or concentrate better.”
3.4. Actively Engage and Reenergize

Last but not least, a significant number of students, 23%, experienced the activities as also
helping them to actively engage and reenergize. A few examples from the narratives:
“…it helps my overall work flow…”
“…helping to keep be active and engaged.”
“…using (the activities) to get reenergized.”
4. Discussion
The findings of this study support the notion that mindfulness practices have positive effects
on students’ academic performance and overall well-being. The finding that the majority of
the respondents have experienced the Brain Booth helping them to relax and destress is
consistent with the scholarly literature outlining this outcome after mindfulness practices
(e.g., Azevedo & Menezes, 2020; Bartos et al., 2022; Ramasubramanian, 2017; Schwind et
al., 2017).
This study showed that the mindfulness activities at the Brain Booth were helping students
calm down and diminish anxiety in addition to relaxing and destressing, which means that
the offered activities are addressing both factors affecting academic performance the most,
namely, stress and anxiety. The study showed that the Brain Booth was helpful with fostering
singular thoughtful focus, which addresses the lack of concentration exhibited through ‘mind
wondering.’ The Brain Booth’s offerings of many different activities available to students at
any time distinguishes this from other studies on focus improvement (e.g.,
Ramasubramanian, 2017; Bartos et al., 2022). Such a setting makes it more convenient for
students to participate in such activities, hence provides an opportunity for self-help without
a need for a constant formal class or session meetings at regulated times. In addition, the
Brain Booth was experienced as reenergizing and supporting active engagement. In the
future, the researchers plan to design and conduct a study on exploring if any of the specific
offered activities are correlated with any of the specific findings of this study.
An interesting outcome of the study was the fact that the activities mentioned as helpful by
name the most were the activities not including any digital components or screen time. Also,
some specific statements in the narratives were very explicit in support of that notion, for
example, “they” (the activities) “are a fun distraction that doesn’t require my eyes to be glued
883
to a screen.” Considering these, the researchers plan to increase the stations with such
activities in the Brain Booth to be able to accommodate more students at the same time
interested in those activities. At the same time, the Brain Booth will continue to provide the
full plethora of offered activities, because this is an important point that makes the initiative
unique and innovative. It is also well regarded by the Brain Booth visitors, as stated directly
for example: “I really like using the brain booth and I think it’s a great idea. There are many
options for people to choose what they like best, so it’s an inclusive space.” The study showed
that the Brain Booth is helping students, at their point of need, with some of the major factors
that affect their academic performance the most, as well as with their overall well-being. To
sum it all, as one of the participants stated, “Always feel much better after spending time
there.” As a future step, the researchers will submit a grant proposal to conduct a feasibility
study and develop a replicable model of embedding the Brain Booth initiative in high school
settings as well.
Acknowledgments
The authors gratefully acknowledge the assistance with the systematic literature review of
librarian Ann Fuller, and three undergraduate research assistants who were supported with
funding by the University Office of Research and the Faculty Research Committee at Georgia
Southern University.
References
Azevedo, M. L. de, & Menezes, B. C. (2020). Effects of the Mindfulness-Based Cognitive
Therapy Program on stress, self-efficacy and mindfulness in university students. SMAD
Revista Electrónica Salud Mental, Alcohol y Drogas, 16(3), 1-11.
doi:10.11606/issn.1806-6976.smad.2020.165513
Bamber, M. D., & Schneider, J. K. (2022). College students' perceptions of mindfulness-
based interventions: A narrative review of qualitative research. Current Psychology, 41,
667-680. doi:10.1007/s12144-019-00592-4
Bartel, K., Huang, C., Maddock, B., Williamson, P., & Gradisar, M. (2018). Brief school‐
based interventions to assist adolescents’ sleep‐onset latency: Comparing mindfulness
and constructive worry versus controls. Journal of Sleep Research, 27(3), 1–9.
doi:10.1111/jsr.12668
Bartos, J. L., Posadas, P. M., & Krageloh, C. (2022). Perceived benefits of remote yoga and
mindfulness program for student musicians during COVID-19. The Humanistic
Psychologist, 13(1), 3-4. doi:10.1037/hum0000277
Colaianne, B. A., Galla, B. M., & Roeser, R. W. (2020). Perceptions of mindful teaching are
associated with longitudinal change in adolescents’ mindfulness and compassion.
International Journal of Behavioral Development, 44(1), 41–50.
doi:10.1177/0165025419870864
884
Chun, T., Cooksey, A., & Kleinmeyer, L. (2021). Incorporating Mindfulness in the School
Library. Knowledge Quest, 50(1), 16-22.
Duffy, B., Rose-Wiles, L. M., & Loesch, M. M. (2021). Contemplating library instruction:
Integrating contemplative practices in a mid-sized academic library. Journal of Academic
Librarianship, 47(3), 102329. doi:10.1016/j.acalib.2021.102329
Egan, H., O'Hara, M., Cook, A., & Mantzios, M. (2022). Mindfulness, self-compassion,
resiliency, and wellbeing in higher education: A recipe to increase academic performance.
Journal of Further and Higher Education, 46(3), 301-311.
doi:10.1080/0309877X.2021.1912306
Finkelstein-Fox, L., Park, C. L., & Riley, K. E. (2018). Mindfulness and emotion regulation:
Promoting well-being during the transition to college. Anxiety, Stress & Coping, 31(6),
639–653.
Fung, J., Kim, J. J., Jin, J., Chen, G., Bear, L., & Lau, A. S. (2019). A randomized trial
evaluating school-based mindfulness intervention for ethnic minority youth: Exploring
mediators and moderators of intervention effects. Journal of Abnormal Child Psychology,
47(1), 1–19. doi:10.1007/s10802-018-0425-7
Gibson, N. S., & Regan, M. T. (2021). On the Journey to a contemplative library: Reflections
from a professional community of practice. International Information & Library
Review, 53(3), 64–271. doi:10.1080/10572317.2021.1949523
Hartel, J., Nguyen, A. T., & Guzik, E. (2017). Mindfulness meditation in the
classroom. Journal of Education for Library and Information Science, 58(2), 112-115.
doi:10.12783/issn.2328-2967/58/2/6
Karadjova, K.G. (2018). Mindfulness and Gamification in the Higher Education Classroom:
Friends or Foes?, The International Conference on Higher Education Advances
(HEAd'18), June 2018, Valencia, Spain. Valencia: Editorial Universitat Politècnica de
València.
Karadjova, K.G. (2019) “Mindfulness Experiences at the Library Brain Booth,” in R. Moniz
and J. Covlin (Eds.), Recipes for Mindfulness in Your Library: Supporting Resilience and
Community Engagement, (pp. 37–43). Chicago, IL: ALA.
Karadjova-Kozhuharova, K.G., & Baker, R. (2022). Mindfulness And Technology: Can They
Be Two Sides Of The Same Coin?, EDULEARN22 Proceedings, pp. 9074-9082.
doi:10.21125/edulearn.2022.2181
Loucks, E. B., Nardi, W. R., Gutman, R., Saadeh, F. B., Li, Y., Vago, …& Harrison, A.
(2021). Mindfulness-Based college: A stage 1 randomized controlled trial for university
wellbeing. Psychosomatic Medicine, 83, 602-614. doi:10.1097/PSY.0000000000000860
Mahmood, L., Hopthrow, T., Randsley de Moura, G. (2016). A moment of mindfulness:
Computer-Mediated mindfulness practice increases state mindfulness. Public Library
Science. 11(4). doi:10.1371/journal.pone.0153923
Placito-De Rango, D. (2018). Situating the post-secondary instructor in a supportive role for
the mental health and well-being of students. International Journal of Mental Health and
Addiction, 16(2), 284-290. doi:.1007/s11469-017-9740-4
885
Ramasubramanian, S. (2017). Mindfulness, stress coping and everyday resilience among

emerging youth in a university setting: A mixed methods approach. International Journal
of Adolescence and Youth, 22(3), 308-321. doi:10.1080/02673843.2016.1175361
Reeve, J., Jindal, R., Bartle, C., & Stokes, N. (2021). Using creative and contemplative
pedagogy to promote learning efficacy and wellbeing within first-year, undergraduate
students. Gateway Papers: A Journal of Education and Pedagogic Research, 2(1), 5.
doi:10.3943/gp.34
Ruhlmann, E. (2017). Mindful librarianship. American Libraries, 48(6), 44-47.
Schwind, J. K, McCay, E., Beanlands, H., Martin, L.S., Martin, J., & Binder, M. (2017).
Mindfulness practice as a teaching-learning strategy in higher education: A qualitative
exploratory pilot study. Nurse Education Today, 50, 92-96.
doi:10.1016/j.nedt.2016.12.017
Seemiller, C., & Grace, M. (2016). Generation Z goes to college. San Francisco, CA: Jossey-
Bass.
Vitha, M. F. (2022). Contemplative Fridays: Mindfulness pedagogy in the general chemistry
classroom. Journal of Chemical Education, 99(7), 2441-2445.
doi:10.1021/acs.jchemed.2c00159
Warren, S., & Deckert, J. C. (2019). Contemplative practices for self-care in the Social Work
Classroom. Social Work, 65(1), 11-20. doi:10.1093/sw/swz039
886
Peer coaching in a leadership development program: the role of

developmental relationships
Department of Management, Ca’ Foscari University of Venice, Italy.
Abstract
Notwithstanding the increasing widespread of the peer coaching technique for
leadership development in the business settings, there is a remarkable scarcity
of theoretical and empirical research on the adoption of this technique and on
the measurement of its effects, especially in the higher education. This study
describes the implementation of peer coaching in a leadership course
delivered in a public Italian University to Master’s degree students and
presents two scales to measure the effectiveness of and the satisfaction with
this technique. The results from the adoption of these scales and the students’
reflections on their learning experience provide suggestions on how to
encompass developmental relationships within the design of a leadership
learning journey in the higher education context.
Keywords: leadership development; peer coaching; experiential learning.

Peer coaching in a leadership development program
1. Introduction
Higher education institutions and, in particular, business schools recently have devoted
attention to the design of educational programs aiming at developing students’ leadership
skills vital for contemporary organisations (Allen et al., 2022). However, to date, most of
leadership programmes in higher education still strongly rely on cognitive training, limiting
the transferability of the learning into the participants’ real-life context (Day et al., 2014).
Social cognitive and constructivist learning orientations (Allen et al., 2022) maintain that
behavioural change in a learning programme can be effectively attained providing
participants opportunities to interact with the environment and derive meaning from their
experience. In this regards, developmental relationships are identified as a key component of
effective leadership development (Leskiw & Singh, 2007). Specifically, experiential learning
advocates the value of consulting with knowledgeable peers (Schön, 1983), as it allows
individuals to assist each other in learning and drawing insights into personal behaviour
(Parker et al., 2008). In this regards, peer-coaching has been increasingly employed in
leadership development programs (Eriksen et al., 2020). In a peer coaching session, the peers
alternate between being the coach and being the coachee and reciprocally listen to each other
and ask open-ended or probing questions to trigger partner’s reflections and increase his/her
self-understanding (Parker et al., 2008). Notwithstanding the increasing widespread of the
peer coaching technique in the business settings, there is a remarkable scarcity of theoretical
and empirical research on the adoption of peer coaching and the measurement of its effects,
especially in the higher education (Eriksen et al., 2020).
The next sections will review the literature on peer coaching and will describe in detail the
implementation of the peer coaching technique in a leadership course delivered in a public
Italian University to master’s degree students. The presentation of the methodology adopted,
the related peer-coaching measurement scale developed and the analysis of the students’
reflections on their learning experience will provide suggestions on how to encompass
developmental relationships within the design of a leadership learning journey in the higher
education context.
The foundation of peer coaching as a way for managers and staff to advance their careers
draws on the breakthrough work in clinical supervision by Goldhammer (1969) (Britton &
Anderson, 2010). Peer coaching is a sustained, structured interaction that is typically dyadic
in nature (D’Abate et al., 2003). The coach-coachee relationship is lateral and mutual and it
is centered on the development of a particular set of skills or on performance enhancements,
as it entails the coach supporting the coachee’s improvement by providing feedback (Hagen
et al., 2017).
888
Peer coaching is considered an effective learning technique, as it allows individuals to

develop their skills and knowledge in a particular area (Hagen et al., 2017). This process
allows two or more colleagues to reflect on current practices, expand and refine skills, share
ideas, teach one another, and solve problems in the workplace. Peer coaching has become
increasingly popular amongst teachers to enhance their teaching skills but also among
students to enhance their personal skills (Slatter & Simons, 2001; Lu, 2008). Furthermore,
those who participate in peer coaching experiences are better able to find their own answers
to problems whether they occur at work or even in their personal life (Murrihy, 2009).
Consistent evidence displays three key components of peer coaching which are trust,
reflection, and solid communication skills. The literature explicitly states how crucial it is to
build a peer coaching relationship based on trust (Jackson, 2004; Robertson, 2005).
According to Wenger et al. (2002), trust in peer coaching grows over time, similarly to a
community of practice where members get to know one another better (Wenger et al., 2002).
Reflection is another essential component of peer coaching. According to Jackson (2004),
peer coaching is fundamentally a reflective activity, hence participants must be adept at
reflection (Loughran, 2002; O'Connor & Diggins, 2002). According to Jan Robertson's
(2005) coaching leadership model, coaches can better enable partners' ability to critically
reflect on their issues and topics by developing their reflective interviewing abilities. The
capacity to actively listen is fundamental to have fruitful reflective conversations (Robertson,
2005). Finally, solid communication skills include the ability to give non-evaluative
criticism. Rice (2012), conducting research on a group of faculty members in higher
education, discovered that people who engaged in "formative discourses", while receiving
peer coaching, appeared to have a positive opinion of the process. Thus, through peer
coaching, leaders can gain deeper insights, develop valuable skills, and create meaningful
connections with their peers.
3. Method
3.1. Structure of the Leadership development programme

The Leadership course analyzed in this study is delivered, as elective, in a public University
in Italy to Master’s degree students. The course lasts six weeks during which students
participate in 15 in-person classes for 30 hours overall. After the sixth week students take the
exam and subsequently, on a voluntary basis, they can continue their leadership learning path
for other 16 weeks. The methodological approach implemented in the course draws on the
experiential learning cycle (Kolb, 1984), the intentional change process (Boyatzis, 2006) and
the whole person learning pedagogy (Hoover et al., 2010). During the in-class lectures,
students familiarise with the theoretical framework of emotionally intelligent and resonant
leadership (Goleman et al., 2013), understanding in-depth the characteristics of each
leadership style (Visionary; Coaching; Affiliative; Democratic) and the techniques to put
889
each style into practice in real-life contexts. At the end of the course (week 6), students are
asked to submit a personal leadership learning plan reporting: i) the developmental
objectives, namely the specific leadership styles they want to develop; ii) the concrete actions
they wish to undertake to practice their leadership in real-life settings; iii) the contexts and
the persons with whom to experiment the new behaviours; and iv) a timeframe for action. In
the subsequent 16 weeks students are continuously stimulated to put into practice their
personal leadership learning plan in a variety of environments and with different people.
During their learning journey they are engaged in at least three peer coaching sessions that
consist of a conversation with a classmate to share their experimentation of the leadership
behaviours. When in pair, students have the opportunity to learn alternative ways to improve
a specific behaviour and progressively become aware of the learning outcomes attained.
During the course, the instructor introduces the peer-coaching technique to the students and
provides a template to conduct the sessions. After each session students are invited to report,
on a digital platform, their reflections on: i) the main topics discussed; ii) the perceived
benefits of the session; iii) the coach’s questions that helped them to construct meaning from
the learning experiences. At the end of the 16th week, a survey is administrated to the students
to collect their opinions on the peer coaching experience.
3.2. Sample
A group of 112 Master’s degree students who completed all the activities of the leadership
course serves as a sample of this study. The sample has an average age of 23.93 (SD 1.19),
and 66.1% were female. Most of the sample had previous working experiences (59.8%).
3.3. Measures
To measure the effectiveness and the level of satisfaction of the peer coaching activity, we
developed two scales.
Peer coaching reflection. Based on the work of Parker et al. (2008), we developed four items
measuring the extent to which students make sense of their learning process and the progress
attained through the peer coaching experience. Sample items are “The conversation with my
coach helped me to become aware of the learning progress I was making over time during
the course” and “The conversation with my coach helped me to identify the behaviors that
led to positive results in my leadership learning experience”. Items were measured on a 5-
point scale in which 1=totally disagree and 5=totally agree. We performed an Exploratory
Factor Analysis (EFA) using Maximum Likelihood and Promax rotation, which shows
adequate loading magnitude (from .63 to .77), and inter-item correlation (from .42 to .57,
p<.01). The four items account for 49% of the variance, Cronbach’s alpha is .79.
Peer coaching satisfaction. Drawing on Baron and Morin (2009), five items capturing
students’ level of satisfaction with the peer coaching activity were developed and measured
890
on a 5-point scale in which 1=totally disagree and 5=totally agree. Sample items are “I felt
comfortable sharing my experience with my peer coach” and “I enjoyed the sessions with my
peer coach”. EFA showed the items account for 48.2% of the variance, with adequate factor
loadings (ranging from .62 to .75) and inter-item correlations (from .33 to .63). Cronbach’s
alpha is .81.
4. Results
When asked about the extent to which peer coaching helped them make sense of their learning
experience and develop a stronger awareness of their learning achievement, students tend to
agree on the positive influence of this technique (M=4.11, SD=0.61). Verbal comments
provided by the student reported that: “It helped me to compare myself with a totally foreign
person. The comparison made me think about many aspects that I did not consider before.”;
“I was able to express myself freely without feeling intimidated, I always felt listened to,
understood. My partner helped me figure out how to change some of my behaviors. She has
allowed me to discover further perspectives and ways of acting.”; “When I had to tell a
situation in which I had been able to put into practice one of the styles seen in the classroom,
it was useful to me because I thought about the results obtained and I was able to make a
comparison with the past ones and make sure to make the good decision next time.”; “At the
end of each session, thinking back on what was said, I was always amazed by all the things
that I managed to highlight and on which I reflected”.
This empirical evidence shows that peer coaching has several benefits in terms of
development, self-reflection and awareness. Students particularly appreciated looking back
at their experience and making sense of it analysing their progresses and results, and
discussing with peers about possible alternative behavioural strategies.
Concerning the level of satisfaction with the peer coaching sessions, students agreed they
perceived a high level of interpersonal comfort and commitment in their relationship with
their peers (M=4.43, SD=0.51). In their verbal comments they report: “Peer coaching was
undoubtedly helpful, particularly since you have to communicate with individuals you do not
know”; “I was compelled by this to be as specific as I could be about my goals and
personality traits. I was encouraged by peer coaching to communicate more precisely and
avoid using unclear words.”; “The peer coaching exercise was very beneficial. Speaking
with strangers really allows me to be honest”.
These results show a high level of satisfaction with the peer coaching experience,
highlighting the positive environment built by the process, and underlining the impact
perceived by the participants on themselves.
891
5. Discussion
There are several contributions offered by this study to the research on leadership
development and to the design of educational initiatives.
First, this study highlights the relevance of a learning technique based on an interpersonal
relationship which supports the leadership development process. The peer coaching sessions
help participants to give meaning to their learning experiences and at the same time represent
an appointment which requires them to have put into practice the leadership behaviors they
identified as critical in their learning plan. Since experimenting the leadership behaviors in
real and different contexts represented one of the most relevant difficulties for those who
attended the course, the mutual discussions with the peer-coach allowed the participants to
take inspiration from their peer’s experiences and helped them to: identify more situations in
their real life to use as learning environments; identify alternative behaviors to practice in
specific situations; see their behaviors and their impact from a different perspective; reflect
on future behaviors to adopt in similar situations; appreciate their growth over time; share
difficulties and understand that these difficulties are often common. This is consistent with
previous studies highlighting the role of peer coaching in providing support and
encouragement during the development path, and in allowing learning from one another
(Anderson et al., 2005; Donegan et al., 2000).
Second, this study offers two scales to measure the peer coaching process from the
perspective of those who experienced it. Measuring the perceived effectiveness and
satisfaction of the peer coaching experience helps to design learning initiatives that keep the
different dimensions of this process under control, and allows to avoid the risk of unfruitful
relationships. In addition, these measures help the participants to become aware of the results
of their peer coaching experience and to reflect on the characteristics of the relation
undertaken, in order to improve it if needed.
Third, this study provides evidence of an educational course characterized by an integration
of different learning tools to support personal development. Indeed, this course combines
theoretical, experiential and interpersonal dimensions, and consequently encompasses three
different learning paradigms to make leadership development more successful. This
combination of dimensions and related tools can enhance the learning of effective leadership
behaviors by creating a process where the participants understand the characteristics of the
leadership behaviors, define a personal learning plan to put them into practice, experiment
them in real life contexts and interact with others on this experience obtaining feedback aimed
to improve the subsequent experimentation.
One limitation of this study is related to the use of self-reported measures by the participants
and their consequent potential bias. Although the satisfaction coming from the peer coaching
experience requires a personal evaluation, the measurement of the level of reflection on the
892
learning experience induced by peer coaching might benefit from a third-party assessment.
Moreover, future research may consider implementing quasi-experimental design to
investigate the impact of the peer coaching activity on the change participants experienced
in their leadership behaviours.
References
Allen, S. J., Rosch, D. M., & Riggio, R. E. (2022). Advancing leadership education and
development: Integrating adult learning theory. Journal of Management Education,
46(2), 252–283.
Anderson, N. A., Barksdale, M. A., & Hite, C. E. (2005). Preservice teachers’ observations
of cooperating teachers and peers while participating in an early field experience. Teacher
Education Quarterly, 32(4), 97–117.
Baron, L., & Morin, L. (2009). The coach-coachee relationship in executive coaching: A field
study. Human Resource Development Quarterly, 20(1), 85–106.
Boyatzis, R. E. (2006). An overview of intentional change from a complexity perspective.
Journal of Management Development, 25(7), 607–623.
Britton, L. R., & Anderson, K. A. (2010). Peer coaching and pre-service teachers: Examining
an underutilised concept. Teaching and Teacher Education, 26, 306– 314.
D'Abate, C.P., & Eddy, E. R. (2007). Engaging in personal business on the job: Extending
the presenteeism construct. Human Resource Development Quarterly, 18(3), 361-383.
Day, D.V., Fleenor, J.W., Atwater, L.E., Sturm, R.E., & McKee, R.A. (2014). Advances in
leader and leadership development: A review of 25 years of research and theory. The
Leadership Quarterly, 25(1), 63–82.
Donegan, M. M., Ostrosky, M. M., & Fowler, S. A. (2000). Peer coaching: Teachers
supporting teachers. Young Exceptional Children, 3(9), 9–16.
Eriksen, M., Collins, S., Finocchio, B., & Oakley, J. (2020). Developing students’ coaching
ability through peer coaching. Journal of Management Education, 44(1), 9–38.
Goldhammer, R. (1969). Clinical supervision: Special methods for the supervision of
teachers. New York, NY: Holt, Rinehart, & Winston.
Goleman, D., Boyatzis, R.E., & McKee, A. (2013). Primal leadership. Unleashing the power
of emotional intelligence. Boston MA: Harvard Business Review.
Hagen, M. S., Bialek, T. K., & Peterson, S. L. (2017). The nature of peer coaching:
definitions, goals, processes and outcomes. European Journal of Training and
Development, 41(6).
Hoover, J. D., Giambatista, R. C, Sorenson, R. L., & Bommer, W. H. (2010). Assessing the
effectiveness of whole person learning pedagogy in skill acquisition. Academy of
Management Learning and Education, 9(2), 192-203.
Jackson, P. (2004). Understanding the experience of experience: a practical model of
reflective practice for coaching. International Journal of Evidence Based Coaching and
Mentoring, 2(1), 57–67.
893
Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and

development. Prentice Hall.
Leskiw, S., & Singh, P. (2007). Leadership development: learning from best practices.
Leadership & Organizational Development Journal, 28(5), 444-464.
Loughran, J. J. (2002). Effective reflective practice: In search of meaning in learning about
teaching. Journal of Teacher Education, 53(1), 33–43.
Lu, H. L. (2010). Research on peer coaching in preservice teacher education–A review of
literature. Teaching and teacher education, 26(4), 748-753.
Murrihy, L. (2009). Coaching and the growth of three New Zealand educators: A
multidimensional journey (Unpublished doctoral thesis). University of Waikato,
Hamilton, New Zealand.
O’Connor, A., & Diggins, C. (2002). On reflection: Reflective practice for early childhood
educators. Lower Hutt, New Zealand: Open Mind.
Parker, P, Hall, D., & Kram, K. E. (2008). Peer coaching: a relational process for accelerating
career learning. Academy of Management Learning & Education, 7(4), 487–503.
Rice, G. (2012). Formative dialogues in teaching: Nonthreatening peer coaching. The Journal
of Chiropractic Education, 26(1), 62–67.
Robertson, J. (2005). Coaching leadership: Building educational leadership capacity
through coaching partnerships. Wellington, New Zealand: NZCER Press.
Schön, D.A. (1983). The reflective practitioner: How professionals think in action. New
York, NY: Basic Books.
Slater, C. L., & Simmons, D. L. (2001). The Design and Implementation of a Peer Coaching
Program. American Secondary Education, 29(3), 67–76.
Wenger, E., McDermott, R., & Snyder, W. M. (2002). Managing organizational knowledge
through communities of practice. Boston, MA: Harvard Business School Press.
894
The current HE classroom: promoting new types of learning,

executive function processes and strategies to foster students’
motivation and academic success
Genny Villa
Université de Montréal, Canada.
Abstract
Students' academic success in this digital, globalized era requires their
mastery of processes such as goal setting, planning, prioritization,
organization, flexibility to change, storage/manipulation of information in
working memory, and self-monitoring. These processes are called executive
function (EF) processes. It is important to integrate strategies that
systematically address these processes in the classroom to help students
understand how they think and how they learn. This paper provides a paradigm
for understanding/helping students integrate strategies involving EF
processes; it describes how strategic, systematic instruction and adaptations
to classroom-work and tasks may benefit all students, while effectively
addressing the needs of students who exhibit significant weaknesses in these
processes. Furthermore, individuals and organizations involved in HE express
the need for other important types of learning that do not readily emerge from
Bloom's taxonomy: e.g., learning to learn, leadership, interpersonal skills,
ethics. This paper introduces D. Fink’s proposal for a broader taxonomy of
significant learning.
Keywords: Pedagogical engineering approach; executive function processes;
Dee Fink’s Taxonomy of Significant Learning; new learning/teaching
strategies.

The current HE classroom
1. Introduction
Research (e.g., Bandura, 2000; Brunstein, Schultheiss, & Grassman, 1998; Helliwell, 2003;
Kasser & Ryan, 1996; Meltzer, Reddy, Sales Pollica, & Roditi, 2004b; Pajares & Schunk,
2001) shows that academic success for all students, is inextricably linked to their motivation,
effort, persistence, academic self-concept, and self-efficacy. Their academic success in this
digital and globalized era requires their mastery of processes such as goal setting, planning,
prioritization, organization, flexibility to change, storing and manipulating information in the
working memory, and self-monitoring. Such processes are referred to as executive function
(EF) processes.
As educators, we need to realize that it is increasingly important not only to integrate
strategies to systematically address these processes in the classroom to help students
understand how they think and how they learn, but also that teaching, like all professions,
should allow research to inform its practice. In this paper we provide a paradigm for
understanding and teaching students strategies that integrate executive function processes. It
describes how strategic and systematic instruction and adaptations to classroom work and
assignments can benefit all students, while effectively addressing the needs of a large number
of students who exhibit significant weaknesses in these processes.
As is currently known, individuals and organizations involved in HE are expressing the need
for other important types of learning that do not readily emerge from Bloom's taxonomy, e.g.,
learning to learn, leadership and interpersonal skills, ethics, communication skills, character,
tolerance, and the ability to adapt to change. This state of affairs requires a pedagogical
engineering approach and new types of learning that go far beyond the cognitive domain of
Bloom's taxonomy and even, beyond cognitive learning itself. It is also in response to this
need that we present here Fink’s proposal for a new and broader taxonomy of significant
learning.
2. Executive Function
Executive Function (EF) refers to a family of control functions that are essential to
concentrate, pay attention and think, when going on automatic, acting on intuition or the
initial impulse may be ill-advised (Burgess & Simons 2005; Diamond, 2012). These
functions depend on a neural circuit in which the prefrontal cortex plays a prominent role
(Anderson, Jacobs, & Anderson, 2008; Craik, Bialystok, Gillingham & Stuss, 2018). There
is general agreement (e.g., Miyake et al., 2000) that there are fundamental EFs: Attention;
Inhibition or inhibitory control; working memory; and cognitive flexibility. These form the
basis for higher-order EFs such as reasoning, problem solving, and planning (Christoff,
Ream, Geddes, & Gabrieli, 2003; Collins & Koechlin, 2012; Lunt et al., 2012). Executive
896
Genny Villa
function processes provide us with the “big picture” of a situation, a theme or a task at hand,
and allow us to switch flexibly between this big picture and its relevant details.
There exist six processes that researchers (e.g., Christoff, et al. 2003; Collins et.al., 2012;
Diamond, 2012 Diamond, 2012; Lunt et al., 2012; Meltzer, et al., 2004b; Meltzer & Krishnan,
2007) have found to be key to executive function: goal setting; planning and prioritizing;
organizing; retaining and manipulating information in working memory; flexible switching;
and self-monitoring and self-control. These processes are referred to as executive function
processes, and are of vital importance to all aspects of academic performance because, since
as we all know, students' academic success in this digital and globalized age requires not only
their technological expertise but, more importantly, their mastery of the above-mentioned
processes.
As educators, we may have occasionally noticed that “although some students demonstrate
strong conceptual reasoning skills, these same students are inefficient with their work and
have difficulty demonstrating what they know in the classroom; their study skills and test
performance are compromised; and their academic grades do not reflect their actual
intellectual capacity. What we are usually unaware of is that for these students having
executive function weaknesses, their conceptual reasoning skills may be stronger than their
production and productivity” (Meltzer, 2010, p.28). Meltzer et al., (2007) studies found that
students with executive function challenges, have particular problems with academic tasks
that involve the coordination and integration of different subskills, such as beginning writing
tasks summarizing information; taking notes; studying; planning, executing, and completing
projects on time; maintaining attention during long, detailed assignments; and remembering
to submit their work on time. Furthermore, their problems with attention, working memory,
and inhibition create additional challenges, and their productivity with classwork and
homework is affected.
2.1. Why Are Executive Function Processes So Important?

Research (e.g., Bandura, 2000; Helliwell, 2003; Kasser & Ryan, 1996; Sheldon & Elliot,
1999), as already mentioned, has largely supported that academic success for all students, is
inextricably linked to their motivation, effort, persistence, academic self-concept, and self-
efficacy. These cognitive and motivational processes are connected with students’ use of
executive function strategies, as well as with their academic performance (Meltzer &
Krishnan, 2007). As such, strategies that address executive function processes provide an
entry point for improving academic performance. Along with Meltzer (2010) we argue that
when students learn and apply these strategies effectively, they become more efficient and
thus begin to succeed academically. Academic success in turn boosts self-confidence and
academic self-concept, which results in more focused effort so that students’ hard work is
897
targeted strategically toward specific goals. The learning environment and the instructional
methods and materials all play an important role in mediating this cyclical relationship.
For all students, but particularly for students with learning and/or attention difficulties,
effective strategies and focused effort will help them to bridge gaps between their skills and
the academic demands they face (Meltzer et al.,2004b). In other words, their academic
performance is often dependent on their knowledge of and willingness to use strategies.
Students, therefore, need to understand their own learning styles. This will help them discover
which strategies work well for them, as well as why, where, when, and how to apply each
specific strategy. This understanding is referred to as metacognition, or the ability to think
about one’s own thinking and learning. Metacognition underlies students’ use of executive
function processes.
Executive function challenges may take different forms:
• Problems with Task Initiation involve procrastination or failing to do what we know we

are supposed to do and, trouble transitioning from one task to another.
• Organizing, Planning and Prioritizing Weaknesses show up when the student has trouble
committing to plans; waits until the last minute to start a project; underestimates time to
complete a task; gets caught up in details and misses the “big picture”.
• Poor Impulse Inhibition is made evident when the student talks before thinking; does not
consider long-term impact of an action; and interrupts or blurs out socially inappropriate
comments.
• Trouble Shifting Tasks implies having difficulty stopping an activity and moving to the
next; having trouble tolerating unpredictable events or schedule changes; and becoming
"stuck" on a particular line of thinking.
• Trouble with Emotional Control is associated with being over-reactive to small
problems; easily upset and having trouble calming down; and unclear about what triggers
an emotional reaction.
• Poor working Memory encompasses trouble with multi-step routines; problems
following complicated group conversations and trouble multi-tasking at work.
• Trouble Keeping Material Organized show up when students have trouble keeping track
of belongings, files; and their home and work environment is messy or disorganized.
• Poor Self-Monitoring Skills imply difficulty noticing careless errors; trouble reflecting
on what works and what does not work; and poor awareness of others.
898
Genny Villa
3. Importance of Teaching Explicit Strategies
3.1. For Goal Setting

Currently, to achieve an optimal balance between a curriculum-centered approach of
education and a student-centered approach, teachers need to share the goal-setting process
with their students, to help them become independent learners. Research findings (e.g.,
Pintrich & Schunk, 1996; Schunk, 1995, Meltzer, 2010) support the importance of teaching
goal-setting strategies and also show (1) a strong cyclical relationship between the ability to
set personal goals and sustain higher levels of motivation as well as the development of
positive self-efficacy (which promotes success); (2) that students may be more motivated
and interested when they engage in tasks of their choosing, and that they may work harder
on self-made goals than they do when they work toward the plans and expectations of others;
(3) that when teachers share the goal-setting process with students and give them appropriate
guidance, students can learn how to set reasonable goals, which increases their motivation
and willingness to persist.
Researchers (De la Paz, 2007; Graham, 2006 in Meltzer, 2010) suggested that when students
learn to set effective personal goals, and when their learning is supported by meaningful
feedback (self-assessment and teacher feedback) as well as by appropriate learning
experiences and strategies, they show improvements in their academic performance.
According to De la Paz, and Graham, given a complex process such as writing, students are
more easily able to manage the multiple demands of the task when they learn how to self-
regulate their efforts through goal setting and planning. Research evidence also shows that
this is true both for typically developing students and for students with learning challenges
(Carlson, Booth, Shin, & Canu, 2002, in Meltzer, 2010). Indeed, while all students benefit
from learning how to set goals, students who struggle with academic tasks benefit even more.
3.2. Some Strategies to Help Students Setting Goals

Meltzer (2010) argues that “given the pace of the curriculum, the diversity of learning styles
represented in the classroom, and the large volume of content, many educators find it difficult
to make the time to teach goal setting, as it is not a component of the content-centered
curriculum” (p.81). Nevertheless, setting goals should be taught through classroom
accommodations and direct, systematic instruction.
The following common instructional accommodations provide students with scaffolds that
help them with goal setting. These accommodations constitute “best practices”.
• Rubrics: Allow students to have a clear idea of what the teacher is looking for and how
performance may be assessed.
899
• Visual representations of projects completed by previous students (pictures or actual

examples), can lead current students through a sample prior to giving them a similar
assignment.
• The use of calendars and phased timelines- Calendars provide visual reminders of due
dates, as well as schemas for setting and achieving short-term goals that contribute to the
successful attainment of the larger, final goal (Newport,2006; Villa, 2018, 2021).
• Direct, systematic instruction allows students benefit from explicitly learning how to set
goals. It entails three key components considered the most effective in goal setting:
• Self-understanding/Self-knowledge is an important component of a resilient learning
style, which in turns ensures that students set reasonable and appropriate goals;
• Grasp of the “big picture,” or schema, with respect to broad goals for the year or the
term, along with more specific content-or lesson-related goals, and
• Valuing the task –research (e.g., Villa, 2018, 2021) shows that it is essential to
understand that individuals are motivated to achieve goals that are in line with their own
strong desires and values; and that individuals have a deep-rooted desire to have an effect
on their environment and to obtain results that they value within their own contexts.
Furthermore, self-determination theory suggests that people are motivated to act in a manner
that is consistent with their values (Deci & Ryan, 2000). These are powerful drives that exist
in all learners.
3.3. Strategies for Planning and Prioritizing

Planning and prioritizing are essential for success in today's classrooms. Current emphasis
on long-term projects and expectations for independent work have increased the requirement
for all students to plan and prioritize independently. Thus, students' ability to arrange tasks
in order of importance and use strategies to complete them on time is directly related to their
success and their ability to achieve their goals in all academic areas. Effective planning and
prioritizing leads to efficient time management, which in turn increases productivity, relieves
stress, and has a positive effect on student learning and performance (Misra & McKean, 2000
as cited in Meltzer 2010).
Time management involves interactions among several different executive function
processes, including goal setting, prioritizing goals, planning goal achievement, and
prioritizing tasks to accomplish objectives (Britton & Tesser, 1991). These processes are
fairly complex for most students; therefore, it is important to begin by focusing on the
foundational processes involved in time management: (1) Knowledge of time; (2) knowledge
of task; (3) Prioritizing tasks; and (4) Monitoring progress. (Villa, 2021; Newport, 2006;
Wolters, 2003).
900
Genny Villa
Students who implement time management strategies are often considered to be more self-
regulated, more aware of their thinking processes, and more able to manage their learning
across contexts than those who procrastinate (Wolters, 2003). Efficient time management is
a combination of innate abilities and learned strategies. However, the challenges of time
management have become increasingly complex. Daily packed schedules and “timeless”
distractions like video games and social networking sites (e.g., Facebook and MySpace), can
easily reduce a student’s task efficiency and productivity.
3.4. Strategies for Accurate Time Estimation

Students often make independent choices about the order in which they will approach their
tasks and how they will schedule their time. Although they typically have established
routines, students often do not reflect on the efficiency of their habits. (Villa, 2021).
Educators can support the ongoing development of accurate estimation skills by providing
structured opportunities for reflection after tasks (Teaching assistants may guide this
exercise). For students to allot adequate time to complex projects, studying for tests, or even
accomplishing everyday tasks (e.g., eating lunch, includes preparing food, or waiting in line
to pay for it) they need to learn strategies for doing so. Perfecting the ability to estimate the
time required for a task is a gradual process that improves with practice and age.
Educators can help students become accurate time estimators with estimation strategies such
as:
• Dividing a large-scale project into its smaller components, and then estimate the
time it will take to complete each part.
• How long will it take breaking the task down into multiple steps/estimate the time
to allocate to each step.
By helping students get a clear understanding of the different types of activities (obligation,
aspiration, and negotiation) in their daily lives, teachers can provide them with a method of
prioritization that results in the creation of a productive and efficient schedule.
4. Concern Regarding University Teaching

Research (e.g., Fink, 2003, 2013; Saroyan, & Amundsen, 2004) reports concerns regarding
learning goals not going beyond a understand-and-remember type of learning; professors
having difficulty integrating additional activities to the lectures and discussions; and
programs not considering the challenge and complexity of good teaching. Furthermore, the
“get busy with research and publishing” pressure is considerable, even beyond tenure, to
maintain strong research (and teaching) records.
901
4.1. Lecturing
What kinds of results does lecturing, even good lecturing, produce? Research (e.g., Saroyan,
2000), shows that lecturing has limited effectiveness in helping students: (1) Retain
information after a course is over; (2) Develop an ability to transfer knowledge to novel
situations; (3) Develop skill in thinking or problem solving; (4) Achieve affective outcomes,
such as motivation for additional learning or a change in attitude. However, lecture-based
instruction has been shown to be most effective when the goal is to transmit information, and
when organization and clarity are desired (Saroyan, 2000).
4.2. What are some common faculty concerns?

Research (e.g., Dee, F.L. 2001, 2013) have found that educators are mostly concerned about:
(1) Low student attendance; (2) Low energy in class discussions; (3) Focus on grades rather
than learning; (4) Larger textbooks = More material to cover; (5) Loss of joy in teaching; and
(6) Feeling unsupported to embark on change. Fifty-five faculty participating in a workshop
we carried out last September 2022, also reported their concerns regarding Motivating
students and focus on Grades (that may be related to (1), (2) and (3) above, while shared (See
Figure 1. below) and added to this list concerns such as: Ability to publish and Secure grants;
Research; Budget/Money; Teaching objectives; Learning objectives; Students’ time
management; Teaching load; Time for practical activities; Technology; Collaboration,
CVTC, among others.
Figure 1. Common faculty concerns. Source: Villa’s workshop, Sept. 2022).
4.3. Common student concerns

These same participant professors expressed what, according to them, are common student
concerns (See Fig. 2. below), which remarkably highlights other aspects that are not
mentioned in research reports (e.g., Courst & McInerney, 2003; Fink, 2013) such as Time
(Time management; free time, to study, time investment, time pressure, no time to do sports,
902
Genny Villa
etc.); Self-management skills; Getting a job/Securing a job; Girlfriend/Boyfriend; Work-Life

balance; Family responsibilities; Tuition cost, etc.
Common Student Concerns reported by research include: (1) No “Hands-on Learning”:
Transmission of information via lectures and workbooks; (2) Uninteresting courses; (3)
Classes reduced to note-taking and cramming for exams; (4) Difficulty perceiving value and
application of course content; (5) Larger textbooks = More material to cover, more
memorization, more cramming; (6) Belief that teachers do not care much about them,
promoting their knowledge or interacting with them; (7) No motivation to learn due to
perceived lack of practical and relevant content; (8) Not confident in their ability to approach
a problem and figure it out on their own (Courst & McInerney, 2003; Fink, 2013; Villa,
2021).
Figure 2. Faculty perception of common student concerns. Source: Villa’s workshop, Sept. 2022).
4.4. Net Results: Low intellectual effort by students

Besides the already expressed concerns regarding their students, it is worth mentioning that
according to teachers’ expectations, a full-time student enrolled in five three-credit-hour
courses would need to study 30 hours/week. However, most students spend 6 hours/week—
for all their courses combined! It seems that they spend much more time socializing, working,
or watching TV (Gardiner, 1994 as cited in Fink, 2013, p. 6).
4.5. Improving Teaching in Higher Education

To have faculty actually embark on efforts to improve their teaching, and therefore their
students’ learning, they would need to see that there are ways of teaching, significantly
different, from what they are doing now, and that these new and different ways of teaching
will result in good things happening -and new kinds of learning. In addition, they must ensure
that they are provided with the guidance, mentoring and training they need to know how to
teach differently; and that the institutions in which they work recognize their efforts and
provide them with encouragement and support.
903
5. Call for New Kinds of Learning

Research and leaders in business, industry and government have identified the so-called
“critical competencies”, for the current societal needs, which include: (1) Conscientiousness,
personal responsibility, and dependability; (2) Ability to act in a principled, ethical fashion;
(3) Skill in oral and written communication; (4) Interpersonal and team skills; (5) Skill in
critical thinking and in solving complex problems; (6) Respect for people different from
oneself; (7) Ability to adapt to change; (7) Ability and desire for lifelong learning (e.g.,
Gardiner, 1994; Martin-OCDE, 2018; Luna Scott-UNESCO, 2015).
Individuals and organizations involved in higher education stress out that our current
dynamic new world requires new kinds of learning. In this digital and globalized era students
need to learn how to learn; to acquire and develop leadership and interpersonal skills, ethics,
communication skills, character, tolerance, and the ability to adapt to change. Therefore,
there is a pressing need for new kinds of learning that go well beyond the cognitive domain
of Bloom’s taxonomy (1956, 2001), and even beyond cognitive learning itself. We need a
new and broader taxonomy.
5.1. Defining Learning as Change

For learning to occur, there has to be some kind of change in the learner. No change, no
learning. Learning is seen as a central underlying mechanism to acquire knowledge and skills,
as well as long-term behavior change, which should positively impact performance
(Campbell 1990; Hesketh & Neal 1999; Fink, 2003, 2023; London & Mone 1999 as cited in
Attri, 2018). Significant learning requires lasting change that is important in terms of the
learner’s life.
Fink (2003, 2013), presents a new vision of teaching in which students are engaged in their
learning, resulting in a high-energy classroom and students experiencing meaningful and
lasting change. In addition, it is presented as a value in life, since learning enhances students'
lives and prepares them to participate actively and effectively as valued members of society.
This new vision involves: (1) Significant learning; (2) Integral course design: and (3) Better
organizational support. A particular perspective on learning, defined in terms of change,
guided D. Fink (2003) in the construction of this new taxonomy. Thus, for learning to occur,
there has to be some kind of change in the learner. No change, no learning. With this
perspective in mind Fink created a taxonomy with six kinds of significant learning (see
Fig.3).
5.2. Major Categories of the Taxonomy of Significant Learning (L. Dee Fink, 2003, 2013)
Each category of significant learning contains several more specific kinds of learning that are
related in some way and have a distinct value for the learner.
904
Genny Villa
• Foundational Knowledge: Ability to understand & remember specific information

and ideas. Value: Provides basic understanding necessary for other kinds of
learning. (e.g., Pre-requisites).
• Application: Engage in some new kind of action: intellectual, physical, and/or
social. Engage in various kinds of thinking critical, creative, and practical. Value:
Allows other kinds of learning to become useful.
• Integration: Able to see and understand the connections between different things.
Sometimes they make connections between specific ideas, between various learning
experiences (e.g., between courses or entire disciplines), or between different areas
of life (e.g., between school and work or between school and leisure life). Value:
Gives students a new form of power: Intellectual power.
• Human Dimension: Learning about themselves and others; how and why others act
the way they do; enables students to function, and interact with others more
effectively. Discover the personal and social implications of what they have learned.
New understanding of themselves (self-image) or new vision of what they want to
become (self-ideal). Value: informs students the human significance of what they
learn.
• Caring: A learning experience changes the degree to which students care. It is
reflected in the form of new feelings, interests, or values. Value: Fosters in students
the energy they need for learning more about something and making it a part of their
lives.
• Learning how to Learn: Learn something about the process of learning itself.
Learning how to be a better student; engage in a particular kind of inquiry; become
a self-directing learner. Value: enables students to continue learning in the future
and to do so with greater effectiveness
Figure 3. Taxonomy of Significant Learning. (Fink, 2013)
905
Regarding the course outcomes, it is quite reasonable to expect that educators who decide to
adopt Fink’s Taxonomy and set their general course objectives in terms of significant
learning would greatly facilitate that at the end of the course in question, students will:
Understand and remember key concepts, terms, relationships, and so on; know how to use
the content; be able to relate this subject to other subjects; understand the personal and social
implications of knowing about this subject; care about the subject (and about learning more
about it); and know how to keep on learning about this subject after the course is over.
6. Conclusion
Faculty members are increasingly aware that something is missing in their teaching practice.
Although they want their students to achieve higher levels of learning, they continue to use
teaching practices that are not always effective in promoting such learning. Traditionally,
they have relied heavily on lectures as the main form of teaching. There is a need to change
the nature of the role of professors from being primarily lecturers to being professors
primarily designers of significant learning and environments. It is undeniable that teaching,
like all professions, should allow research to inform its practice. This requires facilitating that
faculty acknowledge and accept research-based evidence showing that: (1) academic success
for all students, is inextricably linked to their motivation, effort, persistence, academic self-
concept, and self-efficacy; (2) academic success is closely connected to the strategies that
support executive function processes to help students understand how they think and how
they learn, while effectively addressing the needs of those students who show significant
weaknesses in these processes; 3) academic success is also closely related to the effectiveness
of strategic and systematic teaching and adaptation of work and tasks in the classroom; (4)
there is a need for new and multiple kinds of specific learning by students, that is, something
more than understanding and remembering subject-related information.
By integrating the Taxonomy of Significant Learning into their teaching practice, professors
may realize that they can actually foster the acquisition and development of skills that do not
readily emerge from the Bloom’s taxonomy. Accordingly, pre- and in-service teacher
education and updating programs should integrate these topics in their curricula, since as
Hargreaves (2003) argues, and as educators all have noticed at some point, teachers tend to
teach in the same ways they were taught.
“We won’t meet the needs for more and better higher education until professors become
designers of learning experiences and not teachers.” —Larry Spence (2001).
References
Anderson, V., Jacobs, R., & Anderson, P. J. (Eds.). (2008). Executive functions and the
frontal lobes: A lifespan perspective. Taylor & Francis.
906
Genny Villa
Anderson, L. W., and Krathwohl, D. R., eds. 2001. A Taxonomy for Learning, Teaching, and
Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. New York:
Addison Wesley Longman.
Attri, R.K. (2018). The models of skill acquisition and expertise development: A quick
reference of summaries. Speed To Proficiency Research. Singapore.
Bandura, A. (2000). Self-Efficacy: The Foundation of Agency. In W. J. Perrig (ed.), Control
of Human Behavior, Mental Processes, and Consciousness. Mahwah, N.J.: Erlbaum.
Brunstein, J.C., Schultheiss, O. C., & Grässmann, R. (1998). Personal Goals and Emotional
Well-Being: The moderating role of motive dispositions. Journal of Personality & Social
Psychology,75(2), 494-508.
Caine, R.N. (2006). Meaningful learning and the executive functions of the brain. New
Directions for Adult and Continuing Ed. 110, 53-61
Christoff, K., Ream, J. M., Geddes, L., & Gabrieli, J. D. (2003). Evaluating self-generated
information: anterior prefrontal contributions to human cognition. Behavioral
neuroscience, 117(6), 1161.
Collins A, Koechlin E (2012). Reasoning, Learning, and Creativity: Frontal Lobe Function
and Human Decision-Making. PLoS Biol 10(3): e1001293.
Cozolino, L. (2013). The Social Neuroscience of Education. Optimizing attachment and
learning in the classroom. W.W. Norton & Co.
Craik, F. I. M., Bialystok, E., Gillingham, S., & Stuss, D. T. (2018). Alpha span: A measure
of working memory. Canadian Journal of Experimental Psychology / Revue canadienne
de psychologie expérimentale, 72(3), 141–152. https://doi.org/10.1037/cep0000143
Fink, D. L. (2003). Creating Significant Learning Experiences. An Integrated Approach to
Designing College Courses. Jossey-Bass. San Francisco.
Fink, D. L. (2013). Creating Significant Learning Experiences. An Integrated Approach to
Designing College Courses. Revised and updated by L.D. Fink. Jossey-Bass. San
Francisco.
Diamond, A. (2012). Activities and Programs That Improve Children’s Executive Functions.
Current Directions in Psychological Science, 21(5), 335–341.
https://doi.org/10.1177/0963721412453722
Hargreaves, A. (2003). Teaching in the knowledge society: Education in the age of insecurity.
New York: Teachers College Press.
Helliwell, J.F. (2003). How is life? Combining individual and national variations to explain
subjective well-being. Economic Modelling, 20, 331-360.
Kasser, T. & Ryan, D. (1996). Further examining the American dream. Differential
correlates of Intrinsic and extrinsic goals. Personality and Social Psychology Bulletin, 22,
280-287.
Luna Scott, C. (2015). The futures of learning 2: What kind of learning for the 21st century?
UNESCO.
Lunt, L., Bramham, J., Morris, R. G., Bullock, P. R., Selway, R. P., Xenitidis, K., & David,
A. S. (2011). Prefrontal cortex dysfunction and “Jumping to Conclusions”: Bias or
deficit? Journal of Neuropsychology, 6(1), 65–78. doi:10.1111/j.1748-
6653.2011.02005.x
907
Martin, J. (2018), Skills for the 21st century: Findings and policy lessons from the OECD
survey of adult skills, OECD Ed. Working Papers,166, OECD Publishing, Paris,
https://doi.org/10.1787/96e69229-en
Meltzer, L. (2010). Promoting Executive Function in the Classroom. The Guilford Press.
New York.
Meltzer, L., Reddy, R., Sales Pollica, L., & Roditi, B. (2004b) Academic success in students
with learning disabilities. The roles of self-understanding, strategy use and effort.
Thalamus, 22(1), 16-32.
Meltzer, L., & Krishnan, K. (2007). Executive function difficulties and learning disabilities.
Executive function in education: From theory to practice, 77, 105.
Meltzer, L. (2010). Promoting executive function in the classroom. Guilford Press.
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D.
(2000). The unity and diversity of executive functions and their contributions to complex
“frontal lobe” tasks: A latent variable analysis. Cognitive psychology, 41(1), 49-100.
Newport, C. (2006). How to become a Straight-A Student: Unconventional Strategies Real
College Students Use to Score High While Studying Less. Broadway Books, New York.
Pajares, F. & Schunk, D. H. (2001). Self-belief and school success. Self-efficacy, self-
concept and school achievement. In R. Riding & S. Rayner (Eds.), Self-perception
(pp.239-266). Westport, CT: Ablex.
Saroyan, A. (2000). Addressing the needs of large groups: The lecturer. In J. Bess (Ed.),
Teaching alone/ teaching together: Transforming the structure of teams for teaching (pp.
87-107). SF: Jossey-Bass.
Saroyan, A., & Amundsen, C. (2004). Rethinking teaching in higher education: from a course
design workshop to a faculty development framework. Sterling, Va.: Stylus Pub.
Sheldon, K., M., & Elliot, A.J. (1999). Goal striving, need satisfaction, and longitudinal well-
being: The self-concordance model. Journal of Personality and Social Psychology, 76,
482-497.
Villa, G. 2018. e-Learning Culture: From theory to practice. In V. Kapur & S. Ghose (Eds):
Dynamic Learning Spaces in Education, pp. 31-49. Springer Nature, Singapore (August).
Villa, G. (2021). Essential executive functions in the classroom: Promoting metacognition
and effective use of practical strategies in higher education. Design and development of
a course for the Master's program in Neuro-psychopedagogy. Corporación Universitaria
de la Costa (CUC) and Colciencias- Ministerio de ciencia, tecnología e innovación.
Colombia.
Villa, G. (2022). The future engineering classroom: Introducing new types of learning,
executive function processes and the effective use of strategies. 3-hour workshop
facilitated during the 25th Int’l Conference on Interactive Collaborative Learning -ICL,
in Vienna, Austria, September 28.
908
The value of the teaching quality innovation projects in the

universities. The case of Quid Sapienza
Department of Communication and Social Research, Sapienza University of Rome, Italy.
Abstract
The research presents the results of a survey conducted in Sapienza, University
of Rome, aimed at studying the use of Innovative Didactics methods among
teachers trained in the context of a teaching quality innovation project
(QUID). The goal is to understand the usefulness of teacher training to
disseminate innovative practices and the effectiveness of the most advanced
student-centered teaching and learning methods centered on the use of
technology, also by comparing the results with other similar studies reported
in the literature. The study highlights the value of the QUID project as a tool
for raising awareness and disseminating innovative practices. In fact, the
training of teachers is more effective to promote their awareness of adopting
ID methods to promote involvement and better performance of the students.
Conversely, the pandemic period has not positively influenced the progress
process in the use of innovative teaching practices, but only in the use of
technologies.
Keywords: didactic innovation projects, innovative didactics, quality of
educational innovation, university teaching practices,

The value of the teaching quality innovation projects in the Universities. The case of Quid Sapienza
1. Introduction
In the last decades, the attention toward Innovative Didactics (ID) by the Italian Universities
has grown, as well as abroad. As a recent survey shows (Mazza & Valentini, 2021), between
2016 and 2019, numerous teaching innovation projects were launched. Half of those analyzed
in the mentioned study is concentrated between 2017 and 2018, in response to the indications
of the European Commission regarding the desire to establish the European education area
by 2025. Sapienza, University of Rome was among the first Italian universities to promote a
working group and a project on the Quality of Educational Innovation (QUID). In the
theoretical framework, we have adopted an extended concept of ID, which implies processes
that stimulate the active, creative, and collaborative dimensions. Although Innovative
Didactics (ID) and distance learning do not necessarily coincide, digital technologies can
support these dimensions. For this reason, we have also focused on research that have studied
the use of digital technologies to develop cooperative and collaborative learning processes e
the teaching strategies adopted by academics. The second part presents the results of a survey
conducted in Sapienza, University of Rome, aimed at understanding the use of ID methods
among teachers trained in the context of the QUID project.
There are a variety of definitions of ID in the literature. A synthetic framework, within the
studies conducted by the Joint Research Center Institute for Prospective Technological
Studies of the European Commission, is offered in Kampylis, Bocconi & Punie, 2012. We
have adopted an extended concept of ID: as a process of overcoming the traditional practice
of teaching based on a frontal and transmissive approach in favor of student-centered learning
processes that stimulate the active, creative and collaborative dimension (Sancassani et al.,
2019). In previous studies (Mazza & Valentini, 2021; Mazza & Valentini, 2023) we have
critically examined the Italian and non-Italian literature on models of didactic innovation,
and the main theoretical frameworks concerning innovative didactics (among others,
Mezirow, 2001; Meyers & Jones, 1993; Watkins & Mazur, 2013). Furthermore, starting from
the classification proposed by Fedeli & Frison (2018), we have focused attention on
participatory and interactive strategies and methods to facilitate learner-centered learning.
This review shows that ID can be promoted regardless of technology. At the same time,
digital technologies can facilitate student learning and action. This could happen both in more
learner-centered approaches, but also in more transmissive teaching.
Among others, Calvani (2005) and Ranieri (2004) have focused on the use of digital
technologies to develop cooperative and collaborative learning processes. Trentin (2017),
Rivoltella (2012), and other scholars have studied innovative teaching methods in hybrid
learning modalities, based on the integration of physical and online teaching spaces.
910
Moreover we have analysed some recent studies that have investigated the teaching strategies
used in the pandemic emergency by the Italian academics because during this period the
diffusion of the use of digital technologies has obviously increased, but has not necessarily
increased the use of innovative teaching strategies (Mazza & Valentini, 2023; Bruschi, 2020;
Consiglio et al., 2020; Ramella & Rostan, 2021; Tamborra, 2021). According to Tamborra
(2021), digital technologies have been used to reproduce frontal and transmissive teaching
without innovation, risking keeping the typical practices of face-to-face teaching unchanged.
Conversely, other research has shown greater innovation, especially when ID practices or
training were already established (Mazza & Valentini, 2023).
Considering the theme of this paper, it is useful to deepen some results of the research
presented in Ramella & Rostan (2020), based on a survey conducted on a sample of 3.398
academics from various Italian universities. Their study shows that innovative didactics is
adopted and prevalent with respect to the “frontal lesson” strategy since before the pandemic,
but during the emergency an impoverishment and simplification of innovative teaching
methods has emerged. They identifies three innovative approaches. Firstly, the use of the
transmission-dialogical strategy (based on traditional lectures, but enriched by discussions
between academics and students) has doubled. Secondly, the strategy of interactive
transmission (based on the active involvement of students through exercises, laboratories,
and group work) has remained constant. Thirdly, the diffusion of the collaborative-innovative
strategy (based on group work, peer-to-peer discussion, and evaluation methods, activities
aimed at transversal skills, and projects aimed at stimulating students’ creativity and
problem-solving skills) has halved.
Given the theoretical framework on ID and what emerged from the review of research on
teaching strategies used in the pandemic emergency, we focused our attention on the
University of Rome, which has decided to invest, since the academic year 2016-2017, in the
QUID project. It aims to disseminate ID models, methods, and techniques among its
professors to ensure the growing adoption of ID in the university. The project has reached its
seventh edition and there are now over 500 professors who have acquired ID knowledge and
skills. In particular, the study presented in this work aims to: 1) verify the usefulness of the
QUID as a tool for knowledge and dissemination of innovative practices; 2) understand
which innovative teaching strategies the professors who took part in the QUID use in the
classroom and with what degree of effectiveness; 3) detect the strengths and weaknesses in
the use of the ID that the professors have found.
3. Methodology
In this study, the interviewees who participated in the project up to the fifth edition (the
academic year 2021/22) were interviewed - therefore at the end of the pandemic. The decision
911
to focus attention on this type of professor stems from the awareness that they are the most
expert in ID and that, for some reason, they are the ones who make more active use of it, as
well as possessing the greatest and most adequate knowledge and skills to be assessed. Out
of 350 teachers and researchers, 54% participated in the research (189), of which 37.4%
worked in the Health area (HeS), 32.3% in Hard Sciences (HaS), 18.2% % in socio-political
sciences (SPS), and 12.1% in the humanities (HS). The interviewees were asked, through an
interview with a semi-structured questionnaire distributed through a web survey, to evaluate
both the effectiveness of the experience in the QUID project and the adoption of Innovation
Didactics (ID) and to report the limitations and benefits highlighted in classroom teaching.
A mono and bivariate analysis of the collected data was then performed and the utility (UR),
ID use (IDR) and effectiveness (ER) rates were constructed to measure respectively how
much the QUID experience was useful to teachers, how much the application of the identity
document in the classroom and the effectiveness they consider to use the identity document
in their teaching activity. These indicators were developed by comparing, respectively, the
average value of the perceived usefulness of innovative didactics with respect to the
organization of teaching and the relationships with colleagues; the frequency of use of ID
didactic tools and teaching methods; and the effectiveness of ID methods to promote the
learning of content. Finally, a linear regression analysis was performed using the Pearson
regression coefficient (r), between -1 and 1 to verify the degree of significance of the
application of the ID by the teachers with respect to the different scientific areas in which
they operate. This made it possible to understand whether some scientific areas use ID more
and more effectively and how much QUID’s experience in using innovative teaching
methodologies has influenced them.
4. Outcomes
Research has shown that the QUID Project has been very useful for university professors to
deepen their knowledge of ID methods and techniques and to better understand how to apply
them in the design and implementation of their teachings (see Table 1). The recorded utility
rate is 85.7% with the highest peaks in the Healthcare area (93.3%) and in the political and
social sciences (91.4%.).
It is no coincidence that the Pearson regression coefficient is also very high in all scientific
areas and above all in those in which teachers have considered QUID more useful (U)
(rUxSPS=0.997, rUxHeS=0.993, rUxHS=0.990, rUxHaS=0.983).
This result of the QUID is confirmed by the use that the interviewees declare to make of the
ID in the classroom. The average utilization rate (IDR) is 56%. Although it is lower than the
awareness gained in teaching innovation, it still demonstrates a commitment to its
application. In detail, among the professors who use ID, 18.2% do it continuously and as a
912
priority, 59.4% in a limited way and limited to some activities, and only 22.4% sporadic and
occasionally. But how is the ID used (see Table 2)?
Table 1. The value of the QUID experience as an incentive to use the ID.
MR%
32.5
improve the planning and organization of their teaching activities %
24.6
useful for exploring new methods and getting up to speed on new ID techniques %
10.1
meet colleagues and exchange ideas and teaching experiences %
10.1
train new hires in ID methods %
strengthen relationships with colleagues to think about forms of collaboration in the
classroom 9.1%
align with international standards of use of ID 7.3%
improve the coordination of teaching contents between the teachings of the course of
study 6.3%
TOT. Responses 370
Source: Sapienza (a.a 2021-2022).
Above all to verify and evaluate the learning results in the classroom, but the share of
respondents who use innovative methods to involve students, encourage them to participate,
and stimulate interactive and participatory learning, placing them at the center of training, is
also very significant. In this sense, they apply ID most effectively and adequately to pursue
its objectives, especially in the Hard Sciences.Training objectives that they also claim to
achieve, given that the effectiveness rate stands at 99.3% with the highest peaks in political-
social sciences (99.9%) and in Healthcare (99.8%), followed by Humanities (99.6%) and the
Hard Sciences (99.3%). Respondents see significant improvements in the learning outcomes
obtained in classrooms where they use ID and better development of their skills thanks to the
improvement of participation, involvement, and peer comparison (see Table 3).
913
Table 2. How do they use the ID in the classroom?
MR%
methods of verification and/or ongoing evaluation 38.0%
quiz at the beginning, during, or at the end of the lesson 35.6%
realization of group work and project work 24.3%
intermediate tests 23.0%
written exercises 11.8%
short presentations in the classroom 5.3%
r (use ID for macroarea): rIDxHeS=0.981; rIDxHas=0.996; rIDxSPS=0.968;
rIDxHS=0.942.
student-centered and peer activism methods 37.8%
collaborative learning and community discussion 22.5%
group work 19.2%
ongoing self-assessments 17.9%
peer-to-peer assessments among students 17.2%
flipped classroom 17.2%
role-playing and simulations 6.0%
r (use ID for macroarea): rIDxHeS=0.994; rIDxHas=0.996; rIDxSPS=0.984;
rIDxHS=0.967.
stimulation/incentive methods 24.2%
discussion of case studies and problem-solving 49.5%
joint lessons between colleagues or with expert testimony in the classroom 13.4%
document reading 10.3%
ex-classroom lesson (visits to museums, organizations, institutions, companies, etc.) 8.2%
help of videos 7.2%
participation in conferences and seminars 6.2%
research and study activities in the classroom 5.2%
r (use ID for macroarea): rIDxHeS=0.978; rIDxHas=0.991; rIDxSPS=0.986; rIDxHS=0.930.
TOT. Rensponses = 468
Source: Sapienza (a.a. 2021-2022).
914
Table 3. Teachers' considerations on the effectiveness of the application of ID in teaching.
effective for: MR%

improve student performance and learning outcomes 17.8%
encourage more interactivity in the classroom and encourage participation 16.4%
generate engagement and implement students' emotional involvement beyond
the classroom 13.5%
enhance critical thinking and the development of abstract reasoning 11.7%
enhance transversal and vocational skills 11.4%
foster more collegiality between teachers in setting up the content and more
interdisciplinary connections between the knowledge acquired 9.3%
get more feedback and better understand student needs 7.6%
make teaching activities more engaging through the use of technologies 7.0%
improve the quality and effectiveness of assessment tools 3.1%
implement interactions with stakeholders and teachers and students of other
universities and create contacts with the labor market 2.2%
r (use ID for macroarea): rIDxHeS=0.964; rIDxHas=0.985; rIDxSPS=0.970; rIDxHS=0.819.
TOT. Responses 951
Source: Sapienza (a.a 2021-2022).

The study highlights the value of the QUID project as a tool for raising awareness and
disseminating innovative practices. Recent studies conducted during the pandemic had, for
example, reported low values in the use of student-centered methods and peer activism (for
example, the use of these more advanced teaching methods had increased on average from
2.9% to 7.7% in the previous two years (2019-21) (Mazza & Valentini, 2021). Similarly, the
progressive improvement confirms what has emerged in the literature, i.e. that the pandemic
period has in no way positively influenced the progress process in the use of innovative
teaching practices (but only in the use of technologies), while on the contrary, the training of
teachers was more effective. Finally, this research highlights how, although in absolute
values, the more advanced methods, i.e. the student-centred and peer activism methods are
significant in percentage terms, they are not dominant with respect to the other two types
(“methods of verification and/ or ongoing evaluation” and “stimulation/incentive methods”)
which however express lower levels of innovation. This limit is confirmed by the
considerations of the students. University-wide structural surveys have been initiated to
assess student satisfaction with ID. These surveys mostly refer to the use of technology in
distance or blended learning. The growth in student participation in opinion surveys indicates
an increasing attention to teaching and a corresponding demand for quality that must be taken
into account in order to implement improvement actions (Sapienza, 2022). In 2020-21, the
positive evaluations expressed by attending students on overall satisfaction with courses
915
amounted to around 87%, an increase over the previous year. It should be noted that students
on Master’s courses express more variability in their judgments, perhaps due to a greater
experience of university life. These opinions could be taken into consideration precisely in
order to identify specific aspects for improving teaching quality. The satisfaction ratio
expressed on various aspects of the educational offerings reveals a high level of satisfaction
with the use of technology in teaching. At the University level, on average, the ratio of highly
satisfied students to dissatisfied students is 3.8%, an increase over the previous year. This
ratio, when calculated with regard to the evaluation of online teaching activities (multimedia
films, hypertext units, etc.), reaches levels well above this value: 8.5% for Bachelors courses
and 12.2% for Master’s courses. This means that teaching supported by technology manages
to meet students’ learning expectations to a greater extent. These tools will have to be further
developed in order to build the basis on which cooperative and collaborative learning
experiences can grow. It is therefore clear that, although teachers are aware of the importance
of adopting ID methods that favor more student involvement and better performance, it is
necessary for them to further enhance the methods of use of innovative teaching methods,
also (but not only) using digital technologies so appreciated by the students.
References
Bruschi, B. (2020). Università dopo l’emergenza: quali opportunità? Il caso dell’Università
di Torino. Scuola democratica, Learning for Democracy, 3, 583-590. doi:
10.12828/99906.
Calvani, A. (2005). Reti, comunità e conoscenza: costruire e gestire dinamiche collaborative.
Trento: Erickson.
Consiglio, S., Gambardella, D., Serpieri, R., Rebeggiani, E., Grimaldi, E., Pirone F. &
Lumino, R. (2020). La didattica universitaria nella transizione al digitale. QTimes, 3.
Fedeli, M, & Frison, D. (2018). Metodi per la facilitazione dei processi di apprendimento nei
diversi contesti educativi. Form@re - Open Journal per la formazione in rete, 18 (3),
153-169. doi: 10.13128/formare-24001.
Kampylis, P. G., Bocconi S. and Punie Y. (2012). Towards a Mapping Framework of ICT-
enabled Innovation for Learning, European Union.
Mazza, B., Valentini, E. (2023). Innovation in Teaching Methods and Techniques.
Experiments and Uncertainties of Academics Before and After the Pandemic in the Italian
Universities. Italian Journal of Sociology of Education, 15(1), 51-80. doi: 10.14658/pupj-
ijse-(2023)-1-3.
Mazza, B., & Valentini, E. (2021). Un modello comunicativo per la didattica universitaria
innovativa: la centralità di partecipazione, coinvolgimento e componente emotiva.
Sociologia Italiana-Ais Journal of Sociology, 18, 2021, 29-52. doi: 10.1485/2281-2652-
202118-2.
Meyers, C., & Jones, T. B. (1993). Promoting Active Learning. Strategies for the College
Classroom. San Francisco. Jossey-Bass.
916
Mezirow, J. (1991). Transformative dimensions of adult learning. Jossey-Bass.Nipper, S.

(1989). Third generation distance learning and computer conferencing. In R.D. Mason &
A.R. Kaye (Eds). Mindweave: Communication, computers and distance education (pp.
63-73), Oxford: Pergamon Press.
Ramella F., & Rostan M. (2020). Gli accademici italiani e la didattica a distanza durante
l’emergenza Covid-19. Working papers CLB-CPS. Università di Torino.
Rivoltella, P.C. (2012). La comunicazione e le relazioni didattiche. In P.C. Rivoltella, P. G.
Rossi, L’agire didattico. Manuale per l’insegnante (pp. 151-166). Brescia: La Scuola.
Sancassani, S., Brambilla, F., Casiraghi, D. & Marenghi, P. (2019). Progettare l’innovazione
didattica. Torino: Pearson.
Sapienza, 2022, Valutazione delle opinioni degli studenti sulla didattica. A.A. 2020/21,
https://www.uniroma1.it/sites/default/files/user/8443/relazione_opis_2020_2021_26apri
le2022_1.pdf.
Trentin, G. (2017). Connettività, Spazi Ibridi e Always-on Education. Rivista AEIT.
Watkins, J., & Mazur, E. (2013). Retaining students in science, technology, engineering, and
mathematics (STEM) majors. Journal of College Science Teaching, 42(5), 36-41.
917
The search for values as a didactic tool - an interdisciplinary

perspective
Department of Surgical Sciences, Sapienza University of Rome, Italy.
Abstract
The present article explores the didactic potentialities of an interdisciplinary
approach intended for university, master or doctorate courses where the
development of professional strategies and the acquisition of English for
Specific Purposes (ESP) are the main target. By exploiting a heuristic method
provided by semiotics, namely Floch’s Axiological Square, students can
acquire the toolkit that permits them to analyse the process of value creation
as well as the discursive procedures and lexical peculiarities typical of
corporate communication. Although the focus is on pharmaceutical discourse,
we are convinced that the methodological apparatus illustrated here is
potentially applicable to other ESP domains and that it highlights the
importance of dialogue between apparently distant disciplines for better
educational outcomes.
Keywords: Floch’s Semiotic square; learning by doing; ESP;
interdisciplinarity.

The search for values as a didactic tool – an interdisciplinary perspective
1. Introduction
The present contribution1 is part of a vaster project aimed at exploring the heuristic and
didactic potentialities of an interdisciplinary approach (Swales, 1990; Perrenoud, 1998;
Bhatia, 2008). This is a proposal intended for university, master, or doctorate courses where
the acquisition of English for Specific Purposes (ESP) is aimed at developing skills that
permit the production of effective communicative strategies capable of creating and
presenting a credible and convincing corporate image. The focus here is on pharmaceutical
discourse but we are convinced that the methodological apparatus proposed can be exported
into other ESP domains.
Within the realm of specialist-language education, recourse to the analytical and practical
tools made available by disciplines apparently remote from each other may impact,
positively, upon processes of student learning and motivation (Holmbukt & Larsen, 2016).
We wish to highlight the contribution Floch’s semiotics of marketing might well make to
corporate communication. The dissection of the semiotic mechanisms underpinning the
production of meaning may foster and enhance the motivation of ESP learners.
The objective is, therefore, twofold: on the one hand, to provide learners with some of the
tools they need to understand value-endowing discursive procedures; on the other, to allow
them to “learn by doing” (Ellis, 2003; Steffens et al., 2015; Niemeier, 2017), that is, to acquire
some of the rhetorical strategies and lexical peculiarities typical of corporate communication
indirectly. The learning outcomes expected upon completion of the didactic activities
illustrated below (in section 3) are:
• Acquisition of a theoretical tool for text analysis (semiotic square) and its application
to various domains beyond its original scope;
• Ability to identify the value sets encoded in texts and to replicate them for effective
communication;
• Ability to recognise the dynamics of meaning-making processes;
• Acquisition of the ESP and discursive strategies typical of pharmaceutical
communication.
1
The authors conceived and discussed the paper together. In particular, Andrea Fenice is responsible for Sections 3 and 4; Renzo
Mocini for Sections 1 and 2.
920
2. Methodology
In order to show how our approach may be implemented, two pharmaceutical websites were
selected: astrazeneca.com and pfizer.com2. This choice was dictated by criteria of
opportunity as the companies chosen seemed particularly suitable for illustrating the
educational potential of an interdisciplinary approach that brings marketing semiotics and
ESP in a dialogue. Eco (1988) claims that semiotics aims to describe the workings of
communication and signification. According to Greimas (1966), the elementary structure of
signification is produced according to a semiotic square (see Figure 1), which allows us to
represent semantic categories by implementing three opposite logical sets: presupposition or
contrariety, complementarity or implication, and negation or contradiction. Floch (1990)
applies this tool to marketing, describing how companies invest their products with a set of
values. Starting from the opposition between “existential values” and “utilitarian values”, he
posits that promotional strategies fall into one or more of the four categories associated with
the vertices of a square: Practical, Ludic-Aesthetic, Utopian and/or Critical.
• The Practical set of values is associated with utilitarian values and highlights the
functional features of a product.
• On the contrary, the Utopian set corresponds to existential values and aims at
exalting general principles such as life, identity, adventure, freedom and so on.
• The Ludic set, also called aesthetic, focuses on the qualities a product possesses,
which are capable of enhancing leisure and/or generating aesthetic appeal. This set
of values refers to pleasure, enjoyment as well as luxury, beauty, and other non-
utilitarian characteristics.
• Finally, the Critical set of values emphasises an object as such. This mode presents
the product in purely concrete, pragmatic terms by focusing on elements such as the
cost-benefits ratio or value for money.
Crucially, Floch’s square, like the semiotic square in general, is not arranged statically, but
is a dynamic configuration of concepts. This means that the semantic space between them is
as important as the vertices themselves. In practical terms, a textual configuration not only
can, but is likely to use more than one value set and mixed value strategies as well. This is
also why we strongly believe in the semiotic square as a teaching tool capable of stimulating
reasoning rather than mnemonics. Students do not learn a series of “labels”, static categories
to which to assign textual strategies, but they are encouraged to develop critical thinking and
question all the assumptions they make, avoiding simplistic often inaccurate binary solutions.
2
The analysis that follows is based on the version of the two websites mentioned (last accessed on the 29th of January 2023).
921
Within the domain of corporate communication, this semiotic tool can be didactically
exploited to investigate the language and discourse strategies used by pharmaceutical
companies to showcase their research products and frame information disclosure. After all,
the square describes only the deeper (more abstract) levels of signification, while “at a more
superficial level, the positions are converted into values that ultimately will be desired and
pursued by the subjects” (Floch, 2001, p. 113). Therefore, the same values can be discursively
realised in different ways, also depending on the semiotic mode selected: verbal, non-verbal
or both. This way, the square allows for “the recognition of pertinent expressive features in
the production of a specific meaning effect” (Floch, 2001, p. 137). Against this background,
students will be guided to use the axiological square to analyse the various value-endowing
devices employed by pharmaceutical companies on their websites. Figure 1 below displays
the four Flochian strategies and their corresponding value sets.
Figure 1. Floch’s axiological square.
The practical value set is based on utilitarian values and corresponds to what Floch calls a
“referential strategy”: a “set of procedures aimed at presenting the discourse in question as
true” (Floch, 1990, p. 274, our translation and emphasis). Clearly, not only marketing, but
any other form of communication including pharmaceutical discourse can adopt this strategy
of factual presentation of reality, by foregrounding objectivity, providing data, tables, graphs,
and referencing scientific sources.
Conversely, a utopian value set corresponds to a mythical strategy based on fundamental,
existential values. It can be found almost without variation in the discourse of pharmaceutical
companies. Life, health, the power of science and social well-being are some of the most
common utopian values called upon to enhance their communiqués.
922
Floch associates the ludic or aesthetic set of values with an oblique form of advertising, which
relies on less immediate communication that required active interpretation, a “cognitive
effort” to understand indirect messages. The label “aesthetic” refers to non-utilitarian values
that stress the aesthetic/ludic pleasure experienced by the subject. Just as the practical set of
values was an ostentation of reality, its contradictory opposite – the aesthetic set – is a strategy
of concealment. Therefore, the enunciator can “strike the ludic chord” of the enunciatee
implicitly – or as Floch puts it, obliquely – through aesthetically pleasing images, or enticing
calls for action.
Finally, the critical value set denotes a substantial strategy. Looking at Floch’s examples for
marketing (quality/price, cost/benefits ratios) this set does not seem to apply to
pharmaceutical discourse. However, one should make distinctions between the meanings of
“substantial”, going beyond the common meaning (ample, sizeable), and focusing on its
etymological meaning: “pertaining to the substance or essence”, “involving an essential part
or point” (Etymonline). Marrone (2007) proposes a revised square in which the substantial
strategy consists in a sort of “close-up” of the object per se and its perceptible characteristics.
Thus, the emphasis is on the object itself, a “hyper-realistic” presentation where some
identifying characteristics are selected and magnified in their perceptual nature, often in
isolation from their context.
3. Didactic activities
The didactic activities outlined here are not meant to act as a thorough and exhaustive
examination. Rather, they simply provide a sample analysis to illustrate the teaching
application of the Flochian axiologies. Once the basic concepts of Floch’s square have been
explained by the teacher, the students are asked to analyse selected portions of
pharmaceutical websites to look for the specific value strategies adopted by the enunciator.
As expected, these websites regularly adopt the practical value set. Given the abundance and
easy identification of this kind of strategy, the students should start their analysis by
identifying verbal and visual elements leading to a referential strategy. Significantly, the
expression “what science can do”, which we find on the homepage of AstraZeneca, is an
eloquent example of this communicative solution. In general, most webpages of
pharmaceutical companies feature articles with more or less technical or general-purpose
health information. On pfizer.com, for example, a menu provides links to various informative
sections, such as “how drugs are made”, “product list”, “coronavirus resources”. Similarly,
AstraZeneca’s website displays charts and visual data summaries all intended to pinpoint
relevant medical facts. Since it is frequent and easy to detect, this kind of pragmatic strategy
is a good starting point for students to accustom themselves to the characteristics of Flochian
analysis. While analysing communicative strategies, students can – osmotically as it were –
923
identify instances of ESP and recurring syntactical structures. However, the teacher should
point out that, although the provision of information is the overt aim of these websites, the
practical value set is not necessarily the only strategy involved.
Once the students have familiarised with the referential strategy, we suggest that the teacher
guide them to make comparisons with its complementary communicative solution, the
substantial strategy. Both are easily found together in visual discourse. We argue that very
similar images can implement one or the other strategy depending on the (mainly verbal)
context in which they are placed. The article “Harnessing the power of cell therapy” on the
AstraZeneca website, for example, is accompanied by the silhouette of a woman’s body full
of animated pictures of magnified cells, proteins and other microscopic elements. This picture
possesses almost no informative content. Despite using actual images obtained using a
microscope, it is a mere collage whose only purpose is to attract the reader’s attention. In this
case, the magnification is simply a “hyper-realistic zoom”, a focus on the object per se, which
indicates a substantial strategy. However, when the article describes the CAR-T cell therapy,
the pictures of magnified cells refer directly to the scientific information that is being given.
Here we can identify a vertical movement on the square towards a practical set of values
since the picture is a visual aid to the verbal explanation of the research data provided.
An eloquent illustration of the ludic set can be found in the lexical choice concerning the link
buttons. Normally, these are variously labelled as “read more”, “find out more”, etc. On
astrazeneca.com, however, we also find buttons such as “explore our latest film” or “explore
all our science centres”. The same call for action is present on the Pfizer homepage where
the first button reads “Explore our science”. These labels frame the search for information as
a quest, in a sort of mild gamification where the user is rewarded with various kinds of data.
Interestingly, on the Pfizer website, the button “Explore our science” leads to a section called
“science will win”, which refers to the “war against diseases” but also subtly reinforces the
semantic domain of “game”. It is clearly a different kind of ludic value set compared to that
used by Floch to analyse the promotion of a product for its aesthetic or enjoyable features.
However, in both cases, the enunciatee is presented with an indirect message which requires
active participation to be decoded. Here, the metaphor of exploration is used to tell a story,
adding an aesthetic component to an informative text, using a mixed strategy that exploits
contrasting value sets on the diagonal of the square.
During the final phase of the learning unit, the teacher can guide the students through more
complex analyses, focusing on less obvious cases where multiple strategies can be detected.
When entering the AstraZeneca homepage, for instance, the user is welcomed by a
conspicuous teal banner. On the left, there is the aforementioned white caption reading “What
science can do”; in the right-central area, we find a mid-ground close-up shot of a man’s
silhouette full of colourful lights. A closer examination shows that the blue light in his head
is the schematization of a digital network with floating 0s and 1s, while the yellow and pink
924
lights in his chest are folded proteins and a double DNA helix. In the bottom-right-hand
corner, there is a white “i” information icon. A click of the mouse reveals a short paragraph:
Disease understanding: Connecting broader omic-technologies can identify the more
complex and transient molecular changes that underpin the course of disease.
AstraZeneca is turning this data into knowledge, to help identify novel drug targets.
Here, the focus of the textual segment is clearly the image, which reveals a clear substantial
strategy: it is aesthetically appealing, but the students should be warned that this does not
necessarily imply the choice of an aesthetic strategy. In actual fact, the values are not
ludic/aesthetic (non-utilitarian); rather, the caption stresses the usefulness of science,
indicating a practical and/or critical position on the square. Which of the two is dominant?
We suggest the latter. Although the paragraph quoted provides hard scientific facts, evoking
objectivity and research practices through the use of a highly specialised lexicon (e.g., omic-
technologies, transient molecular changes), two communicative aspects need to be
underlined. Firstly, this excerpt exhibits factual information, but it does so without entering
into detail. It is the written equivalent of the aforementioned “zoom effect” focusing on a
very specific aspect of medical research without any precise context. We might argue that
this excessive close-up actually diminishes the strength of the informative content. The
phrase “turning this data into knowledge” does not explain much of the actual process, and
it is not even clear what “this data” or “knowledge” refer to. Secondly, the excerpt appears
only if the user passes the mouse over the information icon. This means that the written
message is backgrounded in favour of visual communication. Referring back to the image,
the same strategy can be identified: although the visual elements shown are the result of
advanced technology and scientific research (actual pictures of DNA and folded proteins),
they yield no actual scientific information. Again, we can find the “hyperrealism” which
Marrone considers an aspect of the substantial strategy, rather than the factual presentation
of reality indicating a practical set of values. Therefore, the analysis performed using the
square reveals a vertical semantic movement from a practical to a critical value set.
As a final example, the students might consider that, with the exception of titles and menu
items, AstraZeneca’s homepage opens with a claim followed by a call to action:
We are transforming the future of healthcare by unlocking the power of what science
can do, for people, society and planet.
Discover more about our company [Link button]
The initial claim enacts a mythical strategy, presenting a utopian value set. Students should
be invited to focus on the language used: on the one hand, the presence of words referring to
existential values: future, people, society, planet; on the other, the use of the continuous
present “we are transforming” that suggests both a constant effort and a future plan. The
925
enunciator is implicitly creating a narrative where the company plays a proactive role in
shaping the world. The verb unlock adds to this strategy: “unlocking the power of science”
creates a mythical, almost heroic configuration with the company as the protagonist of a
story. Further evidence of this strategy can be found in the short videoclip accompanying the
claim. This is no place for a multimodal analysis, which would go far beyond the scope and
aims of this paper. Suffice it to mention that the voiceover introduces the video telling us that
“This is the adventure of what science can do”, and claims that science has taken them “to
places [they] never thought possible” making them “pioneers”. Using this perspective, one
can explain the peculiar verb used in the link button, discover, which reinforces the narrative
of the explorer the term adventure connotes. This is also an instance of the different though
associated ludic strategy created by the “explore” link buttons mentioned above.
4. Conclusion
Contemplating Floch’s square and the generative potential of the values presented at the four
vertices as they interact vertically and transversally with each other to produce semantically
nuanced texts, it should appear clear that the discourse found on pharmaceutical websites
rarely pursues a single strategy. Therefore, Floch’s square should be presented to ESP
students not as a simple list of strategies but as a tool capable of facilitating the interpretation
and replication of the values encoded in texts. In other words, the square is a dynamic tool
and should be used to highlight logical relations rather than to assign mere labels to segments
of discourse. When used heuristically, the semiotic apparatus proposed here can favour the
acquisition by learners of discursive and linguistic skills crucial to effective corporate
communication in English. The illustrative analysis carried out has striven to corroborate the
fact that the Flochian method appears to be applicable beyond its original scope, confirming
the pedagogical potential and value of “any form of dialogue or interaction between two or
more disciplines” (Moran, 2002, p. 16) and calling for further research in this direction.
References
AstraZeneca, Homepage. https://www.astrazeneca.com/, last accessed 28/01/2023.
Bhatia, V. K. (2008). Genre Analysis, ESP and Professional Practice. English for Specific
Purposes, 27, 161-174. doi: 10.1016/j.esp.2007.07.005
Eco, U. (1988). Le signe. Bruxelles: Éditions Labor.
Ellis, R. (2003). Task-based language learning and teaching. Oxford: Oxford UP.
Floch, J. M. (1990). Sémiotique, marketing et communication. Italian tr. Semiotica, marketing
e comunicazione. Dietro i segni, le strategie. Milano: Franco Angeli.
Floch, J. M. (2001). Semiotics, Marketing and Communication. New York: Palgrave Macmillan.
Greimas, A. J. (1966). Sémantique structurale. Recherche de méthode. Paris: Larousse.
926
Holmbukt, T.E., & Larsen, A.B. (2016). Interdisciplinary teaching as motivation: An

initiative for change in post-16 vocational education. Nordic Journal of Modern
Language Methodology, 4(1), 67-82. doi: 10.46364/njmlm.v4i1.325
Marrone, G. (2007). Il discorso di marca. Modelli semiotici per il branding. Roma: Laterza.
Moran, J. (2002). Interdisciplinarity. London: Routledge.
Niemeier, S. (2017). Task-based grammar teaching of English: Where cognitive grammar
and task-based language teaching meet. Tübingen: A. Francke Verlag.
Etymonline – Online Etymology Dictionary (n.d.). Substantial. In etymonline.com,
https://www.etymonline.com/word/substantial, retrieved 02/02/2023.
Perrenoud, P. (1998). From formative evaluation to a controlled regulation of learning
processes. Towards a wider conceptual field. Assessment in Education, 5(1), 85-102. doi:
10.1080/0969595980050105
Pfizer, Homepage, https://www.pfizer.com/, last accessed 29/01/2023.
Steffens, M. C., von Stülpnagel, R., & Schult, J. C. (2015). Memory Recall After ‘Learning
by Doing’ and ‘Learning by Viewing’: Boundary Conditions of an Enactment Benefit.
Frontiers in psychology, 6, 1907. doi: 10.3389/fpsyg.2015.01907
Swales, J. (1990). Genre Analysis: English in Academic and Research Settings. Cambridge,
UK: Cambridge University Press.
927
Design and manufacturing of a LIGHTBOARD - Combining the

peer-to-peer idea with project based teaching
Anja Pfennig
School of engineering and life science, HTW Berlin, University of Applied Sciences, Berlin,
Germany.
Abstract
Lecturers in higher education find it challenging to involve students into
critical thinking, carefully pondering solutions to mechanical engineering
problems. Project based teaching gives the opportunity to gain both, higher
learning outcome and self-reliant study skills. During fourth semester
mechanical engineering a guided one semester project (180 hours workload,
6 ECTS) related to a complex real engineering problem focusing on customer
demands and possible solutions. It comprised of the design and manufacturing
of a lightboard – a “learning glass panel” consisting of a high quality
optiglass panel which is surrounded by LED lights. As on regular whiteboards
a lecture may be given and filmed directly writing on the 16:9 screen glass
panel with fluorescent pens – the lecturer always facing the audience. This
lightboard project directly involves students in the production of a teaching
device (peer-to-peer approach) and the teaching method is regarded beneficial
in terms of students` learning outcome and self-reliance as well as engineering
skills. However, the work load is very high and grading is insufficient.
Therefore, the method and the role of a lecturer as facilitator is discussed.
Keywords: Lightboard, project-based learning, peer-to-peer teaching.

Design and manufacturing of a LIGHTBOARD
1. Introduction
Globalization and ready accessibility of information nowadays encourages HTW Berlin to
move to a competency-based teaching model helping graduates to acquire autonomy and to
appropriate the learning process: Huguet et al. (2017), Pfennig (2020), Pfennig (2022).
Practicing problem-based learning in education enables students to gain skills above and
beyond sheer theoretical and factual knowledge: Balvea and Albert (2015) and reinforces
their natural desire to learn: Gomez-Pablos et al. (2016). In general, project based learning
methods: Efstratia (2014), Balve and Albert (2015) improve
• interpersonal and interaction skills,

• the capacity of dealing with conflicts and uncertainty,
• communication and presentation skills,
• the ability to work autonomously and assume responsibility,
• the capacity of reasoned decisions,
• and the ability to dive quickly into new areas of knowledge and apply this
knowledge in practical situations
The peer-to-peer approach applies well in project based teaching where advanced students
generate teaching material for first and second year students: Pfennig (2020), Pfennig (2019).
Class results indicate that involving students directly into teaching activities (preparation of
lecture videos) can be very effective in getting students to engage in critical thinking: Lord,
(2012) entailing deeper learning outcomes: Goto and Schneider (2010). Lecture videos
produced via the peer-to-peer approach during student projects: Pfennig (2019) are
successfully implemented in inverted classroom teaching scenarios in a first year materials
science course at HTW Berlin: Pfennig (2020). Inverted classroom teaching covers the
scientific input via defined self-studying phases enabling communicative teaching
approaches (group work, discussions, hands-on problems, etc.) during face-to-face time:
Pfennig (2020), Pfennig (2022), Setren et al. (2019).
Most self-study phases of the inverted classroom lectures are accompanied by lecture capture
videos because lecture videos provide both, an audio and visual stimulus: Gulley and Jackson
(2016). But, -with exception of screen casts- the lecturer generally turns her or his back
towards the students which is rated disruptive by students. Therefore, lightboard video
recording offers a promising low threshold solution to teach short sequences online and
simultaneously face students. A lightboard is a “transparent blackboard” consisting of a high
quality optiglass panel surrounded by LED lights: Pershkin (2020). The lecturer may directly
write on the lightboard with fluorescent pens, being filmed in front. During post processing
the video is mirrored and the video lecture topics may be directly used e.g for self-studying
(Figure 1). As the production of lightboard videos is easy and post productions not time
consuming at set of lecture films were produced partly on student demand. These can directly
930
Anja Pfennig
be implemented in the self-study phase of inverted classroom teaching scenarios

correspondent to video captures, online lectures, written information and other teaching
ressources: Pfennig (2018), Pfennig (2019-3), Pfennig (2020), Pfennig (2022). However,
since there was no device readily available, a completely new lightboard was designed and
manufactured as a one semester student project and implemented in teaching at HTW Berlin
starting 2019.
Set up Micro
lecture
Figure 1. Examples of lightboard videos at HTW Berlin: left: production, right: lightboard video ready to use.
2. Project Lightboard – design

The fourth semester of mechanical engineering comprises of a 6 ECTS course
“interdisciplinary engineering project” to be conducted in 6 months. Students chose among
20 to 40 projects facilitated by different lecturers, that is: all students voluntarily chose the
“lightboard-project. The entire scenario accounts for 180 hours in total (contact time in
meetings is approximately 20 hours), 70 for self-studying and design, 8 hours for presentation
and 82 hours of lab work on the lightboard. The lightboard project was selected by 9 students
(generally too many, but all of the students were very eager to work on and contribute to the
project). The project was sub-classified into 3 phases: kick-off: directly at the beginning of
the semester, design: 8 weeks (+4 weeks time to adjust) and manufacturing: 12 weeks.
To keep all students in the lightboard project, they agreed on 3 teams of 3 students each
working on different designs in a competitive manner with the best design to be transfered
into the product. Teams were formed easily according to students` preferences, because these
students knew each other for more than 2 years. Work packages were agreed on 1. layout of
the design, 2. order and setting of the room for the lightboard, 3. manufacturing and redesign
and 4. documentation. The first milestone, the layout of the designs was set for a period of 6
weeks, finish after 12 weeks. This implied intensive work, but offered time for the finding of
the design, feasibility study and evaluate order and/or shipping time of components. The role
of the lecturer was to facilitate, answer questions whenever these aroused, manage conflicts
once students asked for advice or help and keep track of mile stones and dead-lines.
931
3. Project Lightboard – manufacturing

3 different designs were presented differing in complexity of the design. After presentations
and approximately 4 hours of discussion the students agreed on a design comprising of
electrical adjustment of height (Figure 2). The group defined new competencies and
responsibilites with 1 student in charge of welding, 2 for orders, 2 for room installation, 2 for
construction and 2 for documentation.
Figure 2. Images of lightboard procution and glass micro flaws/precipitations.
During manufacturing of the lightboard e.g. welding, construction, electrical installation it

was soon clear that the motor of the chosen design was not deliverable. The group agreed on
another design with the same student competencies. Another drawback was the low quality
of the glass panel consisting of inacceptable micro flaws and precipitations (Figure 1, right).
All the time the students were facilitated by the lecturer and project leader via skype, slack
and email contact. However, the rebuying of the second glass panel was done by the project
leader/lecturer. Due to the redesign the time scheme was extended 4 weeks.
The manufacturing report was delivered on time and applied for instruction manual for other
projects (HTW Berlin delivered the design in 2020 to a university in southern Germany). The
film documentation lacked of professionalism.
4. Discussion: lessons learnt, evaluation and self-reflection

Project based teaching allows students to learn science and learn how to be independent
thinkers. Students should be capable to take responsibility of their own learning process and
find their own study pathway. The project based method implements a structural teaching
change with grades not being the main focus: Balvea and Albert (2015) emphasizing in non-
technical skill development although the capability of planning actions to solve situations
might stay ominous: García (2016). However, as traditional courses do not always prepare
932
Anja Pfennig
individuals to be competitive, and consequently may have little value or relevance to students
after they graduate: Sarta (2014) project based learning prepares engineering students for
market needs in a globalized world.
The lightboard project directly involves students in the production of a teaching device
implementing their skills received in former classes. Therefore, these students were not only
engaged in the design and manufacturing itself but also in the process of project work. The
sustainability of the product, the implementation of the lightboard as teaching device further
encouraged students. Students had to arrange with different engineering related tasks such as
the design, manufacturing, welding, ordering components, handling equipment and reporting
properly on their project progress. But also social skills were accomplished, such as:
communication with sales person, team building, reporting properly written and orally,
discussions of project related problems, discussing social related problems during workflow,
to deal with unforeseen problems during the project postponing of orders and finding experts
to support (e.g. welding, calculation, etc.). For them it was the first time to actually be
involved in a real product related work giving them the feeling of importance of themselves
as engineer, person and project college.
When implementing the project based teaching method, the attitude of lecturers is of great
importance to the success of the method: Lasauskiene et al. (2015). It is strongly suggested
to assign a sufficient substructure for both the teacher and the student. Every student should
relate to a duty according to his or her own ability: Ergül et al. (2015), where their
characteristics and qualities, environmental conditions along with fundamental principles
should be taken into account: Kaya et al. (2014). Transparent steady course demands and
sufficient high quality learning material need to be provided to make the method successful
– even in terms of better grades: Kaya et al. (2014). Tasks were clear, grading was transparent
all the time (Figure 3), contact hours set, the project leader available online all the time
offering valuable advice and helping with viable tasks or decisions. Still, grading of the
course (comprising of: engineering skills, creativity, feasibility, communication,
management, documentation, reporting regularly was mixed (Figure 3): 1 students scored:
A+, 1 student: A, 1 student: A-, 2 students B and 4 students: B- (note, students were members
of different groups). The B- was mainly related to lack of participation and communication
as well as missing of deadlines and quality of the reports.
These deficits could have been detected earlier if fellow students had reported the difficulties.
Students often lack of help and support among each other when carrying out the tasks:
Gomez-Pablos (2016) and may not know how to interact effectively so that lecturers and
administration should discuss and reinforce social skills along with providing guiding
templates to help staying focused.
933
4,0
3,5 quality of design

Number of Students
3,0 30%
report 20%
No. of students
2,5
2,0 15%
20%
1,5 15%
communication
participation in
1,0 practical work
0,5 reliabilty
0,0
A+ A A- B+ B B- C+ C C- D F
Grades
1.0 1.3 1.7 2.0 2.3 2.7 3.0 3.3 3.7 4.0 5.0
Figure 3. Distribution and constitution of student grades: project lightboard.
Four draw backs during the project were vital to its ongoing: First, the insufficient quality of
the glass panel which was costly in terms of time. Second, the framing was calculated
according to a feasibility study and highest load and strain. However, aluminum profiles
distort during welding leaving the frame of the glass panel bent (it does not impair the
functionality of the lightboard). Third, the electrical motor for the electrical height adjustment
was not deliverable so that the group hat to change to a different design. Forth, was related
to the project leader because here the lack of tight guidance during the project and the
renunciation of close reporting intervals was disadvantageous in terms of project planning.
Therefore, one of the most important lessons learnt is to not accept more students that were
intended initially. And during this project, closer guidance and facilitation by the lecturer is
highly advised to keep students working on the main tasks fulfilling the work packages.
Decisions have to made earlier to direct students and it is also necessary to set non-negotiable
deadlines. These were not given ahead of the project start because a lightboard has never
been designed before (now there are custom-ordered designs available) and no one new about
the project outcome. However, the creativity of the engineering design is not suppressed with
strict framing, moreover it may help students to focus on the project outcome. The most
important lessons learnt are:
• Closer facilitation towards fulfilling milestones
• Clear advice on task (deliverables of the work packages)
• Feasibility study and component availability right from the beginning
• Mid-term reports and/or weekly reports to keep close the project proces
• 6 students are a good working group size to handle well for lecturers
934
Anja Pfennig
5. Conclusion
The peer-to-peer approach was combined with project based teaching to design and
manufacture a lightboard that may be used by every lecturer at HTW Berlin to generate low
threshold lecture videos. Advantage of lightboard videos over lecture capturing is that
students are faced directly during online teaching and are therefore secured in their teaching
process. A guided one semester project (180 hours workload, 6 ECTS) related to a complex
real engineering problem focused on customer demands and possible solutions. The
lightboard project directly involves students in the production of a teaching device and
therefore these students were not only engaged in the design and manufacturing itself but
also in the process of project work and sustainability of their project outcome. 4 work
packages were defined. Results were compiled and presented at milestones. Students had to
arrange with different tasks: design, manufacturing, welding, ordering components,
communication with sales person, team building, reporting properly, discussions of project
related problems, discussing social related problems during workflow, handling equipment,
dealing with the postponing of orders, finding experts to support (e.g. welding, calculation,
etc.). Lectures learnt to facilitate properly are: offer closer guidance, make decisions earlier
to direct students and to set non-negotiable deadlines. Quintessence is that the creativity of
an engineering design is not suppressed but supported by close guidance allowing for
students to focus on the project outcome. During the project based work students took over
the responsibility for their own learning process. Although grades were not sufficiently high,
the teaching method is still assessed as beneficial in terms of lightboard design and
construction, learning output of students and lecturer, understanding of project work,
concentration and attentiveness as well as both joy of studying and working as a future
engineer.
Acknowledgements
This project was performed with financial contribution of the Lehrefonds at HTW Berlin.
The author wants to thank Jörg Maier-Rothe for sufficient help during the project and the
students participating and widely contributing to the success of the project: JENS RADKE,
ALEXANDER RICKERT, VIKTOR GAUSS, ALEXANDER GLASUNOW,
ALEXANDER SCHOTT, SAGAR KUPPAST, DANIEL JONATHAN SETIAWAN, LEON
WECHSLER, EREN TEKGÜL.
References
Balvea,P., Alberta, M. (2015). Project-based learning in production engineering at Heilbronn
Learning Factory. 5th Conf. on Learning Factories 2015, Procedia CIRP 32, 104 – 108.
935
Efstratia, D. (2014). Experiential education through project based learning, ERPA 2014.
Procedia - Social and Behavioral Sciences 152, 1256 – 1260.
Ergül, N.R., Kargınb, E.K. (2014). The Effect Of Project Based Learning On Students’
Science, LINELT 2013. Procedia - Social and Behavioral Sciences, 136, 537 – 541.
Garcíaa, C. (2016). Project-based learning in virtual groups - collaboration and learning
outcomes in a virtual training course for teachers. Procedia - Social and Behavioral
Sciences, 228, 100 – 105.
Gomez-Pablos, V.B., Martín del Pozo, M., García-Valc, A., Munoz-Repiso, M. (2017).
Project-based learning through the incorporation of digital technologies: An evaluation
based on the experience of serving teachers. Computers in Human Behavior, 68, 501-512.
Goto, K. & Schneider, J. (2010). Learning through teaching: Challenges and opportunities in
facilitating student learning in food science and nutrition by using the interteaching
approach. Journal of Food Science Education, 9(1), 31-35.
Gulley, O.D., Jackson, A.L. (2016). A case study on using instructor-recorded videos in an
upper level economics course. International Review of Economics Education, 23, 28–33.
Huguet, C.; Pearse, J.; Noè, Leslie, F.; Castillo, R.N.; Diego, M.; Jimenez, H.A.; Patiño A.,
Mónica, A. (2017). Use of emerging technologies in flipped classes. 3rd International
Conference on Higher Education Advances (HEAd´17), 23-25 June 2017, 891-898
Kaya, H., Bodura, G. (2014). Nursing students' opinions regarding project based learning.
ERPA 2014. Procedia - Social and Behavioral Sciences, 152, 379 – 385,
Lasauskiene, J., Rauduvaite, A. (2015). Project-Based Learning at University: Teaching
Experiences of Lecturers. Procedia - Social and Behavioral Sciences 197, 788–792. 7th
World Conference on Educational Sciences, (WCES-2015), 05-07 February 2015.
Lord, T. (2012). 101 reasons for using cooperative learning in biology teaching. The
American Biology Teacher, 63(1), 30-38.
Pershkin. (2020) https://lightboard.info/
Pfennig, A., Maier-Rothe, J. (2019). “Making of" - Successfully Planning and Implementing
peer-to-peer Lecture Films. PEOPLE: Internat. Journal of Social Sciences, 5(1), 795-
815.
Pfennig, A. (2020). Matching Course Assessment of a First Year Material Science Course to
the Blended-Learning Teaching Approach. International Journal of e-Education, e-
Business, e-Management and e-Learning (IJEEEE 2020), 10(1), 53-59.
Pfennig, A. (2022). Successfully Inverting the Classroom - Recommendations to Teach (fully
online). Proceedings 6th Int. Conf. Ed. & Multimedia Techn. (ICEMT 2022), 324-330.
Sarta, G. (2014). The effects of the development of metacognition on project-based,
Learning, ERPA 2014. Procedia - Social and Behavioral Sciences, 152, 131 – 136.
Setren, E., Greenberg, K., Moore, O., Yankovich, M. (2019). Effects of the Flipped
Classroom: Evidence from a Randomized Trial, SEII Discussion Paper #2019.07, MIT
Department of Economics, National Bureau of Economic Research.
936
The impact of technology adoption in teaching and learning

within ODeL
Department of Mechanical Engineering, University of South Africa, South Africa.
Abstract
The emergence of the fourth industrial revolution (4IR) instigated various
institutions of the world to move towards technology integration in teaching
and learning. The purpose of the current study was to assess how impactful
was Wolfram System Modeler (WSM) software to students in their studies by
utilising the technology acceptance model. MS forms online survey was utilised
to collect data from participating students of the Solid Mechanics (SME3701)
module. SME3701 module was targeted whereby 95 students participated.
Data analysis was done through IBM SPSS Statistics. The findings of the study
indicated that students found the software WSM interesting as evidence of the
module performance as compared to before the introduction of the technology.
The module performance improved drastically as evidence of the students
‘acceptability of the technology.
Keywords: Teaching and learning; Wolfram System Modeler.

The impact of technology adoption in teaching and learning with ODeL
1. Introduction
The emergence of technology during the current digital age has transformed the distance
education space into what is currently. Technology is playing a fundamental role to deliver
modern engineering education to the next generation (Abbad, 2011). Apparently,
appropriate, and yet effective utilization of technology is tremendously important for distance
education (Amanullah, Chandran, and Stojcevski, 2014). In the view of Arinto (2013), the
transition of distance education emerged as far as correspondence to open distance learning
(ODL) and eventually to open distance e-learning (ODeL). ODeL platform allows teaching
and learning to take place due to technology enablement without the confinement of the
physical building (Arinto, 2016). The teaching and learning in an ODeL platform permit
learning to take in both synchronous and asynchronous (Alfonso, 2012). The University of
South Africa (UNISA) is one of the distance education universities that experienced the
mentioned evolution of distance education to now Ode since 1950 (Arinto, 2016; Arinto,
2013). Numerous universities all over the world are gradually shifting from a predominantly
print-based mode of delivery to an online mode characterized by using virtual learning
environments and various web technologies (Arinto, 2016).
Notably, the evolution of the Fourth industrial revolution is exponentially growing more than
ever before while disrupting industries all over the world. 4IR is characterized by 3D printing,
the Internet of Things (IoT), and the fusion of technologies (Xu, Min, Jeanne M. David, and
Suk Hi Kim, 2018) influencing the education sector to follow suit. Educational technology
can be defined as the study and ethical practice of facilitating learning and improving
performance by creating, using, and managing suitable technological processes and
resources. To recognize the value of such technology, it is essential to familiarize ourselves
with the technology adoption process and the factors affecting it (Kannan, Punithavathi, and
Sambandam, 2018). Hence, the current study utilizes TAM to assess technology adoption in
the institution of higher learning as is preferred in various sectors.
2. Methodology
This Tavallaei and Talib, (2010) mention that research methodology refers to the researcher’s
general approach in carrying out the research project. As of the current study, researchers
earmarked research problems to be solved within the institution of learning utilizing TAM in
view of collecting quantitative data by means of an online survey (Microsoft forms) from the
Advanced Diploma students at an institution of higher learning in South Africa. The
quantitative research approach (Creswell, 2014) was found to be suitable for collecting
quantitative information from the students utilizing the TAM survey. The collected data was
focused on the Solid Mechanics IV module in the Advanced Diploma course in Mechanical
Engineering for the first semester, of the 2022 academic year. The module’s main content is
938
vibration analysis. For the year 2022, seven assessments were given to students, four of the
assessments were based on Wolfram System Modeller (WSM). During the assessments,
students were required to analyze a physical model problem, and solve it using mathematical
equations. They would then construct the physical model onto WSM, insert boundary
conditions for the model operation, and then simulate to obtain and interpret the mathematical
analysis on a graph.
Research samples are participants from whom data is collected (McMillan & Schumacher,
2014). In view of the current study, the module has 158 registered students, however only
138 fully participate in the assessments given by the lecturer. For this survey, 95 students
voluntarily participated, as was aligned with convenience sampling which deals with the
availability of selected participants (Creswell, 2014; McMillan & Schumacher, 2014). The
online survey form was structured into 3 categories, sections A, B, and C. Where section A,
comprised demographic information, section B concentrated on quantitative data regarding
Perceived ease of use (PE), Perceived usefulness (PU), Attitude (AT), and Intention to Use
(ITU). While section C collected the student’s personal views and comments about WSM. A
5-point Likert scale was utilized to collect the quantitative data in section B. McMillan and
Schumacher (2014) viewed Likert scale questionnaire as the supreme and most extensively
used form of a scaled questionnaire. While SPSS (Statistical Package for Social Sciences)
was used to analyze the quantitative data.
The validity and reliability of the research instrument provide lead to meaningful
interpretations of data (Creswell, 2014) and for decision making. Statistical analysis was done
to determine the internal consistency between the questions in the questionnaire. The
Cronbach Alpha reliability test tabulated in Table 1 resulted in a correlation coefficient of
more than 0.70 which is viewed as a high internal consistency. If the items are strongly
correlated with each other, their internal consistency is high, and the alpha coefficient will be
close to one as is of the current study (0.918) (Fontanilla, 2016).
Table 1. Cronbach’s Alpha Reliability Test.
Cronbach's Alpha Cronbach's Alpha Based on Standardized Items N of Items
0.918 0.923 17
3. Results & Discussion

The results of the current study are comprised of demographic (section A), quantitative
survey (section B), and qualitative analysis (section C) as explained in the above section.
939
3.1. Demographic results

Demographic data were gathered for all the 95 participants considered for analysis with
(26%) female and (74%) male participants in the survey. With 0% in the 15-19 age group,
50.5% in the 20-29 age group, 42.1% in the 30-39 age group, 6.3% in the 40-49, and only
2.1% in the 50+ age group. The number of e-learning courses taken by the student during the
semester, with 3.2% having taken 1 module, 7.4% with 2 courses, 27.4% with 3 courses,
21.1% with 4 courses, and 42.1% with above 5 courses. Only 54.7% had e-learning courses
that incorporated computer simulations, these 52 students were registered for Solid
Mechanics. 33.7% had 2 courses with computer simulations, 9.5% had 3 courses, 1% had 4
courses, and 1% had 5 of these courses. Out of the total number of participants, 89.5% were
exposed to computer simulation software, while 9.5% have not been. 83.2% had personal
computers that meet the minimum requirements for the software to run, while 8.4% had not
and 7.3% of students answered may be.
3.2. Student ease of use of WSM

Figure 1 shows the perceived ease of use of WSM. The response of the students was rated on
a scale from 1 to 5, with 1 being strongly agreed and 5 being strongly disagreed. Finding
WSM easy to use, 47.8% of students strongly agreed. The average percentage of students
learning to use WSM was 48.8% strongly agreeing. The interaction with WSM being clear
to the students was on an average percentage of 48.2% strongly agreeing. The average for
students being able to solve complex engineering problems related to Solid Mechanics using
WSM was an average percentage of 53.6% strongly agreeing.
Perceived usefulness plotted in Figure 2 for using the WSM to enhance the student’s
effectiveness in learning, an average percentage of 48% of strongly agree was recorded.
Using WSM to improve the student’s course performance was averaged to a percentage of
47.4% strongly agreeing. For using WSM to increase the student’s productivity in their
course work, the average percentage was 50.4% strongly agreeing, and finding WSM useful
in their studies, was averaging 45.2% strongly agreeing.
Attitudes from Figure 3, the student’s results for disliking the idea of using WSM in their
studies 73.8% strongly disagree. students found a generally favorable attitude toward using
WSM was 45.6% strongly agreeing. For students believing that it’s a good idea to use WSM
for their course work 41.4% strongly agreed and among those finding WSM, a foolish idea
81.4% strongly disagreed.
Intention to Use results is shown in Figure 4. The student’s response to WSM making
teaching and learning interesting was 41.4% strongly agreed. Of students who intend to
continue using WSM during the semester were 43.2% strongly agreeing. Students who in
intend to continue checking announcements from WSM frequently averaged 46.4%. Students
940
who intended to recommend WSM to their peers in the industry averaged 46.4%. And
students who intend to be heavy users of WSM averaged 51.4%.
Figure 1. Perceived ease of use (PE).
Figure 2. Perceived usefulness (PU).
941
Figure 3. Attitude (AT).
Figure 4. Intention to Use (ITU).
3.3. Student’s personal views and comments about WSM

With the integration of WSM into the module, the gap between experiments and theoretical
studies was bridged. With the complexity of the vibrations system, WSM was able to improve
the student’s understanding of the module by displaying these vibration systems, simulating
the model according to various boundary conditions, and plotting various graphs to interpret
the model.
942
Challenges were encountered by students when their computers were not able to meet the
minimum requirement for the software to operate. Third-party software had to be installed to
assist the software in running, this then resulted in extra resources being used for this. Cloud-
based operations are recommended as they are much easier for students to access. Due to the
complexity of the software, resources were provided by the lecturer, however, this does not
suffice, as students must learn to navigate the software before attempting to construct the
model. This is evident that a support structure must be put in place to assist students more.
According to the lecture’s experience with the integration of WSM into the module, students
who frequently consulted were able to troubleshoot the challenges incurred.
Despite all the challenges, students’ attitude towards the software was good. The student’s
reaction toward WSM taking the place of the theoretical approach to solving engineering
problems, was that they acknowledged that the theoretical approach had to be done, and it
was complemented by the software’s ability to interpret the theoretical condition they
inserted.
It was noted by the lecturer that since incorporating WSM in the module, the student’s
performance has increased. Students found that WSM contributed to their performance by
giving more knowledge, however, some did mention that due to the pressure and the short
duration of the semester, they were not able to use the software effectively. They also
emphasize that WSM should have its separate module so that when it’s integrated into the
different modules, they are able to use it effectively.
When the students were asked, “Will you recommend WSM to other students?” 65% said
yes. The remaining number includes those who did not know and those who felt that they
had not used the software effectively to be able to recommend it to other students.
4. Conclusion
The current study utilised the technology adoption model to access how students viewed the
integration of WSM into the SME3701 module at the diploma level in the department of
Mechanical Engineering. As alluded to earlier, TAM focussed more on Perceived ease of use
(PE), Perceived usefulness (PU), Attitude (AT), and Intention to Use (ITU). The research
findings clearly indicate that the introduction of WSM software into the module brought
some hope to the students as they received the initiative with open hands. It was also good
that almost all students registered for the module has access to the computer and could easily
interact with the software. The students were now able to simulate, plot graphs, and interpret
the data without any challenges. The lecturer was able to provide background information
regarding the purpose of the software and how it would assist them to understand the module
better. As result, the students perceived the software and the intervention positively. They
also find it userfriendly and developed a positive attitude towards it. The increase in the pass
943
rate of the module was a good indication that students found WSM very useful. When they
were asked a question if they would refer other students to this software almost 65 % agreed
to indicate their intention to further utilize the system.
The findings also indicated that the age group of these students is aligned to the current
technological dispensation as consistent with Porter and Donthu (2006) in their study. Young
students find technology interesting, and it enhances their learning experiences.
TAM, in this regard, managed to assist in determining how students found the adoption
students WSM in their SME3701 module as supported by Park (2009). Both quantitative and
qualitative results concur with each other that student welcomed the integration of the WSM
in their module as their positive performance speaks a volume.
The adoption of technology in teaching and learning can assist academics in better presenting
their content to students since most institutions are now embracing online learning. It is
therefore recommended that a similar study can be adapted to other modules in the faculty to
encourage other academics still reluctant to utilize technology for teaching and learning to
follow suit. The adoption of technology in teaching and learning is slowly becoming the
reality in this dispensation of 4IR technologies.
References
Abbad, M., (2011), April. A conceptual model of factors affecting e-learning adoption. In
2011 IEEE Global Engineering Education Conference (EDUCON) (pp. 1108-1119).
IEEE.
Alfonso, G. J. (2012). UP Open University: Thoughts about openness in a digitized world
[Powerpoint slides]. Presentation at the UPOU Roundtable Discussion, UPOU Oblation
Hall, Los Banos, Laguna.
Amanullah, M.T.O., Chandran, J. and Stojcevski, A., (2014). Technology adoption in
engineering design for distance education. International Journal of Quality Assurance in
Engineering and Technology Education (IJQAETE), 3(2), pp.54-64.
Arinto, P.B., (2013). A framework for developing competencies in open and distance
learning. International Review of Research in Open and Distributed Learning, 14(1),
pp.167-185.
Arinto, P.B., (2016). Issues and challenges in open and distance e-learning: Perspectives from
the Philippines. International Review of Research in Open and Distributed Learning,
17(2), pp.162-180.
Creswell, J.W., 2014, Research design: Qualitative & quantitative approaches, Sage, London.
(3rd ed.). Thousand Oaks, CA: Sage
Fontanilla, H.S., (2016). Comparison of beginning teachers' and experienced teachers'
readiness to integrate technology as measured by TPACK scores (Doctoral dissertation,
Brandman University).
944
Kannan, R.J., Punithavathi, P. and Sambandam, N., (2018), November. Technology

Adoption Models–Adoption of ICT in Educational Institutions in India. In 2018 World
Engineering Education Forum-Global Engineering Deans Council (WEEF-GEDC) (pp.
1-6). IEEE.
McMillan, J. & Schumacher, S. 2014. Research in Education James McMillan Sally
Schumacher. In Pearson Ed ed. J. McMillan & S. Schumacher (eds.). United States of
America: Pearson New International Edition Research in Education Evidence Based
Inquiry. 159–163.
Park, S.Y., (2009). An analysis of the technology acceptance model in understanding
university students' behavioral intention to use e-learning. Journal of Educational
Technology & Society, 12(3), pp.150-162.
Porter, C.E. and Donthu, N., (2006). Using the technology acceptance model to explain how
attitudes determine Internet usage: The role of perceived access barriers and
demographics. Journal of business research, 59(9), pp.999-1007.
Tavallaei, M. and Talib, M.A., (2010). A general perspective on role of theory in qualitative
research. Journal of International Social Research, 3(11).
Van Vugt, M., Hogan, R., & Kaiser, R. B. (2008). Leadership, followership, and evolution:
Some lessons from the past. American Psychologist, 63(3), 182-196. doi: 10.1037/0003-
066X.63.3.182.
Xu, Min, Jeanne M. David, and Suk Hi Kim. "The fourth industrial revolution: Opportunities
and challenges." International journal of financial research 9, no. 2 (2018): 90-95.
945
Moodle-based e-learning courses for introduction to critical

thinking in a multidisciplinary perspective
Jean-François Bodart1, Anne-Frédérique Paul-Antoine2, Aline Stevenoot3, Corinne
Vézirian4
1
Univ. Lille, CNRS, UMR 8576–UGSF–Unité de Glycobiologie Structurale et Fonctionnelle,
F-59000 Lille, France, 2Univ. Lille, Laboratoire HT2S, EA 3716 – CNAM, F-59000 Lille,
France, 3Univ.Lille., TVES, ULR 4477, F-59000 Lille, France, 4Univ. Lille, INSPE Lille
HDF, F-59000 Lille, France.
Abstract
A course design to introduce to critical thinking was built in the context of the
need of developing these skills, with aiming at proposing this course beyond
disciplinary knowledges. Critical thinking may be considered a process that
transcends disciplines, committing students to be actors of their learning. Due
to constraints related to the high number, the diversity of students (<500
students/session), this course was proposed as an online learning environment
on Moodle. The course contains learning modules including resources and
interactive components. We discuss here the strengths and limits of this course
design. Optimization and opportunity to adapt the framework of online
learning environments for students to exercise critical thinking are considered.
Keywords: Critical thinking; e-learning; Moodle; learning environment.

Moodle-based e-learning courses for introduction to critical thinking
1. Introduction
1.1 Context
The need for the development of critical thinking relies upon the confrontation with
economic, social, human and environmental issues, in a state of political and technical
reconfiguration. Critical thinking might be considered as a set of requested tools for students’
own development but also for caring for others and our environment. Being a set of attitudes
and habits related to discernment, thinking and vigilance over one's judgments and those of
others, it shall be included in all academic program. Indeed, critical thinking is a central
component for personal and professional autonomy, for social integration and significant
contribution at scientific level. One shall note also that critical thinking dynamics is never a
definite asset and should be constantly nurtured. Therefore, the development of critical
thinking has been promoted both at the French national level - being an institutional
injunction (Kammerer, 2021) - and at the international level, being promoted by the United
Nations Educational, Scientific and Cultural Organization (UNESCO) or the Organization
for Economic Cooperation and Development (OECD) guidelines.
Critical thinking is widely acknowledged as a state of mind and a set of practices that nurture
and reinforce each other’s. Its place in education is crucial as underlined by the analysis of
Sellars et al. (2018) or Southworth (2022). These skills are called to play a major role i.e., for
sustainable development or promotion of peace.
Our institution offers a wide range of educational programs in the disciplinary fields of Law,
Economics, Management, Life and Health Sciences, Sciences and Technologies, Humanities
and Social Sciences. More than 79,000 students are spread across 6 campuses located on
different sites. Our course was developed facing two challenges: (1) to provide a course
conceived as an introduction and an awareness-raising to critical thinking, while (2) offering
this teaching in a multidisciplinary manner, so that it is de-compartmentalized and accessible
to all undergraduate students (second and third year), regardless of their discipline or campus.
1.2 Critical thinking nature and training

While the nature of critical thinking itself might be debated (Sellars, 2018), one can point out
that substantial contributions have been elaborated to provide a common language around
the concept and the specific capacities, attitudes and cognitive strategies intrinsically linked
to the exercise of critical thinking (Ennis, 1987; Paul et al., 1989).
Critical thinking is often referred as a reflective process in which the propensity for self-
correction is essential (Ennis, 1987; Lipman, 2003) and being best practiced when students
have an in-depth knowledge in the considered field (McPeck, 1981).
948
Jean-François Bodart et al.
In order to offer an educational program in a multidisciplinary manner, we focused on the

non-disciplinary dimensions of critical thinking since it can be referred to a global set of
minds and a wide set of skills that are applied to any subject and/or action, as highlighted by
Dekker (2020). Indeed, key factors in critical thinking development may rely upon a
multidisciplinary curriculum, a student-centered pedagogy, and the diversity within the
academic community engaged in the teaching (Dekker, 2020). We also based our approach
by integrating a critical thinking definition encompassing those from primary schools to
universities (Eduscol 2016). The latter stresses out (1) attitude such as curiosity, autonomy,
lucidity, modesty and listening, which are often omitted when considering critical thinking
(Thonney and Montgomery, 2019), as well as (2) the necessity to discriminate between facts
and interpretation, knowledge and emotions, inquiries and exchanges.
1.3 Moodle-based e-learning courses for critical teaching

We faced pedagogical, practical and administrative challenges: how to provide an awareness-
raising courses to critical thinking, while offering it to a very large audience and taking in
account (1) the heterogeneity of the students’ profiles (i.e., disciplinary fields, campus
localizations) and (2) a sustainable offer regarding the dimension of the administrative staff
in support? In a context of the massification of higher education, we aimed at conceiving
these courses to be open to every second- and third-year undergraduate student, while being
accessible at any time, in order to increase opportunities for students to enroll and
conveniently follow these courses.
E-learning appears as a counterintuitive solution for introducing to critical thinking since it
relies in part on social interactions and skills. However, critical thinking e-learning
environments have been experienced and their efficiency analyzed (Haghparast et al., 2014,
Chou et al., 2019, Puig et al.,2020). Thus, we chose to develop an autonomous and
asynchronous courses without individual following, with a fully automated examination
process. Moodle was chosen being an open-source learning management system that allows
users to develop e-learning, providing amenability and integration of various types of
learning sources (Hirschel 2012).
Here we discuss several aspects like the amenability of e-learning, student autonomy, success
rates, interest generated among colleagues, and the fact that this teaching can serve as a basis
for the construction of other teaching units where critical thinking plays a predominant role.
2. Courses design
This optional courses were proposed to second- and third-year undergraduate students. The
multidisciplinary academic team, which conceived the courses are mainly in the disciplinary
fields of Sciences and Technologies, and Humanities and Social Sciences. We also articulate
949
our constraints with the three pillars of pedagogical alignment in the construction of this
teaching: objectives, teaching activities and assessment (Biggs, 2003). The courses aims, its
organization, functioning, timeline and the evaluation process were clearly outline in an
introductory section. A forum was open to answer the different questions and requests of the
students.
Several aims were defined (1) to reinforce the overall reflexive capacity to increase the
freedom of thought and the ability to make choices, (2) to perceive the limits of one's natural
abilities to evaluate information in certain circumstances and to demonstrate both modesty,
lucidity, caution and listening, according to the information and circumstances; (3) to acquire
new tools and criteria to evaluate the qualities of information and to distinguish facts,
knowledge, opinions and beliefs; (4) to raise the awareness of taking into account the
diversity of perspectives to help develop constructive and respectful communication
strategies with others. Activities were built according to these objectives.
Figure 1. Overall scheme of teachings (academic year 2022-2023). Courses were provided online using Moodle
platform. Five sections were proposed and self-assessment of the understanding of the courses to the students.
The courses were made up with five main sections: (1) how to define critical thinking, (2)
stimulation of reflexive activity by watching a classic of American cinema: “12 angry men”
(the latter movie can be referred to as an example to illustrate the manifestations of critical
950
thinking (Boisvert, 2015)), (3) to address the place of reasoning in discussions, (4) the
definition of a fact and finally (5) how to exert critical thinking in a media society. Overview
of the architecture, calendar and assessment schedules is provided in figure 1. Each part of
the courses were built upon various activities posted on a Moodle page: guided analyses of
films, texts, images, online tests (online multiple-choice quiz, with immediate feedback), etc.
Students were asked to complete all of these activities while carefully following instructions.
Three times during the semester were organized on-line meeting with students using zoom
software.
Learning outcomes were assessed as follows: two continuous assessment evaluations (CC1
and CC2): CC1was related to part 1, part 2, and part 3 of the courses while CC2was related
to parts 4 and 5 of the courses. A "second chance" exam was proposed before the end of the
semester to students who did not have the overall average on the first two exams as following
the rules of the University. During the duration of the evaluations (48 hours), the on-line
pedagogical resources were hidden. The tests allowed the teachers to verify that the students
had consulted the available resources, that they had understood and assimilated the key
concepts explained in these resources, and which students were then able to implement
simple skills in simple case analyses. During the academic year 2022-23, 466 students
enrolled and participated to the learning outcomes evaluation. Furthermore, 40 students were
enrolled but did not participate in the exams.
3. Evaluation
We performed courses evaluation in order to improve the teaching and to enable teachers to
undertake a reflective approach to the courses. This evaluation combined quality assessment
and analysis of students’ academic results at the end of the courses. To avoid and minimize
the risks of complacency bias or any Hawthorne-type effect, the evaluations were carried out
by an independent service of the university, devoted to pedagogy and Innovation support.
The latter are discussed in the following section. The courses quality assessment was based
upon an anonymous questionnaire addressed to the students using the LimeSurvey software.
It was made up of 28 closed questions (regarding general appreciation, contents, animation,
interactions and activities) and 4 open questions (Why do you choose to follow these courses?
What are the strengths of these courses? What proposal would you make to improve the
courses? Do you have any further remark?)

Our institution is acknowledged for both the excellence and diversity of its research and its
educational programs, as well as for its territorial anchoring. It aims to engage students to be
actors, as citizens, and to position themselves through skills such as those of critical thinking,
951
that implement both careers and personal lives. Several teaching formats that mobilize critical
thinking are proposed in the disciplinary fields but few courses are offered to respond both
(1) to the heterogeneity and diversity of our students and (2) to the need to develop critical
thinking skills. Therefore, we specifically developed a course entitled “introduction to critical
thinking”, built in the specific scheme of courses designed for transversality and enrollment
of large groups (up to 600 students). The latter courses, which were optional and proposed to
second- and third-year undergraduate students immediately found its audience: 420 students
in 2021 and 506 students in 2022. The profiles of these students exhibited diversity, since
these students engaged in degrees in different disciplinary fields such as law, political
sciences, economics and social administration, education and training sciences, sociology,
economics and management, information sciences or linguistic studies.
Regardless of the courses attractiveness, we observed attendance to the tests and
examinations that were above the average (personal observations). More than 90% students
finished the courses in 21-22 and 87% in 22-23. In 22-23: 466 students participated in the
evaluation tests. Out of the 466, 71 students did not get the average at the end of the
evaluation test and therefore performed the so-called 2nd chance test. At the end of this test,
only 25 students did not validate the courses.
By relying on the results and observations from questionnaires, interviews and analysis of
student productions, several teaching strengths emerged: (1) asynchronous Moodle based-
teaching unit (mentioned by 88 students out of 190 responses; “distance”, “no time
constraint”, “freedom of organization”, autonomy were appreciated, and could be correlated
to the reason why the students chose this courses (53 students out of 196 responses), (2)
contents (18 students out of 190 highlighted the nature of the activities and their diversities
(videos, online tests,…)).
This diversity of different didactic sensitivities and approaches of the teachers appeared in
the feedback from the evaluation, as well as a strength and as a source of complexity. While
proposing rich and original activities, it appeared that the latter could be adapted and be more
concise or request a smaller workload. Some activities could be more efficient if shorter,
according to the students, in order to allow them to develop synergy between the time
management and the dynamics of the courses. We undertook an adaptation of the distribution
of the workload.
More video resources and more interactive resources could accentuate the originality of the
offered supports and to promote dynamism. More sensitive themes and more current events
are a stimulating development track, thus illustrating the need to resolve the cognitive gap
inherent in critical thinking between emotion and reasoning.
We also noted that “12 Angry Men”, the cinematographic work that introduced the courses,
was well perceived, as well as its input. It drove the viewer to widen their scope and to put
952
in perspective their own opinions. Beyond prejudices and motivations, it is here the
mechanism of critical thinking that is highlighted (Boisvert, 2015).
Several areas for improvement have emerged and remarks that rose during the quality
assessment of the courses mirrored considerations that had been at the core of the creation of
these courses. The areas for improvement raised by the students mainly concern the
evaluation methods (46 students/181 responses, complexity of the questions, other methods,
clarity of the questions) and the workload represented by the activities (22/181).
We also identified several limits. The first limit is related to the importance of regulated
discussions (an essential didactic tools for the confrontation of facts, ideas and opinions).
This dimension is hindered by the intrinsic potentials and limitations of Moodle. Several
verbatim from students expressed their desire to participate in these intellectual exercises in
the context of the courses, but the pedagogical team was not able to overcome the constraints
that a large audience generates on Moodle. In contrast, the meeting organized online had very
mitigated success (attendance was very scarce, never reaching more than 30 students) and
revealed mostly administrative questions, the need for collective reassurance and little
exchange about critical thinking. One might also interpretate the low number of students in
the online sessions as a sign that the students have taken the courses in complete autonomy.
The availability of teachers and administrative staff has not been questioned, indicating that
the interactions setup and the information made available are adequate. Together with mail
exchanges, this online meeting appeared to be sufficient for students’ assistance and guidance
throughout the courses. Both online meetings and evaluation emphasized that the themes and
methods used during the courses were discussed in larger settings (friends, family, other
courses).
The second limit was related to the difficulty to evaluate skills related to critical thinking
online. The Gordian knot of the pedagogical relationship remains the act of evaluation, so
the creation of the different controls generated questions and dissatisfaction, which found an
echo among the students both for the length and level of difficulty. A few questions were
considered based more on memorization skills than critical thinking and “disconnected from
the content of the courses”. These questions were revised and modified. Evaluating open-
mindedness, curiosity, and modesty remain the challenge to be taken up.
Another point of consideration was the architecture of the courses on Moodle, since it is a
crucial point for the operability of these courses. We aimed for clarity and efficiency. The
inherent limits of Moodle can make one fear a lack of fluidity and logistical heaviness. The
courses quality assessment resonates these concerns, but the quality of the courses structure
and the clarity of the instructions was also highlighted. This contrast might be related to the
heterogeneity of students’ relation towards Moodle and e-learning.
953
The conclusions of the courses evaluations lead us to several observations, including that the
students appreciated developing their critical thinking and became aware of the abilities that
are requested to maintain a critical thinking. These courses has raised an interest in other
departments in our university, which asked for their students a free access to evaluate the
interest of these students in such program. Therefore, teaching supports and assessment tests
could be used, adapted and might be proposed in another format, to increase and maintain
students' enthusiasm in practicing critical thinking, and ensuring that they can deepen the
concepts presented and apply them in cases of increasing complexity according to their
discipline, which could optimize their learning. Two constraints potentially limit its
expansion within our institution, linked to the human resource that is requested both at the
pedagogical and the administrative levels for maintenance of online teaching and satisfying
levels of interactions with students.
Acknowledgments
The authors would like to thank Yanele Poteaux, Céline Meurice and Aurélie for their
pedagogical assistances and fruitful discussions.
References
Biggs, J.B. (2003). Teaching for quality learning at university. Buckingham: Open
University Press/Society for Research into Higher Education.
Boisvert J. (2015) Critical thinking: definition, illustration and applications. Revue
Québécoise de psychologie, 36(1), 3-33.
Chou, T.-L., Wu, J.-J., & Tsai, C.-C. (2019). Research Trends and Features of Critical
Thinking Studies in E-Learning Environments: A Review. Journal of Educational
Computing Research, 57(4), 1038–1077. https://doi.org/10.1177/0735633118774350
Dekker T.J. (2020). Teaching critical thinking through engagement with multiplicity.
Thinking Skills and Creativity, Volume 37, https://doi.org/10.1016/j.tsc.2020.100701
Eduscol (2016) Former l’esprit critique des élèves, Retrieved from
https://eduscol.education.fr/1538/former-l-esprit-critique-des-eleves
Ennis, R. H. (1987). A taxonomy of critical thinking dispositions and skills. In Baron, J.,
Sternberg, R. (Eds.), Teaching thinking skills: Theory and practice (pp. 9–26). New York,
NY: W. H. Freeman.
Haghparast, M., Nasaruddin, F.H., Abdullah, N., Cultivating Critical Thinking Through E-
learning Environment and Tools: A Review, Procedia - Social and Behavioral Sciences,
Volume 129, 2014, Pages 527-535, doi.org/10.1016/j.sbspro.2014.03.710
Hirschel, R. (2012). ‘Moodle : Students ’ perspectives on forums, glossaries and quizzes’,
Jalt Call, vol. 8, no. 2, pp. 95-112.
Kammerer B. (2021) L’esprit critique est-il (encore) une valeur républicaine ? L'école des
parents, 2021/1 (n° 638), p. 41-43. DOI : 10.3917/epar.638.0041.
954
Lipman, M. (2003) Thinking in education. New York : Cambridge University Press.

McPeck J.E. (1981) Critical thinking ad education. New York, NY: St. Martin’s press.
Paul, R., Binker, A., Martin, D., & Adamson, K. (1989). Critical thinking handbook : High
school (a guide for redesigning instruction). Foundation for Critical Thinking. ISBN-0-
944583-03-2
Puig, B., Blanco Anaya, P., & Bargiela I.M. (2020) Handbook of Research in Educational
Communications and Technology pp 345–362 . A Systematic Review on E-learning
Environments for Promoting Critical Thinking in Higher Education DOI: 10.1007/978-
3-030-36119-8_15
Sellars, M., Fakirmohammad, R., Bui, L., Fishetti, J.; Niyozov, S.; Reynolds, R.; Thapliyal,
N.; Liu-Smith, Y.-L.; Ali, N. (2018) Conversations on Critical Thinking: Can Critical
Thinking Find Its Way Forward as the Skill Set and Mindset of the Century? Educ. Sci.,
8, 205. https://doi.org/10.3390/educsci8040205.
Southworth, J. (2022). Bridging critical thinking and transformative learning: The role of
perspective-taking. Theory and Research in Education, 20 (1), 44–63. https://doi
10.1177/14778785221090853
Thonney T. and Montgomery J.C. (2019) Defining Critical Thinking Across Disciplines: An
Analysis of Community College Faculty Perspectives, College Teaching, 67:3, 169-176,
DOI: 10.1080/87567555.2019.1579700
955
Tailoring professional development to postgraduate students: it’s

not the chicken, it’s the egg
Marianne Savory
Work and Employment Relations Department, Leeds University Business School, UK.
Abstract
Graduate employability and professional development are more commonly
discussed in an undergraduate context. Yet many postgraduate students seek
professional development as part of their programme of study. There are
challenges in providing an in-programme tailored, person-centered
professional development offer at post graduate level when student cohorts are
often large and international. This case study explores a professional
development provision embedded into a master's programme, centred on
developing self-reflection through experiential learning approaches. The
students tailor their path through taught workshops and lived experiences, to
recognise gaps and take actions to fulfil their development needs; during their
studies, and as reflective practitioners of the future. The paper concludes there
is a need for self-reflection to be practised and assessed as part of a
professional development provision at master's level, but that measuring the
impact of developing this skill in an individual cannot be accurately captured
immediately post-graduation.
Keywords: Postgraduate; professional development; employability; self-
reflection; tailored.

Tailoring professional development to post graduate students
1. Introduction
Graduate employment outcomes, captured in the Graduate Outcomes Survey (GOS) and
other employability measures are of significance to the ‘triple helix’ of higher education
institutions (HEIs), industry and government (Etztowitz, 2003). These outcomes have
societal and economic impacts and are cited as one of the top reasons students chose to go to
university by Universities UK (2022). Employment outcomes are used in the Financial Times
business school rankings as a measure of return on investment. Such crude calculations
overlook additional gains for individuals, some of which may not be recognised until later in
their working lives. Capturing postgraduate outcomes in this way is less common.
‘Employability’ on the part of the individual depends on the knowledge, skills and attitudes
and the way those assets are used and presented to employers (Hillage & Pollard, 1998).
McQuaid and Lindsay (2005, p. 205) argued such a “hollowed out” definition of
employability focuses on the supply side only, ignoring labour market demand that enables
those with the requisite competencies to become employed. Winterton and Turner (2019, p.
538) show inconsistencies in the attributes employers claim to want from graduates.
Notwithstanding these concerns, most HEIs develop employability practices that focus on
enabling students to achieve GOS measures of ‘success’ by developing skills and work
readiness techniques. Graduate skills and attitudes most desired by employers are prominent
in HEI curricula, but overwhelmingly focused on undergraduate provision.
Graduates are not “oven ready chickens” perfectly prepared for the graduate jobs market
(Atkins, 1999) and two graduates from the same programme do not necessarily have the same
professional aspirations. Rather than seeking to deliver the chicken, HEIs might better focus
attention on the egg: the egg as the individual; the nucleus, with all the potential to develop
into a functional individual who can contribute to, and gain satisfaction from, the world of
work. The outcomes for the egg are in many ways yet to be determined, depending on nurture,
nourishment and support. A successful HEI employability offer should be centred on
developing self-reflection and personal action and in turn, providing the egg with what is
needed. This kind of employability input should start much sooner than a student’s arrival in
a master’s programme, as early as primary education according to Kashefpakdel, Rehill &
Hughes (2018). The role of HEIs is to enable students to identify and explore their
professional development needs and provide a menu of opportunities to meet these.
Considering ‘sought after’ skills can be useful in framing professional development provision
for post graduates. However, re-focusing such provision to a person-centred approach, with
the student owning their professional development and tailoring their learning through
effective self-reflection and action, offers a more flexible and aspirational approach to
employability. In this way students can determine what ‘success’ means to them as
individuals rather than trying to conform to a ‘one size fits all’ measure.
958
Marianne Savory
2. Professional development provision

For the postgraduate taught programme in human resource management at a Russell Group
business school with a large and international student cohort, the professional development
provision evolved over a decade into an offer tailored to individual needs. The provision is
embedded and recognised as a key component of the programme and while not credit bearing
in itself, it is part of the formal timetable and is assessed through the core HRM academic
module of the programme. Professional development provision runs parallel to the core
module, providing professional development opportunities and experiences in the context of
the academic programme as part of co-curricular activities. Students attend professional skill
workshops where they are introduced to a development area and provided with context (made
aware of why each is being included in their learning offer). The workshops are often active
learning experiences in themselves (team challenges, Lego activities, business simulation,
action maze, group discussions, for example). They attempt to develop personal and
professional skills in the student such as teamworking, presentation, cross cultural
communication, commercial awareness, resilience and reflection, alongside digital and
technical skills such as video presentation, project management and performance
management. Collectively, these workshops provide a buffet of opportunity for students to
experience and practise each skill.
Simultaneously, students are part of ‘activity teams’ that provide vehicles for self-directed
experiential learning in team working and collaboration, cross cultural awareness,
communication and presentation skills, all set in a framework of self-reflection. Students are
ultimately assessed in the core academic modules through team presentations on an academic
issue, an individual video presentation and, most significantly, through an individual
reflective essay. In this way students experience ‘authentic assessment’ (Gulikers, Bastiaens
& Kirschner, 2004, Pg. 69) that is meaningful and contextualised. Motivation for this
approach is captured well by Fonteijn and Decker (2021, para. 6): “this is all necessary
because we don’t want our students to feel that their education is separate from their personal
growth and that they are merely jumping through academic hoops until graduation, when
their lives start.”
The provision also involves a range of optional peripheral opportunities (also contained
within the programme rather than centrally positioned) such as e-mentoring, industry
projects, networking/alumni events and professional accreditation by MOS (Microsoft Office
Specialist) the CIPD (Chartered Institute of Personnel & Development). The central
university careers service and the HRM programme intersect to bring value in terms of
supporting students with the techniques and etiquettes necessary to enter the graduate labour
market, such as job searches, application tuition and mock interviews. The focus here is on
the process of gaining entry to the graduate labour market, while the content for the
applications and interviews is primarily gained in the above provision.
959
3. Intentional design principles

The design of the postgraduate professional development deliberately involved the following
principles:
• Experiential, active and uncomfortable

• Engagement, feedback and belonging
• Integrative learning and assessment
3.1. Experiential, active and uncomfortable

Kolb’s learning theory (Kolb, 1984) informs the experiential and active approach that
underpins the professional development provision in this instance. Experiential teaching and
learning practices of this nature are encouraged by business school accrediting bodies like
AACSB (LeClair, 2018), but come with warnings of ‘explicit moral duty’ on the part of the
HEI to protect all stakeholders from what can create uncomfortable and new learning
experiences (Lund Dean, Wright & Forray, 2020). However, in general terms, and depending
on a students preferred learning style, being challenged to learn in an uncomfortable space
has the potential to create excellent learning outcomes.
This approach also requires effort and a potentially a different skill set on the part of the
teaching team too, in order to effectively scaffold and protect students as they learn in this
experiential way as part of their activity teams. Here the personal tutor as activity team
mentor shows their value. The cautions are worth adhering and mitigating where possible as
the potential gains to the student are significant. Activity teams promote ‘real play, not role
play’ (Forster & Robson, 2019, Pg. 358) as students explore their team working and
collaboration abilities; taking ownership of their learning. Significantly, active learning is
noted as one of the ‘Top 15 skills for 2025’ by the World Economic Forum (Future of Jobs
Report, 2020, Pg. 36.).
3.2. Engagement, feedback and belonging

The reflective essay as an assessment, guarantees an element of student engagement with this
format, but the potential value is much greater than this one grade on a student's transcript.
The process of understanding and practicing in small episodes (through the workshops) then
experiencing ‘for real’ in activity teams when preparing team presentations, informs the
reflective essay that follows, where reflections and future actions are captured by the student.
Students are encouraged to seek feedback from their peers to inform these reflections,
allowing reflection in action and on action (Schon, 1983). Formative feedback is also gained
from the professional skills workshops, personal tutors who mentor the activity teams, 1:1
meetings with the professional skills tutor and peers on the programme, allowing students to
explore different team roles/contributions and build resilience and empathy in their
960
Marianne Savory
understanding of team contributions and effective team working. Here students are co-
creators of their own learning.
There are also gains in terms of ‘belonging’; both from a sense on the part of the student in
the team and on the programme, but also ‘belonging’ in terms of the programme’s ownership
of the teaching and learning process.
3.3. Integrative learning and assessment

Importantly, our model seeks to comfort criticisms from traditional educationalists that
competence-based approaches to education are crowding out knowledge in the curriculum
and that the true purpose of HEIs is to develop a students’ ability to critique rather than simply
serve the employers’ agenda (Young, 2007). Our model for professional development allows
both agendas to co-exist and not one at the expense of the other. Research-led academic
content remains untouched, but the compulsory co-curricular tailored offer that runs parallel,
develops the student's professional skills and both intersect to provide experiences and
opportunities for students to develop. There are signposts, links and references to the
academic content in the professional skills offer, encouraging students to think, learn and
reflect on the whole programme, including the experiences of critically engaging with the
academic content of the core programme. This integrative learning approach is assessed by
the reflective essay that requires students to synthesise and make connections in their
experiences, knowledge and skill development across the entire programme through
articulation of personal critical reflective comment. This may also be an example of ‘synoptic
assessment’ more commonly adopted in 16-18 year educational programme design but may
feature more in HEI educational design going forward if student learning is indeed enhanced
by this approach. (Constantinou, 2020).
4. Challenges
The international make-up of the HRM master's programme student cohort is similar to many
postgraduate business school cohorts across the UK. This presents additional gains, but also
different challenges and some questions yet to be answered. Working collaboratively in
cross-cultural teams and the wider programme cohort, presents the opportunity for students
develop a ‘global mindset’ (part of most leading business school education strategies),
through exposure to other students from different nationalities, cultural backgrounds and
other diversifying factors. However, these gains are limited by a student's language ability,
pre-conceived ideas of how to learn in an educational environment and a range of (conscious
or unconscious) bias. Perceptions (cultural or otherwise) of the value of co-curricular
workshops and activities can also impact on attendance. Arguably, these factors are also areas
of our responsibility to include and address in programme as part of the student's
development. Further, the skills we look to develop in the postgraduate professional
961
development provision – to question, to be curious, to work effectively as part of a team,

scaffolded by self-reflection – neglects the geography of where students will ultimately be
seeking employment. Country context matters here in terms of economic, political and
cultural influences and expectations. Are the skills we look to evoke desirable for graduates
in all country labour markets?
Evaluating the success of this professional development approach is hampered by the same
challenges referred to in the opening paragraph of the paper; the desired outcome is to
develop the student into a self-aware, reflective and lifelong learner and this is not easily
captured or measured at the immediate end point of graduation. However, to inform our way
forward, baseline data of student’s perceptions of their professional skills have been collected
at the entry point to the programme and future work will capture perceptions upon completion
of the programme, allowing initial evaluation.
5. Conclusions
There is a strong case for an element, or elements, of self-reflection to feature in all HEI post
graduate professional development provision, whether or not it is assessed, to ensure students
authentically recognise the value of their experiences and take away genuine learning that
can influence their future actions. Without this, students risk becoming robots, chasing only
graded activity as a homogeneous mass. Indeed, if every graduate emerged with the same
repertoire of employability skills any market advantage would disappear (Akins, 1999, p.
272). Self-reflection and trained articulation will allow them to stand out as unique and
demonstrate their potential to employers.
However, the success of the professional development offer discussed above (the value to
the individual), is only recognised if the skills of reflection are sufficiently developed, and
this may be an area for further consideration. Here, the narrative needs particular attention
because students need to understand the pedagogical approach sufficiently to appreciate why
and how we are meeting their professional development needs in the programme. The aim is
for them to understand the value of the professional skill workshops, the activity teams, the
assessments and their intended outcomes. Being more explicit in this respect is intended to
surface skill requirements allowing students more effectively to recognise and to articulate
the skills/assets they have gained and their transferable nature (Daubney, 2021). Moreover,
the contribution of a HEI to a postgraduate’s professional development officially lasts 12
months but should continue to deliver beyond this to develop students’ self-reflective abilities
and effectively enthuse them to become reflective practitioners (Schon, 1983). That
aspiration is clear, but the challenge is in capturing and measuring this as an output.
Given the pace of change apparent in the current labour market, the skills, competencies and
knowledge required of our postgraduates will continue to change. Employability is “a
962
Marianne Savory
continually re-constructed concept” (Williams, et al., 2019), and perhaps the only constant is
the need for our postgraduates to be self-aware and reflective, in order to move ahead with
purpose and advantage. Professional development is a continuous process, potentially
without end. Being reflective and acting continuously to develop professionally, coupled with
the ability to articulate how experiences have prompted actions to overcome gaps and the
subsequent strengths now beheld, will make the graduate unique and when the time is right
for them, ready to hatch. As a lifelong reflective practitioner, they will however, need
continuously to “self-baste” (Atkins, 1999).
References
Atkins, M.J. (1999). Oven‐ready and self‐basting: taking stock of employability skills.
Teaching in Higher Education, 4(2), 267-280. doi: 10.1080/1356251990040208.
Daubney, K. (2021). Surfacing the employability value of the Humanities. Higher Education
Policy Institute, 29 September [Online] [Accessed 5th February 2023].
https://www.hepi.ac.uk/2021/09/29/surfacing-the-employability-value-of-the-
humanities/
Constantinou, F. (2020). What is synoptic assessment? Defining and operationalising an as
yet non-mainstream assessment concept, Assessment in Education: Principles, Policy &
Practice, 27(6), 670-686. doi: 10.1080/0969594X.2020.1841734
Fonteijn, H., & Dekker, T. (2020). Now is the time for global citizenship education to come
of age. Times Higher Education, 20 May [Online] [Accessed 31st January 2023].
https://www.timeshighereducation.com/opinion/now-time-global-citizenship-education-
come-age
Forster, G., & Robson, A. (2019). Developing the ‘Oven-Ready’ Postgraduate: Squeezing a
Quart into a Pint Pot to Meet the Employability Agenda. In: A. Diver (Ed.), Employability
via Higher Education: Sustainability as Scholarship (pp. 349 - 368). Cham: Springer.
https://doi.org/10.1007/978-3-030-26342-3_23
Gulikers, J.T.M., Bastiaens, T.J., & Kirschner, P.A. (2004). A five-dimensional framework
for authentic assessment. Educational Technology, Research & Development, 52(3), 67–
86. https://doi.org/10.1007/BF02504676
Helyer, R. (2015). Learning through reflection: the critical role of reflection in work-based
learning (WBL). Journal of Work-Applied Management, 7(1), 15-27.
https://doi.org/10.1108/JWAM-10-2015-003
Hillage, J., & Pollard, E. (1998). Employability: Developing a Framework for Policy
Analysis. Research Briefing No. 85. London: Department for Education & Employment.
Kashefpakdel, E., Rehill, J., & Hughes, D. (2018). What works? Career-related learning in
primary schools. London: The Careers and Enterprise Company.
1145_what_works_primary_v6_digital9.pdf (educationandemployers.org)
Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and
development (Vol. 1). Englewood Cliffs, NJ: Prentice-Hall.
963
LeClair, D. (2018). The connective power of experiential learning. AACSB Insights, 3

October. Connecting Through Experiential Learning | AACSB
Lund Dean, K., Wright, S., & Forray, J. M. (2020). Experiential Learning and the Moral Duty
of Business Schools. Academy of Management Learning & Education, 19(4), 569–583.
https://doi.org/10.5465/amle.2018.0335
McQuaid, R., & Lindsay, C. (2005). The concept of employability. Urban Studies, 42(2),
197–219.
Moon, J. A. (1999). Reflection in Learning and Professional Development: Theory and
Practice. London: Routledge. https://doi.org/10.4324/9780203822296
O’Leary, S. (2012). Impact of Entrepreneurship Teaching in Higher Education on the
Employability of Scientists and Engineers. Industry and Higher Education, 26(6), 431–
442. https://doi.org/10.5367/ihe.2012.0128
Riebe1, L., Girardi, A. & Whitsed, C. (2016). A Systematic Literature Review of Teamwork
Pedagogy in Higher Education. Small Group Research, 47(6), 619–664.
https://doi.org/10.1177/1046496416665221
Schon, D. A. (1983). The Reflective Practitioner: How Professionals Think in Action, New
York, NY: Basic Books.
Universities UK (2022). Graduate employment: its limits in measuring the value of higher
education. [Online] [Accessed 31st January 2023].
https://www.universitiesuk.ac.uk/what-we-do/policy-and-
research/publications/graduate-employment-its-limits-measuring
Williams, S., Karypidou, A., Steele, C., & Dodd, L. (2019). A personal construct approach
to employability: comparing stakeholders’ implicit theories. Education + Training, 61(4),
390-412.
Winterton, J. & Turner, J. (2019). Preparing graduates for work readiness: an overview and
agenda. Education + Training, 61(5), 536-551. doi 10.1108/ET-03-2019-0044
World Economic Forum (2020). Future of Jobs Report, October 2020 [Online] [accessed 6th
February 2023] https://www.weforum.org/reports/the-future-of-jobs-report-2020
Young, M. F. D. (2007). Bringing knowledge back in. From social constructivism to social
realism in the sociology of education. London: Routledge.
964
Academic dishonesty, essay mills, and Artificial Intelligence:

rethinking assessment strategies
Simon Sweeney
School for Business and Society, University of York, UK.
Abstract
This paper considers how universities should respond to the threat from
academic dishonesty, including essay mills, and Artificial Intelligence
undermining assessment processes, and institutional credibility. The article
describes steps taken to combat the use of essay mills, but also tools like
ChatGPT, apparently able to generate essays that appear credible, and
sufficient to offer a solution to an imminent deadline. The paper argues for
reconsideration of the traditional essay, which may already be of questionable
value. We should look to alternatives. The paper discusses whether universities
should ban ChatGPT and similar tools, or accept them and design assessment
processes that are more difficult to fake. The paper proposes a modified
version of the essay, namely a reflective report, and explains why this is a more
creative and idiosyncratic approach, better suited to today’s learners, and
better aligned with employers’ expectations regarding employability and the
skills graduates need.
Keywords: Academic dishonesty; essay mills; artificial intelligence (AI);
assessment; reflective statement, employability.
______________________________________________________________

Academic dishonesty, essay mills, and Artificial Intelligence: rethinking assessment strategies
1. Introduction
This paper is based on experience of teaching a postgraduate module, International Political
Economy and Business (IPEB), a long-established, popular, and successful module within a
suite of international business (IB)-related degree programmes. It has relied on a 2,500-word
essay as its standard form of assessment. Results used to be consistently excellent but growth
in cohort size, following expansion of the university’s postgraduate intake, coincided with
higher failure rates and some evidence of increasing academic dishonesty (AD) among
successive cohorts.
Academic dishonesty (AD) is a ‘deceitful or unfair act intended to produce a more desirable
outcome on an exam, paper, homework assignment, or other assessment of learning’ (Miller,
et al., 2017:121). AD takes many forms, but in recent months alarm has focused on the threat
from Artificial Intelligence (AI) and the risk that students would resort to chatbots to help
with essay writing (Moser, et al., 2022, Sparrow, 2022). This paper discusses different forms
of AD and measures to reduce its incidence. AI is a new challenge and has prompted a change
in the assessment – perhaps not before time given the limitations of the standard essay as a
form of assessment, and its questionable contribution to graduate employability.
The paper is intended as a contribution to a live debate in higher education (HE), one of vital
importance given the challenge that AD and now AI presents to the integrity of universities’
assessment strategies and awards standards.
2. Essay mills and how to reduce their use

The problem of essay mills was first raised in our School in 2016 by student representatives
in a postgraduate staff-student liaison committee meeting. The reps told us that use of essay
mills was a growing problem, especially among students whose first language was not
English. My university, like many others, offers master’s courses overwhelmingly comprised
of high fee paying international students, some with rather poor English language
competence.
It is not only overseas students who resort to buying essays online, an apparently more
‘secure’ form of cheating than old-fashioned plagiarism via copy and paste. The latter will
usually be caught by plagiarism detection software like Turnitin, but a one-off essay
commissioned from an essay mill is harder to spot (Ross, 2021).
Our response to the essay mill threat was first to take down the posters in the students’ first
language that were on Department notice boards advertising ‘help with your essays’ and
‘academic tutoring services’. We also stepped up publicity regarding in-house study skills
and writing support, and ensured the university provided more resources. In addition, we
worked harder to alert students to the risks from essay milling, which at the very least could
966
Simon Sweeney
mean failing a course, but also students being blackmailed to re-use a service they had fallen
prey to (Draper, et al., 2021; Yorke, et al., 2022). In the most unscrupulous cases, once a
student provides their contact details, they are exposed to threats, and extortion.
Local providers targeting our students were evident, but essay mill services are widely
advertised online, even alongside universities’ official promotion of their courses. Essay
mills are beyond national jurisdiction, so even where they are banned by law, this has no
effect (HM Government, 2021).
My unsophisticated detection system for spotting milled essays relies on judgement and
experience, but this is not foolproof. Nevertheless one strategy did reduce the incidence of
milling. The assignment brief was amended to say that all references cited must be to authors
on the module reading list or in the course book, an edited collection written by various
experts (Baylis, et al., 2020). International organisations (UN, IMF, WTO, WHO, EU, etc)
are also legitimate sources. This proved an effective deterrent. It is presumably harder for
essay mill writers to use the named sources, and if they tried, clear anomalies were easily
identified, such as random placement of references with no relation to what the original
author actually wrote. We also reinforced the formative assessment process, emphasizing the
need to focus on presenting argument supported by sources from module readings.
With large cohorts, and to be frank, too many students with poor English and a basic lack of
competence, many students are tempted to engage in AD. The closed reading list reduced the
incidence of essay milling, saving students money, but the weakest students still failed to
write a passable essay. Some would engage in the random referencing trick, which is
especially unwise when the marking team knows the reading list well.
So, the essay mill may be the cheating instrument of choice after copy and paste, but it has
limitations, and carries severe risk. Our task is to become more adept at identifying AD, and
appropriate action is needed to protect the assessment process and the integrity of awards.
3. Assessment – rethinking what we ask students to do

There are many ways to assess students’ learning, from the traditional combination of closed
examination and essays, to group work, literature review, project report, multiple choice
questions, long answers, short answers, open questions, twenty-four hour open book
examination, presentation, videos and podcasts, group work, role-playing, reflective
statements, and more besides (Pugh, et al., n.d.). Viva voce is a long-established form of
assessment and standard for doctoral candidates. If conducted properly, it can be very
effective but it is quite unsuited to large cohorts, and out of favour where students operate in
an increasingly litigious and customer-focused marketized system, such as now applies in the
United Kingdom (Nixon, et al., 2016; Molesworth, et al., 2009).
967
Examinations and essays are still the commonest forms of assessment in many institutions.
In Britain, reliance on examinations and essays stems from the grammar and private school
educational models of the 1950s and 1960s, consolidated at the university level, especially
in arts, humanities, and social science disciplines. Assumptions regarding formal academic
writing became established in elite institutions, Oxford, Cambridge, the Ivy League in the
United States, Les Grandes Écoles in France, and the crème de la crème elsewhere.
We should escape from formulaic essays shaped by a formative assessment based on ‘an
essay outline’. This consists of planning an essay structure and adding some indicative
references. But students are often not very good at this, so we end up doing it for them. The
submissions that follow are often underwhelming, although some are excellent. The weakest
suggest that today’s students are not ideally equipped to write traditional academic essays,
especially overseas postgraduates adjusting to very different expectations from from their
experience at home, both in school and university. Writing in a foreign language compounds
the challenge.
The traditional essay may suit training for future academics, and perhaps journalists, but it
far less helpful for careers in business, or in specific sectors such as games development, an
actuary office in an insurance company, banking, or research in a pharmaceutical company.
Essays are not much use in sport, food production and processing, nor in tourism and
hospitality. We should find better ways to demonstrate employability, namely the skills that
employers say graduates need (Swain, 2022; NACE, 2023).
Employers look for graduates with core competences around communication, various
software, IT, and data handling, and teamworking, even project management. They also want
evidence of creativity, and critical and analytical thinking.
Instead of the formal academic essay, students need something more open-ended and better
attuned to individual idiosyncrasy. Assessment should provide more scope for analysis and
critical thinking than essays typically reveal. Moreover, assessment needs to accommodate
changing learning styles shaped by technological innovation and social media.
A student, perhaps stressed and tired after working shifts in a part-time job to support their
studies, and approaching a submission deadline, might try to complete an assignment
dishonestly, perhaps through traditional plagiarism via copy and paste, or submitting an entire
essay found on the Internet with a changed title and some course sources added as references.
Or they might contact an essay mill offering ‘help with your assignment’, masquerading as
honest tutoring (Stokel-Walker, 2022). Or, they might seek help from AI, the latest threat to
academic integrity.
968
Simon Sweeney
4. Artificial Intelligence tools

The supposed threat to HE from AI has been headline news in recent months. According to
some, chatbots like ChatGPT or GPT-3 can answer a wide range of challenges that risk
undermining much of the teaching and learning process (Sparrow, 2022). ChatGPT, or its
Google rival Bard, can provide seemingly plausible answers to almost any problem
(Kleinman, 2023). But they turn up misinformation, and invent quotes from sources, adding
apparent veracity (Delouya, 2022). So while chatbots utilize algorithms that synthesise data
from a huge number of sources to produce answers to problems, they are unable to distinguish
truth from fiction. They are not concerned with truth. Chatbots may generate grammatically
well-formed answers to questions, which with the addition of a few plausible-looking
references may be enough to avoid detection even by an experienced marking team.
A student website reports that ChatGPT cannot yet deliver an entire essay of 2,000 words,
but it can provide answers to targeted questions, text that can be assembled into a reasonable
looking essay once intext references are added, so might get a pass (Snepvangers, 2023).
Alarmingly, the article constitutes advice on how to use ChatGPT to gain a modest pass.
AI is developing and is likely to get better at solving complex problems, and it is already an
integral part of the graduate workplace (Tomlinson, 2022). AI is used in finance, banking
and insurance, leisure and tourism, retail, and research and development of new products and
services, including in manufacturing and in the IT sector. It has has been used to influence
political campaigns and to predict the future. It is therefore unlikely, and unrealistic, that
universities should ban its use. We cannot bar students from using the Internet, and equally
we cannot prevent them using AI. Australian universities have led the way in warning
students about using AI-generated text, and have opted for a return to examinations and pen
and paper (Cassidy, 2023). But another strategy is to accept AI as something we have to live
with. This has important implications for assessment.
Of course, we use a range of assessments in a degree programme. Some modules are better
suited to assessment by essay than others. But I have progressively lost confidence in the
formal essay, and am adapting it to focus on the learning process, and on ways of thinking.
Perhaps this will help combat the essay mill and chatbots too, by encouraging freedom of
expression in assessment. If we step away from demanding conformity to imposed
frameworks this may reduce anxiety and lessen the temptation to engage in AD.
AI instruments are readily accessible. They may become as integral to the student experience
as any favoured search engine. Universities cannot shut out something already present and
available. They will have to accommodate this reality, this rapidly developing and adaptable
technology (Gold Penguin, 2023).
969
5. The reflective report – a variation on the traditional essay

My proposal is to retain the module focus on key readings that assist understanding of current
challenges around international relations, climate change, finding a balance between
economic imperatives and environmental sustainability, and discussing alternative forms of
governance. This reflects the importance of geopolitics as a focal area for management and
business studies education (Mollan and Sweeney, 2022). But assessment must take account
of how students study, different learning styles and previous educational experience, the
multiplicity of sources they may use, and the limitless nature of the internet including
Artificial Intelligence. So the assessment will become a reflective essay, as follows:
Assignment: International political economy and business: a reflection on what I
have learned from the module (2,000 words).
Consider what you have learned from the module. Identify ideas that have changed
your thinking, or provided new insights about the international political economy
and business.
Refer to at least five key readings, explaining how they have shaped your learning.
Provide a critical and reflective assessment of your learning. What has been the most
rewarding aspect of the module? What has changed or reinforced your thinking? Is
there anything you will think about more deeply, and for more time, in the future?
Use Harvard referencing for all sources mentioned in your report.
You must demonstrate learning from the module, and from module sources,
including the course book.
You may write in an informal style, using first person pronouns (I/we).
You may use Internet sources but you must reference these, and provide a critical
commentary on any material you use.
While reflective essays have become a fairly commonplace form of assessment in recent
years, the origins of the approach are centuries old and credited to the French thinker Michel
de Montaigne (1533-92) (Halpin, 2014). We anticipate that reflection along the lines
suggested above will give students an open opportunity to draw on any aspect of the module,
to demonstrate learning, and importantly, to give personal responses to the module and issues
raised. This allows for creative and thoughtful reflection, and will ensure individual,
unimitable, idiosyncratic responses. The reflective statement recasts the essay by focusing on
process, how insight is gained, and how thinking develops (Tomlinson, 2022).
We also anticipate less or even no recourse to essay mills. The assignment brief is designed
to make the assessment almost unrealizable though recourse to AI. But if students do use
970
Simon Sweeney
internet-enabled assistance, the assignment brief indicates that this must be referenced,
commented upon and critically appraised.
While the assignment insists on evidence of engagement with module reading, it does not
preclude Internet searches. Students are instructed to comment critically on what they find.
6. Conclusion
This paper has described a response to the threat from essay mills and other forms of
academic dishonesty (AD). The article recognizes the risk that students may resort to artificial
intelligence (AI) to compose essays and other written submissions. The paper recognizes that
most universities use a wide range of assessment methods, but examination and formal essays
are still favoured methods of assessment in many institutions especially in humanities and
social science disciplines.
Weaknesses in the traditional essay approach are highlighted, including the view that essays
are not the best way to develop the skills and competences identified as contributing to
graduate employability. A different kind of written assignment, involving personal response
to module content and reflection on the learning process can allow a more creative,
idiosyncratic and relevant task, better matched to contemporary learning styles and new
technologies. A reflective statement can better identify core take-aways from a module. It
constitutes an easier, more relevant and more creative form of assessment.
Finally the paper maintains that it is more fruitful and realistic for universities to embrace
emergent technologies such as AI, rather than prohibit their use. The essence is to ensure
students’ critical engagement with material encountered during their studies, and making this
central to the assessment process. This will better prepare graduates for entry into fulltime
employment.
References
Baylis, J., Smith, S., & Owens, P. (2020) The Globalization of World Politics: An
Introduction to International Relations. Oxford: Oxford University Press.
Cassidy, C. (2023) ‘Australian universities to return to ‘pen and paper’ exams after students
caught using AI to write essays’. 10 January. The Guardian. Link.
Delouya, S. (2022) ‘I asked ChatGPT to do my work and write an Insider article for me. It
quickly generated an alarmingly convincing article filled with misinformation’. 19
December. Business Insider. Link.
Draper, M., Lancaster, T., Dann, S., Crockett, R., & Glendinning, I. (2021) ‘Essay mills and
other contract cheating services: to buy or not to buy and the consequences of students
changing their minds’. International Journal for Educational Integrity. 17 (13). Link.
971
Gold Penguin (2023) ‘Can ChatGPT + Quillbot create undetectable AI content?’ Blog by
Justim. 9 April. Link.
Halpin, D. (2014) Essaying and Reflective Practice in Education: The Legacy
of Michel de Montaigne. Journal of Philosophy of Education, 49(1),
129–141. doi:10.1111/1467-9752.12098
HM Government (2021) ‘Essay mills to be banned under plans to reform post-16 education’.
Press Release, 5 November. Link.
Kleinman, Z. (2023) ‘Google launches ChatGPT rival called Bard’ BBC Online. 6 February.
Link.
Miller, A.B., Murdock, T.B. & Grotewiel, M.M. (2017) ‘Addressing academic dishonesty
amongst the highest achievers’. Theory into Practice, 56 (2) 121-128.
Molesworth, M., Nixon, E., & Scullion, R. (2009) ‘Having, being and higher education: the
marketisation of the university and the transformation of the student into consumer’.
Teaching in Higher Education, 14(3) 277-287.
Mollan, S. & Sweeney, S. (2022) ‘Geopolitics and business: implications for management
education’. HEAd’22 24 May. Conference Proceedings. 1295-1302. Universitat
Politecnica de Valencia, Valencia. Link.
Moser, C., den Hond, F., & Lindebaum, D. (2022) ‘Morality in the age of artificially
intelligent algorithms’. Academy of Management Learning and Education 21(1) 139-155.
NACE (2023) What is Career Readiness? National Association of Colleges and Employers.
Link.
Nixon, E., Scullion, R. & Hearn, R. (2016) ‘Her majesty the student: Marketized higher
education and the narcissistic (dis)satisfactions of the student-consumer’. Studies in
Higher Education, (0):1-27.
Pugh, S. (ed) (n.d.) A Compendium of Assessment Techniques in Higher Education: from
students’ perspectives’. Leeds: Leeds Institute for Teaching Excellence. Link.
Ross, J. (2021) ‘We’ll write for you, plagiarism-free – essay mills’ latest ploy’. Times Higher
Education, 17 November. Link.
Snepvangers, P. (2023) ‘I got ChatGPT to write my essay to see if I could get a 2:1 from a
Russell Group uni’ The Tab (UK). 3 February. Link.
Sparrow, J. (2022) ‘Full-on robot writing: the artificial intelligence challenge facing
universities’ 18 November. The Guardian. Link.
Stokel-Walker, C. (2022) ‘Inside a Highly lucrative, ethically-questionable essay-writing
service’. 9 September. Input. Link.
Swain, R. (2021) ‘What skills do employers want?’ Prospects. April. Link.
Tomlinson, R.C. (2022) ‘Contract Cheating: the place of the essay in the age of the essay
mill’. 27 September. Council for the Defence of British Universities. Link.
Yorke, J., Sefcik, L., Veeran-Colton, T. (2022) ‘Contract cheating and blackmail: a risky
business?’ Studies in Higher Education, 47(1) 53-66.
972
Thinking outside the box – virtual labs in engineering education

Julia Salzinger1, Ika Kurniawati1, Lars Abrahamczyk2, Rüdiger Höffer1
1
Department of Civil and Environmental Engineering, Ruhr-Universität Bochum, Germany,
2
Department of Civil Engineering, Bauhaus-Universität Weimar, Germany.
Abstract
An important part of engineering education is training in laboratories. If done
correctly, such a hands-on approach is much more effective than pure lectures,
where knowledge is only passively absorbed. The traditional lab is usually on-
site, requires students to be on campus, and learning content is limited to the
particular focus of that institution. For this reason, five European universities
have formed a partnership and developed a joint platform that offers
experiments that can be experienced in virtual labs and accessed repeatedly.
This way students profit from the expertise of different institutions, while
gaining intercultural work experience. Additionally, virtual labs reduce the
immense expense of student mobility, are more inclusive towards minority
groups and lower costs for universities in the long term.
Keywords: virtual labs; engineering education; structural engineering, virtual
access to experiments.

Thinking outside the box - virtual, intercultural labs in engineering education
1. Introduction
The COVID-19 pandemic had a huge impact on teaching and teaching methods: from one
day to the next physical attendance on campus and in the classroom was no longer an option.
This posed various challenges to the different disciplines as new, digital teaching formats had
to be created in a rather short amount of time. While the humanities and social sciences were
able to hold synchronous classes online relatively easily via conference tools, or convert
teaching materials into asynchronous e-learning modules, the engineering disciplines were
faced with the challenge that a vital part of their curricula consists of laboratory experiments.
These are often devised for a better understanding of theoretical knowledge taught in lectures
and vital in the education of future engineers. Laboratories can lead to a substantial learning
success among students that is difficult to achieve in other traditional learning environments
or didactic situations. Therefore, implementation of digital learning is only successful if
laboratory experiments are included in digitalization efforts.
Another setback for a comprehensive education was that suddenly exchange programs could
no longer take place. Giving students the chance to extend their education to fields not offered
at their home university has been part of the teaching program of several European
universities. This led to a partnership with a strong exchange between participating
departments. Such partnerships among European universities had always required students
to travel to different locations to have the chance to participate in experiments offered by
different universities. However, the overall expense of student mobility is immense (Johri &
Jesiek, 2014) and may exclude students from lower income backgrounds (Ackers, 2010;
Rodríguez et al., 2011) or countries with with a lower income level (e.g. eastern European
countries) unless fully sponsored. Students with a disability are also at a disadvantage.
The pandemic offered the chance to rethink teaching options and make excellent education
more easily available to all students. The PARFORCE project has executed this by bundling
the expertise of different partners across Europe and providing students with access to
different facilities from their home computers or by using facilities at the university in their
home country. Creating virtual labs made it possible for students to expand their knowledge
beyond the area of expertise of their home university and beyond individual personal
restrictions.
2. Project PARFORCE
2.1. Strategy and objectives

Five European universities across different countries have formed a strategic partnership
within the framework of Erasmus+. Two universities are situated in Germany (Bauhaus-
Universität Weimar and Ruhr-Universität Bochum), one in Croatia (University of Osijek),
974
one in North Macedonia (Institute of Earthquake Engineering and Seismology at the Ss. Cyril
and Methodius University in Skopje), and one in Portugal (Universidade de Aveiro). Each
partner university has a different focus. This is often based on their respective geographic
situation or the different demands that are placed on local structures (e.g., earthquake-prone,
growing, i.e. climate-related exposure to fire effects, wind as an impact and as a resource).
Together, they developed a common platform for digital/virtual laboratory experiments to
support higher education in European civil engineering and contribute to the students’
understanding of course content.
The availability of various laboratory experiments that are not part of the standard education
at each university is the main objective of the project. The experiments planned in this project
are: boundary layer wind tunnel experiments, non-destructive and destructive experiments
on a shaking table, and fire resistancy tests. The project applies the concept of immersive 3D
representations for detailed and realistic visualization in virtual reality to the performance of
laboratory experiments in the context of higher education in civil engineering.
2.2. Local labs vs. Virtual labs

A laboratory is understood as a scientific or technological environment in which scientific or
technological research, development experiments, tests or analyses are conducted
(Terkowsky et al., 2020). Technical laboratories are used to build, develop, and test technical
solutions, processes, and products. Laboratories in engineering education can be used for
empirical research to observe, measure, test, analyze, and especially to develop
understanding. Unfortunately, the use of local labs in engineering education is limited due to
severe time constraints and their highly specialized nature. In addition, these laboratories
require students to be physically present on site. Zutin et al. (2010) point out that the
inefficient use of expensive and highly sophisticated and specialized equipment that is
common in traditional laboratories can be overcome by providing online labs to learners as
technology-enhanced learning environments. Virtual labs offer several advantages over
traditional labs, including almost unlimited time availability, interoperability, reusability, and
location-independent access. Online labs do not require the learner to be present on site, but
can provide an experimental experience that is accessible from the learner’s location,
regarlesss of the distance to the institution.
2.3. Shortcomings of labs

Laboratories can offer educators the opportunity to provide students with a learning
environment where they can creatively solve engineering challenges designed to facilitate
competencies required of engineers in the digital age. Degree programs as mechanical
engineering aim to to prepare students for the highly practical nature of their future careers,
in addition to providing them with a scientific background.
975
Planning and conducting experiments in the laboratory are among the typical tasks of
graduates. For this reason, it is important for them to gain experience working with full-scale
equipment while still studying. This includes e.g. printers for additive manufacturing,
prefabrication lines, or equipment used for long-term testing, such as a wind turbine with a
Structural Health Monitoring system installed.
Engineering laboratories have great potential to activate and develop competencies that go
beyond the conventional knowledge and skills that can be acquired in traditional classroom
settings. However, they often fail to provide additional learning objectives (Terkowsky et al.,
2020). Many laboratories present students with tasks and topic descriptions that clearly
describe of a problem and a well-defined solution strategy. Of course, this is in stark contrast
to many challenges in an engineer’s professional life, where neither the problem nor the
solution is explicitly known in advance. Thus, this approach of strongly guiding students
contrasts with the aim to produce competent, self-organized and self-controlled engineers.
3. Virtual labs
Appropriate learning objectives are the starting point when planning an engineering
education environment. Providing engineering labs requires appropriate facilities,
equipment, and staff. As a result, they are often more costly than traditional classroom
environments. Making such labs available as virtual labs involves monetary costs for
technical equipment and infrastructure. In addition, time is required for didactic planning.
However, these costs are quickly offset by the virtual lab's unlimited scalability and
permanent availability.
The use of physical and virtual laboratories in engineering education curricula requires a
rationale for how student work in laboratories and how this can complement the learning
objectives of traditional classroom settings. Therefore, the formulation of learning objectives
should emphasize the importance and potential of laboratories for engineering education.
Virtual labs, such as a virtual wind tunnel or chemistry labs developed at the Ruhr-University
Bochum, are a better choice than local labs. These are web-based applications that simulate
real engineering test facilities in civil, environmental, and process engineering. Essential
parameters such as wind speeds, wind directions or turbulence intensities can be varied in a
boundary layer wind tunnel experiment according to realistic configurations or processes.
Therefore, the test results, such as the modeled structural response etc., can be experienced
virtually (see fig. 1) with the same result as in an on-site lab. The students perform
experiments independently from their home PC and evaluate them according to scientific
methods and requirements.
976
Figure 1. 3D virtual representation of the Boundary Layer Wind Tunnel at the Ruhr-Universität Bochum - remote
access points are marked “i” and show explanations about steering and sensoring of experiments as well as
videos of specific experiments; experimental parameters can be varied within a certain range, corresponding data
from previous experimental campaigns can be downloaded for further processing.
The idea of the virtual labs was motivated by the fact that specialized laboratories are highly
valued by teachers and students. Students in particular appreciate that these labs offer an
intensive learning experience. Virtual labs can thus be understood as a complement to
specialized laboratories with virtual, but realistic experiments in order to make the students’
learning experiences more comprehensive.
Students are expected to plan and conduct experiments on their own. This usually includes
planning the measurement series: e.g. preliminary considerations as to which gas flows and
which liquid flows need to be measured in order to fulfill the task of the specialized
laboratory. After the experiments, the students apply the knowledge gained in lectures. By
studying the experimental script to evaluate the measurement results, they formulate
statements about the quality of the results. This is necessary because measurement errors and
fluctuations can also be incorporated into the model response monitoring within a virtual
experiment in order to reproduce the system realistically. Teachers can tailor the application
to students' prior knowledge.
Using a provided script, students prepare to conduct the experiment in the virtual test facility,
developing the ability to plan a systematic experiment independently. The experiments are
designed so that exploratory learning can be used to observe the phenomena described in the
experimental script, such as the systematic variation of the main operating parameters of the
experimental plant (pressure, temperature, mass flows, etc.). Students should be able to
identify measurement errors and outliers and correct them by adjusting the accuracy and
reliability or taking additional measurements to increase the statistical stability of the results,
if necessary. While the students prepare independently without teacher supervision, the
process of the virtual experiment and the evaluation of the results must be supervised by
competent teachers.
977
Following scientific procedures, appropriate documentation and presentation of both the

experimental procedure and the results are presented by the students in a report, e.g. in the
form of a protocol, possibly with an oral presentation.
The learning objectives are to prepare, plan, execute and evaluate experiments independently,
and assess the results in a knowledge-based manner. Exploratory learning enables students
to subjectively discover new aspects. At the same time, they become familiar with scientific
working processes and acquire sustainable knowledge and skills that go beyond technical and
methodological knowledge – above all, independence in decision-making. This enables them
to apply what they have learned to new situations and scenarios. These skills are necessary
to be able to solve problems in a competent and systematic way later in their careers, and are
far more valuable than inert knowledge passively acquired in lectures.
3.1. Virtual labs and virtual exchange

Virtual labs can be made available to students across different countries and expand the scope
of learning by involving institutions with different expertise. Besides the greater advantage
in knowledge, two other aspects stand out: on the one hand, more students can gain
intercultural experience and on the other hand, more equal opportunities are created,
especially for students who are often excluded from physical exchanges due to the
aforementioned reasons.
When the labs and the assignments are set up so that students from different countries work
together in a study group, they learn to interact and collaborate with people from different
backgrounds. This is a valuable experience given the increasing importance of international
collaborations today. Different perspectives and new insights can be shared through
established interaction. In addition, an intercultural learning environment can help those
learners who are not from the majority group (Gay, 2018) and may otherwise be excluded.
4. Conclusion
Integrating virtual labs into engineering education has several advantages, if done right.
Students can access virtual labs from home; they do not need to be on campus or even in a
particular country. As such, virtual labs are much more inclusive, allowing everyone to
participate regardless of disability, social background, or the need to work in order to support
themselves financially.
In terms of the skills that graduates should have, it has been shown that the learning outcome
is much higher when engineering students have the opportunity to test and apply the
knowledge they have acquired in lectures in a laboratory; especially in a lab that focuses on
a hands-on approach. Availability, didactic variability and cost efficiency are best when the
978
lab is virtual. Exploratory learning can have a better impact on the comprehension of learning
material. This will lead to more independent, self-directed and self-organized future
engineers who have more than just passive, scientific knowledge. Students learn the process
of researching and applying their skills to real-world tasks. It promotes the ability to identify
problems and generate ideas, enabling the learner to understand what needs to be done to find
solutions.
A third advantage of virtual labs is their intercultural nature: not only do they offer students
the opportunity to acquire knowledge not taught at their home university, but they also give
students the chance to gain experience working with people from different cultural and
linguistic backgrounds. They learn to overcome and appreciate culturally determined
differences in approaching tasks and finding solutions.
Thus, it can be said that including virtual labs has brought significant benefits and lasting
improvements to engineering education, broadening the the expertise taught and providing a
lower threshold for participation.
Acknowledgement
The the authors gratefully acknowledge the financial support from the Erasmus Plus - Project
PARFORCE (grant no. 2016-1-DE01-KA203-002905), the methodological support from the
Center for Teaching and Learning (ZfW), and the eLearning team of the Ruhr-Universität
Bochum as well as from the eLearning Lab of the Bauhaus-Universität Weimar supported by
the eTeach Network Thuringia.
References
Ackers, L. (2010). Internationalisation and Equality. Recherches Sociologiques Et
Anthropologiques, 41(1), 83–103. https://doi.org/10.4000/rsa.189
Gay, G. (2018). Culturally responsive teaching: Theory, research, and practice (Third
edition). Multicultural education series. New York, London: Teachers College Press
Teachers College Columbia University.
Johri, A., & Jesiek, B. K. (2014). Global and International Issues in Engineering Education.
In A. Johri & B. M. Olds (Eds.), Cambridge Handbook of Engineering Education
Research (pp. 655–672). New York: Cambridge University Press.
https://doi.org/10.1017/CBO9781139013451.040
Rodríguez González, C., Bustillo Mesanza, R., & Mariel, P. (2011). The determinants of
international student mobility flows: an empirical study on the Erasmus programme.
Higher Education, 62(4), 413–430. https://doi.org/10.1007/s10734-010-9396-5
Terkowsky, C., Frye, S., & May, D. (2020). Forschendes Lernen im Labor: Labordidaktische
Ansätze zwischen hands-on und Corss-Reality. In C. Terkowsky, D. May, S. Frye, & T.
979
Haertel (Eds.), Labore in der Hochschullehre: Didaktik, Digitalisierung, Organisation (1st

ed., pp. 13–34). Bielefeld: wbv Publikation.
Zutin, D. G., Auer, M. E., Maier, C., & Niederstatter, M. (2010). Lab2go — A repository to
locate educational online laboratories. In IEEE EDUCON 2010 Conference (pp. 1741–
1746). IEEE. https://doi.org/10.1109/EDUCON.2010.5492412
980
Students experience of blended teaching formats in pre-calculus

courses
Department of Mathematics, Politecnico di Milano, Italy.
Abstract
This work aims to investigate the difference (if any) between two blended
teaching formats in a pre-calculus course for undergraduate engineering
students. In 2022 students could choose among two settings for the preparatory
course: a face-to-face and an online blended format, both based on a Pre-
Calculus MOOC. At the very end of the course, we collected data from a survey
to which 129 students answered. The analysis of the responses shows that the
main differences between face-to-face and online blended formats concern
intrinsic characteristics of the two settings. On the other hand, the data show
that an accurate planning of the online format allows students’ engagement
even more than in face-to-face blended setting. Moreover, the findings shed a
light on the potentiality of online learning in making the transition from high
school to university smoother.
Keywords: Innovative teaching; MOOC; STEM; preparatory course;
transition; mathematics.

Students experience of blended teaching formats in pre-calculus courses
1. Introduction
One of the main obstacles for students enrolled in first year STEM programs is the Calculus
course. Different researchers in Mathematics Education (Tall, 2004) suggest that the main
difficulties are created by a different approach to mathematics with respect to the one seen in
high school, since there is a shift of the focus from procedural aspects to conceptual
understanding (Clark and Lovric, 2008). Moreover, learning resources are perceived in a
different way (Kock, Brunetto & Pepin, 2019) and students are supposed to study and work
with more independence with respect to high school. In order to smoothen the transition
between secondary and tertiary education, universities usually offer preparatory courses to
first-year students. As Gamer and Gamer (2001) pointed out, student-directed learning
promotes conceptual learning more effectively if compared to teacher-directed approaches.
One possibility to encourage student directed learning is given by blended learning formats.
In this context, Massive Open Online Course (MOOC) is a significant resource that can be
exploited in a blended course: students have to watch videos making sense of their content
without the teachers’ guidance; they have the possibility to self-evaluate themselves with
exercises and to resort online forums and other sources clarifying their doubts. All this
engages students in different mathematical activities that generate a personal production of
meanings and mathematical knowledge.
Blended learning formats recently became even more preeminent and relevant, since
universities are trying to adapt to the “new normal” after the COVID-19 pandemic forced
them to do remote teaching (Bakker et al., 2021). It is well documented that emergency
remote teaching and online learning are quite different (Hodges et al., 2020), but the
exceptional situation of 2020 created room for new teaching experimentations with online
learning, since both students and teachers gained familiarity with new technological tools as
services for web conferences (as Zoom, Webex, Microsoft Teams), software for sharing
boards and contents (as Jamboard, padlet, Google Drive) and so on.
In this work we want to investigate how students experience online and face-to-face teaching
in terms of quality and usefulness. We will focus on a pre-calculus preparatory course,
delivered in 2022 for first year students at Politecnico di Milano, in which students were
given the possibility to attend face-to-face (f2f) classes or online classes.
2. Methodology
The preparatory math course is delivered every year at the Politecnico di Milano before the
beginning of the first semester. Usually, the course is structured in two parts: a MOOC that
students are invited to attend during the summer and eight classes delivered in the two weeks
before the beginning of the academic year. The pre-calculus MOOC is delivered on the POK
platform (www.pok.polimi.it), where students can watch videos about theory explanations
982
and resolutions of exercises, test their basic knowledge in mathematics through quizzes and
interact with other students and tutors through a forum. The MOOC course is structured in
six chapters: arithmetic, algebra, geometry, logics, functions, probability. The eight classes
of four hours each, follow the syllabus of the MOOC course.
In the 2022-23 academic year the attendance part of the preparatory course was organized in
two different formats, that we will call face-to-face (f2f) and online blended classroom. In
the face-to-face format the eight classes were delivered in a traditional classroom setting, that
is tutors and students met in a physical learning space. Each class started with a “warm-up”
activity that was carried out in small groups, followed by the correction of the tasks and the
discussion of other examples and exercises by the tutor. At the end of the class, a formative
assessment test was left to students in order to solidify their knowledge and skills. In the
online format there was not a physical room: the tutor and the students were interacting
using an online platform (Webex). The online class started with asynchronous activities, in
the sense that students were not connected live with the tutor. Students received instructions
about which part of the pre-calculus MOOC had to be studied (videos and quizzes) and,
afterwards, they were prompted to solve the task of the “warm-up” activity, posting the
solutions and questions on the MOOC forum. Later, there was a synchronous part of the
class, delivered through the online platform, in which the tutor discussed the solution of the
warm-up activity, addressed the questions, and deepened the topic. At the end of this part
students were given some time to answer the formative-assessment test (the same the face-
to-face students were doing). We underline that the two formats are equivalent in the sense
that they refer to the same topics and use the same educational resources, but they differ in
terms of educational environment and timing. The MOOC was available for both groups,
students received instructions by email when they enrolled to the preparatory course and they
were reminded about all educational resources daily by verbal communication by the tutor.
The preparatory course, which was not mandatory, was attended by 380 students, on average.
Among them, almost the 15% of students attended the online format, held by one single tutor,
while the huge majority attended the f2f format held by 7 tutors in 7 physical classrooms.
More precisely, two classes had about 20 students while the other 5 had about 60-70 students.
At the end of the course, namely at the end of day 8, students were asked to respond to a
satisfaction survey voluntarily. We collected 129 responses (about the 40% of the students
who attended the last lesson). The survey is composed of 4 parts, we focus on the questions
about the course, and open comments and suggestions. The former is formulated as three (0-
2) likert-scale questions. Table 1 reports the 16 items (i1, i2, i3, …) to which students
answered using 0- “disagree” / “not at all”, 1- “agree”/ “enough”, 2- “strongly agree”/ “fully”.
We classified the 16 items a-posteriori according to the two dimensions we wanted to
investigate: D1) the quality of the course in terms of tutor practice (i12-i16), topics (i5), and
students’ interaction with each other and with the tutor (i10, i11), and D2) the usefulness, in
983
terms of overcoming the transition issues, such as fostering conceptual-mathematics (i6-i9),

prompt autonomy (i1-i4).
Table 1. List of the items divided into the dimensions under analysis.
Quality Usefulness
i5 - During the classes we revised the most i1 - Before the beginning of the refresher course
important topics we studied in high school. I attended the pre-calculus MOOC.
i10 - During the classes, we worked on our own. i2 - During the refresher course I attended the
pre-calculus MOOC.
i11 - During the classes, we discussed and
cooperated. i3 - During the refresher course I was involved
in the Discussion forum on the pre-calculus
i12 - During the classes, the tutor fostered
MOOC.
students’ collaboration.
i4 - During the refresher course, I did the
i13 - During the classes, the tutor involved
formative-assessment quiz.
students.
i6 - During the classes, I found a new language
i14 - During the classes, the tutor’s explanations
to do mathematics.
were clear.
i7 - During the classes, I found new mathematics
i15 - During the classes, the tutor precisely
topics.
answered our questions.
i8 - During the classes, we focused a lot on
i16 - During the classes, the tutor positively
formulas and algorithms to solve exercises.
influenced the independent study.
i9 - During the classes, we analyzed definitions,
theorems, and general properties.
Source: by authors (2022).
In this work we aim at addressing the following research question: “How do students
experience online and face-to-face teaching in terms of quality and usefulness?”. To that end,
the students’ responses were analyzed both quantitatively and qualitatively. For each item,
we resort to descriptive statistics (mean, standard deviation, and histograms) distinguishing
between two groups of students, namely the f2f students’ responses (102) and online
students’ responses (27). Moreover, we employ the one-way analysis of variance (ANOVA)
to determine if data from the two groups have a common mean. We recall that the ANOVA
helps determine if the mean of the two (or more) groups are all the same. The null hypothesis
H0 is that, for each item, the means are the same (Wu & Hamada, 2000). Once the test is
performed, the H0 is rejected if the returned p-value is less than a specific threshold (0.05 by
default). Indeed, the p-value consists of the probability of making a mistake rejecting the H0.
Finally, we analyzed the students’ comments looking for the indicators of quality and
usefulness, to deepen our analysis and to better interpret the previous results. It is worth
984
noticing that among 127 students, only 90 provided a comment. More precisely, almost all
the online students (23 out of 27) left a comment, whilst only 67 (out of 102) of the f2f
students did.
3. Data analysis and discussion

In this section we report data analysis of the 16 items related to the quality and usefulness,
enriched by some of the students’ comments. Table 2 summarizes the results of the statistical
analysis. We observe that there are only two questions (i2, i14) whose responses are
statistically different (p-value < 0.05, marked with * in Table 2). Nevertheless, some
questions (i5, i10, i11) have slightly different results with respect to the two groups (0.05 <
p-value < 0.15, marked ** in Table 2).
Table 2. In this table for each question, we report the p-value obtained by the comparison
between the two groups and the means (standard deviation) of the two groups.
Usefulness i1 i2 i4 i6 i7 i8 i9 i10
p-value 0.48 0.00* 0.17 0.84 0.49 0.49 0.45 0.07**
mean 0.81 1.56 1.59 1.15 0.89 1.18 1.66 1.33

(std) (0.87) (0.64) (0.50) (0.60) (0.70) (0.55) (0.55) (0.48)
online
mean 0.94 0.67 1.73 1.12 0.99 1.10 1.57 1.12

(std) (0.81) (0.66) (0.46) (0.68) (0.67) (0.57) (0.57) (0.55)
f2f
Quality i3 i5 i11 i12 i13 i14 i15 i16
p-value 0.54 0.15** 0.11** 0.49 0.22 0.01* 0.547 0.57
mean 0.63 1.56 1.81 1.74 1.78 1.56 1.78 1.56

(std) (0.74) (0.50) (0.39) (0.44) (0.42) (0.50) (0.42) (0.50)
online
mean 0.53 1.70 1.62 1.66 1.64 1.80 1.83 1.62

(std) (0.71) (0.46) (0.58) (0.53) (0.52) (0.42) (0.40) (0.56)
f2f
Source: by authors (2023).
We observe that the most relevant difference between online and f2f students concerns the
use of online resources, i.e., the Pre-Calculus MOOC, during the two weeks of the
preparatory course (see i2 in Table 2). More precisely, the percentage of f2f students who
985
declared not to use the MOOC is greater than four-times the percentage of online students;
on the other hand, more than 60% of the online students declared to study the MOOC
resources during the course (see Figure 1). We stress that such a difference depends on the
structure of the online course, in which students were recalled to study the videos and to do
activities on MOOC on a daily basis. It is important to underline that the MOOC was
available to all the students and both formats were designed according to the MOOC
structure. This allows us to claim that students resort to online resources only if they are
directly instructed about it. Furthermore, a student of the online course commented: “I really
appreciated the forum. It really helped me to see different strategies to solve some of the
exercises [warm-up tasks]”. This is an indicator that the guided use of the online resources
also allows students to exploit the potentiality of new tools, such as the forum.
Figure 1. The percentage of answers to i2. Source: by authors (2023).
The item i10 regards the self-regulation of the students, namely if they perceived they worked
on their own during the classes. The two groups are slightly different (1.33 for online and
1.12 for f2f) because online students reported more work on their own than f2f students. This
could be explained by the fact that half of the online class was explicitly devoted to
asynchronous and autonomous activities (such as the warm-up exercises). To that end, we
report the following comment by an online student: “[I appreciated] the possibility of
managing on my own the study moments and reviewing the scheduled topics”. The comment
underlines two aspects: the autonomy dimension and the scheduled topics. The student was
able to self-organize their study thanks to the shared planning of the classes. Therefore, we
argue that the majority of first-year students may lack autonomy when they are not guided as
they were used to in high school.
986
The item i14, whose answers are statistically different, concerns the clarity of tutor
explanations. However, from an educational point of view the difference of the two means
(1.57 for online and 1.80 for f2f) are not relevant, because the values are high. We infer that
the different results depend more on the tutor-factor than on the format-factor. Indeed,
looking at the f2f format, the 7 tutors were perceived (by students) positively in different
ways, namely the means associated to each tutor ranges from 1.5 to 2. Moreover, students of
both groups commented with positive feedback about tutors, for instance a f2f student
reported “I do appreciate the clarity of the tutor explanation” and an online student wrote:
“the tutor is very competent, clear and precise”.
We underline that i14 is one of the 5 questions related to the tutor's influence on the course,
so that any possible differences between the two groups’ answers may be affected by the
tutor-factor. However, the means of those questions are not so different. In particular, we
focus on i12 and i13 which aimed at investigating tutor’s influence in fostering students’
collaboration and involving them during classes, respectively. The means of the online
classes (1.74 for i12 and 1.78 for i13) are slightly higher than the f2f classes (1.66 for i12 and
1.64 for i13). Despite the common idea that online classes are not a good setting for engaging
students, this result shows that students may have an active role even in online classes.
Another result that supports such a claim comes from the answers to item i11, in which
students were asked if in classes they discussed and cooperated. The two groups’ means are
1.81 (for online) and 1.62 (for f2f) with a slightly statistical difference (p-value=0.11).
Moreover, the analysis of the students’ comments shows that only one of the f2f students
provided a comment about the students’ collaboration: “I liked to work with my course mates
to solve the tasks”, while online students gave more comments in such direction, for instance:
“I really appreciated the possibility of learning not only from the tutor but from my course
mates’ ideas and advices” and “I liked the continuous discussion and interaction among
mates”. Somehow, students felt they discussed and interacted more online than students did
in the classroom.
4. Conclusion
The spreading of innovative learning and teaching experiences raises questions about their
efficacy and impact with respect to “traditional” experiences. In this work, we do not dwell
on the definitions of such terms, but we are interested in investigating how students perceive
differences between two formats of the same course. To that end, we focus on a preparatory
course for first-year students in a STEM program, addressing the following research
question: how students perceive online and f2f teaching in terms of usefulness and quality.
The usefulness and quality dimensions are meant in the frame of transition between high
school and university mathematics.
987
Overall, the data analysis shows that no perceived differences were reported by the students
for the two formats, which were perceived both as good courses. This was also supported by
a final question in the survey: “do you recommend this course to future first-year students?”,
to which none of the students answered “no”. Nevertheless, we underline that the few
differences can be explained according to the unavoidable characteristics of the course, such
as the tutors who taught the course, and the different instructions the students received.
Concerning the latter aspect, we stress that students need to be guided step by step in the use
of educational resources. Despite having the same topics and tasks of the f2f format, the
online format was not just a migration to a remote teaching method, but it was carefully
planned according to the potentiality and constraints of the online environment. In this way
also online students were able to interact, discuss and cooperate gaining autonomy and being
guided during the transition to higher education.
References
Bakker, A., Cai, J., & Zenger, L. (2021). Future themes of mathematics education research:
An international survey before and during the pandemic. Educational Studies in
Mathematics, 107(1), 1-24.
Clark, M., & Lovric, M. (2008). Suggestion for a theoretical model for secondary-tertiary
transition in mathematics. Mathematics Education Research Journal, 20(2), 25-37.
Gamer, B.E., Gamer, L.E. (2001). Retention of concepts and skills in traditional and reformed
applied calculus. Mathematics Education Research Journal 13(3), 165–184.
Hodges, C.B., Moore, S., Lockee, B.B., Trust, T., Bond, M.A (2020). The difference between
emergency remote teaching and online learning.
Kock, Zeger, J., Brunetto, D., Pepin, B. (2020). Students’choice and perceived importance
of resources in first-year university calculus and linear algebra. In: Barzel, B., Bebernik,
R., Göbel, L., Pohl, M., Schacht, F., Thurm, D.E. (eds.) Proceeding of the 14th
International conference on technology in mathematics teaching – ICTMT14. vol. 48, pp.
91–98. DuEPublico. doi: 10.17185/duepublico/48820
Tall, D. (2004). Building theories: The three worlds of mathematics. For the learning of
mathematics, 24(1), 29-32.
Wu, C. F. J., and M. Hamada. (2000) Experiments: Planning, Analysis, and Parameter Design
Optimization.
988
Should we be afraid of open book exams? Our experience

Antoni Alegre-Martínez1, María Isabel Martínez-Martínez2, José Luis Alfonso-
Sanchez3
1
Biomedical Sciences Department, Faculty of Health Sciences, Cardenal Herrera CEU
University, Spain, 2Public Health Department, Faculty of Nursing and Podiatry, University
of Valencia, Spain, 3Department of Preventive Medicine, University of Valencia, Spain.
Abstract
This study reports our first experience introducing open-book exams in the
Anatomy subject at a medical school. Students were authorized to bring study
material to the exam and consult books, atlases and their own notes. Our
objective was to check whether the test scores were significantly higher or
lower compared to the rote tests taken before it. After statistical analysis, there
are no differences in the mean, median, range, minimum score, or maximum
score. These results are consistent with the consulted bibliography. We can
conclude that there is no need to be afraid of implementing this type of exam
in the future.
Keywords: Open-book; exam; test; rote; memorization; grades.

1. Introduction
At the beginning of this 2022-2023 academic year, a guideline was announced at our
Cardenal Herrera University that many of us did not expect: the midterm continuous
assessment exams could not be rote. This instruction did not affect final exams, but it did put
professors in a position to replace the traditional midterm exams with other types of evaluable
activities, or allow the midterm exam but with the presence of notes: what in the literature
has come to be called open-book exams.
The controversy between open-book and rote exams is not new. The latest recommendations
point to a mixed method of examination since each type of examination assesses different
skill sets in students.(Johanns, Dinkens, and Moore 2017). Other authors recommend using
it as a complement to closed-book questions and believe that the post-covid world could be
a good time to rethink opening up to these possibilities.(Zagury-Orly and Durning 2020). The
grades tend to be generally higher and prepare the student for a proactive knowledge of the
exam content and the application of the subject to solve problems. (Dave et al. 2020). Some
experiences in health students have shown how the use of tools, including Internet searches,
are not very useful in questions about problem solving or those in which evidence has to be
analyzed to issue a clinical attitude, so it is necessary that the student has studied the subject
from memory (Mathieson, Sutthakorn, and Thomas 2020)
Arter reading some examples in the literature, in the Anatomy III and Anatomy IV subjects
of the Medicine degree, we planned this course to carry out an open-book midterm exam and
analyze the results to see if there were differences in results with respect to traditional exams.
Therefore, we allowed students to do the midterm exam using the analog material they
wanted: books, notes, notebooks, sheets, atlases, cards, etc. The only material we did not
authorize was digital material that could be used to directly search the questions on the
Internet. The exam proceeded normally. Finally, in the final exam in January 2023, the exam
was performed from memory in the traditional style, without any materials, thus complying
with the guidelines implemented this year.
2. Objectives
Our goal is to verify whether open-book testing has had an impact on student grades.
Although general research wishes to find differences between groups (alternative
hypothesis), in this specific case the good news would be that the implementation of open-
book exams has not influenced student grades and, therefore, the implementation of these
tests have not affected the performance of our students.
990
Antoni Alegre-Martínez, María Isabel Martínez-Martínez, José Luis Alfonso-Sanchez
3. Material and methods

The first step is to carry out a search of the grades of the students in the subjects of Anatomy
III and Anatomy IV that are taken by the Medicine students of the Cardenal Herrera CEU
University in Alfara del Patriarca (Spain). Anatomy III is taught in the first semester
(September to December, with an exam in January) and Anatomy IV in the second semester
(February to May, with an exam in June). In each subject there is a midterm exam and a final
exam that are evaluated on a total of 10 points, so that for each calendar year we will find
four exams in total. An advantage of this research is that both subjects are taught by the same
professor, which guarantees homogeneity in both teaching and evaluation. The exams were
of the same style over the years: multiple choice questions without open questions, with
incorrect answers subtracting a third of the correct ones, with 50 questions for the final exam
and 40 for the partmidterm exams. These common characteristics made it very easy to
compare the different exams since the methodology has remained constant. To avoid biases
in our results, we eliminated the exams that due to the Covid-19 pandemic could not be done
in the classroom and had to be done virtually or through proctoring. Therefore we discard the
exams "Midterm Anatomy IV 2020 (which in fact was not held), Final Anatomy IV 2020,
Midterm Anatomy III 2020, Final Anatomy III 2021 and Midterm Anatomy IV 2021".
Beginning again with Final Anatomy IV 2021, exams were already carried out under normal
conditions in the classroom and therefore the confounding factors disappear. We will use
Student's t test to compare the means between both groups of exams to check if there have
been differences between both groups.
4. Results
The main search results are shown in Table 1 from the most recent to the oldest. In it we find
the parameters of the 17 exams, the second one being the exam of interest because it is the
one that was carried out as an open-book. At first glance, it may be striking that there are
small variations in the number of students who take the partial exam and the corresponding
final exam. These variations can be explained for several reasons: students who follow the
normal course but finally drop the subject for the extraordinary session in July, or because
they enroll when the course is halfway through, or who do not take the partial exam for some
justified reason, etc.
Regarding the average of the exam, it can be seen that it has been oscillating in values close
to 6 and 7, with excellent years like 2018, and years with very low average grades like in
2022. If we compare the average of the same group of students who took the open-book exam
(6.09) and later the final exam (5.84) we see a slight decrease, although not as pronounced
as the one suffered by the Anatomy III 2021-22 group (which average dropped almost 2
points) or the group from Anatomy III 2019-20 (which lowered the average 0.64 points). It
991
is not possible to establish a clear trend as to whether the final exam has a better mean than
the midterm, since we find different cases every year. Regarding the range, practically every
year there have been students with grades of less than 1 and brilliant students with a 10 or
almost a 10. In the open-book exam there are no differences in this regard
Table 1. Measures of central tendency that describe the different exams. The Open-Book exam
is highlighted with an asterisk and bold characters. Last row shows the average of the †-marked
five midterm exams made before the Open-Book exam for comparison.
Maximum
Delivered
Niminum
deviation
Standard
Average
Median
exams
Range
Final Anatomy III 2023 60 5.84 5.85 1.90 9.0 0.5 9.5
*Midterm Anatomy III 2022 59 6.09 6.40 2.04 9.0 0.8 9.8
Final Anatomy IV 2022 56 6.44 6.45 2.18 9.7 0.3 10.0
†Midterm Anatomy IV 2022 61 5.62 5.70 2.43 9.9 0.1 10.0
†Midterm Anatomy III 2021 60 7.54 8.25 2.28 9.1 0.9 10.0
Final Anatomy IV 2021 48 7.77 8.25 1.85 9.0 0.9 9.9
Final Anatomy IV 2019 47 7.14 7.47 2.03 7.1 2.9 10.0
†Midterm Anatomy IV 2019 49 7.71 8.20 1.85 7.2 2.8 10.0
Final Anatomy IV 2018 61 7.03 7.30 1.89 8.1 1.6 9.73
Midterm Anatomy IV 2018 60 7.89 8.29 1.99 9.2 0.8 10.0
Midterm Anatomy III 2017 61 6.85 7.10 2.01 8.2 1.8 10.0
†Midterms previous 5 years 279 6.69 6.62 2.30 9.9 0.1 10.0
Source: self-made
Figure 1 represents the distribution of student grades in the different exams in a box plot for
quick and easy visualization. The exam marked in dark color is the open-book exam. In
addition, we have marked with a thick line the period in which, due to Covid-19, proctoring
examinations were carried out to avoid confounding variables. At first glance, there is not
much variability between the open-book exam and other exams taken previously.
992
Figure 1. Boxplot of each one of the exams, showing retrospectively the range and median. The second dark on the
left is the open-book exam. Source: Self-made.
It might be thought that the partial exams and the final exams are not comparable, since the
number of questions in the final exams is greater, the amount of knowladge is considerably
larger, and that this could affect the results of the exams. For this reason, the results of the
five previous partial exams have been grouped in Table 1, obtaining an average of 6.69,
somewhat higher than the 6.09 average points reached in the open-book exam. There are
hardly any changes in the 50th percentile, which reaches 6.40 in the open-book exam and
6.62 in the average of the five previous midterms. There are no changes in the minimum
marks (0.1 in the five previous midterms and 0.8 in the open-book exam) or maximum marks
(10 and 9.8 respectively). This comparison can be visualized graphically in a more
comfortable way in Figure 2:
Figure 2. Boxplot comparing the open-book exam scores with the overall scores of the previous five midterm
exams, marked in table 1 with an †obelisk. Source:Self-made.
The mean score for the rote exams was 6.77, while the mean score for the open-book exam
was 6.09. To check if this difference is statistically significant, we have performed the
Student's t-test to compare the different parameters using the categorical classification
variable "open-book exam" or "rote exam". In all applied parameters, Levene's test indicated
that equal variances were assumed and that the two-sided p values for each of the parameters
were not significant. That is, for 15 degrees of freedom, the mean (t = -0.853, p = 0.407), the
median (t = 0.587, p = 0.566), the range (t = -0.465, p = 0.648), the minimum ( t = 0.572, p
= 0.576) and maximum (t = 0.480, p = 0.638) do not represent a statistically significant
993
difference between the two groups. With these data we can conclude the main objective of
the study: the fact of taking an open-book exam did not significantly alter the parameters of
the exam with respect to the exams performed rotely.
5. Discussion
Our findings are consistent with other studies that have analyzed the impact on open-book
exam scores. Spiegel found no significant differences in the two cohorts (open an closed
exams), both in academic performance and knowledge retention.(Spiegel and Nivette 2021).
The acceptance among the students is also high, a study in London in which students were
asked for feedback found 65% of students wanting more open-book exams.(Chadha, Maraj,
and Kogelbauer 2020). Another study reports that 85% of the students preferred an open-
book exam, due to less stress and pressure, valuing positively that the effort was focused on
comprehension rather than on memorization (Quille et al. 2021). However, these exams must
be carefully prepared to ensure the validity, reliability and fairness of the examinations (Er
et al. 2020). There are also many risks and limitations, for example the evaluation of the
information retrieval instead of the knowledge acquired (Dave et al. 2020).
Before concluding, we want to highlight a limitation of our study and that is that it is based
on only one open-book exam (the first of them). However, we wanted to analyze these results
early so that we could consider whether to continue using it or not next year. Another
limitation of this study is that we have focused solely on the exam grade, without taking into
account other results of the learning process or the distribution of other quantitative
parameters such as the number of retaking students, or the relationship with other course
assessments such as the dissection practical exam. For future editions we will analyze this
data and implement some Feedback questions to find out the preferences of the students and
the perceived difficulty.
6. Conclusion
The analysis of the results of this first open-book exam allows us to conclude that there have
been no significant differences between performing the exam in this way and the traditional
rote way, and therefore there should be no fear of continuing to use them in the future.
However, we have only carried out one of these exams, so we will have to continue analyzing
the results in the following courses
References
Chadha, Deesha, Marsha Maraj, and Andreas Kogelbauer. 2020. “Opening up Assessment in
the Age of COVID-19: Exploring the Utility of Online Open-Book Exams.” Advances in
994
Engineering Education 8(4).

Dave, M., S. Ariyaratnam, C. Dixon, and N. Patel. 2020. “Open-Book Examinations.” British
Dental Journal 2020 229:3 229(3):149–149. doi: 10.1038/s41415-020-2008-6.
Er, Hui Meng, Vishna Devi Nadarajah, Pei Se Wong, Nilesh Kumar Mitra, and Zabibah
Ibrahim. 2020. “Practical Considerations for Online Open Book Examinations in Remote
Settings [Version 2].” MedEdPublish 9(1). doi: 10.15694/MEP.2020.000153.2.
Johanns, Beth, Amber Dinkens, and Jill Moore. 2017. “A Systematic Review Comparing
Open-Book and Closed-Book Examinations: Evaluating Effects on Development of
Critical Thinking Skills.” Nurse Education in Practice 27:89–94. doi:
10.1016/J.NEPR.2017.08.018.
Mathieson, Georgia, Roxanne Sutthakorn, and Olivia Thomas. 2020. “Could the Future of
Medical School Examinations Be Open-Book - a Medical Student’s Perspective?”
Https://Doi.Org/10.1080/10872981.2020.1787308 25(1). doi:
10.1080/10872981.2020.1787308.
Quille, Keith, Keith Nolan, Brett A. Becker, and Seán McHugh. 2021. “Developing an Open-
Book Online Exam for Final Year Students.” Annual Conference on Innovation and
Technology in Computer Science Education, ITiCSE 338–44. doi:
10.1145/3430665.3456373.
Spiegel, Tali, and Amy Nivette. 2021. “The Relative Impact of In-Class Closed-Book versus
Take-Home Open-Book Examination Type on Academic Performance, Student
Knowledge Retention and Wellbeing.”
Https://Doi.Org/10.1080/02602938.2021.2016607 48(1):27–40. doi:
10.1080/02602938.2021.2016607.
Zagury-Orly, Ivry, and Steven J. Durning. 2020. “Assessing Open-Book Examination in
Medical Education: The Time Is Now.”
Https://Doi.Org/10.1080/0142159X.2020.1811214 43(8):972–73. doi:
10.1080/0142159X.2020.1811214.
995
Experiencing third spaces in between university and society:

transdisciplinary learning experiences in a shopping center
Thorsten Philipp1, Katarina Marej2
1
The President’s Office, TU Berlin, Germany, 2Department of Sociology and Social Work,
Universitetet i Agder, Kristiansand, Norway.
Abstract
What happens when a university leaves its campus and enter spaces that seem
reserved for completely different functions and areas of life - such as
shopping? What opportunities and potentials arise from such a shift for the
dialog between science and society? Is it an innovative way for science
communication to reach groups other than the education-oriented middle
class? What skills do academics need to succeed in such a setting? These and
similar questions formed the starting point of the Berlin project “Mall Anders
- Open Learning Lab for Science and Society”. For seven months in 2021/22,
TU Berlin used a retail space in a shopping mall to test new ways of
communicating science and co-producing knowledge in a dialogue between
the university, business, civil society, politics, and culture. The relocation of
university life to a public shopping center enabled broad public involvement
in teaching and learning, high activating potential in overcoming the rifts
between university and society and new impulses for science communication,
participatory research, and student-organized teaching.
Keywords: Transdisciplinary learning; knowledge transfer; didactics;
transformative science; participatory research; science communication.

Experiencing third spaces in between university and society
1. Introduction
A central question in the debate about participatory ways of knowledge production concerns
the forums and infrastructures of collaborative research and learning alliances on major
societal problems (Bergmann et al. 2012; Fam et al. 2019; Gibbons et al. 1994; Thompson
Klein/Philipp 2023). How can inter- and transdisciplinary spaces be created that mediate
between academia and society – and how should they look like in practice? What are the
requirements of experimental learning facilities that follow a transdisciplinary objective, and
which kind of learning dynamics do they enable?
To plausibilize these problems and related questions on hybrid areas, transfer practices, and
thirdspace (Soja 2007), which increasingly accompany the debate on academic research and
teaching, this article makes the case for a specific pioneer project in Berlin: Mall Anders -
Offenes Lernlabor für Universität und Gesellschaft (Mall Anders - Open Learning Lab for
University and Society). Mall Anders (ambiguously translatable as “The Other Mall” or “This
Time Differently”) was a transdisciplinary teaching and learning space in the core of a
shopping center that operated over seven months in 2021/22. This article presents (1) idea,
genesis, and shape of the project, (2) theoretical foundations and didactic methods applied
within it, and (3) central learning experiences for different participants at the interface
between university and society.
2. “Mall Anders” - Origin and Design of a Public Learning Lab

Mall Anders was established in fall 2021 in the middle of WILMA Shoppen, a retail shopping
center in Berlin-Charlottenburg. Its genesis was the result of TU Berlin’s transfer policy and
continuous efforts to strengthen the cooperation between the university and representatives
from business, politics, civil society, culture, and media. With the mediation of the municipal
administration of Charlottenburg-Wilmersdorf, the university obtained a 380-square-meter
retail space whose previous user, a retailer specializing in gift items, had gone bankrupt
during the first phase of the pandemic and left the entire interior (shelves, lighting, cash
register systems, etc.) along with plenty of merchandise in the shop.
A service-learning course at the TU Berlin had the task to redesign the retail space with the
given materials. The Natural Building Lab (2021) of the Faculty of Architecture, which
specializes in practices of circular building, offered students an interdisciplinary design
seminar with the creative task of exploring the space, its context, its intended purpose, and
the range of materials left behind with the task to prepare it for a transdisciplinary learning
site. After eight weeks of analysis and reflection, the students decided on a flexible, easily
changeable interior design that consisted almost entirely of upcycling the given materials:
shelf combinations became room dividers and projection surfaces for video installations,
drawers were transformed into seatings, stuffed animals served as seat upholstery –
998
civilization trash became interior architecture (Mall Anders 2022). Mall Anders inspired the
discourse of sustainable development both materially and didactically (cf. Philipp 2023) and
thereby aimed at contributing practical knowledge to the ongoing transdisciplinary debate
3. Theoretical Foundations and Expectations

Transdisciplinarity has been negotiated and elaborated since the 1990s (Vilsmaier et al.
2023). As academia is increasingly expected to be impactful for society, new responsibilities
arise that force researchers to consider their role in the research process. In the face of
systemic uncertainties and normative plurality, scholars alone can neither represent the
various perspectives and interests nor make reliable predictions about future developments.
In addition to ‘applied science’ and ‘professional consultancy', Funtowicz & Ravetz (1993)
recommend a post-normal science. This calls for constructive stakeholder involvement and
thus aims at the establishment of democratic research structures which consider the concerns
of those people ‘affected’ by the research results. It is therefore a prerequisite of
transdisciplinary learning to recognize the plurality of knowledge resources (Philipp &
Schmohl 2023) and the necessity of horizontal and dialogical processes.
Mall Anders aimed precisely at this goal, to provide space for the interplay of plural
knowledge resources, to allow heterogeneous forms of knowledge to meet and to enable
interaction. The claim to test different ways of knowledge communication and production,
however, presupposed a reflexive opening of academic knowledge culture in order to unfold
effectiveness (Marej 2020). Mall Anders therefore rejected traditional transfer logics of
unidirectional knowledge transmission and aimed at the multidirectional flow of knowledge
between diverse members of society. By confronting academic knowledge with local
everyday and practical knowledge, the experimental space triggered processes of reflection
and learning about tacit forms of knowledge (Polanyi 1966) as well as power differences of
situated knowledge (Haraway 1988). Against this background, the question arose where and
how such a place could be created and which challenges would emerge. On the one hand,
non-participation or fake participation (cf. Arnstein 1969), which would only serve academic
self-assurance, should be avoided. On the other hand, the widespread problem was that only
members of the educated middle classes participate and that other population groups are not
included. Therefore, first, a place in the city was sought that was accessible to everyone and
had no educational barriers: a shopping mall. Moreover, the content of the project was
conceptualized in a democratic way with a plurality of themes, methods and techniques.
Mall Anders opened its doors in December 2021. Its program was conceived through an open
call for projects: students, teachers, and other staff members of Berlin’s universities, and
public actors from the arts and civil society were invited to use the space temporarily by
proposing an interactive educational aim and concept. The applicants had to explain the
999
communicative methods and potentials they were aiming for with their project. There were
no thematic requirements. The use of the site was free of charge for all groups and was
organized, supervized, and supported by a team of student assistants and early career
researchers.
During its seven-month duration, Mall Anders hosted over 250 events. The content spectrum
was broad and comprised sustainability (e.g. circular economy or mobility), urban
development, digitalization, pop music research, and linguistics up to ancient cultures and
numerous other topics and disciplines. The methodological approaches included workshops,
(interactive) exhibitions, science slams, short lectures, quizzes, infomovies, creative
workshops for children, artistic performances, and individual counseling (see: Berlin
University Alliance 2023).
The primary user groups were: (1) students, who used the space in the context of project
workshops or student research, e.g. to present their own research or to collect data in
conversation with visitors, etc. (2) Senior researchers, who presented their research on
current issues such as the future of liberal democracies or the latest developments in catalysis
research. Their favorite communicative methods were exhibitions, panels, storytelling
events, and science slams. (3) Representatives of civil society, who addressed either locally
relevant topics such as urban transition, mobility, land use etc. or overarching societal
challenges such as conflict resolution techniques. Furthermore, Mall Anders was also an
arena of (4) artistic intervention: the co-production of knowledge at the intersection of art,
technology, and science (Chemi/Du 2017) created knowledge exchange through artistic
products, artistic practices, and interpersonal exchanges. Finally, the (5) curators of the site
used it as a display and experimental space of current science theory debates. The whole
project was an open invitation to reflect on the cognitive, communicative, and organisational
tasks of science and academic learning through the confrontation with new audiences. This
enterprise included theoretical preparations and structured reflection workshops, but also
daily tasks such as creating an inviting atmosphere for everyone involved and explaining the
learning space and its program to the visitors.
Different forms of learning and interaction defined Mall Anders: in (1) everyday mode during
the few time slots when there was no program announced, a team member welcomed visitors
at the entrance area to answer questions. The space was open to the public on a daily basis,
and Mall Anders’ visual identity of color coding, palm trees, lounge chairs, and modular-
mobile furnishings attracted people to enter, linger, and interact. Another free form of
interaction was what we called (2) Open House or Project Display. Mall Anders became a
publicly accessible co-working space for students, scholars or artists who transferred their
work to the mall and thereby provided insights into their work processes and research
practices. In addition, Mall Anders served as a (3) forum for exchange and reflection for
workshops and conferences with experts on transdisciplinarity and science communication.
1000
This interaction largely took place between actors of the academic and other professional
communities. Visitors participated by joining the conference, sharing thoughts and topics.
The most frequented formats were the ones explicitly oriented towards interaction: events
curated by cooperation partners (4), which significantly determined the program. The
cooperation partners announced their respective event via the website and through their
networks, they planned and designed their learning arrangements, and prepared how to
address passers-by and visitors. The events ranged from one-hour to multi-day exhibition,
dialog, experiment, and workshop formats.
4. Learning Experiences
Both after each event and at several larger workshops, we reviewed critically the procedure
the results and processed them for later events. In two debriefing reflection-workshops after
seven unusual months of university life in a shopping mall, we identified four dimensions of
learning experiences.
(1) Visitors and passers-by often reported that Mall Anders was in many ways an irritation
in the flow of consumption, a disruption of spatial expectations, a surprising learning
experience in a place where this was not to be expected. First-time visits were often
stimulated by interest in individual topics, by curiosity about the unusual store and its
aesthetics, or by personal contact. In the conversation with academic representatives, some
visitors recognized Mall Anders as a space of information, an opportunity to reflect their own
knowledge resources and their own educational biography. Thereby, Mall Anders was a
means of educational reflection.
(2) Students, professors, and university members revealed to us that the communication
framework of the mall exposed the limits of their communicative ability when it came to
opening up research questions to a new, unknown audience. How do I talk about my research
if I am not confronted with scientists? How do I address my audience if I do not know
anything about their personal and professional backgrounds? How do I arouse their interest?
For many university members, the unusual formats entailed significant challenges. In this
respect, Mall Anders was also a reflection on the limits and challenges of science
communication in a time, in which audience formations are constantly changing and the
legitimacy of science is increasingly put in doubt (Bucchi/Trench 2021).
There were also learning experiences on the side of the (3) shopping center management.
The non-commercial use of store by a university helped to overcome the embarrassment of
unused areas. It was capable to attrackt a new audience and media attention, at least
temporarily. On the other hand, the work of students in the shopping center and the
unconventional nature of some interventions inevitably led to conflicts with other store
operators. Mall Anders rather unintentionally initiated the discussion about to what extent
1001
shopping centers are willing to leave behind the outdated doctrine of an “architecture of
seduction” (Venturi/Brown/Izenour 1979: 20) and instead embrace the social interaction with
their visitors as a process of ‘collective creativity’. However, for the latter, visitors need to
be recognized as co-creators rather than as consumers.
A continuous learning experience happened alsow within our own (4) team of initiators and
curators: the colleagues who daily managed Mall Anders, opened and closed its glass doors,
curated the program, advised the scientists and students, answered the visitors’ questions,
endeavored to connect with the other stores, and much more. One central learning experience
was dealing with the spatial and social threshold between inside and outside. Many passers-
by were aware of the challenge of exposing themselves to the risky, unknown space - but
how to react? The team learned to shape the threshold permeable: constant attention and
unobtrusive personal contact were the keys to signal accessibility and arouse curiosity.
Balancing between ‘welcome culture’ and restraint was a challenge. Children, meanwhile,
were the ones least shy about entering our space, and furthermore technical and scientific
installations and experiments and other playful interaction offers in the entrance area gained
attention immediately.
With all these communicative dynamics, Mall Anders turned out to be an example of
boundary work (Gieryn 1983), as it operated on the demarcation of science from non-science
and the interrelatedness of various forms of knowledge resources such as professional
knowledge, scientific knowledge, embodied knowledge (Gustafson 1999; Freiler 2008), etc.
The specific combination of learning lab, upcycling practice, and maker education in a
shopping mall turned Mall Anders into an experience of public irritation. The building type
shopping mall was originally entirely oriented towards consumerism and capitalist
exploitation logic (Kroes 1996: XIV; Coleman 2006: 145-167). However, it has long since
ceased to be exclusively dedicated to shopping and has become a place for leisure and urban
life. The fact that it could also serve as a place of university life and science-society
interaction therefore appears to be a reasonable extension.
As an in-between learning space (Vilsmaier 2015), as a transdisciplinary forum of debate,
Mall Anders enabled the work on rhizomatic intersections (Deleuze/Guattari 1980) of
problem contexts and thereby dispensed hierarchical structures. Originally intended as a
means of science communication, as a service of university representatives for society, Mall
Anders turned out to be a learning experience for the universities themselves. At the
crossroads of science and society, it was not only a space of risk and rehearsal, but also a
marketplace of competence acquisition for scholars of all disciplines and qualification levels.
In this respect, Mall Anders was also a ‘boundary work’ on the university’s societal role and
its communicative potentials.
1002
References
Arnstein, S. (1969). A ladder of citizen participation. Journal of the American Institute of
Planers 4, 216–224.
Bergmann, M., Jahn, T., Knobloch, T., Krohn, W., Pohl, C., & Schramm, E. (2012). Methods
for transdisciplinary research: A primer for practise. Frankfurt am Main: Campus.
Berlin University Alliance (2023). Mall Anders – Offenes Lernlabor für Wissenschaft und
Gesellschaft. Retrieved from http://mall-anders.berlin
Bucchi, M. & Trench, B (2021). Rethinking science communication as the social
conversation around science. Journal of Science Communication (JCOM) 20 (3): Y01.
Chemi, T., & Du, X. (Eds.) (2018). Palgrave Studies in Business, Arts and Humanities. Arts-
based Methods and Organizational Learning: Higher Education Around the World.
Cham: Springer.
Coleman, P. (2006). Shopping Environments. Evolution, Planning and Design.
Amsterdam/Boston: Elsevier.
Deleuze, G., & Guattari, F. (1976). Rhizome. Paris: Les Éditions de Minuit.
Fam, D., Neuhauser, L., & Gibbs, P. (eds.) (2018). Transdisciplinary theory, practice and
education: The art of collaborative research and collective learning. Cham: Springer.
Freiler, T. J. (2008). Learning through the body. New Directions for Adult and Continuing
Education 119: 37–47.
Funtowicz, Silvio O., Ravetz, Jerome R. (1993). Science for the post-normal age. Futures,
25(7), 739–755.
Gibbons, M., Camille L., Helga N., Schwartzmann, S., Scott, P., and Trow, T. 1994. The new
production of knowledge: The dynamics of science and research in contemporary
societies. Los Angeles: Sage.
Gieryn, T. F. (1983). Boundary-Work and the Demarcation of Science from Non-Science:
Strains and Interests in Professional Ideologies of Scientists. American Sociological
Review, 48(6), 781–795.
Gustafson, D. L. 1999. Embodied learning: The body as an epistemological site. In M.
Mayberry and E. C. Rose (eds.), Meeting the challenge: Innovative feminist pedagogies
in action, 249–73. New York: Routledge.
Haraway, D. (1988). Situated Knowledges: The Science Question in Feminism and the
Priviledge of Partial Perspective. Feminist Studies, 14, 575–599.
Kroes, R. (1996). If you’ve seen one, you’ve seen the mall. Europeans and American mass
culture. Urbana: University of Illinois Press.
Mall Anders (2022). Raus aus dem Elfenbeinturm und rein in die Mall. Retrieved from
https://www.mall-anders.berlin/ort.
Marej, K. (2020). Creating impact in civic education by transformative research. Indications
for professionalisation. JSSE 19(2), 24–46.
Natural Building Lab (2021). Mall Anders. Retrieved from
https://www.nbl.berlin/circles/mall-anders.
1003
Philipp, T. (2023). Nachhaltigkeit durch Pluralität der Wissensressourcen: Prämissen und

Praktiken Transdisziplinären Lernens. In W. Leal (Ed.), Lernziele und Kompetenzen im
Bereich Nachhaltigkeit. Wiesbaden: Springer (in press).
Philipp, T., & Schmohl, T. (2023). Embracing the Rhizome: Transdisciplinary Learning for
Innovative Problem Solving. In T. Philipp & T. Schmohl (eds.), Handbook
Transdisciplinary Learning. Bielefeld: transcript (in press).
Philipp, T., & Thompson Klein, J. (2023). Interdisciplinarity. In T. Philipp & T. Schmohl
(eds.), Handbook Transdisciplinary Learning. Bielefeld: transcript (in press).
Philipp, T., Marej, K., & Fenster, L. (2023). Didaktische Experimente im Spielfeld zwischen
Universität und Gesellschaft: Ein transdisziplinäres Lernlabor im Einkaufszentrum. In K.
Kiprijanov, T. Philipp, & T. Roelcke (eds.), Transferwissenschaften: Mode oder
Mehrwert? Frankfurt am Main: Peter Lang (in press).
Polanyi, M. (1966). The Tacit Dimension. Chicago: University of Chicago Press.
Soja, E. W. (2007). Thirdspace: Journeys to Los Angeles and other real-and-imagined
places (10. printing). Malden, Mass.: Blackwell.
Venturi, R., Scott Brown, D., & Izenour, S. (1977). Learning from Las Vegas: The forgotten
symbolism of architectural form. Cambridge, Mass.: MIT Press.
Vilsmaier, U., & Lang, D. J. (2015). Making a difference by marking the difference:
Constituting in-between spaces for sustainability learning. Current Opinion in
Environmental Sustainability. (16), 51–55.
Vilsmaier, U., Merçon, J., & Meyer, E. (2023). Transdisciplinarity. In T. Philipp & T.
Schmohl (eds.), Handbook Transdisciplinary Learning. Bielefeld: transcript (in press).
1004
Adaptation of the interprofessional collaborative competency

attainment scale for usage across professions
Monja Pohley, Simon Schmitt, Aldin Striković, Eveline Wittmann
Technical University of Munich, Germany; TUM School of Social Sciences and Technology,
Department of Educational Sciences.
Abstract
Health and nursing professionals, not only in clinical environments but also in
nursing homes or in the home environment, are increasingly influenced by
digital technology, specifically in the smart home. Technology professionals in
IT are affected as well: clients, but also the professionals acting on their
behalf, depend on technological support and consultation, because they
themselves need to make responsible decisions concerning issues such as data
security. More broadly, it is necessary to enhance multidisciplinary
perspectives to collaborate across technological and non-technological
professions. Meeting this challenge requires educational interventions to
promote multi-professional collaboration competencies. Research on
enhancing these competencies requires measurement instruments. For this
purpose, this paper describes the adaptation of a measurement instrument
originating from interprofessional collaboration in health care, the ICCAS,
and its translation for application in a German language context via the
TRAPD method. The results of a pretest evaluating the comprehensibility of
the items are reported.
Keywords: nursing education; technology education; digital transformation;
multi-professional collaboration.

Adaptation of the interprofessional collaborative competency attainment scale
1. Introduction
Florence Nightingale is reputed to be the founder of professional nursing education (Attewell,
1998, Egenes, 2009). Since then, nursing education and training underwent changes,
developed (Scheckel, 2009) and the current and ongoing change in medicine and health
through digital technologies will inevitably affect nursing as well (Barbosa et al., 2021).
While the replacement of professional services in these social fields of action is unlikely
(WHO, 2019, Dengler & Matthes. 2019), networked digital technology and infrastructures
are increasingly being implemented in these fields, including in private homes (Brynjolfsson
& McAfee, 2014).
Unsurprisingly, these changes not only require technological skills and competencies, but
also competencies in areas like collaboration (van Laar et al. 2018). But while the debate in
this regard has mostly focussed on industrial requirements, our interest is at the intersection
of healthcare/nursing and IT/technological professions, including the most private space, the
home. Two professional fields merge in people’s homes, or in nursing homes, with the
implementation of smart home technology (Bartsch & Stilz, 2021; Nettelstroth et al. 2022):
Professionals providing care, healthcare and autonomy support,1 and technological
professionals providing the IT infrastructure, but also technological support and consultation,
for example with regard to data security, data protection and privacy in these sensitive
personal spheres (General Data Protection Regulation 2018).
With the purpose of creating a multi-professional training within higher education to support
multi-professional collaboration at the intersection of healthcare and nursing professionals
and technological professionals (specifically, IT), our goal is to be able to measure whether
this training has the intended effect and, therefore, to provide for valid measurement. Because
of the longstanding work on interprofessional collaboration in healthcare and nursing,
instruments are already available in this field, but may not be valid for the intersection with
technological professionals. For example, they often refer to ‘teams’, whereas work at the
intersection we intend to describe may not be perceived as team work by the professionals.
We chose the ICCAS instrument, which is currently only available in French and English
(MacDonald, et al., 2010). Thus, we aim to adapt it to collaboration across professions while
also translating it into German language following the TRAPD method (European Social
Survey, 2020; Harkness, 2003).
1
For example in Ambient Assistant Living.
1006
2. Multi-professional collaboration between healthcare and nursing

professionals and technological/IT professionals in the context of the digital
transformation
Historically, professional work in nursing and related fields has rather been seen in opposition
to technological professional work, due to two conditions: First, it has been rather unaffected
by digitalisation beyond its use for personal communication. Consequently, health care
professionals have long been rather unfamiliar with work-related technologies such as
electronic health records (Pohley & Wittmann, 2021). This is the case, despite the fact that
an electronic health record can serve as a means of providing personalized care in terms of
care and case management through data extraction and data analysis (Snoddon, 2010;
Wittmann & Weyland, 2020; Striković & Wittmann, 2022). Additionally, when it comes to
nursing people in their own homes, professionals, not only nurses, are increasingly
confronted with smart home technology implemented to maintain the resident’s independent
lifestyle. But secondly, professional work in the field has historically been a predominantly
female profession, originating in religious orders with charity as a common ground (Gauci et
al., 2022; Friese, 2018; 2021). In contrast, professional work in technological areas,
especially IT, has historically been predominantly male, associated more with math and anti-
social imagery and less associated with service in social fields (Ensmerger, 2015).
Multi-professional collaboration, we contend, therefore requires substantial changes in
perspective from both professional viewpoints, including active listening from the part of the
technological professionals, and the readiness and ability to articulate one’s concerns in
relation to technological professions from the part of the healthcare and nursing
professionals. To support multi-professional collaboration competencies by students from
both professional areas, we developed a training based on role-pay located in an innovative
academic smart home simulation lab, using perspective taking on multiple instances, added
by an introduction into the varying professional perspectives on the smart home, as well as
reflection and discussion (Striković & Wittmann, 2022). Following the curriculum-
instruction-assessment triad (Achtenhagen, 2012; Pellegrino, 2012), in order to assess the
outcome of the training we require an instrument that can validly capture multi-professional
collaboration across the healthcare and nursing as well as the technological professional
fields – we are not aware of any such instrument.
3. Adaptation of the Interprofessional Collaborative Competency Attainment

Scale
Because of the longstanding work on interprofessional collaboration in healthcare and
nursing, instruments for assessing interprofessional collaboration competency are already
available in this field, but may not be valid for the intersection with technological
1007
professionals. One instrument identified through a systematic literature review conducted by

Striković and Wittmann (2020) is the Interprofessional Collaborative Competency
Attainment Scale (ICCAS). The ICCAS seems to be suitable for adapting it to collaboration
across professions since MacDonald and colleagues (2010) propose that it “can be applied to
a variety of interdisciplinary tasks and settings, such as social work and human services
education” (p. 304) as well. Moreover, the ICCAS is also a validated questionnaire
(Archibald et al., 2014) and has been replicated and revised (Schmitz et al., 2017).
Unfortunately, the questionnaire is only available in English and French (MacDonald et al.,
2010), which means that it not only has to be adapted but also translated into German before
being used for assessment of multi-professional competency in a German context. For this,
we chose the TRAPD method consisting of five steps: The first step is to translate the items
independently. This also includes the deliberate modification of the ICCAS items’ content
with regard to multi-professional contexts. Second, the two different translations are
discussed under the supervision of a reviewer. If there are remaining points of discussion
after the review phase, the third step is to settle them with an adjudicator. The fourth step is
to conduct the pretest whereas the fifth step is to document the whole process thoroughly
(European Social Survey, 2020; Harkness, 2003).
The authors speaking German as their first language consider themselves as suitable
translators, reviewer and adjudicator, due to their individual backgrounds:
1. Translation: MP and SiS have a professional qualification in health care and
computer science, respectively.
2. Review: AS’ research focuses on modelling and measuring multi-professional
competencies – specifically in nursing education. Furthermore, he has a background
in electrical engineering and IT.
3. Adjudication: EW has advanced experience in developing questionnaires.
Regarding the adaptation of the ICCAS, there were some challenging phrases and terms. For
instance, the original questionnaire often refers to an interprofessional team or team members
whereas work at the intersection we intend to describe may not be perceived as team work
by the professionals. For example, on the one hand, technological professionals provide the
IT infrastructure, but also technological support and consultation; on the other hand, nurses
act on the behalf of the patients in order to support their autonomy. Hence, in such a multi-
professional context, when fulfilling customers’ needs technological professionals have to
consider clients’ autonomy, too. Thus, the goals of these two professional groups overlap
whereas they do not coincide as it would be in a team situation. As a consequence, we
determined ,representative of another profession’ to be more appropriate than
,interprofessional team (members)’. Moreover, some items included health specific terms
expressing the patient-centered care. In order to adapt the instrument to the broader context
1008
of collaboration across professions, we used the term ,problems’ instead (see Striković &
Wittmann, 2022).
4. Conclusion
As we have argued, the digitalisation necessitates multi-professional collaboration. With the
purpose of evaluating a multi-professional training within higher education, we chose the
ICCAS in order to adapt it to collaboration across professions while also translating it into
German language following the TRAPD method. The results of a pretest evaluating the
comprehensibility of the items are reported.
References
Achtenhagen, F. (2012). The curriculum-instruction-assessment triad. Empirical Research in
Vocational Education and Training, 4, 5–25. doi: 10.1007/BF03546504
Attewell, A. (1998). Florence Nightingale (1820-1910). Prospects, 28, 151–166. doi:
10.1007/BF02737786
Archibald, D., Trumpower, D., & MacDonald, C. J. (2014). Validation of the
interprofessional collaborative competency attainment survey (ICCAS). Journal of
Interprofessional Care, 28(6), 553–558. doi: 10.3109/13561820.2014.917407
Barbosa S., Abbot, P.A., Sasso, G. (2021) Nursing in the Digital Health Era. Journal of
Nursing Scholarship, 53(1), 5-6. doi: 10.1111/jnu.12620
Bartsch, S. & Stilz, M. (2021). Verantwortlich entscheiden (lassen). Smarte
Haushaltstechnologien als Anlass, über eine zukunftsfähige Verbraucherbildung
nachzudenken. In M. Friese (Ed.), Care Work 4.0. Digitalisierung in der beruflichen &
akademischen Bildung für personenbezogene Dienstleistungsberufe [Care work 4.0.
Digitalization in vocational & academic education for personal service professions] (pp.
185–197). Bielefeld: wbv.
Brynjolfsson, E. & McAfee, A. (2014). The second machine age: Work, progress and
prosperity in a time of brilliant technologies. New York, NY: W. W. Norton & Company.
Dengler, K., & Matthes, B. (2019). Digitalisierung in Deutschland:
Substituierbarkeitspotenziale von Berufen und die möglichen Folgen für die
Beschäftigung. In R.Dobischat, B. Käpplinger, G.Holzberger & Münk, D. (Eds), Bildung
2.1 für Arbeit 4.0?. [Education 2.1 for Work 4.0?] (pp. 49–62). Springer VS: Wiesbaden.
doi: 10.1007/978-3-658-23373-0_3
Egenes, K.J. (2009). History of Nursing. In G. Roux & J. Halstead (Eds.), Issues and Trends
in Nursing. Essential Knowledge for Today and Tomorrow (pp. 1–26). Sudbury, MA:
Jones and Bartlett.
Ensmerger, N. (2010). Making Programming Masculine. In M. Thomas (Ed.), Gender
Codes: Why Women are Leaving Computing. Hoboken, NJ: Wiley. doi:
10.1002/9780470619926.ch6
1009
European Social Survey (2020). ESS Round 10 Translation Guidelines. London: ESS ERIC
Headquarters. https://www.europeansocialsurvey.org/docs/methodology/ESS_R10_Tra
nslation_Guidelines_GESIS-final-alert-1.pdf. Accessed 27 February 2023.
Friese, M. (2018). Reformprojekt Care Work. Professionalisierung der beruflichen und
akademischen Ausbildung [Care work reform project: Professionalization of vocational
and academic training]. Bielefeld: wbv. doi: 10.3278/9783763959754
Friese, M. (2021). Care Work 4.0. Überblick und Einführung in die Beiträge. In M. Friese
(Ed.), Care Work 4.0. Digitalisierung in der beruflichen & akademischen Bildung für
personenbezogene Dienstleistungsberufe [Care work 4.0: Digitalization in vocational &
academic education for personal service professions] (pp. 13–28). Bielefeld: wbv. doi:
10.3278/6004710w
General Data Protection Regulation (2018). https://gdpr-info.eu/
Gauci, P., Peters, K., O’Reilly, K., & Elmir, R. (2022). The experience of workplace gender
discrimination for women registered nurses: A qualitative study. Journal of Advanced
Nursing, 78, 1743–1754. doi: 10.1111/jan.15137
Harkness, J. (2003). Questionnaire translation. In J. Harkness, F. van de Vijver, & P. Mohler
(Eds.), Cross-Cultural Survey Methods (pp. 35–56). Hoboken, NJ: Wiley.
MacDonald, C. J., Archibald, D., Trumpower, D., Casimiro, L., Cragg, B., & Jelley, W.
(2010). Designing and operationalizing a toolkit of bilingual interprofessional education
assessment instruments. Journal of Research in Interprofessional Practice and
Education, 1(3). doi: 10.22230/jripe.2010v1n3a36
Nettelstroth, W., Dietrich, O., Schilling, G., Korflür, I., Löckener, R. & Gebauer, T. (2022).
Mehrwert Mitgestaltung – Der ganzheitliche Weg zur Arbeit 4.0 [Added value Co-design
– The holistic path to Work 4.0]. In R. Dumitrescu (Ed.), Gestaltung digitalisierter
Arbeitswelten. Intelligente Technische Systeme – Lösungen aus dem Spitzencluster it’s
OWL. Berlin, Heidelberg: Springer Vieweg. doi: 10.1007/978-3-662-58014-1_9
Pellegrino, J. W. (2012). The design of an assessment system focused on student
achievement: A learning sciences perspective on issues of competence, growth and
measurement. In S. Bernholt, K. Neumann & P. Nentwig (Eds.), Making it tangible –
learning outcomes in science education (pp. 79–107). Münster: Waxmann.
Pohley, M., & Wittmann, E. (2021). IT-Grundausbildung für angehende Lehrkräfte im
Bereich Pflege und Gesundheit. In M. Friese (Ed.), Care Work 4.0. Digitalisierung in der
beruflichen & akademischen Bildung für personenbezogene Dienstleistungsberufe [Care
work 4.0. Digitalization in vocational & academic education for personal service
professions] (pp. 63–72). Bielefeld: wbv.
Scheckel, M. (2009). Nursing education: Past, present, future. In G.Roux & J. Halstead
(Eds.), Issues and Trends in Nursing. Essential Knowledge for Today and Tomorrow (pp.
27–61). Sudbury, MA: Jones and Bartlett.
Schmitz, C. C., Radosevich, D. M., Jardine, P., MacDonald, C. J., Trumpower, D., &
Archibald, D. (2017). The Interprofessional Collaborative Competency Attainment
Survey (ICCAS): A replication validation study. Journal of Interprofessional Care,
31(1), 28–34. doi: 10.1080/13561820.2016.1233096
Snoddon, J. (2010). Case management of long-term conditions: Principles and practice for
nurses. Oxford: Blackwell. doi: 10.1002/9781444319866.ch2
1010
Striković, A. & Wittmann, E. (2020). Interprofessionelle Kooperationskompetenz bei

Pflegeauszubildenden im Lichte der Forschung [Interprofessional collaboration
competence of nursing students in the light of research]. Jahrestagung der Sektion Berufs-
und Wirtschaftspädagogik der Deutschen Gesellschaft für Erziehungswissenschaft 2020,
Universität Osnabrück / Universität Münster (Virtuelle Tagung), 09.09.2020.
Striković, A., & Wittmann, E. (2022). Collaborating Across Occupational Boundaries:
Towards a Theoretical Model. Vocations and Learning, 15(2), 183–208. 10.1007/s12186-
022-09284-w
Van Laar, E., van Deursen, A. J., van Dijk, J. A., & de Haan, J. (2018). 21st-century digital
skills instrument aimed at working professionals: Conceptual development and empirical
validation. Telematics and informatics, 35(8), 2184–2200. doi:
10.1016/j.tele.2018.08.006
Wittmann, E. & Weyland, U. (2020). Berufliche Bildung im Kontext der digitalen
Transformation [Vocational education in the context of digital transformation]. Zeitschrift
für Berufs- und Wirtschaftspädagogik, 116(2), 269–291. doi: 10.25162/zbw-2020-0012
Wittmann, E. & Neuweg, G.-H. (2021). Die digitale Transformation als Herausforderung für
den Hintergrund unseres Wissens [The digital transformation as a challenge for the
background of our knowledge]. In K. Beck & F. Oser (Eds.), Aktuelle Resultate und
Probleme der Berufsbildungsforschung (pp. 265–277). Bielefeld: wbv.
WHO (2019). Future of digital health systems: Report on the WHO Symposium on the Future
of Digital Health Systems in the European Region: Copenhagen, Denmark, 6–8 February
2019. Copenhagen: WHO Regional Office for Europe.
https://apps.who.int/iris/handle/10665/329032 Accessed 09 February 2023
1011
Story/No Story: narrative design exercise for shared ways of

seeing
John Stevens
School of Design, Royal College of Art, United Kingdom.
Abstract
A case example is presented of an innovative teaching and learning
experience, devised somewhat emergently for a mixed class of 30
undergraduate design students in Beijing, China. It shares the benefits of a
mash-up approach drawing on fundamentals of design practice translated into
an intensive five-day observation-to-animation exercise, with unexpected
learning value to students of three different design-related disciplines.
This paper outlines the instructions given, the process followed, the resulting
designed outcomes, and student feedback gathered some three and a half years
later. These testimonies evidence the lasting impact of this activity on the
students despite its short duration, and specifically the value of observation,
creative experiment, facilitating peer relationships, collaboration, and peer
learning across the three disciplines. All of these elements were explicitly
absent from other learning modules in their curriculum. The scenario provides
a framework to inspire and guide similar collaborative activities across design
disciplines.
Keywords: narrative design; observation; animation, interaction design;
drawing, sensemaking; action learning.

Story/No Story: a narrative design exercise for shared ways of seeing
1. Introduction
As this is a qualitative personal account, this paper is written in the first person, giving the
author’s subjective perspective on a particular teaching situation, accompanied by samples
of the visual output of the students and of their recollections of the project.
While visiting Beijing from the UK I was invited to lead a short course for a mixed class of
30 Animation, Information & Art Design (Infodesign), and Photography undergraduate
students. I had complete freedom to cover any areas I chose, with no learning outcomes
specified by the hosting faculty. However, as I had come expecting to run a product
innovation course, this left me with a short time to plan. Having no experience of animation
I consulted a friend and former colleague, Dr Hugo Glover, who gave me very helpful
suggestions for accessible and stimulating approaches to animation.It was a new challenge
to me to devise a learning exercise that would be equally engaging, relevant and stimulating
to students of all three loosely connected disciplines. Looking for common ground between
them, I devised an activity that would emphasise practises common to all – namely observing,
visualising, and storytelling – but not overly familiar or unchallenging. I was confident that
I could draw on my own expertise in innovation design, of products, services and digital
experiences, to conceive such an activity, but was uncertain how successful it would be.
The main activity outlined here required the students to work together in mixed teams of
three or four people, to examine a simple (to them), everyday task in detail, by describing in
text, then enacting it, recording on video, translating to a storyboard then into a
stylised animation. By moving from the detailed recording of the actions to a free, abstracted
visualisation, the exercise balanced the contrasting elements of technical exactitude and
creative fluidity.
2. Educational context
Animation as a practice demonstrates and builds skills of visual narrative and observation
(Williams, 2009; Dobson et al. 2018; Glover, 2020). Constructivist theories of learning
suggest students acquire knowledge by active engagement, connecting existing knowledge
with new stimuli (Bada & Olusegun, 2015). In design pedagogy the studio learning
environment is considered fundamentally important for enabling active engagement for tacit
knowledge acquisition, not only ‘hard’ technical skills, such as drawing and modelmaking,
but the softer skills of observation and creative association, as well as informing development
of a personal perspective or ethos (Wilson & Zamberlan, 2017).
In design pedagogy, drawing objects is a well established studio activity to observe and
interrogate, principally for a better understanding of their material and formal qualities but
also to practise and build this skill of noticing, that is, to develop attentiveness to details of
1014
John Stevens
form and materiality. The principle can be extended from object to experience; if through
drawing we learn to “see things more clearly” (Edwards, 1997), it follows that in enacting
and studying a short temporal experience and focussing on the minutiae of the actions
involved, students would learn about the specific chosen experience but more importantly,
would develop technique for observation, relevant to all three specialisms. While such
detailed scrutiny is important in design knowledge, a less formal, less deliberate form of
noticing is also important in creative practice for making serendipitous connections. The term
encounter (Adamson et al, 2011) is helpful to convey such a subtler kind of openness to
connections and insights that may yield creative progress:
“Knowledge is always fragmentary, partial and provisional, and only comes into its own
through the unexpected challenges, confirmations, elaborations and unsettlings that result
from encounter.” (Adamson et al 2011)
Similar claims are found in theories of sensemaking – the process of giving meaning to
experience – (see eg Weick 1995), of which the activity of design is an example (Stevens
2013). Sensemaking is triggered when a person becomes aware of a disruption, a deviation
from the expected (Weick, 1995, p. 5). This enables one to “comprehend, understand,
explain, attribute, extrapolate, and predict” (Starbuck & Milliken, 1988, p. 51); to understand
connections among, for example, people, places, and events (Klein et al., 2006); or to explain
surprises or discrepancies.
The exercise was planned therefore to promote shared ways of seeing and acting – to
encompass in a very short time these complementary practices of observing and noticing,
analysis and creative synthesis, and (by fostering their collective and individual agency)
action-based learning (van Lier 2007). In so doing, the students collectively produced an
abstracted narrative film of striking quality.
3. Exercise Instructions and Sample Outcome

• Class divided into 8 teams of 3-4, at least one from each discipline.
• “Imagine someone carrying out a simple task over a few minutes in the home, at
work, or a public place, (eg boil an egg, fasten a tie, hang a picture on a wall). It
should be something you can do yourself, and include physical objects.”
• “Make written notes in as much detail as possible: What steps are required? Any
emotions like frustration, delight? Parts that are easy, difficult, unthinking, require
other objects, special skill or knowledge etc..?”
• Each team member then enacts the task, recorded in simple video using phones.
• Storyboard sketches of key steps: draw up a storyboard using stills from the videos.
• Edit the video footage into an abstract 16 second movie. “It does not have to be
instructive. Be as expressive and stylised as you like.”
1015
• Divide the 16” video equally across the team

• Working solo, trace the video frame by frame (rotoscoping) on tracing paper or
digitally, and embellish with any visual style (fig 1) then animate the frames.
• Each team re-assemble their 3 or 4 segments back into one 16” movie (fig 2).
The finale of the course was to show all eight animations to the class as a single film, titled
Story/No Story (Stevens 2019) to which I added text titles and an upbeat soundtrack. In the
end, the students, myself and the hosting faculty were very pleased and impressed with the
results, both in terms of learning outcome and in the resulting output of the students.
Figure 1. Frame-by-frame tracing of live action (rotoscoping) using paper over a tablet (top) and directly on
screen (bottom). Photo credit: author.
1016
John Stevens
Figure 2. Sample frames from the final compiled animation show the variety of styles and media. Credit: student
participants, see Acknowledgements.
4. Student Feedback
The following are extracts from reflections of six student participants, now graduates, who
kindly responded in English text to my emailed request for feedback.
4.1. What do you remember most from the class?

“What impressed me most was the group animation cooperation, in which everyone built
different visual styles through the way of real shooting and turning to tracing. When these
styles were played together, they brought me a novel experience …The whole class was full
of lively atmosphere and innovative and open teaching contents.” Yuxin (Animation).
“The most impressive thing for me was that the teacher encouraged group members to create
an animation together. At first, we doubted ourselves about this, worried that our styles would
have conflicts if combined… Finally, to our surprise, we did fantastic work, brightening our
eyes. … After I graduated, looking back, this class was the most impressive to me because it
was a brand-new attempt for me to learn how to make animation like that. I hope more classes
like this can be added to students' schedules in the future.” Yuelin (information & art
design).
1017
“It was the first time for me to make animation informally, so the part where the team did
different styles of animation impressed me the most. The teacher asked us to pick a small
routine in the daily behaviour and record it as detailed as possible. As first it may seem
tedious and superfluous, but soon we realized that it is a delicate observation method to every
detail, which can help us to create animation more vividly.” Zhonghe (animation).
“In the last class, when everyone's animations were connected and played together, it was
very touching. Another point is that I have always been very timid to brainstorm and express
ideas quickly. But in the first class, I could imagine an interaction product… with students
from different majors and exhibited our ideas in class, which impressed me a lot.… Thank
you, you brought us a wonderful course that I still remember many details even [after] four
years passed.” Hanjie (information & art design).
“I attended this course a long time ago, but the experience is still memorable because it was
the first time I had worked in a group for a project. Plus, the diversified academic background
among the members and experiencing the complete design process was truly inspirational.
At the time… sketching, sculpture, and Chinese painting were my daily routine. However, it
was the first time I took the hybrid class that consisted of three different classes. It was
unfamiliar but exciting. I got to meet new classmates and established friendships with them
over time. This was a very exciting process.” Lingjun (Photography).
“I remember clearly… our group made a short animation video and a poster about beauty
and makeup, trying to break and re-think about this daily routine though this whole creative
process. The process of research and design inspired me a lot in my later practices. I found
that the design methodologies I learned from the class are actually widely used by people
doing interaction or industrial designs. I majored in animation, so that these methodologies
do not usually work straightforwardly in my own creative practice. However, the way the
class taught me to observe life, find pain points and dig into it — the way of thinking really
helped a lot in my own studies.” Rui (animation).
4.2. How did you feel being in a mixed class of photography, animation and digital
designers?
“In a hybrid classroom, the integration of different majors can provide suggestions from
different perspectives for the implementation of project cooperation, and these suggestions
can create new sparks.” Yuxin.
“I think it's cool! Because we are from different majors so that we have different perspectives
and different methods of solving a problem. We learned from each other and leveraged our
respective strengths to complete our final work. For example, I learned how to make a frame-
by-frame animation under the guidance of my team member who majored in animation.”
Yuelin.
1018
John Stevens
“It was an amazing experience to have classes with students from different majors. Different
people grasp different knowledge and have different understanding on one topic. Therefore,
when facing the same design, they choose different angles. They were all very enlightening.”
Zhonghe.
“Being in a mixed class is, at first, painful, but I definitely learned a lot from this diversity.
At the time … most of us did not have a very clear idea of what our majors were really about
and what we were learning… I think that being able to have a mixed class in freshman year
really expanded my vision not just in an interdisciplinary perspective, but gave some more
depth to my understanding of my own major — animation, which became crucial to my later
trans-disciplinary studies and career path.” Rui.
“When cooperating with other members, I have sensed the charm within a cross-major
environment. We come from different fields of studies, but gathered as a whole, displaying
our own unique strengths. We help and encourage each other during the process, as well as
reminding each other of the designing procedures and details. As time progresses, the
fascinating aspects of cooperating with people that come from different academic
backgrounds have taught me the basics, and the importance of personnel speciality during
the designing process.” Lingjun.
5. Conclusion
The structure of the unit was highly successful in acheiving a range of learning outcomes,
and is easily repeatable via simple instruction, with hardware and software commonly already
used by many students. The same structure could, I suggest, also be valuable for younger
learners or non-design students, as the rotoscoping technique is highly accessible to less
expert or able students, and produces vary satisfying results.
The teams achived a high quality output, demostrating the complementary practices of
observation, analysis and creative synthesis, and an intense combination of group and solo
work, and of intra- and interdisciplinarity. From these written reflections, and in the class
atmosphere at the end of the module, it is clear the intensity and intimacy of the challenge
had a highly emotive resonance. Participants found collaboration daunting at first, then
enjoyed the lively and energetic atmosphere in the class, and ultimately made valuable
learnings and lasting friendships, and produced an artwork they were immensely proud of.
Acknowledgements
I’m grateful to Dr Hugo Glover, Senior Lecturer, Motion Graphics and Animation,
Northumbria University for his invaluable guidance on animation pedagogy, to the host staff
at Tsinghua University, Ping Li, Ruobing Cai, and Prof Lie Zhang, and to the students for
1019
their hard work and for making me so welcome: Zhang Zhonghe, Zhu Yanru, Zhang
Yueying, Du Yilin, Wu Yudan, Yang Linyue, Wang Rui, Lin Jiahua, Guo Xinyi, Liu Chang,
Wang Yiliu, Yu Hanjie, He Shixian, Chen Long, Liu Lingjun, Han Xiaoxuan, Xiao Xiaodi,
Hu Yifei, Liu Siyi, and Tan Xinran.
References
Adamson, G., Riello, G., & Teasley, S. (Eds.). (2011). Global design history. Taylor &
Francis.
Bada, S. O., & Olusegun, S. (2015). Constructivism learning theory: A paradigm for teaching
and learning. Journal of Research & Method in Education, 5(6), 66-70.
Dobson, N., Roe, A. H., Ratelle, A., & Ruddell, C. (Eds.). (2018). The animation studies
reader. Bloomsbury Academic.
Edwards, B. (1997). Drawing on the Right Side of the Brain CHI ‘97 Extended Abstracts on
Human Factors in Computing Systems. Proceedings from CHI EA ‘97, New York, NY,
USA Atlanta, Georgia.
Glover, H. (2020). Animatory thinking: An enquiry into tacit knowledge within animation
practice, Doctoral Thesis, Royal College of Art (United Kingdom)
Klein, G., Moon, B., & Hoffman, R.R. (2006). Making sense of sensemaking: 1. Alternative
perspectives. Intelligent Systems, IEEE 21(4), 770–773.
Starbuck, W.H., & Milliken, F.J. (1988). Executives’ perceptual filters: what they notice and
how they make sense. In The Executive Effect: Concepts and Methods for Studying Top
Managers (Hambrick, D.C., Ed.), pp. 35– 65. Greenwich, CT: JAI Press.
Stevens, J (2013). Design as communication in microstrategy: Strategic sensemaking and
sensegiving mediated through designed artifacts. Artificial Intelligence for Engineering
Design, Analysis and Manufacturing, 27, pp 133-142
Stevens, J (2019) Story / No Story Animated short, retrieved March 2023 from
https://researchonline.rca.ac.uk/5270/
Weick, K.E. (1995). Sensemaking in Organizations (Foundations for Organizational
Science). London: Sage.
Willcocks, J. (2015). The power of concrete experience: museum collections, touch and
meaning-making in art and design pedagogy. Engaging the senses: object-based learning
in higher education. Farnham: Routledge, 43-56.
Wilson, S.E. and Zamberlan, L. (2017), Design Pedagogy for an Unknown Future: A View
from the Expanding Field of Design Scholarship and Professional Practice. Int J Art Des
Educ, 36: 106-117. https://doi.org/10.1111/jade.12076
1020
Collaborative knowledge construction during computational lab

activities in Financial Mathematics
Alice Barana1, Marina Marchisio1, Adamaria Perrotta2, Matteo Sacchet1
1
Department of Molecular Biotechnology and Health Sciences, University of Turin, Italy,
2
School of Mathematics and Statistics, University College Dublin, Ireland.
Abstract
Nowadays, there is a growing interest in including some aspects of
computation and computational thinking within science subjects, in particular
Mathematics. Moreover, the COVID pandemic has increased the opportunities
for remote-based work across economies, including education. This implies
that workers and students should be able to collaborate remotely using
collaborative technologies. In this paper, we show how tailored student-led
computational practices designed for a Computational Finance module
provide opportunities for the co-creation of knowledge in Financial
Mathematics in a Computer-Supported Collaborative Learning environment.
We analyzed students’ answers to a weekly survey using Gunawardena et al.
(1997) Interaction Analysis Model for collaborative knowledge construction.
The results show that, in a large number of discussions, the highest levels of
the collaborative learning process were achieved.
Keywords: Collaborative knowledge construction; computational thinking;
computer supported collaborative learning; inclusive computational
practices; interaction analysis model; financial mathematics.

Collaborative knowledge construction during computational lab activities in Financial Mathematics
1. Introduction
Nowadays, there has been growing interest in including some aspects of computation and
computational thinking within science subjects, in particular in Maths pathways. The use of
computation can deepen the learning and experience of Mathematics; at the same time,
Mathematics also provides a meaningful context which computational thinking can be
applied to (Lockwood et. al., 2019). In this setting, Computational Finance is a new and
highly interdisciplinary Math subject, relevant to master Financial Mathematics and
computational thinking and fundamental for high-level quantitative finance jobs. Despite the
incredible growth of Financial Mathematics programs in the last few years, it is worthy to
notice that the Computational Finance curriculum is under-researched. Moreover, the
COVID pandemic has caused a rapid increase in the demand from employers for remote-
based work across economies (World Economic Forum, 2020). The same happened in the
education sector. This implies that workers and students should be able to collaborate
remotely, using digital tools for communication and shared workspace.
In this paper, we show how tailored student-led computational practices designed for a
Computational Finance module delivered at University College Dublin provide opportunities
for the co-creation of knowledge in Financial Mathematics in a Computer-Supported
Collaborative Learning environment (CSCL). We analyzed the students’ responses to weekly
surveys to address the following research question: do the activities designed for the
Computational Finance module allow students to create shared knowledge in financial
Mathematics?
CSCL is a field of research, emerging in the ‘90s, focused on how technology can facilitate
the sharing and creation of knowledge and expertise through peer interaction and group
learning processes (Resta & Laferrière, 2007). CSCL grounds on socio-constructivist
theories, where learning is seen as a construction process performed by communities of
practice, social units participating in a common situation with a shared goal. Communities of
practice are built through the coordinated use of collaborative technologies, tools that enable
individuals to jointly participate in the construction of a shared knowledge (Roschelle &
Teasley, 1995). The design of CSCL environments is crucial to promote the process of
collaborative knowledge construction. Donnely et al. (2014) identified four features that
should be considered when creating CSCL activities: (1) tasks should be meaningful to
engage students in exploration and relevant problem solving activities; (2) simulations and
dynamic visualizations can create emerging conditions for collaborative learning; (3)
students need to be encouraged to collaborate in their work; (4) it is important to cultivate
the use of metacognitive strategies over time, to enhance students’ capacity to self and co-
1022
Alice Barana, Marina Marchisio, Adamaria Perrotta, Matteo Sacchet
regulate themselves in socio-cultural environments. An example of similar CSCL for

Mathematics can be found in (Barana & Marchisio, 2016).
The “inclusive computational practices” definition (Caballero & Hjorth-Jensen, 2018) is
perfectly aligned with the above description. In fact, this definition includes a wide range of
high-level coding activities like having students work in groups with simulations to
understand the main characteristics of a mathematical model; giving students pieces of code
to complete or modify in order to adapt them to a different problem; critically inspecting and
judging computational inputs and outputs; advising students on open-ended group projects
where they write code from scratch. Inclusive computational practices have been heavily
used to design physics undergraduates’ modules in Michigan State University and Georgia
Tech (Caballero et al., 2012; Caballero & Hjorth-Jensen, 2018). Here computation and
computational thinking are a central element and not a tool in the design process.
The process and outcomes of collaboration can be analyzed using different frameworks. This
study adopts the Interaction Analysis Model (IAM) for collaborative knowledge construction
designed by Gunawardena et al. (1997). According to Lucas et al. (2014), interaction is “the
process through which negotiation of meaning and co-creation of knowledge occurs and
should be viewed as the totality of interconnected and mutually responsive messages, an
entire gestalt formed by the online communications among participants”. Thus, the co-
creation of knowledge is conveyed by interaction. Based on this definition, Gunawardena et
al. (1997) developed a IAM oriented at capturing the knowledge construction process when
a community of practice learns using collaborative technologies. It is composed of 5 phases:
(1) sharing and comparing of information; (2) discovering and exploring dissonance or
inconsistency among ideas or statements; (3) negotiation of meaning/co-construction of
knowledge; (4) testing and modification of proposed synthesis or co-construction; (5)
agreement statement(s)/applications of newly constructed meaning.
This model was originally created to analyze the interactions during an online debate among
researchers and education professionals, i.e. with peer experts within an asynchronous setting
where the technology is a communication tool. Gunawardena et al. (1997) themselves argued
(and encouraged) that the same method could be applied also with non-experts (such as
students) and in different CSCL settings. Recently, the IAM has been used to analyze face-
to-face interactions of students working in synchronous settings with peer non-experts and
using technology as a cognitive tool (see for example Zabolotna et al., 2023). In this paper,
we show how tailored inclusive computational practices may provide opportunities for the
co-creation of knowledge in Computational Finance in a CSCL environment.
1023
3. Research methodology: setting, data collection and analysis

This research study has been conducted in the AY 2020/2021 in the Computational Finance
module ACM30070, core for the stage 3 of the BSc in Financial Mathematics (FM), School
of Mathematics and Statistics, UCD. The module is also optional for stage 3 of the BSc in
Applied and Computational Mathematics (ACM). In 2020/2021, 50 students attended the
module, with 35 FM and 15 ACM. This module was offered as a pilot in 2016/2017. Then,
it was redesigned and improved to include labs including computational practices and group
activities in a CSCL environment where students actively participate in the co-creation of
their knowledge. Those practices are structured to reciprocally use computational thinking to
enrich the mastery of FM and the theory of FM to enrich students’ computational thinking.
An extensive description of the module design and lab activities can be found in (Perrotta,
2021) and (Perrotta & Dolphin, 2021). To facilitate students’ learning and ensure engagement
and participation, a tutor and a teaching assistant moderated the labs, actively intervening
only if needed or required by students. Each lab lasted 2h. During the first hour, students
worked in groups on computational modeling, pseudo-coding, data analysis and other related
computer-based activities. In the second hour, each group chose a representative to present
the group outcomes to the whole class. The tutor guided groups in presenting their results
and encouraged dialogue between groups in order to come to a conclusion. The class was
divided into 7 groups (6 of 7 students and 1 of 8), which stayed the same for the whole term.
Students were grouped according to: their GPA, (homogeneous, i.e. similar GPA on AVG),
their pathway (5 FM and 2 ACM), and gender balance (at least 2 women per group). To
positively set students’ expectation in view of co-creation of knowledge, students were made
aware of their similar GPA, so that they were all expected to be able to contribute in the same
way within their group. After each lab, students were invited to fill out a Google Form survey
to critically reflect on the activities done in class. Each student completed 10 surveys (1 per
week). The survey contains qualitative and quantitative questions, their answers constitute
the dataset for this research study. In this paper, we analyze the (open-ended) answers to the
following questions:
1. Were the expected outcomes and goals of today's lab clear to you? If yes, please list
which are, in your opinion, the key concepts of today's class. If not, please state
what was unclear to you.
2. Can you describe in detail in your own words how peer feedback and discussions
have or have not helped you in supporting your learning today? Please refer
explicitly to the exercises we have covered and make examples.
3. How useful do you think the computational part was for improving your
understanding today? Please refer explicitly to the exercises we have covered and
make examples.
1024
The Survey was filled out individually and it stayed open for one week, so this activity can
be classified as an individual, asynchronous, computer-mediated one. We highlight that the
dataset refers to the Jan-May 2020 period. Before March 2020, lab activities took place f2f,
in Active Learning rooms. When restrictions were put in place, labs were live-streamed on
Zoom, with groups working in breakout rooms with shared screens in the first hour.
For the analysis of this pilot study, we selected 6 students out of 50, who worked in 6 different
groups. The selection was made in order to represent three different proficiency levels: two
high-level students, two medium achievers and two low-level students. There are 5 males
and 1 female in the sub-sample. We analyzed the 6 students’ answers to the 10 weekly
surveys for the question listed in section 3 using the IAM framework. For each student and
each survey, one phase of knowledge co-construction was selected, namely the highest phase
that could be inferred from all the answers to the questions of the survey. The goal of our
analysis was identifying which was the highest phase reached by the groups during one lab,
in order to infer the level of knowledge co-construction achieved by that group during the
lab. Since the complete discussions were not available and knowledge co-construction is a
progressive process, when one phase was identified from the survey’s answers, we deduced
that the lower phases were covered during the discussion. So, when more than one phase was
detected, we chose the highest one. In case of doubts between two or more phases, we chose
the lowest one. The analyses were performed independently by the authors, so that all
answers were analyzed by two researchers, and then discussed by all the authors altogether.
Then, the phases’ frequencies in the 10 labs were computed.

Table 1 shows the frequencies of the highest phases occurred during the discussions that we
identified from the answers of the three questions listed in Section 3.
Table 1. Phases of knowledge construction reached during the lab discussions.
Phase Frequency Percentage
1 20 35.1%
2 11 19.3%
3 13 22.8%
4 2 3.5%
5 12 19.3%
From these results, we can notice that all the phases of knowledge co-construction described
in the IAM occurred during the discussions. Phase 1 is the most frequent, which is in line
1025
with similar studies. We assigned to phase 1 those answers which simply show that students
compared their solutions or ideas without moving to a deeper discussion. One example of
phase 1 is the following, where it seems that during the lab time the group discussion was
limited to sharing different points of views.
“As mentioned above, my classmates helped me get a better grasp of the delta of an option.
It was helpful to hear their points of view and their thoughts [...] “
Phases 2, 3 and 5 have a similar frequency. We assigned to phase 2 those answers which
allowed us to detect disagreement or discussion about different emerging ideas. One example
of phase 2 is the following, where it seems that during lab time there was discussion about
different ideas and approaches, but students did not reach a common solution:
“Peer feedback was good this week, especially with question 8, where multiple different
people brought up differences between the two techniques, so we all learned of differences
we wouldn’t have thought of. However, with question 7, we were all quite confused [...] and
did not get the answer out.”
Phase 3 was identified when there was evidence of common knowledge or an agreed solution
achieved through collaborative work or discussion after a deep reflection. One example is
the following, where the transition from phase 2 (identification of disagreement) to phase 3
(agreement of an explanation) is clearly visible.
“In Q3, we discussed the different possible parameters and variables. Some said that
volatility should be a variable, others said a parameter. We agreed that in the real world, it
is of course varying constantly but in the BS model it is treated as a parameter.”
Phase 4 is the least common, maybe due to the fact that many groups that reached phase 4
then moved to phase 5, and therefore they are included in the number of answers assigned to
phase 5. The following answer provides an explicit example of phase 4, since we can observe
how the students of this group used the technologies to test and find a confirmation of the
knowledge shared during the previous discussion:
“We were able to discuss more things about the model. We actually plotted a comparison of
the binomial price and the Black-Scholes price for different values of n and saw that as n got
larger the binomial price (which is only an approximation) came much closer. We also
plotted the error and observed that the binomial price eventually seemed to converge to the
binomial price.”
Phase 5 was detected when there was evidence of modification of individual understanding
as a consequence of the interaction, such as in the following answer:
“[…] the computational aspect that we focused on was the implementation of Fincad. [...].
The most educational part for me was to notice he vast difference in options prices when the
1026
time to maturity and the spot price is 10 euro above the strike price. The call price was
roughly 1 euro and the put price was so small it is almost negligible and can be rounded to
0. This prompted much speculation and debate in our group as to the reason why (We believe
it's because the price of the share at maturity was unlikely to fall below the strike price,
resulting in most likely a loss). We also used Fincad to evaluate the put-call parity with a
time to maturity of two years now. By calculating the respective put and call prices using
Fincad, we were able to see this was roughly true to approximately 0.362 of a difference
between the two sides of the equation. This computational aspect allowed me […] to solidify
the truth of the put call parity.”
This answer perfectly shows all the collaborative knowledge construction process occurred,
from phase 1 to phase 5. We can see how the group passed through phases 1 and 2 (comparing
and contrasting different ideas on why they found a difference between call and put prices),
then moved to phase 3 (agreeing on a hypothesis on why the phenomenon occurred), then 4
(testing the shared solution with Fincad) and lastly to phase 5 (applying the new knowledge
that the discussion helped them gain, reaching a more solid comprehension of the theory).
5. Conclusions and future improvements

We performed a pilot study to investigate the effectiveness of tailored inclusive
computational practices in providing opportunities for the co-creation of knowledge in
Computational Finance in a CSCL environment. We selected 6 students out of 50 and we
analyzed the answers to the weekly survey questions listed in section 3 through the IAM
framework. The results are shown in Table 1 and are in line with the available literature using
the same framework in different CSCL environments. From these results, we can notice that
all the phases of knowledge co-construction described in the IAM occurred during the
discussions. Despite phase 1 being the most frequent (35.1%), which is in line with previous
studies (Lucas et al., 2014), relevant number of discussions which developed beyond phase
3: 3.5% stopped at phase 4, the 19.3% reached phase 5. This result improves the available
literature and lets us conclude that the proposed activities have been successful in activating
the highest level of co-construction of knowledge. The technology and computation used in
this module had a key role, since they mediated and fostered the whole learning process. This
result supports Donnely et al.’s (2012) thesis: the collaboration within specific problem-
solving and computational thinking activities creates suitable conditions for co-learning
construction in CSCL environments. We are now planning to extend the analysis to the entire
dataset to further investigate how computation intended as combined with a collaborative
learning environment fostered the whole learning process. Another possible development of
our research is focusing on the impact of group composition in the co-construction of
knowledge: indeed, we noticed groups constituted by students with lower GPAs collaborated
more than students with higher GPAs, and this was reflected in the attainment of higher
1027
phases. This study has a limitation: we have analyzed students' answers instead of recorded
conversations, so students may have reached higher phases but without being able to express
this in words. For this reason, we are planning to collect a new dataset based on video-
recordings of activities and analyze discussions transcripts with the IAM framework.
References
Barana, A., & Marchisio, M. (2016). From digital mate training experience to alternating
school work activities. Mondo Digitale, 15(64), 63–82.
Caballero, M.D., Kohlmyer, M.A. & Schat, M.F., (2012), Implementing and assessing
computational modeling in introductory mechanics, Physics Education Research, 8,
020106.
Caballero, M.D., & Hjorth-Jensen, M., (2018). Integrating a Computational Perspective in
Physics Course, New Trends in Physics Education Research, 47-76, Nova Science.
Donnelly, D. F., Linn, M. C., & Ludvigsen, S. (2014). Impacts and Characteristics of
Computer-Based Science Inquiry Learning Environments for Precollege Students.
Review of Educational Research, 84(4), 572–608.
Gunawardena, C. N., Lowe, C. A., & Anderson, T. (1997). Analysis of a Global Online
Debate and the Development of an Interaction Analysis Model for Examining Social
Construction of Knowledge in Computer Conferencing. Journal of Educational
Computing Research, 17(4), 397–431.
Lockwood, E., DeJarnette, A.F., & Thomas, M., (2019). Computing as mathematical
disciplinary practice. Journal of Mathematical Behaviour, 54, 100688.
Lucas, M., Gunawardena, C., & Moreira, A. (2014). Assessing social construction of
knowledge online: A critique of the interaction analysis model. Computers in Human
Behavior, 30, 574–582. https://doi.org/10.1016/j.chb.2013.07.050
Perrotta, A. (2021). A learner-centered approach to design a Computational Finance module
in higher education, HEAd'21 - 7th International Conference on Higher Education
Advances, Editorial Universitat Politècnica de València, 405-412.
Perrotta, A. & Dolphin, R. (2021). Combining Student-Led Lab Activities with
Computational Practices to Promote Sensemaking in Financial Mathematics”,
Proceedings of the Eighth Conference on Research in Mathematics Education in Ireland
(MEI8), In M. Kingston, & P. Grimes (Eds.), 348–355.
Resta, P., & Laferrière, T. (2007). Technology in Support of Collaborative Learning.
Educational Psychology Review, 19(1), 65–83.
Roschelle, J., & Teasley, S. D. (1995). The Construction of Shared Knowledge in
Collaborative Problem Solving. In C. O’Malley (Ed.), Computer Supported Collaborative
Learning (pp. 69–97). Springer Berlin Heidelberg.
World Economic Forum. (2020). The Future of Jobs Report 2020.
Zabolotna, K., Malmberg, J., & Järvenoja, H. (2023). Examining the interplay of knowledge
construction and group-level regulation in a computer-supported collaborative learning
physics task. Computers in Human Behavior, 138, 107494.
1028
Learning agroecology through the serious game SEGAE in an

online lesson: unveiling its impact on knowledge articulation
Mireille De Graeuwe, Lavena Van Cranenbroeck, Rémy Parent, Clément Serdobbel,
Yves Brostaux, Kevin Maréchal
Gembloux Agro-Bio Tech, University of Liège, Belgium.
Abstract
Facing global challenges, the currently dominating agricultural system has
shown its limitation. Therefore, agroecology appears as an attractive
alternative. Its implementation has been shown to benefit the environment,
while harboring economic potential. For a transition towards more
agroecology, this new paradigm needs to be taught to students and
professionals in the agricultural sector. However, traditional learning
methods are not fully adapted to teach these concepts because of the poor
interactivity and/or lack of interdisciplinarity.
To help teach agroecology, the “SErious Game for AgroEcology” (SEGAE)
was developed. SEGAE is a simulated mixed crop–livestock farm complex
models in which players can learn by doing.
To access SEGAE pedagogical interest, university students, coming from four
specializations (Agronomy; Chemistry; Environment and Forest) participated
to a theoretical class which was tighted to a play session of the game. The
students then answered a knowledge survey on agroecology and a feedback
survey based on flow theory. Results showed that students did significantly
increase their knowledge of agroecology. Students particularly succeeded
when answering open-ended questions, which required them to articulate
knowledge. Morevoer, a large majority of students enjoyed the game (83%)
and thought playing this game increased their knowledge (91%).
Keywords: Serious game; agroecology; interdisciplinarity; farming system.

Learning agroecology through the serious game SEGAE in an online lesson
1. Introduction
Acknowledging the documented impacts of agriculture (Campbell et al., 2017), the transition
towards more sustainable agro-food systems is urgently needed (IPES-FOOD, 2018). One
tricky equation in this will be to strike a good balance between warranting sufficient
production of food for a growing population and remaining within planetary boundaries
(Gerten et al., 2020). Associated with the concept of food sovereignty, agroecology is
increasingly advocated as a promising alternative (Wezel et al., 2014) . More specifically,
agroecology-led initiatives are designed to meet the triple challenge of proposing a food
system that is at the same time more sustainable, efficient, and socially fair (Gliessman, 2014;
Godfray et al., 2010).
Agroecology is commonly defined as “the ecology of sustainable food systems” (Gliessman,
2014). Agroecology is defined in this article as “the study of the interactions between plants,
animals, humans, and the environment within agricultural systems” (Dalgaard et al., 2003, p.
42).
However, even if the advantages of agroecology have already been demonstrated in
numerous studies (van der Ploeg et al., 2019; Wezel et al., 2014), it remains a topic which
raises many questions in terms of learning, in academic institutions. When it comes to food-
related and agronomic purposes, students are indeed still mostly trained to be specialized in
particular fields, such as biotechnology, economics, and soil sciences. However, agroecology
is a notion that brings together a complex of components in interaction, such as
environmental, social, and economic sciences (Francis et al., 2011). Agroecology requires
being taught in a different way, in order to give a more systemic view of its various
components (Francis et al., 2008). As judiciously pointed to in David & Bell (2018, p. 615),
the teaching of agroecology “is not only about content, it is also about process”.
Serious games are a way to satisfy the teaching requirements of agroecology because digital
tools allow the simulation of complex models which allow players to develop their
understanding of systems approaches (Wu & Lee, 2015). They are referred to as being
entertaining tools with an education purpose. They improve the implication, the engagement
and the immersion of the players, while having a fun aspect. New skills are acquired by
participants who receive immediate feedback based on the choices and strategies adopted.
Accordingly, the serious game SEGAE (SErious Game for AgroEcology learning) has been
developed to facilitate the teaching of agroecology, accounting for the need to convey both a
systemic, as well as, an inter(trans)disciplinary perspective (De Graeuwe et al., 2020). Within
the realm of the present study, the objective is to explore the relevance of using this game in
short lessons of 4 hours, with a single teacher. More particularly, we make the assumption
that serious games, and more precisely SEGAE, are an effective learning tool, adapted to a
short lesson, for improving the agroecological knowledge of university-level students. This
1030
Mireille De Graeuwe et al.
acquisition of knowledge should be more manifest in open-ended questions, which require

having acquired a systemic vision. A related and complementary hypothesis is that SEGAE
positively contributes to knowledge acquisition through providing fun during the game
session.
To determine the performance of SEGAE with respect to the education of agroecology, two
types of surveys have been conducted. Firstly, to measure the acquisition of agroecological
skills, a knowledge survey was proposed before and after the lesson. Secondly, to assess the
student’s perception of the serious game, flow theory has been mobilised (Fu et al., 2009)
and used as a feedback survey. Both surveys are the replica of those used in De Graeuwe et
al. (2020).
2. Materials and method

The study was carried out in Belgium during March 2021. Due to the Covid-19 pandemic,
all activities took place online. Students from the 3rd bachelor year, following an agricultural
engineering education in four specializations (Agronomy; Chemistry; Environment and
Forest) followed this lesson. In a period of 4 hours, several activities were organized (1) a
theoretical class, (2) a serious game class and (3) two survey’s times.
2.1. Theoretical class

Once the pre-survey was completed, a 2-hour lesson was given to introduce the theoretical
notions of agroecology. This introduction contains three distinct modules which are "Soil-
Plant-Ecology", "Animal" and "Socio-economic". The goal of this introduction was to give
students basic knowledge of agroecology and its effects, introduce a system approach to
evaluate a combination of practices on the production system and also to study different
options to solve a given problem with limited resources.
2.2. Serious game class

After the theoretical class, another 2-hour lesson was given to the students. This time is
dedicated to the handling of the serious game. The whole group was divided into several
subgroups from 20 to 30 students, randomly mixing the different specializations. During this
class, three scenarios were proposed to them, starting with a "sandbox", followed by a
"system approach" and ending with an "indicator oriented” (for more information, see De
Graeuwe et al. (2020, p. 6)). After that, a discussion about the results and limits of the serious
game with the teacher followed.
1031
2.3. Survey’s times

Before starting classes, it was asked to student to fill a pre-survey which was divided in two
sections. The first one was about the respondent's profile questions such as age, nationality
and specialization of study were asked. The second one contained 15 multiple choice and 6
open-ended questions about agroecology knowledge.
In order to be able to measure the acquisition of competence, the same knowledge questions,
as in the pre-survey were asked in the post-survey. Moreover, a feedback section was added
to assess whether the SEGAE game was stimulating as a learning tool. It included 4 open-
ended questions and 45 statements. The eight factor of EgameFlow were used to characterize
the state of flow that student experienced : (1) concentration, (2) clear goal, (3) feedback, (4)
challenge, (5) autonomy, (6) immersion, (7) social interaction and (8) knowledge acquisition
(for more details, see De Graeuwe et al. (2020, p. 7)). Each flow factor was assessed with 2-
5 statements. Students ranked each statement on a 4 Likert scale.
2.4. Analysis of survey results

All the questionnaires were anonymised. The study included 68 students for the pre-survey
and 72 for the post-survey. After data cleaning, 50 students were retained for the analysis of
knowledge questions (17 from Agronomy; 6 from Chemistry; 10 from Environment; and 17
from Forest). Aberrant data was removed from the database. Students who did not answer all
the open-ended questions in both surveys (even if it was required) were discarded for the
knowledge section. For the flow assessment, the 72 students of the post-survey were all
retained. All statistical data analyses were performed using R software v.4.2.2 (R Core Team,
2022) and a 𝞪 = 0,05 for significance level.
Each student's pre- and post-scores were calculated and registered. For the multiple-choice
questions, students received a score of 1 when they had a correct answer, 0 when they did
not answer, -1 if the answer was not correct. For open-ended questions, students are given a
score between 0 and 3 depending on the number of correct answers given, compared to those
expected. They are given a score of -1 for each incorrect answer. The result of each question
is added up to calculate the total score. The maximum score that could be reached is 39 and
it was converted into percentages. Final scores were used to calculate descriptive statistics
(e.g average and median). Then, a paired t-test was performed with the null hypothesis of
absence of increase of the scores between the before/after the 4-hour lesson. Subsequently,
the scores of each specialization were calculated and the same statistical analysis as
previously described was performed.
The score for the feedback questions is calculated by converting all answers to each of the
45 statements with a scale from 1 to 4 (1=strongly disagree, 2=disagree, 3=agree, 4=strongly
agree). Each flow factor score was calculated by adding up the results of all statements
1032
corresponding and divided by the number of statements included in the flow factor itself.
Final flow scores were used to calculate descriptive statistics. Mean scores were calculated
for the group of all students and by specialization.
3. Results
3.1. Results of knowledge acquisition

On the knowledge pre-survey, the student’s general mean was 14.7 % of correct answers
(Table 1). The scores differ significantly among the type of questions: Open-ended questions
reached the highest scores (21.3%), while close-ended questions recorded the lowest (9%).
The scores did not differ significantly among the specializations. On the knowledge post-
survey, the students’ general mean was significantly (p<0.05) higher: 18.5%. Their
progression was very highly significant for open-ended questions (p<0.001).
Table 1. Mean of students’ scores (% of correct answers) on the knowledge survey during the
pre-survey and post-survey, and the increase after the 4-hour lesson (percentage points).
Grouping of Data (and number of students) Pre-Survey Post-Survey Increase

All students 14.7 18.5 3.8*
By specialization
Chemistry (6) 14.5 18.4 3.9
Forest (17) 13.4 19.1 5.7
Agronomy (17) 16.1 17.3 1.2
Environment (10) 14.6 19.2 4.6
By type of questions
Close-ended questions 9 9.5 0.5
Open-ended questions 21.3 28.8 7.5***
Legend: *** p < 0.001; ** p < 0.01; * p < 0.05; no symbol p > 0.05.
Overall, 31 of 50 students increased their scores, 5 had the same scores, and 14 decreased
their scores slightly (Figure 1).
1033
Figure 1. Comparison of students’ mean scores on the knowledge pre- and post-surveys, by their specialization:
Agronomy ■, Chemistry ●, Environmental + and Forest ▲.
3.2. Perception of SEGAE and its evaluation through flow

In the feedback post-survey, 83% of the students “agreed” or “strongly agreed” with the
question “I enjoyed the game without feeling bored or anxious”. Therefore, the SEGAE
game is globally appreciated by students. Moreover, 91% of students “agreed” or “strongly
agreed” with “The game increased my knowledge”. The eight factors have an average of 2.93
points (i.e., ≈ “agree”) out of 4 (Figure 2). The factors “social interaction” and “feedback”
are rated particularly low compared to the others. For each specialization, the flow average
was carried out and compared to the general mean. Only chemistry students have a slight
difference, with an average of around 3.25 points out of 4 (Figure 2).
Figure 2. Radar chart on chemist mean scores for the eight flow factors compared to global mean score.
1034
4. Discussion and conlusion

Students significantly increased their knowledge in agroecology using the game. This result
corresponds to 60% of the mean increase previously recorded for a 5 day workshop in a
similar curriculum (6.4 percentage points for “multidisciplinary” see De Graeuwe et al.
(2020, p. 13)). More particularly, students increased their performance with respect to
answering open-ended questions, those which required them to analyse problems in their
global context and as parts of an interconnected whole. The game thus appears to improve
the ability of students to articulate their existing and newly acquired knowledge. The game
makes it easier to link different knowledge by mobilizing it simultaneously. However, the
metacognitive involvement of each student depends of their profound enjoyment and
concentration (i.e. a good level of ‘flow’). The results show that SEGAE provides such a
context for metacognitive learning. The knowledge increase recorded in open-ended
questions is assumed to reflect the increase in the meta-knowledge of students. Accordingly,
following a 4-hour lesson with a single teacher seems promising in terms of learning.
Every specialization increased their mean. The agronomy specialization has the lowest
increase. Agronomy students already had knowledge about agroecology and breeding
systems. Therefore, it is important to note that their scores were already higher in the pre-
survey, thus reducing their possible progression in the post-survey. To the contrary, the other
student groups, only had limited knowledge on the fundamentals of agroecology. Therefore,
it could be argued that this limited knowledge opened the space for a greater progression,
than student specialized in agronomy. Overall, the means pre- and post- surveys are
extremely low; particularly for close-ended questions. Two explanations can be highlighted
to explain those results.
First, the unplanned digitalization of this lesson due to covid-19 period involved poorer
interactions between the students and the teacher. This is reflected in low reported level of
flow factors: “social interaction” and “feedback”. Moreover, surveys were done online
without any supervision, which caused problems to receive answers of these surveys in time.
For further study, the answering time to complete the survey could be analyzed to identify
aberrant data. Also, a replica could be performed with a larger number of students to validate
results found.
Secondly, low knowledge results and low levels of surveys’ submissions can be explained
by the language used for the surveys. They were in English, while most of the students being
francophones. As such, this may have created difficulties with comprehension of the
questions and thus slowing down the progression rate in the surveys. For further study, 4
steps are recommended: (1) translate surveys in French; (2) select a 5 to 10 % of final sample
size; (3) administer the questionnaire to the selected percentage and (4) assess understanding
of these students about each question (Roopa & Rani, 2012, p. 276).
1035
Concerning the feedback survey, the height factors of flow showed that students assessed the
game positively. A large majority (>80%) of them reported that they enjoyed the game and
felt that they increased their knowledge thanks to it. These results reinforce the use of serious
games and particularly SEGAE to learn agroecology. Indeed, this game allows students to
learn through an active posture which helps to overcome the knowledge-action gap.
Especially as this knowledge-action gap has been demonstrated as a critical competence in
learning agroecology (Østergaard et al., 2010) .
To conclude, this study confirms that the format of the game SEGAE in a 4-hour lesson with
a single teacher is a relevant tool in learning agroecology. Students appreciate learning using
this game. Therefore, the development of serious games is an alternative for the future of
agroecology learning. As the activities took place online rather than face-to-face (as initially
planned), the acquisition of agroecology knowledge might have been impacted. The same
study carried out in a non-covid period would allow to determine the influence of online
lesson.
References
Campbell, B. M., Beare, D. J., Bennett, E. M., ,… Shindell, D. (2017). Agriculture production
as a major driver of the Earth system exceeding planetary boundaries. Ecology and
Society, 22(4), art8. doi:10.5751/ES-09595-220408
Dalgaard, T., Hutchings, N. J., & Porter, J. R. (2003). Agroecology, scaling and
interdisciplinarity. Agriculture, Ecosystems & Environment, 100(1), 39‑51.
doi:10.1016/S0167-8809(03)00152-X
David, C., & Bell, M. M. (2018). New challenges for education in agroecology. Agroecology
and Sustainable Food Systems, 9.
De Graeuwe, M., Jouan, J.,., … Godinot, O. (2020). Learning Interdisciplinarity and Systems
Approaches in Agroecology : Experience with the Serious Game SEGAE. Sustainability,
12(11), 4351. doi:10.3390/su12114351
Francis, C. A., Jordan, N., Porter, P., ,…Langer, V. (2011). Innovative Education in
Agroecology : Experiential Learning for a Sustainable Agriculture. Critical Reviews in
Plant Sciences, 30(1‑2), 226‑237. doi:10.1080/07352689.2011.554497
Francis, C. A., Lieblein, G., Breland, T. A., Salomonsson, L., Geber, U., Sriskandarajah, N.,
& Langer, V. (2008). Transdisciplinary Research for a Sustainable Agriculture and Food
Sector. Agronomy Journal, 100(3), 771‑776. doi:10.2134/agronj2007.0073
Fu, F.-L., Su, R.-C., & Yu, S.-C. (2009). EGameFlow : A scale to measure learners’
enjoyment of e-learning games. Computers & Education, 52(1), 101‑112.
doi:10.1016/j.compedu.2008.07.004
Gerten, D., Heck, V., Jägermeyr, J., Bodirsky, B. L., Fetzer, I., Jalava, M., Kummu, M.,
Lucht, W., Rockström, J., Schaphoff, S., & Schellnhuber, H. J. (2020). Feeding ten billion
people is possible within four terrestrial planetary boundaries. Nature Sustainability, 3(3),
200‑208. doi:10.1038/s41893-019-0465-1
1036
Gliessman, S. (2014). Agroecology : The Ecology of Sustainable Food Systems, Third

Edition. CRC Press. doi:10.1201/b17881
Godfray, H. C. J., Beddington, J. R., Crute, I. R., Haddad, L., Lawrence, D., Muir, J. F.,
Pretty, J., Robinson, S., Thomas, S. M., & Toulmin, C. (2010). Food Security : The
Challenge of Feeding 9 Billion People. Science, 327(5967), 812‑818.
doi:10.1126/science.1185383
IPES-FOOD. (2018). Breaking away from industrial food and farming systems : Seven case
studies of agroecological transition. http://www.ipes-food.org
Østergaard, E., Lieblein, G., Breland, T. A., & Francis, C. (2010). Students Learning
Agroecology : Phenomenon-Based Education for Responsible Action. doi:
10.1080/13892240903533053
Roopa, S., & Rani, M. (2012). Questionnaire Designing for a Survey. The Journal of Indian
Orthodontic Society, 46, 273‑277. doi: 10021-1104
van der Ploeg, J. D., Barjolle, D., Bruil, , … Wezel, A. (2019). The economic potential of
agroecology : Empirical evidence from Europe. Journal of Rural Studies, 71, 46‑61.
doi:10.1016/j.jrurstud.2019.09.003
Wezel, A., Casagrande, M., Celette, F., Vian, J.-F., Ferrer, A., & Peigné, J. (2014).
Agroecological practices for sustainable agriculture. A review. Agronomy for Sustainable
Development, 34(1), 1‑20. doi:10.1007/s13593-013-0180-7
Wu, J. S., & Lee, J. J. (2015). Climate change games as tools for education and engagement.
Nature Climate Change, 5(5), 413‑418. doi:10.1038/nclimate2566
1037
Managing european interuniversity collaboration: a bottom-up

approach to identify digital education challenges from below
Alvaro Pina Stranger1, German Varas2, Gaëlle Mobuchon2
1
CREM-UMR 6211, Université de Rennes, Rennes, France, 2 ISTIC, Universite de Rennes,
Rennes, France.
Abstract
The European Comission has engaged with cooperation and collaboration
between European universities to ensure digital education sustainability. In
this study, we research inter-university collaboration from “below”, i.e.,
considering the actors directly involved in educational activities, such as
teachers, students, and academic staff. In this bottom-up approach, and under
the frame of an Erasmus+ project, called OpenU, we managed a consortium
of European universities which conducted a series of educational experiments
that accounted for specific challenges, mainly, from organizational,
pedagogical and technological dimensions. Challenges identified in the
organizational dimension include HEIs’ information flow, staff mobility,
alliances, and multistakeholder content. In the pedagogical dimension,
challenges include intercultural and adaptable content, pedagogical support,
and diploma recognition. Finally, in the technological domain, challenges
include advertising and support, and proper infrastructure. Our research aims
to contribute to the discussion about the relevance of evidence-based data to
feed education policies.
Keywords: Digital education; european higher education institutions; inter-
university collaboration; bottom-up approach; Open-U project; Erasmus+.

Managing european interuniversity collaboration
1. Introduction
The digitalisation of education, fostered by the increasing use of information and
communication technologies, has promoted different collaboration activities for higher
education institutions (Amemado, 2010). The European Commission has encouraged the
creation of “digital programmes” between European universities, through the development
of innovative joint pedagogical activities, e.g., distributed learning models. In this context,
cooperation activities in the educational field are normally encouraged at the policy level
relying on a top-down approach (Carpentier, 2012), which often entails implementation
tensions among “down-stream collaborators”, specially when policies “from above”
overlook the different contextual realities of education-related stakeholders.
In the present work, we contend that policy decisions affecting educational cooperation may
be efficiently fed through a bottom-up approach. In contrast to the top-down perspective, the
bottom-up approach implies researching (innovative) pedagogical methods coming “from
below”, i.e. the people directly impacted by educational policies, such as teachers and/or
headmasters directly (Carpentier, 2012). To research collaboration “from below”, we have
managed a consortium network of seven European HEIs, which has allowed us to identify
critical challenges emerging from inter-university collaboration in the domain of digital
education. As a result, challenges were organized around three main dimensions:
organizational, i.e., aspects related to the logistical part of the cooperation; pedagogical, i.e.,
aspects related to didactics and learning; and technological, i.e., elements related to the
technologies used for cooperation.
Through our study, we argue that a bottom-up approach to collaboration may be a powerful
evidence-based tool to feed education policy reforms.
2. Theoretical background: Inter-University Collaboration and the Bottom-up

Approach
Inter-university collaboration is often implemented following education policies, through a
process which is complex since several stakeholders are involved. The diversity of actors, for
example, in terms of limited resources and institutional restrictions, can lead to “policy
failure” if actors “from below” are not “understood” and “known” properly. In this sense, it
is important to understand collaboration, clarify its determinants and explore ways to make
it more transparent and effective (Viennet & Pont, 2022).
Effectiveness of education policies is often researched following two approaches: top-down
and bottom-up (Napoli, 2021). While the top-down research considers policy from the point
of view of decision-making and observes how it is implemented by the actors, the bottom-up
analysis considers that policy is first and foremost implemented by the actors who interpret
1040
Alvaro Pina Stranger, German Varas, Gaëlle Mobuchon
the decisions in light of the situations they encounter with the beneficiaries of the policies
(Napoli, 2021). From this approach, the aim is then to direct attention to the individuals
situtated “at the bottom of the pyramid”, as they play an active role and exert influence by
making changes to the policy (Carpentier, 2012).
In the domain of education, specifically, the bottom-up approach aims to improve everyday
teaching practice through participatory research and development (Straub & Vilsmaier,
2020). In this sense, bottom-up approaches emphasize the active participation of teachers and
(co-) ownership of the initiation, development and implementation of innovations. Thus,
bottom-up approaches are dynamic, iterative-cyclical and open-ended processes.
In line with the literature on bottom-up methodologies, we have empirically researched
collaboration among European HEIs to shed light on the challenges related to the creation
and implementation of innovative pedagogies, such as online and distributed learning, which
are currently at the core of education policies in Europe.
3. Methodology
Our objective was to research collaboration “from below”, that is to say, to identify HEIs’
challenges related to the creation and implementation of innovative pedagogies, such as
blended learning, online learning, distributed learning, among other forms.
3.1. Participants
To research collaboration “from below”, we coordinated a network of seven “experimenting
partners”, in the framework of OpenU, an Erasmus+ project which aimed to “foster European
cooperation, innovation and sustainability in higher education (…), provid(ing) a digital
infrastructure for higher education policy experimentation in blended learning, mobility and
networking” (see https://openu-project.eu/).
To attract researchers from the OpenU network, we designed a “Pre-Call” and a “Call”
template, which were revised and iterated at least three times by organizers. The Pre-Call
aimed to target internal “experimenters”, either academics or staff participating already in
international projects in the field of education (e.g., UNAEUROPA, EIT Digital, among
others). The Call aimed to promote the activity at a general level, including researchers from
different departments and units. Through both calls, interested researchers had to express
their motivation, describe additional collaboration networks, and provide a preliminary
description of their potential projects.
As shown in Table 1, 8 projects were finally developed in partnership with contributors from
and outside the consortium (marked with *) while four experimentations were developed
without partners because they consisted in pedagogical activities whose aspect to be tested
1041
would not require further cooperation. In all experimentations, different actors (teacher, staff,
and student) and settings (micro-course, course, program) were considered.
Table 1. Projects including leaders and internal and external (*) contributors.
Source Target
Projects Under Experimentation Management
(leader) (contributor)
1. “Educating the Trainers — Blended Content
Aalto UCM, FUB
Production Catalyst”
2. “Scientific posters across boundaries: design of

UCM Rennes, Trento
distributed group research activity”
3. “Joint digital, interactional teaching formats –How
to implement collaborative online and blended UCM FUB
courses”
4. “eTandem - Online Language Partnerships” FUB Bologna, *Edinburgh
5. “Designing and supporting Virtual Mobility
KULeuven *Others
activities”
6. “Definition of users’ needs in the digitalization of
Paris1 JUKrakow
EU HEIs”
7. “Preparation and delivery of an international
KULeuven, *Wurzburg
collaborative MOOC: an analysis on the pedagogical Bologna
*diParma, *Hamburg
and technical implementation”
FUB, *Edinburgh,
*Helsingin y liopisto,
8. "One Health in Bloom" Bologna
JUKrakow, KULeuven,
UCM, Paris1
9. “Gamification tools in Higher Education:
Implementation of the Escape Room in the Pharmacy UCM No partner
Degree”
10. “Students as main actors of European HEIs:
general survey of student population in the aim of
Paris1 No partner
establish needs, aspirations, fears and hopes in the
digital turn of EU HEIs”
11. “Distributed training of students for the quality
UPM No partner
improvement of their bachelor's and master's theses”
12. “Technology Watch to find Solutions to Social
UPM No partner
Challenges of our Society”
3.2. Data and data analysis

Challenges were identified along the experiments conducted by the network of partners.
Experiments were conducted directly by the actors in the field (teachers, students, university
members, etc.). A wide variety of methodological techniques were included, for example,
interviews, focus groups and surveys. Experimenters used different devices (video
conferences, platforms) or tests including synchronous and asynchronous activities to
achieve their objectives. Experimentation leaders reported on the experiments, following a
pre-defined template, including the following information:
1042
• Identification details (leading partner and contributors)

• Experimentation name
• Dates of execution
• Level of application (session, module, course, program, other)
• PAs
• Abstract
• Context
• Research justification
• Questions and objectives
• Methodology
• Results and discussion
• Conclusions
The article-based format of the reporting allowed us to identify three patterns of results in
relation to organization, pedagogical and technological aspects. Reported results were
analyzed and more specific patterns related to challenges emerging from digital cooperation
were identified.
4. Results
The analysis of the experiments conducted within the framework of the Open-U project
allows us to clearly identify three dimensions impacting the collaboration between
universities in the domain of digital education: organizational, pedagogical and
technological.
4.1. Organizational dimension

In the organization dimension, we found four main aspects: information flow, staff mobility,
alliances, and multistakeholder content. In terms of information flow, i.e the way information
moves throughout the education system, researchers at Paris 1 found that students and
teachers tend to be unaware of the collaborative projects conducted by the university, which
clearly results in a disconnection between academics and students. Regarding staff mobility,
researchers at KU Leuven observed that an effective e-mobility policy requires an
“interdisciplinary approach” articulating systematically actors with different hierarchies and
from different departments or faculties. Regarding HEI alliances, several experiments (UPM,
Aalto, FUB) have revealed the importance of including partners other than universities, with
the development of an ecosystem of partners to share relevant information, for example.
Finally, collaborative creation of content and pedagogical activities was identified as an
important sub-dimension to maintain sustainability in the inter-university relationship.
Researchers from Bologna and UPM have shown that the creation of educational content
1043
should involve the design of a transnational collaborative environment, involving the best
professionals of the partner network, and using distributed repositories accessible to any
member of the community, either student, teacher or staff.
4.2. Pedagogical dimension

In the pedagogical dimension, we found three aspects: intercultural and adaptable content,
pedagogical support, and diploma recognition. Regarding intercultural and adaptable
content, Paris 1 researchers found that the adaptability of learning content and assessment
methods are crucial for a “digital university”. Interestingly, researchers from FUB, Bolgona
and KU Leuven argue that networked education implies both new course content and new
learning and teaching methods that should take into account language differences and cultural
exchanges. Regarding pedagogical support, Aalto researchers found that there is a need to
support teachers in the design and delivery of e-learning or blended learning activities (e.g.
through video recording). They argue that when organisations start to produce their first units
of e-learning or blended learning content, they generally do not understand the process
involved and the key elements needed at each stage to succeed. With regard to the recognition
of qualifications, UPM researchers stressed that one of the first steps for the creation and
implementation of distributed learning should be the exploration of local regulations on the
use of educational infrastructures, such as ECTS, for example. These researchers propose
that consortia providing short collaborative modules could benefit, for example, from the use
of one ECTS, which could be inserted into pre-existing courses of formal masters courses
currently running at partner universities.
4.3. Technological dimension

Regarding technological dimension, we found two aspects: advertising and support; and
proper infrastructure. In relation to adversiting and support, KU Leuven researchers
suggested that digital infrastructures could be used to market learning and vocationalization
opportunities to educational developers and teaching staff. FUB researchers emphasised the
importance of the possibility of offering real-time support to teachers and students. They
point to the need for sustainable support services for teachers (technical, didactic and
pedagogical) and for student mobility. Regarding infrastructure, most researchers (e.g. Paris
1, KU Leuven and UPM) identified the need to develop an infrastructure with a high level of
interoperability, i.e. that can be connected to other European platforms, databases and
repositories. Easy access to these platforms, “with minimal registration requirements”, was
also highlighted by Bologna and Aalto researchers.
1044
5. Discussion
The analysis of the experiences of the Open-U project highlighted a number of challenges of
virtual collaboration at the organizational, pedagogical and technological level. In terms of
the organizational dimension, Coombe (2015) highlights that the most frequently mentioned
barriers to inter-university collaboration are conflicting interests and expectations of partner
universities, lack of allocated time, lack of support or inconsistent leadership, geographical
distances and incompatible and bureaucratic systems. The challenges related to faculty
workload and lack of institutional support seem to be confirmed by the literature. Indeed,
Caluianu (2019) showed that increased workload and lack of administrative support are
challenges faced by teachers in adopting this type of teaching.
Pedagogically, our findings on intercultural and adaptable online content are confirmed by
the literature, which recognizes that there is a need to integrate and connect the inter-
university online curriculum with international dimensions through international curriculum,
co-design and collaboration (Devonshire & Siddall, 2011). Secondly, about the challenge of
multi-actor content, the literature emphasizes that inter-university collaborations from
different countries require an additional effort from educational teams, because this implies
that they go out of their comfort zone and work against barriers (differences in academic
standards, languages, academic calendars and time zones) that hinder this collaboration.
Interestingly, the temporal challenges encountered in implementing these innovative
modalities are consistent with those identified in the literature.
In terms of the technological dimension, literature has highlighted the importance of
instructors and students being familiar with the technology used, as this plays an important
role in the success of exchanges (Avgousti, 2018). Among those technological aspects, the
platform, ICT tools, student learning materials, software and formats, and characteristics of
course management should be considered (Soto-Acosta et al., 2014). As shown from the
literature, the quality of the ICT infrastructure and the lack of technical experts may impact
the successful implementation of distance education policies.
6. Conclusion
We have explored collaboration among HEIs to shed light on the challenges related to the
creation and implementation of innovative pedagogies, which are currently at the core of
education policies in Europe. To this end, we have examined European inter-university
collaboration from a bottom-up approach, in which a consortium of institutions have
accounted for specific challenges emerging from three dimensions: a) organisational (HEI
information flow, staff mobility, alliances and multi-stakeholder content), b) pedagogical
(intercultural and adaptable content, pedagogical support, recognition of diplomas), and
technological (publicity and support, adequate infrastructure). Thourgh our results, our
1045
research aims to contribute to the discussion about the relevance of evidence-based data to
feed education policies.
References
Amemado, D. (2010). Changements et évolution des universités conventionnelles sous
l’influence des technologies de l’information et de la communication (TIC): Le cas du
contexte universitaire nord-américain. Doctoral dissertation, Université de Montréal.
Avgousti, M. I. (2018). Intercultural communicative competence and online exchanges: A
systematic review. Computer Assisted Language Learning, 31(8), 819-853.
Caluianu, D. (2019). When more is less: Unexpected challenges and benefits of
telecollaboration. In A. Turula, M. Kurek and T. Lewis (eds), Telecollaboration and
virtual exchange across disciplines: in service of social inclusion and global citizenship
(7-13). Research-Publishing.net.
Carpentier, A. (2012). Les approches et les stratégies gouvernementales de mise en oeuvre
des politiques éducatives. Éducation et francophonie, 40(1), 12-31.
Coombe, L. (2015). Models of interuniversity collaboration in higher education – How do
their features act as barriers and enablers to sustainability? Tertiary Education and
Management, 21, 1-21.
Devonshire, E., & Siddall, P. (2011). Joining forces: Collaborating internationally to deliver
high-quality, online postgraduate education in pain management. Pain Research &
Management : The Journal of the Canadian Pain Society, 16(6), 411-415.
Napoli, J. (2021). Analyser la mise en œuvre des politiques d’éducation compensatoire au
prisme de la sociologie des organisations. Recherches en éducation, (45), 7-19.
Soto-Acosta, P., Pérez-González, D., & Popa, S. (2014). Determinants of Web 2.0
technologies for knowledge sharing in SMEs. Service Business, 8, 425-438.
Straub, R., & Vilsmaier, U. (2020). Pathways to Educational Change Revisited-
Controversies and Advances in the German Teacher Education System. Teaching and
Teacher Education, (96), 1-13.
Viennet, R., & Pont, B. (2017). Education policy implementation: A literature review and
proposed framework. OECD Education Working Papers. OECD Publishing, Paris.
1046
Deep learning: a study on marine renewable energy and

sustainability education in an Irish context
Pauline Logue1, Róisín Nash2, Micheál Walsh1, Michael Faney1, Dearcán Ó Donnghaile1
1
Department of Creative Education, Atlantic Technological University, Ireland, 2Department
of Natural Resources and the Environment, Atlantic Technological University, Ireland.
Abstract
In 2020-2022 a select group of Irish educators, marine renewable energy
proponents and sustainability stakeholders entered into dialogue with a view
to enhancing post-primary educational resources. The paucity of educational
resources in this field was highlighted, most notably Irish language resources.
This dialogue led to the development and piloting of a bilingual (Irish and
English) cross-curricular programme of learning - a Transition Year Unit -
targeted at students aged 15-17. This pilot study aims to evaluate the
effectiveness of the learning unit with respect to enhancing knowledge in these
novel areas. The methodology is a mixed methods case study. Data gathering
processes include student questionnaires, stakeholder focus groups, and expert
interviews. The main finding is the importance of stakeholder input into
curriculum development to ensure the effectiveness of the Transition Year Unit
of Learning, and to enhance learner engagement. Furthermore, the study
recommends that dissemination include a web-based ‘Deep Learning’
educational platform with downloadable resources to enhance nationwide
impact.
Keywords: Global citizenship; marine renewable energy; sustainability
education; transition year programme.

Deep learning: a study on marine renewable energy and sustainability education in an Irish context
1. Introduction
There is evidence of a growing body of literature in the fields of sustainable development
education (SDE) and global citizenship education (GCE) (Gorben & Yemini 2017). Marine
renewable energy (MRE) is one dimension of SDE which entails harnessing sustainable
energy from wind, tides, waves and ocean currents. MRE is much debated, not least because
of the complex technical, ethical, environmental and ecological challenges that it entails
(Borthwick 2016; Weiss et al. 2020; Kulkarni & Edwards 2022). MRE education, which
addresses the complexities of MRE, is pertinent for Ireland, an island nation with limited
energy resources and a view to increasing indigenous production of renewable electricity.
Irish MRE research, education and training is currently facilitated by an number of Irish
organisations, not least the Marine Institute, BlueWise Marine and Sustainable Energy
Authority of Ireland (stakeholders aligned with this study). The Marine Institute (2017), for
example, speaks of the ‘marining’ of education. A prior study by the authors of this paper,
incorporated exploratory dialogue with these and other selected Irish organisations with an
interest in MRE. That study identified the need for educational resources for Irish language
medium post-primary schools (Gaelscoileanna) and for Irish MRE educational research
outputs (Logue et al. 2023). Building on this research, the aim of this paper is to evaluate the
effectiveness of one strategic educational initiative pertaining to MRE education, piloted in
a post-primary Irish language medium school (Gaelscoil), in the west of Ireland, in 2022.
The methodology is a case study and the data gathering methods include a stakeholder focus
group, student questionnaires and expert interviews.
2. Context
In 2015 the United Nations published ‘Transforming our World: The 2030 Agenda for
Sustainable Development’, in which sustainable development goals (SDGs) were identified.
Following this, the Irish Government began a process of curriculum reform and re-design in
order to incorporate SDE into the Irish curriculum (Department of Education and Skills
2020). The Irish Teaching Council also set out its expectation of GCE training in initial
teacher education (Teaching Council 2020, p.14). In light of these developments, in the
period 2020-2022, a collective of Irish educators, MRE stakeholders, and academic
researchers, in the west of Ireland, entered into exploratory conversations around enhancing
bilingual (Irish and English) educational resources, in the areas of SDE, GCE and MRE, in
Irish post-primary schools1. It was established during these exploratory conversations that
Irish language medium schools were the most negatively impacted schools, given a
significant lack of Irish-language educational resources relating to SDE, GCE and MRE
1
The project stakeholders and funders are Atlantic Technological University, Marine Institute/Foras naMara, BlueWise Marine Ltd,
Sustainable Energy Authority of Ireland, Údarás na Gaeltachta, and Coláiste Chroí Mhuire gan Smal, Spiddal, Ireland.
1048
Pauline Logue, Róisín Nash, Micheál Walsh, Michael Faney, Dearcán O Donnghaile
nationally. This dialogue led to the development and piloting of a bilingual (Irish and
English) cross-curricular programme of learning, namely a ‘Transition Year Unit’ (TY Unit),
targeted at senior level post-primary students in the Irish education system, aged 15-17 years.
The TY Unit was in the form of a 10-week planning scheme, incorporating subject
specification learning outcomes, STEM-related curriculum content, teaching and assessment
strategies, weekly lesson plans, bilingual lesson resources and wider supporting educational
resources. It was agreed by the stakeholders that this MRE educational output would be
piloted in one west of Ireland, Irish language medium school (Gaelscoil) in close proximity
to a marine research testing site, where MRE public debates are in evidence.
3. Methodology and Methods

A case study methodology was employed and data gathering was a three-phase sequential
process involving a stakeholder focus group, expert interviews, and student questionnaires.
3.1. Qualitative Research: Focus Group

Phase one consisted of a stakeholder focus group comprised of six funding representatives
from Irish organisations with expertise in marine science, marine renewable energy research,
STEM education, and the Irish language. The focus group was held online, using MS Teams.
It commenced with a brief presentation on the research aims, after which guiding questions
were used to explore participants’ perspectives on MRE education, SDE and GCE
educational resource development, and bilingual resource development. As part of the focus
group, a first-version TY Unit (iteraton one) was critically assessed. This first iteration was
informed by literature research and the specifications of the Irish post-primary STEM
curriculum. This iteration was circulated to the focus group participants prior to the focus
group in order to maximise the quality of the discussion. The primary role of this discussion
was to provide guidance on the further enhancement of the TY Unit. Following the focus
group, the original TY unit was re-designed by the research team (iteration two).
3.2. Qualitative Research: Expert Interviews

Phase two entailed five expert interviews which were conducted with selected Irish experts
in the fields of marine renewable energy, oceanography, and sustainability. The first
interview was a pilot interview which was co-reviewed by academics with attention to
validation and enhancement. As with the prior focus group, the interviews were online, and
recorded, using the MS Teams platform. A primary focus of the interview was a critique of
the TY Unit (itertation two), which had been circulated in advance of the interview, along
with guiding questions. Subsequently, the TY Unit was amended (iteration three).
1049
3.3 Mixed Methods Research: Student Questionnaire

In phase three, this third version of the TY Unit was further reviewed by representatives of
the host case study pilot school (including management, STEM teachers and students).
Following this, a final version was drafted (iteration four) that was a agreed by all parties.
Subsequently, the TY Unit was delivered by a member of the research project team: a
qualified bilingual post-primary teacher-researcher. The pilot delivery of the TY Unit took
place in the host school weekly, over a 10-week period, February to May, 2022. Purposeful
and convenience sampling was employed. The TY students in the pilot school (Gaelscoil)
were invited to participate, and all opted in voluntatily as research participants to the study
(n=30). Demographically, in terms of nationality, 28 students identified as Irish, one
identified as Brazilian and another as Irish/Somalian, the gender profile was 17 female and
13 male, and the age range was 15-18 years. Ethical approval for data gathering was obtained
from the host school, in loco parentis, since some of the participants were as yet ‘minors’. A
student questionnaire, generated using MS Forms, and containing 30 questions, was utilized
to capture students’ perspectives. The questionnaire included open and closed questions, of
a variety of types - Likert Scale, yes/no, selection options, and opinion/example-type
questions, including specific questions relating to MRE education- coded below as MREE
[Qs 8-19]. The questionnaire, which was anonymous, was distributed at the commencement
of the TY Unit (a baseline questionnaire) and again at the conclusion of the unit of learning,
(a final questionnaire), in order to monitor learning.
4. Findings and Analysis

A concise analysis of the dominant findings of the study is presented below with reference
to the data and to relevant literature. Firstly a brief account of the data analysis methodology
is provided with respect to the qualitative research methods in the case of the focus group
and expert interviews. Following this, a mixed methods analysis is provided with respect to
the student questionnaires. themes and related sub-themes were generated: teaching of MRE,
public relations within MRE, marine industry perspectives, and sustainability and society
4.1. Qualitative Research: Data Analysis Methodology

A thematic analysis was applied to the qualitative data arising from the focus groups and
expert interviews, based on an adaptation of the inductive six-stage model of coding and
analysis articulated by Braun and Clarke (2021). This six-stage model includes:
familiarization with data, generation of initial codes and categories, identifying themes,
reviewing themes, defining and naming themes and producing the report. Manual colour
coding of data transcripts and related notations were used to identify first level codes and
categories within the data sets. Co-coding was facilitated by an academic researcher (>10%
of data). MS Word and MS Excel tools were utilized to chunk, cluster and refine related
1050
themes to the point of saturation, where all codes were accounted for in a final re-organisation
of data. At this point the four dominant themes were geerated (see above).
4.2 Qualitative Research: Focus Group and Expert Interviews

In the following discussion of the generated themes, stakeholder participants are coded as S
and numbered (S1, S2, S3, etc.), the pilot expert interview is coded as P, and further expert
interviewees are coded as E, and numbered (E1, E2, E3, etc.).
With respect to the teaching of MRE in the TY Unit, sub-themes included teaching
methodologies, factors in delivery of content, types of resources and important
concepts/messages. Active, collaborative and reflective teaching and learning approaches
were advocated, as well as field trips and guest inputs, demonstrating an awareness of the
target audience (E4). One interviewee emphasised the need for educational programmes and
resources to “develop learners’ understanding of the community that they're in, of the
properties of the local environment, the local heritage of how it relates to them, and how they
live their lives” (P). This point was built upon by another interviewee who spoke of “thinking
global and acting local”(E1). Also, empowerment of learners was highlighted, specifically
youth (E4).
Two sub-themes were generated relating to MREE and public relations, namely, the wider
promotion of MREE and the need for sensitivity in regard to inter-relations with public
stakeholders, e.g. local community representatives, fisheries representatives, local heritage
groups, environmental groups. Local sensitivity and the need for local trust-building was
identified as a significant factor: one stakeholder spoke of the need to develop learners’
understanding of their community they live in and stressed necessary community education
in relation to MRE, including test sites and local heritage interests (S2).
With respect to marine industry perspectives, sub-themes included marine types and
technologies, career opportunities, factors impinging the marine environment, and marine
legislation and policy. One stakeholder identified specific technological curriculum content,
devices and technology necessary to inform learners on the TY Unit (S3). Examples were
spelt out: engineering, finance, computers, artificial intelligence, and graphic design (S1).
Potential marine careers were highlighted, and the need for the TY Unit to get learners
“interested in research and marine careers more generally, and maybe increase their
knowledge because a lot of the time they wouldn't be aware of the different careers that are
in the marine area”(S3). The multifactted and multi-disciplinary nature of MREE was
discussed, including the need for integrated mathematics, science, engineering, and local
heritage, in the context of a life-long learning model, commencing at pre-school level (E2).
Finally, regarding sustainability and society, two sub-themes were generated: society and the
marine, and integrated sustainability factors, the latter emphasising GCE, SDE, the marine
1051
ecosystem and climate action. The need for reformed legislation was noted (E2; E3). One
interviewee stated that “Ireland is actually changing at the moment so we're reforming our
legislation that covers marine development”” (E3). In the context of climate change, local
knowledge and local politics was commented upon, in the context of developing the TY Unit:
“…try and bring it down to a local level to show students what they can and can't have control
over” (E4).
4.3. Mixed Methods Research: Student Questionnaire

Focusing on MRE (Q.s 8-19), it was demonstrated that the TY Unit resulted in increased
MRE knowledge, including greater marine or ocean literacy precision. Learners explicitly
acknowledged increased learning as a result of engaging in the TY Unit (Q.10). With respect
to Q.8, while in the baseline questionnaire students had a vague and somewhat confused
understanding of MRE potential, in the final questionnaire, students cited a number of very
specific possibilities for MRE in Ireland, given its geographic location, as part of the reason
for it being important. In addition, they cited the ability to reduce use of fossil fuels, reduce
dependence on other countries for energy, the need for a more sustainable source of energy
and the goal to reduce fossil fuel usage in general (Q.12). These more insightful responses
accounted for half of the cohort (50%), with the remainder also giving answers containing
some insight (Q.12). With respect to the environmental impact of MRE (Qs.13 & 14), in the
basline questionnaire, 70% of respondants indicated that they did not know if there was an
impact on the environment, as a result of MRE. In contrast, in the final questionnaire, while
there was still no consensus regarding the impact of MRE on the marine ecosystem - with
‘yes’, ‘no’ and ‘maybe’ equally represented – answers were more nuanced. Students could
now provide more technical arguments on both sides and certain answers given were more
detailed, reflecting increased knowledge in the area: “…some of the underwater equipment
can be dangerous for marine life. For example, propellers that use the waves” and “…if
anything, some of the machines can serve marine life in an advantageous way such as one
machine that acts like a reef and helps ecosystems in that way”. In general, more expanded
understandings of environmental concerns (including marine) was in evidence in responses
to Q. 17. This more nuanced and informed understanding of the complexities involved
mirrors recent literature, where the complexities of sustainability ethics, and the need to
negotiate trade-offs between competing sectors and interests are highlighted (Kulkarni and
Edwards, 2022; Galparsoro, et. al, 2022). See Figure 1 where non-renewable energy (NRE),
offshore fishing (OF), plastic pollutants (PP) climate change (CCC), all /other options are
explored and where a shift in understanding is evident.
1052
Figure 1. Environmetal concerns today: baseline and final questionnaire
5. Conslussions and Recommendations

The main conclusion arising out of this pilot study is the effectiveness, on balance, of the
research project’s TY unit of learning as a MRE and sustainability education resource. Its
overall capacity to enhance knowledge of MRE education among post-primary students in
the pilot school (Gaelscoil) was demonstrated. This was largely attributed to educational
inputs from marine experts and educators, who contributed to the curriculum design and
content of the newly-developed TY programme. The study acknowledges the need for a
balance between theoretical and practical learning in post-primary educational initiatives. It
recommends that the educational resource outputs from the study will be more widely
disseminated among Irish post-primary schools, not least the Irish language medium schools
(Gaelscoileanna), in the form of a web-based ‘Deep Learning’ educational platform with
downloadable resources to enhance nationwide impact. Also, ongoing development of
further wide-ranging bilingual educational courses and resources, for the post-primary sector,
is recommended, such as first year induction or ‘taster’ courses, and Junior Cycle short
courses on MRE education, in the context of GCE and SDE. While this study goes some way
to addressing an identified gap in the literature, and in Irish education, it is acknowledged
that the sample size is relatively small and, as a case study, the findings are localized and
non-generalisable. Furthermore, the focus of this study is primarily MRE- a specialist
environmental area, not readily applied to other environmental contexts. However, many of
the research project’s educational outputs are flexible and adaptable, and may be readily
adapted to other post-primary contexts in Irish education.
1053
References
Borthwick, A.G.L. (2016). Marine Renewable Energy Seascape. Engineering. Volume 2.
Issue 1. Pp. 69-78. At Marine Renewable Energy Seascape (pnnl.gov). Retrieved
21/04/2023.
Braun, V., & Clarke, V. (2021). Thematic Analysis: a Practical Guide. SAGE Pub. Ltd.
Department of Education and Skills. (2020). ‘Education for Sustainability’. The National
Strategy for Education on Sustainable Development in Ireland. Government of Ireland.
Galparsoro, I., Menchaca, I., Garmendia, J.M., Borja, A., Maldonado, A.D., Iglesias, G., &
Bald, J. (2022). Reviewing the ecological impacts of offshore wind farms. Ocean
Sustainability.1:1. doi.org/10.1038/s44183-022-00003-5. Retrieved
Goren, H. & M. Yemini. (2017). Global citizenship education redefined – A systematic
review of empirical studies on global citizenship education. International Journal of
Educational Research,Volume 82, 2017, Pages 170-183.
https://doi.org/10.1016/j.ijer.2017.02.004
Kulkarni, S.S., & Edwards. D.J. (2022) A bibliometric review on the implications of
renewable offshore marine energy development on marine species. Aquaculture and
Fisheries, Vol. 7. Issue 2. https://doi.org/10.1016/j.aaf.2021.10.005
Logue, P., Nash R., Walsh, M., Faney M. & Ó Donnghaile, D. (2023). Deep Learning: The
Development of Cross-disciplinary Post-primary Educational Resources to Promote
Marine Renewable Energy and Sustainability Education. Paper. ESAI Conference 2023.
Stranmillas College, Belfast, Ireland.
Marine Institute (2017). Marine Institute Strategic Plan 2018-2022. At Marine Institute
Strategic Plan 2018-2022; Building Ocean Knowledge, Delivering Ocean Services.
Teaching Council (Ireland). (2020). CÉIM: Standards for Initial Teacher Education. At
ceim-standards-for-initial-teacher-education.pdf (teachingcouncil.ie).
United Nations. (2015) Transforming Our World: The 2030 Agenda for Sustainable
Development. At 21252030 Agenda for Sustainable Development web.pdf (un.org).
1054
Plant and plan, care and grow. A hands-on exercise using the
(inner) sustainable development goals to teach research
methodology to final year sociology students
Department of Sociology and CICS.NOVA, Universidade de Évora, Portugal.
Abstract
Etymologically, the word seminar comes from the Latin semen, which means
“seed”. Inspired by such linguistic archeology this text describes an exercise
developed with undergraduate sociology students, who were literally invited
to sow a seed and observe its growth over the course of a semester in which
they must design a sociological research project. Transversally, the students
perceived the exercise of germinating a plant as a metaphor for the
development of the research project as a living and dynamic reality,
highlighting the beginning, growth and maturation as key moments.
Additionally, students emphasized that observing the germination and
development of the plant allowed them to critically reflect on the different
stages of the research project, while allowing their own inner development,
namely with regard to the dimensions of “being”, “thinking”, “relating”,
“collaborating” and “driving change”.
Keywords: Inner development goals; social research methodology; sociology;
sustainable development goals; SDG 4 - Quality Education.

Plant and plan, care and grow
1. Introduction
Higher education is made of many teaching-learning contexts, including seminars.
Etymologically, the word seminar comes from the Latin semen, which means “seed”.
Inspired by such a linguistic archeology, a hands-on exercise was developed with higher
education students, who were literally invited to sow a seed and observe its growth. The
exercise was carried out with 37 undergraduate sociology students within the scope of the
“Research Laboratory: Project Elaboration” (LabEP) course in the fall semester of the
2022/23 academic year. LabEP is a compulsory subject in the final year of the Sociology
graduation at the University of Évora. The weekly workload is two theoretical hours and two
practical hours, which is equivalent to a total of 6 ECTS, but classes often take the form of a
seminar with strong participation by students. According to the study plan in force, the course
aims to develop cognitive and reflective skills and its general objective is to serve as a
scientific and pedagogical contextual framework for the construction of a sociological
research project. The curricular unit is strongly articulated with “Research Laboratory:
Project Execution” in the last semester. Both courses aim to support the construction and
development of sociological research work and to complement the deepening of general
skills, namely: ability to integrate theoretical, methodological, and empirical knowledge
towards the identification and resolution of sociological problems; to develop rigorous and
innovative analysis; clearly communicate analysis and results, as well as its foundations and
justifications, in contexts of research and professional activity; and to stimulate autonomous
personal learning processes. In line with the idea that higher education institutions play a key
role in promoting sustainable development (SDSN General Assembly, 2017), this course
intends to contribute to this, creatively using the sociological literature that recognizes the
different ways of telling – and thinking – about society (Becker, 2007), and the power of the
sociological imagination (Mills, 1959) to create new and more engaging ways of teaching
and learning sociology (Atkinson and Lowney, 2016; Jones, 2017).
2. Context: Looking for deep learning experiences to foster (inner) sustainable

development
At present, educating for Sustainable Development Goals (SDGs) is a core objective for
higher education institutions (SDSN General Assembly, 2017). The 2030 Agenda envisions
a world transformation (United Nations, 2015), that is inseparable from starting at
universities (UNESCO, 2017). This appeal comes in the wider context of the important
changes that are taking place in teaching and learning environments, namely those that claim
the importance of “deep learning” rather than “surface learning” experiences (Atherton,
2009). As the 17 SDGs become widely publicized and increasingly known around the world,
the non-profit, open-source initiative “Inner Development Goals” (IDGs) gains visibility as
“an essential framework of transformative skills for sustainable development” (Inner
1056
Development Goals, 2023). The IDGs framework consists of five dimensions organising 23
skills and qualities of human growth and internal development. The five-dimensional
framework comprises skills related to “Being — Relationship to Self”, “Thinking —
Cognitive Skills”, “Relating — Caring for Others and the World”, “Collaborating — Social
Skills” and “Acting — Driving change”. According to the initiative, IDGs aims to develop
inner abilities to deal with increasingly complex environments and challenges arising from
the implementation of SDGs (Inner Development Goals, 2023), thus accelerating the work
towards the UN’s Global Goals.
3. Plant a seed to investigate the outside and grow from within
3.1. Planting and planning

At the beginning of the semester, students were invited to carry out the germination
experiment of a plant and carefully observe its growth. To carry out the activity successfully,
the teacher distributed to each student in the classroom two seeds of a dry leguminous plant
(beans), a sheet of kitchen paper and a plastic bag with a zipper closure. Instructions were
given in the classroom: (1) students should start by opening the bag to free up space; (2)
moisten kitchen paper; (3) fold the kitchen paper up to three times the size of the beans; (4)
place the wet kitchen paper in the bottom of the zip-lock bag; (5) place a bean on top of the
paper and another bean on the side of the paper that is in contact with the bag; (6) tape the
bag to a window that receives a lot of direct light and leave it for a few days; (7) observe the
germination and care for the plant, later transplanting it to a pot with soil, water and sunlight.
Figure 1 depicts the teacher's exercise, taped to her home office window. A similar project
was developed by the students, captured in the form of a photograph and submitted online
via Moodle in the course area. This photograph was accompanied by a short reflection text
of up to 500 words in which students were invited to reflect on their own research project
and its relationship with the SDGs and IDGs.
3.2. Observing and anticipating

The activity instruction given to students in September, specifically asked them to carry out
a plant germination experiment, carefully observe its growth, and reflect on the research
project and its relationship to the SDGs and IDGs. Three months later, in December, students
were invited to remember the exercise, to think about the different phases of anticipation,
projection, growth, stagnation and eventual disappearance of the plant and to reflect critically
on the elaboration of the research project as a result of academic investment and process of
personal development throughout the semester.
1057
Figure 1. Plant germination activity. Source: Photograph by RCosta (September 20, 2022).
Transversally, students perceived the exercise of germinating a plant as a metaphor for the
development of the research project as a living reality. Specifically, moments of beginning,
growth and maturation were highlighted in their narratives, as shown in the following
excerpts taken from student’s essays:
“In my opinion, the bean is a metaphor for the project. Just as the bean takes time to sprout
and, after that moment, it grows day by day, the project also takes time. It may take a while
for us to decide what we want to do, the theme, the approach, the methods, the location, but
once the first idea comes to us, whether on a sleepless night or a busy afternoon, I think the
project will flow until it is fully prepared to be executed.”
“The seed germination process finds a metaphor related to the elaboration of the research
project. By sowing and consistently tending a small seed, a life is created. As in the project,
from a small idea, perhaps coming from a moment of inspiration, different ideas and
hypotheses are developed that can generate the final project. With patience and consistency,
the idea is fed, doing research and working on different hypotheses, which will bring the
project to life.”
“[…] in a metaphorical tone, I can associate the germination phases of a plant with the fact
that the elaboration of a project is also done in phases, because just as the plant is sown,
cared for and develops over time and with the help of some external factors. The elaboration
of a project starts from a question at the outset, and then, with time and dedication, it
develops through several stages of investigation, until it reaches its result.”
1058
Figure 2. Plant germination. Source: Photograph by Figure 3. Plant after pot transplant. Source:
CFélix (October 7, 2022). Photograph by CCuco (October 21, 2022).
Figure 4. Plant in strong development phase. Source: Photograph by MVaz (October 18, 2022).
The metaphor that associates the germination of the plant with the elaboration of the research
project gains strength as time progresses in the semester, since the course included the
delivery of three reports in the form of a project in its preliminary, intermediate and final
version. The photographs of the different stages of plant germination accompany this process
of simultaneous growth and maturation (Figure 2, Figure 3 and Figure 4).
1059
3.3. Caring and growing

Often in their written essays students emphasized how the exercise required them to develop
specific skills that, while helping to prepare the research project, ultimately have
repercussions in terms of their own inner development, in particular with regard to
relationship to self, cognitive skills, caring for others and the world, social skills and driving
change. The following excerpts illustrate, consecutively, each of these dimensions.
“[…] the experience of germinating a plant, it is related to the IDG's as it provides us with
the opportunity to be present in a project, which obliges us to have an open presence, without
judgments, awakening an basic mindset of curiosity, acting on a commitment of
responsibility. When carrying out the project, we are under full responsibility for it, ensuring
the commitment we have with it. In this investigation, trust is placed in us, the result of a
mobilization of a group of people, in this case the class, to engage in purposes shared. A
collaborative relationship, on the part of the teacher, in providing us with the necessary
materials for its execution.”
“[…] [the project] fits within the competencies defended by the IDGs, due to the fact that
both need planning, organization and insight to overcome the obstacles that arise. Both
projects “require”, from the people responsible for their elaboration, a more critical type of
thinking capable of developing our cognitive abilities, taking different perspectives,
evaluating the information and, in a way, giving it meaning as an interconnected whole, also
being capable of fostering genuine dialogue and constructively managing conflicts and
adapting communication to different groups, transmitting an environment of trust. In short,
both the aforementioned organizations and the research project push individuals to develop
and deepen their relationship with their thoughts, feelings and body, helping them to be
present, intentional and non-reactive when facing some kind of complexity.”
“When trying to put the beans in the bag with the damp kitchen towel and place them on the
window, I felt that I would have to take responsibility for taking care of my research project
in the same way as this germination, being necessary to be assertive (as in the case of the
necessary water) and patience to wait for its growth.”
“Our growth is also dependent on the conditions that we have at our disposal throughout our
lives, our abilities (to think, to act, to socialize, among others) being what we work to acquire
and, ultimately, they are what define us as human beings. These capabilities are developed
through our interaction with the world around us, whether through socialization,
observation, among others. And like beans, sometimes it is necessary to change the
environment in which we find ourselves so that we can progress in our personal development.
The research project is also dependent on this, for its development it is necessary to carry
out research and interactions, and the work invested by us is what is reflected in its final
result.”
1060
“Relating to the IDGs, this activity also allows us to reflect on how simple it is to generate
life and how rewarding it is to accompany its growth and development. During this week I
found myself checking over and over again the state of the little bean seed in the hope of
seeing a sprout, of observing its development. It's very simple to leave for change, sometimes
just a seed.”
The reflexivity around the failure of a first germination appears as particularly heuristic to
understand the power of the suggested metaphor to deal with error and frustration, but also
with courage, creativity, optimism and perseverance. Through the voice of a student, the
following excerpt shows such a connection remarkably.
“The living things we care for eventually grow and “flourish” when properly cared for. Be
it a plant, an animal, a baby, or even ourselves. We grow when we learn something new,
when we improve our skills and abilities, and for that to happen, we need to take care of our
brains and feed them with knowledge. Be there through reading, listening, socializing. We
only grow when we care and feed our desire to do so. And at a certain point, this growth
becomes invisible to the eyes, but it never goes unnoticed in our lives. Our abilities to be, to
think, to relate, to collaborate and to act are always present in our lives and in our actions,
projects, work, social relationships, and even in the relationship we maintain with ourselves.
They are what define us as human beings and social beings, and we must never stop allowing
ourselves to grow in what is our human essence. I planted a bean, placed it inside a very
damp piece of paper and later in a closed plastic bag, then stuck the bag to my bedroom
window and watched the bean every day. Unfortunately, I didn't see growth, but the
appearance of mold. That's when I realized I had put in too much water, and it was too late
to save it. Sometimes we want so much that something is not missing, that we end up
exceeding the possible limits, and we end up suffocating the things we want to take care of
so much. […] In a second attempt, I put 3 little beans in a glass pot, on top of a damp cotton,
and later I put the pot in the kitchen, near the window, where it will catch the sun throughout
the day, and with the pot open, it will also catch air. We all make mistakes sometimes, and
that's not why we should give up, because starting over is something that makes us grow and
teaches us, if not the right path, at least what was the wrong path. So I realized that when we
fail once, we can only try again, until it works, even if it takes us a long time. We have to let
go of the idea that it's never too late to start over. Then I will look forward to the growth of
my beans, and their “blossoming”, as well as I expect the same from myself every day, that
it grows, and that it “blooms”.”
4. Conclusions
This article explored the pedagogical value of using a plant germination experiment as a
practical exercise in the teaching-learning process of a course aimed at undergraduate
1061
students engaged in developing a sociological research project. Transversally, the students

perceived such an exercise as a metaphor for the development of the research project as a
living and dynamic reality, highlighting the beginning, growth and maturation as key
moments. At the end of the semester, the students emphasized how the observation of the
germination and development of the plant allowed them to critically reflect on the different
stages of elaboration of the research project while allowing their own inner development,
namely with regard to the dimensions of “being”, “thinking”, “relating”, “collaborating” and
“driving change”. Interestingly in the students’ narratives, the reflexivity around failure is
heuristic to understand the power of the suggested metaphor to deal with error and frustration,
but also with courage, creativity, optimism and perseverance.
Acknowledgments
This work is financed by national funds through FCT - Foundation for Science and
Technology, I.P., within the scope of the project «UIDB / 04647/2020» of CICS.NOVA -
Centro Interdisciplinar de Ciências Sociais da Universidade Nova de Lisboa.
References
Atherton, J.S. (2009). Learning and teaching: Deep and surface learning.
https://www.learningandteaching.info/%20learning/deepsurf.htm
Atkinson, M.P., & Lowney, K.S. (2016). In the Trenches: Teaching and Learning Sociology.
W. W. Norton & Company, Inc.
Becker, H. S. (2007). Telling about society. University of Chicago Press.
Inner Development Goals (2023). Inner Development Goals – Transformational Skills for
Sustainable Development. https://www.innerdevelopmentgoals.org/
Jones, A. (2017). Teaching Sociology Successfully: A Practical Guide to Planning and
Delivering Outstanding Lessons. Routledge.
Mills, C. W. (1959). The sociological imagination. Oxford University Press.
SDSN General Assembly. (2017). The role of higher education to foster sustainable
development: Practices, tools and solutions. Position paper. https://www.sdsn-
mediterranean.unisi.it/wp-content/uploads/sites/30/2017/08/Testo-positional-CON-FIG-
1.pdf
UNESCO. (2017). Education for Sustainable Development Goals: learning objectives.
https://unesdoc.unesco.org/ark:/48223/pf0000247444
United Nations. (2015). Transforming our world: the 2030 Agenda for sustainable
development. Resolution adopted by the General Assembly on 25 September 2015.
http://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E
1062
Community intervention model: social entrepreneurship

education as a strategy for a sustainable development
Yadira Zulith Flores Anaya, Martín Arturo Ramírez Urquidy, Roberto Iván
Fuentes Contreras
Faculty of Economics and International Relations, Autonomous University of Baja
California, Mexico.
Abstract
Social enterprises are born to solve social problems without neglecting their
financial, economic, and environmental viability. Consequently, in recent
years, these ventures have gained strength as a viable and innovative
alternative to achieve sustainable development, capable of combining
economic growth, social inclusion, and sustainability for present and future
generations. For this reason, this paper aims to identify the effect that
community interventions in the teaching of social entrepreneurship aligned
with development have had on the intention of providing a sustainable solution
to current social problems. Therefore, in the present investigation, a
qualitative analysis of the self-assessment texts was done by a sample of 138
young people who, in 2022, was trained. The results show a bigram of positive
expressions related to learning the fundamentals of social entrepreneurship.
Likewise, the interventions' favorable effect on young people's commitment to
their community is confirmed.
Keywords: Social entrepreneurship education; sustainable development;
community intervention.

Community intervention model
1. Introduction
The COVID-19 pandemic hurt the world economy and had an unfavorable effect on social
well-being by increasing the levels of global poverty and inequality both between individuals
and between countries, thus speaking of a more significant loss of income, particularly in the
groups with more vulnerabilities; factors that have increased the probability of economic risk
so the households, the private initiative as for the governments (World Bank, 2022). The
recovery becomes fragile and asymmetric when adding to the consequences of this health
contingency, the current levels of global inflation, the contractions in the supply chains, the
pressures on the labor market, and the over-indebtedness of developing countries (UN, 2022).
The measures have focused on consumption and production but must address the
environmental objectives within the 2030 Global Agenda for Sustainable Development
(CEPAL, 2021).
Agenda adopted in 2015 by the 178 Member States of the United Nations, in which 17
objectives were agreed on whose purpose is economic growth combined with the end of
poverty, peace, and prosperity for both people and the planet (UN,s.f.). The project seeks an
achieve sustainable development that has “the ability to make development sustainable to
ensure that it meets the needs of the present without compromising the ability of future
generations to meet their own needs” (UN, 1987, p.15).
Sustainable development is a complex global goal that social innovation must address. In this
sense, social enterprises represent a possible alternative by attending to problems from a
deeper, more effective, and accessible approach (Lubberink, 2019). Type I Social Businesses,
following the classification of Yunus (2010), i.e., companies capable of generating economic
and financial profitability and sustainable social impact in your communities; because social
enterprises provide the necessary added value for economic reactivation post-COVID-19 and
incorporate the people who make up the Base of the Pyramid (BoP) as the primary agents of
this process (Méndez et al., 2022).
Among the consequences of the pandemic found that the most significant income losses
among young people, women, and independent and temporary workers who had a lower level
of schooling were registered; Likewise, the loss of learning in childhood is substantial and
could increase the disparity in the future (ONU, 2022). Therefore, it is necessary to add to
the dynamics of sustainable development the population that conforms to the BoP. That is,
those people who live in a context of poverty, marginalization, or vulnerability (Wagner et
al., 2018).
Universities play a more active role as social innovators, incorporating teaching models of
social entrepreneurship with a focus on sustainable development through academic and
service-learning activities and training in social entrepreneurship. The latter is necessary to
develop sustainable projects that solve specific social problems (Villa et al., 2021).
1064
Yadira Zulith Flores Anaya, Martín Arturo Ramírez Urquidy, Roberto Iván Fuentes Contreras
From their sense of social responsibility, the universities "must equip students to understand
the new economy and react swiftly to its socioeconomic crises" (García & Montoya, 2021,
p. 1236). For this reason, the Faculty of Economics and International Relations (FEYRI) of
the Autonomous University of Baja California (UABC), located on the northern border of
Mexico, started a social service project supporting type II social businesses (social-based
microenterprises) in 1999, in line with the proposal of Professor Yunus (2010). The project
was transformed in 2017, in UABC-Centro Yunus, for social businesses and well-being; and
currently continues to provide micro-business assistance. In addition, since 2021, the Center
has been carrying out community intervention projects aimed at children, youth, and groups
in a context of vulnerability to promote social entrepreneurship from empathy, following the
guidelines of the SDGs. The purpose is to promote present and future actions for sustainable
development and integrate the community into the knowledge transfer process in a
multidirectional sense. Therefore, this document aims to analyze these interventions' effects
on the population served.
2. Social Entrepreneurship by Sustainable Development

The current scenario requires changes in traditional models and generating strategies for
constructing innovative ideas that involve communities traditionally excluded from the
dynamics of growth and well-being (Méndez et al., 2022). It is not enough to create social
value; social enterprises require efficient and effective scaling of value creation and
appropriation (Lubberink, 2019). The groups that are part of the BoP must be the primary
recipients of the benefits produced by said social innovations. Because, in these cases, the
problem that social entrepreneurs want to solve directly affects the community's quality of
life, they must use market mechanisms to reinvest those benefits in society (Simon &
Rodriguez, 2021). Therefore, the venture must be sustainable to the point where it is possible
to identify, build, and consolidate business ideas that involve the creation and incorporation
into the market of goods and services, as long as they consider economic, social, and
environmental impacts (Villa et al., 2021).
2.1. Social Entrepreneurship Education by Sustainable Development

The reasons why social entrepreneurship education has become popular are related to
growing globalization, the rise of the knowledge economy, and the construction of the
relationship between the State, the market, and civil society; for its part, in this
reconfiguration, education is crucial for the development of skills, competencies, capacities
and personal attributes aimed at economic development focused on sustainable development
(Roussakis et al., 2020). Therefore, in recent years, higher education institutions have
gradually incorporated, within their academic agenda, the promotion of education for social
entrepreneurship (García & Ramírez, 2021).
1065
Consequently, through education in social entrepreneurship, the university seeks to impact

sustainable development not only indirectly but also actively participates in achieving the
objectives. By generating strategies that contribute to achieving objectives, such as objective
4, quality education; goal 8, decent work and economic growth; and objective 10, reductions
of inequalities. Social entrepreneurship education comprises co-curricular activities,
practices, and internships and does classify into three approaches: accommodating and
incorporating social entrepreneurship courses into the curriculum; integrating, adding co-
curricular activities; and immersion, which involves students experimenting with reality and
interacting directly with the community (Thiru, 2011).
2.2. Community Intervention Model for University Social Innovation

The social entrepreneurship education strategy focused on sustainable development make at
UABC-Yunus Centre contemplates the three approaches mentioned above: accommodating,
through the offer of a postgraduate degree in social business and the teaching of the
entrepreneurship course at the bachelor's level, with a focus on sustainable and inclusive
business; as well as integrating and immersion when developing community intervention
projects. Table 1 shows the activities carried out in this regard, during the second semester
of 2022, with a scope of 1,407 people trained in social entrepreneurship and SDGs, aged 2 to
25 years. In addition, in Table 1, it is possible to identify the Center's community intervention
model, which dose made up of five modalities: workshops, short workshops; rallies, circuits
of physical activities and reflection; boot camps, massive multidisciplinary training; fairs,
self-organized events by students of the entrepreneurship subject, as part of their final project;
and community projects, which involve immersion in the community and a direct approach
to the problem and the social actors. The model in all its modalities comprises two strategies:
gamification and learning by doing; the users of the courses and approaches work as a team
to generate ideas of social impact with a vision of the SDGs in a simple, practical, and
experiential way.
The approach with the community is in two ways: in one, it is the researchers and the students
who perform training in situ, either in schools that locate on the outskirts of the city or in
community centers; Likewise, the groups of children and youth in a context of vulnerability,
are the ones who attend the center to receive the activities within the university facilities. The
interventions are directed towards childhood and youth, as they represent one of the most
vulnerable population groups within the BoP, as to promote in them at an early age the
importance of generating social innovation consistently from empathy, putting people and
the planet at the center; in addition to fostering their self-confidence and creativity. Further,
it seeks to bring quality education closer, thereby reducing the inequalities in knowledge
access.
1066
Table 1. Community intervention Model of UABC-Yunus Centre, 2022.
Modality and SDGs1 BoP2 GS3 LD4 Scope

Topic
Workshop: 6: Clean Elementary Snakes and Mural of Blue 15
Ideas´ Ocean Water & Schoolers ladders: How Entrepreneurship
Sanitation to take care of Idea
water
Rally with 17 SDGs Elementary Snakes and Scholar Orchard 120

Community Schoolers ladders about
Project: Blue Ideas,
Sustainable Lottery of
Stations financial
terms and
agricultural
products
Rally: 17 SDGs Middle Statistics: 659

Entrepreneurship Schoolers horse race;
& Sustainable Sustainable
Development ideas: hot Conscious
potato; decision making
Financial
Concepts:
Hopscotch
Rally: Career Plan Five High Roulette: Planting sprouts 27

and The 5 Dimensions Schoolers economy and in reused plastic
Dimensions of the SDGs: SDGs; pots
SDGs People, Questions and
Prosperity, answers:
Peace, public
Partnership administration
& Planet and SDGs,
memoranda:
international
relations and
SDGs
Workshop: 17 SDGs College Ocean of Social Canvas 100

Build the Idea of Students problems
Sustainable
Entrepreneurship
Fair: People College Shark Tank Elevator Pitch & 64
Start for Empathy dimension: Students Social Stand
1,2,3,4 & 5
1067
Workshop: 17 SDGs College Design Social Canvas & 35

Build the Students Thinking, Elevator Pitch by
Entrepreneurship empathy Team
Idea from awareness
Empathy
Bootcamp: 17 SDGs College Design Social Canvas & 315
Entrepreneurship Students Thinking, Elevator Pitch by
& SDG empathy Team
awareness
Community SDGs: College Interviews Self-managed 22

Project: 3,11, 12, 13 Students and projects: beach
Sustainable & 14 infographics cleaning, donation
Fibonacci of menstrual
products,
awareness videos,
and paper
collection
campaign
Community Planet Children & Paint reusable Reusable wooden 50

Project: dimension: Youth: plastic pots tree with reused
Sustainable 6, 12,13,14 House and plastic pots;
Christmas & 15 SDGs Home Lottery of community
(2 to 25 agricultural garden
years) products construction
1407
Source: Own elaboration
1 SDGs: Sustainable Development Goals, 2 BoP: Base of Pyramid, 3 GS: Gamification Strategy, 4 LD: Learning
by Doing
3. Methology and results

Based on the example, the intervention model in education for sustainable social
entrepreneurship needs to do analyzed. Consequently, the research aims to identify the effect
of these actions; the first approach is learning. For this, analysis descriptive on the feedback
provided by a sample of 138 young people who received the same gamification strategy and
learning by doing: Bootcamp and Workshop Build the Entrepreneurship Idea from Empathy;
the training consisted of building multidisciplinary teams (university students of economics,
international relations, gastronomy, engineering, and enology); agreed on a vulnerable
population group, and developed an empathy map; they built a social canvas after receiving
the systemic change workshop, designed a prototype using their creativity and presented a
pitch. At the end of the interventions, the young people wrote notes with comments about the
training. From them, text analysis develops about the most frequent words and the words
used with more excellent proximity. In this line, Figure 1 shows the map of the most frequent
1068
words used by young people where they stand out: positive words related to the generation
of social initiatives from empathy: help, understand, solve, pledge, better, and today.
Figure 1. Social Entrepreneurship Education.
Figure 2 shows the semantic learning network generated from the young people's comments,
where the closeness of the verbs learned and committed highlight in addition to the binomials
generated between actions and terms such as learning and social entrepreneurship (impact,
change, society, solutions, can); commitment, and Sustainable Development (community,
putting, develop).
Figure 2. Semantic Learning Network on Social Entrepreneurship and SDGs
4. Conclusion
The results show that interventions in social entrepreneurship education positively affected
learning the fundamentals of social entrepreneurship. Likewise, the interventions' favorable
effect on young people's commitment to their community is confirmed. On this occasion, the
analysis of the effects was exploratory/qualitative because it started from the student's
feedback, and even though it yielded interesting results that will strengthen the methodology,
does expect that the following stages to carry out a quantitative analysis of the interventions
for evaluating the effectiveness and impacts of the methodology in terms of the development
and implementation of sustainable social enterprises.
1069
The next phase of analysis contemplates the evaluation of the impact of the model on the
intention of social entrepreneurship of the groups served, as well as on the materialization of
these ideas in the generation of social enterprises, type I.
References
Comisión Económica para América Latina y el Caribe, CEPAL. (2021). Cómo financiar el
desarrollo sostenible. Informe especial COVID-19, No. 14.
https://repositorio.cepal.org/bitstream/handle/11362/47720/1/S2100878_es.pdf
García, A. & Ramírez, M. (2021). Social Entrepreneurship Education: changemaker training
at the university. Higher Education, Skills and Work-Based Learning, 11 (5): 1236-
1251. https://doi.org/10.1108/HESWBL-01-2021-0009
Lubberink, R. (2019) Social Entrepreneurship and Sustainable Development. In: Leal, W.,
Azul A., Brandli L., Özuyar P., Wall T. (eds). Decent Work and Economic Growth.
Encyclopedia of the UN Sustainable Development Goals. Springer, Cham.
https://link.springer.com/ referenceworkentry/10.1007/978-3-319-71058-7_47-1.
Méndez, J., Méndez, M. & Bolaños, C. (2022). Emprendimientos sociales y su efecto en la
reactivación económica en tiempos de pandemia. Revista de Ciencias Humanísitcas y
Sociales. 7 (3): 31-45. https://doi.org/10.33936/rehuso.v7i3.5149.
Roussakis, Y., Brinia, V. & Dragonas, T. (2020). Education in Social Entrepreneurship for
Sustainable Development, A Case Study in Teacher Education at the University of Atens,
Greece. In: Brinia, V. & Davim, P. (eds). Designing an Innovative Pedagogy for
Sustainable Development in Higher Education. Taylor & Francis Group.
Simón, V. & Rodríguez, M. (2021). The Emergence of Social Entrepreneurship, Individuals
and Social Ventures as Agents of Change. Springer.
Thuri, Y. (2011). Social Enterprise Education: New Economics or a Platypus?. Social and
Sustainable Entrepreneurship, 13: 175-200 https://doi.org/10.1108/S1074-
7540(2011)0000013011.
Union Nations, UN. (2022). The Sustainable Development Goals Report 2022.
https://unstats.un.org/sdgs/report/2022/The-Sustainable-Development-Goals-Report-
2022.pdf
Union Nations, UN. (1987). Our Commun Future. Report of the World Commission on
Environment and Development.
https://www.are.admin.ch/are/en/home/media/publications/sustainable-
development/brundtland-report.html
Union Nations, UN. (s.f.). Sustainable Development. https://sdgs.un.org/
Villa, A., Arias, A. & Peña, M. (2021). Un modelo de formación para desarrollar el
emprendimiento social. Educar. 57(1): 97-116. https://doi.org/10.5565/rev/educar.1153
Wagner, D., Wolf, F. & Boruch, R. (2018). Learning at the bottom of the pyramid. UNESCO.
https://unesdoc.unesco.org/ark:/48223/pf0000265581
World Bank. (2022). World Development Report 2022: Finance for an Equitable Recovery.
doi:10.1596/978-1-4648-1730-4
Yunus, M. (2010). Building Social Business. PublicAffairs: New York.
1070
Exploring second language viewers’ use of cognitive strategies in

learning Chinese through multimedia learning resources with
captions and social annotations
Shuzhou Lin1, Ting Wang1, Wei Wei2, Jiahui Yang3, Sara Cañizal Sardón1
1
University International College, Macau University of Science and Technology, Macau,
China, 2Faculty of Applied Sciences, Macao Polytechnic University, Macau, China,
3
Departamento de Filología Inglesa (Lengua Inglesa), Universidad de Sevilla, Sevilla, Spain.
Abstract
Although linguistic captions are perceived as helpful in developing L2
vocabulary knowledge, the use of social annotations in video learning remains
underinvestigated. This study investigated L2 learners’ use of cognitive
strategies in learning through multimedia resources in a MOOC context when
both caption and social annotations modes were available. Triangulated data
were collected from thirteen L2 MOOC learners from Africa, including think-
aloud data, post-video interviews, and their notes. Findings suggest that 1)
captions and social annotations lead to different cognitive strategies and
videos with social annotations are found more engaging, 2) captions facilitate
L2 viewers to conduct more bottom-up listening strategies at sentence level,
while social annotations facilitate more top-down listening strategies, and 3)
social annotations are used not only to enhance and expand the understanding
of the video, but also to create a sense of belonging and further motivate
viewers to achieve a higher level of engagement.
Keywords: Social annotation; caption; multimedia learning; cognitive
strategies; Cognitive Theory of Multimedia Learning (CTML).

1. Introduction
This study aims to investigate the cognitive strategies adopted by L2 learners in multimedia
learning with captions and social annotations. Social annotations refer to the comments
created collaboratively by users on the same online document (Sun, Hwang, Yin, Wang, &
Wang, 2022), and they focus on the content of the document. In a digital reading and writing
environment, social annotations are perceived as beneficial to learners in terms of 1)
increasing motivation and engagement levels (Yang, Zhang, Su, & Tsai., 2011), 2) better
conditions for improving reading comprehension and writing quality (Li & Lai, 2022; Yang
et al., 2011), 3) better opportunities for peer feedback (Chang & Windeatt, 2016), and 4)
promotion of knowledge sharing (Yang et al., 2011). However, linguistic-oriented captions,
also known as subtitles, are also widely promoted as pedagogical and scaffolding tools for
second language viewers in multimedia learning environments, including L2 subtitles
(Mahdi, 2017), hyperlinks (Garrett-Rucks, Howles, & Lake, 2015), and pictorial information
(Aldera & Mohsen, 2013). Contributions of this study include 1) a better understanding of
how linguistic-related captions and content-related social annotations affect L2 viewers’ use
of cognitive strategies in video learning, and 2) the development of more effective, engaging,
and beneficial multimedia language learning resources equipped with captions and social
annotations based on L2 learners’ cognitive learning processes.
Annotations are often used as visual aids to facilitate learning in multimedia learning
environments (Unal, 2021; Mohsen, 2016). In the context of applied linguistics or
educational research, social annotations refer to 1) personal comments on a specific item
and/or peers’ annotations (Li & Lai, 2022; Sun et al., 2022; Yang et al., 2011), 2) questions
and answers to learning materials (Yang et al., 2011), and 3) spontaneous thoughts and
reflections (Li & Lai, 2022). The taxonomy of social annotations includes views (Li & Lai,
2022), judgments and evaluations (Pham, 2021), and emotions and feelings (Chang &
Windeatt, 2016). Conventionally, social annotations are examined in the context of digital
reading and collaborative writing (Pham, 2021; Chang & Windeatt, 2016). In the fields of
computer science and electronic engineering, social annotations are sometimes referred to as
‘barrage’ or ‘bullet screen’ (Chen, Zhou, & Zhi, 2019).
There are three main discussion threads in previous studies on annotations in multimedia
learning. Apart from the first thread which investigates the link between the incidental
learning of vocabulary knowledge and the availability of annotations in a multimedia learning
environment, the second one targets annotations based on topic and content, and concludes
that 1) topic-level annotations are not as effective as lexical annotations for promoting
vocabulary learning (Unal, 2021), and 2) content-related social annotations significantly
1072
improve learners’ writing quality and peer commenting skills (Pham, 2021). For example,
Unal (2021) investigated how lexical and topic-level annotations and working memory
capacity affect incidental vocabulary learning, and concluded that lexical annotations result
in better performance in vocabulary meaning recall. Finally, the third thread investigates the
impact of social annotations on learners’ emotions, motivation, and engagement. Social
annotations are found to 1) increase learners’ motivation and engagement (Li & Lai, 2022;
Yang et al., 2011); 2) develop learners’ confidence, a sense of community, and mutual trust
(Sun et al., 2022; Chang & Windeatt, 2016). For instance, Li and Lai (2022) compared the
effects of social annotations and online forums on L2 learners’ online collaborative writing,
and found that social annotations not only led to better learning outcomes but also enhanced
learners’ engagement levels.
To fill the gaps in the research of multimedia learning, this study asked: what cognitive
strategies are used by L2 viewers when captions and social annotations are provided in
multimedia learning? The Cognitive Theory of Multimedia Learning (CTML), proposed by
Mayer (2009), was adopted as the theoretical framework of this study. According to CTML,
the cognitive process of multimedia learning includes: 1) selecting, which means selecting
relevant verbal and pictorial information from multimedia resources for processing in
working memory; 2) organizing, which means organizing the selected verbal and pictorial
information into the coherent verbal and pictorial models, thus leading to the creation of
internal connections; 3) integrating, which means building external connections by
integrating the verbal and pictorial models with prior knowledge.
3. Research design
3.1. Research context

Thirteen (6 females and 7 males) African students from a three-year learning Chinese as a
second language program were recruited for this study. The participants were second-year
international students from a vocational college, aged between 18 and 24. Their Chinese
proficiency level was around HSK level 3 (equivalent to The Common European Framework
of Reference for Languages (CEFR) level B1). Based on the MOOC platform, this program
is called Improve Chinese Communication through Multimedia Learning. Several carefully
chosen Chinese language videos are included in this 8-week program, with the goal of
enhancing the communication and proficiency of international students in Chinese. Both
linguistic-based captions and social annotations (comments generated by native Chinese
viewers) are provided along with the videos. Due to the pandemic, participants had been
learning online (on the MOOC platform) for at least 12 months, and they were familiar with
the operating system.
1073

The experiment was conducted on a MOOC platform where students voluntarily video-
recorded the whole learning and data collection process with an extra phone. Think-aloud
interviews, observations, and post-video interviews were used to gather the data for this
study. Participants were informed about the purpose of the study and the procedures of data
collection, and then demonstrated how to do think-aloud and control the operating system on
the MOOC platform. They were required to watch a two and half-minute long Chinese video
about spicy Chinese cuisine. With no time limit, they were allowed to pause, rewind, fast-
forward, and restart the video, take photos or notes, and use online dictionaries on their
cellphones. Moreover, they were given control over access to the type of on-screen text,
which means that they could manually switch between two modes: caption mode and social
annotation mode. In the caption mode, viewers can see the transcripts of the video, while in
the annotation mode, they can access comments made by L1 viewers about the video's subject
matter and their feelings toward it.
Post-video interviews were conducted with the same predetermined questions following each
participant's completion of the video they had just watched in order to gather information on
the participants' cognitive processes. The interview questions were designed based on the
cognitive processes from CTML (Mayer, 2009).
3.3. Data analysis

Participants watched the video about four times on average, thus about four short post-video
interviews were conducted for each participant. Participants reportedly selected and
deselected various types of information in the video while in caption mode and social
annotation mode. Based on the three cognitive processes from CTML (Mayer, 2009), think-
aloud and post-video interview data were coded deductively first. Observation data such as
time spent on each video, the number of pauses, and notes taken were collected and analyzed,
as these learning behaviors in two modes indicate different cognitive processes and strategies
in multimedia learning. For example, the duration of video learning may suggest learners’
motivation and engagement levels under different learning modes.
4. Findings
4.1. Quantitative data

Table 1 presents the results of the descriptive data analysis. Overall, to learn the two-and-a-
half-minute-long instructional video, the duration of video learning in social annotation mode
was longer than in the caption mode. Post hoc analyses were conducted using Bonferroni's
post hoc test. Participants took significantly longer time to learn videos with social
annotations (M = 305 seconds, SD = 62.798) than with linguistic-oriented captions (M =
1074
232.32 seconds, SD = 101.584). Moreover, they also paused the video with social annotations
more frequently (M = 7, SD = 4.619) than captions (M = 1.62, SD = 1.387). No statistically
significant difference was found in the number of notes taken under two different modes.
Since participants were encouraged to watch the videos as many times as they wanted, a
longer time spent on videos with social annotations may suggest a higher engagement level.
Therefore, the descriptive data suggest that captions and annotations lead to very different
cognitive strategies, levels of motivation and engagement.
Table 1. Descriptive data
Mode N Mean SD Sig.
Duration Caption 13 232.31 101.584 0.000
Social annotation 13 305.38 62.798

Pauses Caption 13 1.62 1.387 0.000
Notes Caption 13 1.77 1.878 0.134
4.2. Cognitive strategies to cope with linguistic-oriented captions

The different engagement levels between the two modes can be explained by the different
cognitive strategies reported by the L2 viewers during three periods: selecting, organizing
and integrating.
As for the caption mode, the viewers reported that in the selecting period, they paid attention
to three main resources. First of all, pictorial information from images was used to help them
1) understand the unfamiliar vocabulary and phrases, 2) make sense of the sentence plots,
and 3) facilitate their understanding of the details of the video content. Secondly, audio
information was selected from the narration to 1) learn the pronunciation of unknown words,
and 2) assist in the understanding of video content. Thirdly, textural information from
captions was selected to 1) learn the characters of unfamiliar words and phrases, and 2)
support their understanding of the details.
In the organizing period, the cognitive strategies mainly involved: 1) organizing textual
information from a target word and its surrounding words into a verbal model to make sense
of the less familiar words or phrases; and 2) organizing pictorial and audio information with
linguistic captions to learn, check, or enhance their understanding of less familiar words or
phrases, and content details. For example, one participant reported that:
1075
(S2) When I heard a new word, I paid attention to the images, and focused on what
he meant, but I also needed the characters in the captions, otherwise I couldn’t
understand the new word.
In the final period, what they had newly learned from the video was reported to be integrated
with 1) background knowledge, such as already known words and prior knowledge of
Chinese cuisine, and 2) personal experiences. The integration helped them to further turn
working memory into long-term memory. One participant explained:
(S12) To learned the new word “蒸” (to steam), the images helped me learn the word
but I need to listen to it …When I learned this new word in steamed tofu, I thought
about other delicious foods like dumplings and buns, which were all cooked in the
same way.
4.3. Cognitive strategies to cope with social annotations

Under the social annotation mode, participants showed great interest in the comments
generated by native Chinese viewers. In the selecting period, they paid attention to 1) textual
information from social annotations to enhance and expand their understanding of the video,
and 2) pictorial information from comment-related images to make sense of the comments.
As there were numerous scrolling lines of comments, it was discovered that they selectively
read the social annotations and ignored the audio commentary.
In the organizing period, participants were found to organize the verbal information and
pictorial information in three ways: 1) organizing textural information from social
annotations to understand the content of comments, and learning how to write social
annotations properly to communicate with L1 viewers; 2) organizing textural information
from contradictory social annotations to evaluate the comments and video content; 3)
organizing relevant pictorial information from the images to assist in the learning of social
annotations. The quote below demonstrates how the verbal model and pictorial model were
organized:
(S9) The comment said: “Tofu sweats a lot.” In the video, I can’t see tofu sweating
a lot, so it seems that this viewer has tried cooking tofu before and this comment
tells you the possible problem.
After the verbal and pictorial models based on social annotations were organized, they were
integrated with 1) video content learned earlier, 2) similar or contradictory experiences in L1
and L2 contexts, and 3) cultural and learning experiences. Moreover, more than half of the
participants tried to generate comments to interact with the chef and L1 viewers, and reported
that they were happy to see others holding the same opinions as their own, which gave them
a sense of belonging. The quote below suggests how this external connection was built:
1076
(S7) This comment also questioned the use of sugar. In my country, if we want a
sweet dish we only add sugar, and if we want a sour dish we only use vinegar. We
don’t mix sugar with chilli or vinegar. When I see others having the same question,
I feel that I am not alone.
5. Discussion and implications

To comprehend the captions and social annotations, different cognitive strategies were used
by L2 learners to select, organize and integrate the textual, video and audio information.
In caption mode, participants selected verbal information and pictorial information, and then
organized them into verbal models and pictorial models to learn the unfamiliar words and
some details about the content. This indicates that detailed internal connections between the
verbal and pictorial models were built at both the linguistic and content levels (Mayer &
Moreno, 2003). In relation to the learning of new words, participants were found to select the
pronunciation from the audio narration, characters from captions, and relevant images. They
then organized this information and integrated it with their background knowledge and
personal experience to enhance their understanding of the words. In return, a better
comprehension of the details of the captions at the sentence level added more detail to
viewers’ understanding of the video, such as the ingredients and cooking procedures. In other
words, a bottom-up model can be observed when L2 viewers interplay with linguistic
captions in instructional videos at the sentence level (Vandergrift, 2004).
In the mode of social annotations, participants selected, organized and integrated information
for evaluating the content of social annotations, learning how to make comments in L2
properly, and communicating with L1 viewers. They integrated the verbal and pictorial
presentations of social annotations with knowledge learned earlier in the video, personal
experiences, and other background knowledge, to further confirm and evaluate the social
annotations selected and content details. It suggests that both internal and external
connections were made for social annotation learning (Mayer & Moreno, 2003). The overall
understanding of the video and social annotations from other viewers, including personal and
cultural interpretations of the video, was used to evaluate and confirm some other details in
the video, which fits in the top-down model (Vandergrift, 2004). Both quantitative and
qualitative data suggest that social annotations lead to higher engagement and motivation
levels, which is in accordance with the conclusions of previous studies (Li & Lai, 2022; Yang
et al., 2011). For example, longer learning time and more pauses were recorded under the
mode of social annotation, serving as evidence of higher learning engagement and
motivation. Social annotations can also create a sense of belonging for L2 learners (Chang &
Windeatt, 2016) and encourage them to interact with L1 viewers.
1077
At least two pedagogical implications can be summarized. To begin with, giving learners the
autonomy to select captions or social annotations based on their needs may accommodate
their individualized learning goals. In addition, meaningful, comprehensible, and topic-
relevant social annotations can enhance their understanding of video and lead to higher
learning engagement and motivation levels.
References
Aldera, A. S., & Mohsen, M. A. (2013). Annotations in captioned animation: Effects on
vocabulary learning and listening skills. Computers & Education, 68, 60-75. doi:
10.1016/j.compedu.2013.04.018
Chang, H., & Windeatt, S. (2016). Developing collaborative learning practices in an online
language course. Computer Assisted Language Learning, 29(8), 1271–1286. doi:
10.1080/09588221.2016.1274331
Chen, G., Zhou, S., & Zhi, T. (2019). Viewing mechanism of lonely audience: Evidence from
an eye movement experiment on barrage video, Computers in Human Behavior, 101, 327-
333. doi: 10.1016/j.chb.2019.07.025
Garrett-Rucks, P., Howles, L., & Lake, W. M. (2015). Enhancing L2 reading comprehension
with hypermedia texts: Student perceptions. CALICO Journal, 32(1), 26-51. doi:
10.1558/calico.v32i1.26131
Li, S. C., & Lai, T. K. (2022). Unfolding knowledge co-construction processes through social
annotation and online collaborative writing with text mining techniques. Australasian
Journal of Educational Technology, 38(1), 148-163. doi: 10.14742/ajet.6834
Mahdi, H. S. (2017). The use of keyword video captioning on vocabulary learning through
mobile-assisted language learning. International Journal of English Linguistics, 7(4), 1-
7. doi: 10.5539/ijel.v7n4p1
Mayer, R. E. (2009). Multimedia learning. Cambridge: Cambridge University Press. doi:
10.1017/CBO9780511811678
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia
learning, Educational Psychologist, 38(1), 43-52. doi: 10.1207/S15326985EP3801_6
Mohsen, M. A. (2016). Effects of help options in a multimedia listening environment on L2
vocabulary acquisition. Computer Assisted Language Learning, 29(7), 1220-1237. doi:
10.1080/09588221.2016.1210645
Pham, V. P. H. (2021). The effects of lecturer’s model e-comments on graduate students’
peer e-comments and writing revision. Computer Assisted Language Learning, 34(3),
324-357. doi: 10.1080/09588221.2019.1609521
Sun, C., Hwang, G. J., Yin, Z., Wang, Z., & Wang, Z. (2022). Trends and issues of social
annotation in education: A systematic review from 2000 to 2020. Journal of Computer
Assisted Learning, 1-22. doi: 10.1111/jcal.12764
Unal, B. (2021). Glossing and incidental vocabulary learning in L2 reading: a cognitive load
perspective. International Review of Applied Linguistics in Language Teaching. Advance
online publication, 000010151520200164. doi: 10.1515/iral-2020-0164
1078
Vandergrift, L. (2004). 1. Listening to learn or learning to listen?. Annual Review of Applied

Linguistics, 24, 3-25. doi: 10.1017/S0267190504000017
Yang, S., Zhang, J., Su, A., & Tsai, J. (2011). A collaborative multimedia annotation tool for
enhancing knowledge sharing in CSCL, Interactive Learning Environments, 19, 45-62.
doi: 10.1080/10494820.2011.528881
1079
Diversity in (word) meaning: reducing the risk of bias in foreign

language vocabulary teaching using prototype theory
Anna Malena Pichler
Department of German as a Foreign Language, Technische Universität Berlin, Germany.
Abstract
In multifaceted and pluricultural settings, foreign language teaching is facing
new challenges. This paper proposes a way to bring the concept of
transcultural awareness to vocabulary teaching through the theoretical
framework of prototype theory within Cognitive Semantics. Selected aspects of
prototype theory are used in order to find suitable points of contact that can
account for diversity in meaning. This paper argues towards an awareness of
diversity in semantic meaning, using the understanding of prototype theory
that categories are complex and have fuzzy boundaries. Therefore, learners
can be taught a different and multifaceted understanding of word meaning on
this theoretical foundation. Exploring and using prototype theory in this
manner could pave the way towards teaching diversity in the foreign language
classroom on a lexical level.
Keywords: Vocabulary teaching; transcultural awareness; prototype theory;
diversity.

Diversity in (word) meaning: reducing risk of bias in vocabulary teaching using prototype theory
1. Introduction
Foreign language teaching has come a long way since traditional approaches have slowly
been replaced by newer methods. In a globalised world, culture and language can no longer
be conceptualised within borders of nations. Languages can no longer be restricted to a
tangible culture, rather we operate within a network of cultures that do not have clear-cut
confines. Traditional language teaching has long ignored these realities or conveyed them in
stereotypical ways, but newer approaches start to integrate these inter- and transcultural
dimensions. Research in language teaching is now recognising the importance of imparting
communicative skills, with critical incultural communication theories paving the way for a
transcultural communication approach (Baker, 2022).
Another rather new approach within foreign language teaching on the linguistic side is the
adaption of Cognitive Linguistics. There are various studies that apply Cognitive Linguistics
to foreign language teaching (e.g. Holme, 2009; Littlemore, 2009; Tyler, 2012). A few of
these studies apply Cognitive Semantics to language teaching in general or vocabulary
teaching in specific.There is a theory in Cognitive Semantics that appears to be especially
interesting in the field of vocabulary teaching: prototype theory.
Prototype theory, originating in the area of psychology, mainly accounts for a new cognitive
view on categorisation. Categorisation is an important element within a transcultural
approach to language teaching, since learners are taught to maintain a critical position
towards common categorisations (Baker, 2022). It can therefore be hypothesized that
prototype theory within semantics can account for a form of categorisation as well as word
meaning that focuses on diversity aspects. The aim of this paper is to explore how prototype
theory can account for a manner of vocabulary teaching that focuses on diversity in meaning
and therefore reduces bias in this area.
2. Transcultural awareness in foreign language teaching

In a globalised world, societies are increasingly diverse in terms of culture. This impacts all
aspects of society, including education. Nowadays, it is essential to teach diversity in the
classroom (Organisation for Economic Co-Operation and Development [OECD], 2010).
Since learning about culture is an essential part of foreign language teaching (Altmayer,
1997), referring to diversity becomes central. Studies have shown that students perform
significantly better in a learning environment that “treats diversity as a source of potential
growth rather than an inherent hindrance to student performance” (OECD, 2010). Hence,
diversity should be a central aspect in the foreign language classroom.
This is where the concept of transcultural awareness comes into play. This approach aims to
teach the ability to communicate successfully in a transcultural environment, which covers a
1082
Anna Malena Pichler
much wider field than traditional approaches (Baker, 2022). Rather than being a monolateral
approach, transcultural awareness in language teaching is a multifaceted compilation of
different strategies. It is based on the relatively new concept of transcultural communication
which Baker and Ishikawa (2021) describe as follows: “Transcultural communication
attempts to develop more critical approaches to language and culture […] to better account
for the complex and fluid relationships between languages and cultures in interaction […].”
A central idea here is that cultural boundaries are fluid and transgressive and do not conform
to national boarders (Baker & Ishikawa, 2021). Therefore, language teaching needs to adapt
to these new theories, considering the risk of bias and stereotyping while teaching about
culture. Bias and stereotyping can be significantly reduced if learners are aware of cultural
fluidity and diversity. Baker and Ishikawa (2021) note that we have to “fundamentally
rethink” education since especially in higher education settings we are dealing with a “super-
diversity”. They emphasize that students need to be prepared for “multilingual and
multicultural settings”.
3. Prototype theory in lexical semantics and diversity in meaning

Prototype theory in the field of Cognitive Linguistics can be understood as a theory that
accounts for the description of meaning within a holistic understanding of language.
Originally it was a theory of categorisation, and it only became relevant on a semantic level
with the transfer to semantics (Dörschner, 1996). Rather than a singular theory, it is an
accumulation of various ideas within cognitive studies (Schmid, 2000).
A central idea in prototype theory is that categorisation as a basic human thought pattern
facilitates a structured perception of the world (Windeck, 2019). In its more modern version,
the prototype is defined as the most typical example of a category which is to be understood
in a broad sense. A prototype here is an abstract concept that includes the basic meaning of
a category; the boundaries of the categories are fuzzy (Mangasser-Wahl, 2000a). Lakoff
(1987) notes that linguistic categories work in the same way as conceptual categories,
therefore language uses basic cognitive categorisation strategies. It is assumed that
prototypes work similar to basic categorisation. Humans automatically categorise when
perceiving the world they live in because it works as a cognitive “shortcut” (Mangasser-
Wahl, 2000b). Neurological studies show that categorisation functions as a universal
principle of cognition so it can be assumed that for something as complex as language there
must exist a multilevel and hierarchical system of categorisation (Müller & Weiss, 2000).
Brdar-Szabó and Brdar (2000) assume that a prototype approach can be used in foreign
language teaching as “a basic component of the methodological apparatus of a theory” but
not as a “self-contained language theory”. This assumption leads to the aim of this paper. We
have already seen that prototype theory is a relevant approach to lexical semantics and,
1083
therefore, it can be hypothesized that the application to vocabulary learning in specific can
facilitate a more effective understanding and retention of the items. It can further be
hypothesized that prototype theory can account for a certain diversity in the meaning of
lexical items. This can be accomplished through teaching diversity awareness for word
meanings through the findings of prototype theory.
4. Teaching diversity and reducing bias in vocabulary teaching through

prototype theory
The goal of a transcultural language education approach is to educate the learners to use the
target language effectively in communicative situations that occur in multicultural contexts.
This produces a holistic view of communication that entails a greater range of competencies
than the communicative approach focuses on (Baker, 2022). When Kramsch (2021) is talking
about “Language as Symbolic Power”, this is what we can derive for language teaching: she
gives us a new perspective on language education altogether which she approaches from a
post-structuralist point of view. Language learning is no longer about the target language and
target culture but rather about communicative strategies and seeing what effects an utterance
can have in transcultural settings (Kramsch, 2021). Kramsch (2021) summarises: “A post-
structuralist/post-modernist approach to language education will require developing learners’
interpretative abilities, sensitivity to context and appreciation of symbolic complexity.” She
continues to explain that teachers now are confronted with the task of teaching about
“symbolic conditions” and stresses that there has to be a “re-thinking of the purposes of
language teaching within a (post-)humanistic education” (Kramsch, 2021).
This paper aims to apply this to vocabulary teaching in specific using prototype theory as a
theoretical concept within Cognitive Semantics that is assumed to account for several
problems with this proposition. In the following, several aspects of prototype theory that
could be used in order to apply transcultural awareness aspects to vocabulary teaching will
be presented.
4.1. Categorisation
As illustrated in chapter 3, categorisation is a basic human mechanism when perceiving the
world (Kleiber, 1993). Traditional approaches to language teaching, though, use this fact in
a way that often reproduces stereotypes. In order to reduce this risk of bias and stereotyping,
especially in lexemes which denote or describe humans, human interactions, or cultural
practices, this paper suggests a diversity awareness. When we talk about human or cultural
concepts, categorising is inevitable, but with awareness about the plurality of cultures,
cultural practices and human identities, we might be able to gain a less stereotypical view. If
we look at, for example, the word mother we automatically generate an image in our mind
that may be close to a stereotypical understanding. A prototypical understanding of
1084
Anna Malena Pichler
categorisation can help to guide our imagination: we know that next to the stereotypical
mother (e.g. a woman who gave birth to and is raising at least one child) there are also other
members of the category (e.g. a woman or a non-binary person who is raising an adopted
child) that we are aware of in a setting of diversity awareness.
Prototype theory here has to be applied in a very broad sense. Referring to findings within
this area, it has only been proven that it can be applied to concrete objects. However, there
has been evidence that typicality effects are also found for verbs and abstract notions (Lehrer,
1990). Prototype theory negates classic Aristotelian categorisation and offers a theoretical
frame where the existence of common attributes is not necessary for all members of a
category. Rather than assuming logical categories, categories are described as natural.
Criteria for categorisation are not structured in a binary way, rather it is about variation,
global similarities and a comparison with referential prototypes (Kleiber, 1993). Prototype
theory assumes categories with fuzzy boundaries where members are arranged on a scale
from core to periphery; a category has a prototypical inner structure. The decision that an
entity is a member of a certain category is always made globally and not analytically (Kleiber,
1993). If we adapt this manner of thinking to categories that refer to humans or cultural
practices, we automatically create the possibility of more diversity in our conceptual
categories. This is a kind of awareness that should be taught in the foreign language
classroom in order to help create diversity awareness.
4.2. Semantic explanations

Wierzbicka (1990) observes that within linguistics, it is often assumed that abstract concepts
such as emotions cannot be explained but only be felt. She gives the example of envy. Within
traditional, structuralist approaches, a full semantic description is difficult since it has been a
difficult task to properly describe emotions. Wierzbicka (1990) argues that prototype theory
can resolve this problem. If we grasp emotions through a prototype approach, we do not have
to construct unwieldy traditional definitions. Rather, we can explain the concept using
prototypes. Recipients then can compare their emotions or their understanding of them to the
prototypical description and hence get closer to the semantic meaning. Wierzbicka (1990)
concludes that concepts are often vague but according to prototype theory, the explanation
of semantic meaning does not have to be. These assumptions can be used for vocabulary
teaching. Prototype theory here can pave the way to a construction of semantic explanations
within teaching contexts that are concrete and understandable but also leave room for
possible other meanings in other contexts.
4.3. Economy in speech

The tendency to use language in an economical way is as elemental in human cognition as is
the tendency to categorise. It can often be observed that humans tend to choose single lexical
1085
items over long phrases when it comes to their language usage (Lehrer, 1990). Lehrer (1990)
explains that prototype theory plays a vital role here since “it facilitates this kind of
economy”. Furthermore, Rosch (1978) explains the concept of “cognitive economy” as a
basic principle of category formation. Roelcke (2002) corrects this principle and introduces
the principle of “efficiency of communication” which explains that within communicative
processes we tend to either optimise the effort or the result in order to reach efficiency. Within
prototype theory we can see that the goal is to reduce the high number of characteristics that
we would have to assume in traditional semantics. This kind of economy could be an aspect
taught to language learners. If they are aware of human tendencies to use speech
economically, a better understanding for a possible multitude of word meanings is created.
This means that when learners are aware that economy principles are used when substituting
a bulky phrase by one or more lexical items, it is logical to assume that these items can adhere
to meaning in a broad sense with fuzzy boundaries. The contribution towards diversity
awareness here is similar to the one mentioned above: the learners are made aware of
diversity in meaning.
4.4. Social stereotypes

Lakoff (1987) points out the risk of social stereotypes being used as prototypes to represent
entire categories. Nevertheless, he remarks that such social stereotypes are mostly conscious
and change with time. Because they “define cultural expectations”, they are often used as
arguments in debates. A goal in a transculturally aware language classroom would be to
identify these stereotypes and analyse them critically. Prototype theory can help with this
since it offers a way to identify prototypes. Prototypes in this respect are cognitive reference
points that are identified by speakers in a cultural realm and therefore the typicality
judgements will match. It is then the task of critical analysis within a transculturally aware
classroom to work towards plurality and against stereotyping.
4.5. Diversity in meaning

The assumptions of prototype theory enable the re-integration of aspects of meaning that a
traditional approach to word meaning sees as irrelevant, since in that theory they are not
relevant for categorisation (Kleiber, 1993). Lehrer (1990) argues that an important
consequence of prototype semantics is that “concepts, and therefore word meanings do not
(always or even often) have sharp boundaries”. She further deduces that prototype theory
leads to the assumption that vocabulary is flexible. This enables us to use a finite number of
lexical items to account for an indefinite number of senses and meaning and it also means
that word meanings are often extended for non-prototypical meanings (Lehrer, 1990).
This finding is an important connecting point for transcultural awareness in vocabulary
teaching and therefore for diversity awareness. If we assume that categories and word
1086
Anna Malena Pichler
meanings have prototypical members we automatically surmise the existence of non-

prototypical members as well. Fuzzy category boundaries assume a certain diversity in the
members that we can count to those categories. In this theoretical framework, we do not need
a checklist of characteristics anymore to count an entity as a category member or not. This
can be especially important when talking about lexical items that refer to humans or cultural
practices. Diversity awareness, that prototype theory can account for, can nurture a sense of
diversity in meaning which fosters a broader view on people and culture in multicultural
settings.
Certainly, this entails the danger of further establishing binary oppositions that are explicitly
contrasting with the aims of a diversity approach to education. It has to be the task of language
teachers who enforce diversity awareness to educate the learners towards plurality and not
towards newly created binaries.
5. Conclusion
Baker (2015) proposes a radical change of foreign language teaching in the light of “post-
normative and intercultural perspectives”. He argues that language teaching has to move
away from teaching static linguistic forms and rather needs to equip language learners with
communication strategies that help them communicate in a multifaceted environment. This
paper proposed the idea of bringing this goal to vocabulary learning through the theoretical
framework of prototype theory. It was argued that certain findings of prototype theory,
mainly within categorisation and cognitive economy, can be used to work towards teaching
diversity awareness within vocabulary teaching.
Futher research is needed in order to find out how this can be applied in practice and to build
a broader theoretical foundation. Another question of importance will be the didactic
implementation of teaching a variety of meaning and senses within word meaning so that
diversity can be applied beyond the focus on only prototypical members of categories.
Answering these questions is part of a bigger project in form of a doctoral thesis where a
model of vocabulary teaching based on selected theories of Cognitive Semantics is proposed.
References
Altmayer, C. (1997). Zum Kulturbegriff des Faches Deutsch als Fremdsprache. Zeitschrift
für interkulturellen Fremdsprachenunterricht, 2 (2), 1-25.
Baker, W. (2015). Culture and identity through English as a lingua franca. Rethinking
concepts and goals in intercultural communication. Berlin & Boston: Walter de Gruyter.
Baker, W. & Ishikawa, T. (2021). Transcultural communication through global Englishes.
An advanced textbook for students. Oxon & New York: Routledge.
1087
Baker, W. (2022). Intercultural and transcultural awareness in language teaching.

Brdar-Szabó, R., & Brdar, M. (2000). Grammaticalization and the lexicon: Core-and-
periphery model vs prototype approach. In M. Mangasser-Wahl (Ed.), Prototypentheorie
in der Linguistik: Anwendungsbeispiele – Methodenreflexion – Perspektiven (pp. 139-
160). Tübingen: Stauffenburg.
Dörschner, N. (1996). Lexikalische Strukturen. Wortfeldkonzeption und Theorie der
Prototypen im Vergleich. Münster: Nodus Publikationen.
Holme, R. (2009). Cognitive linguistics and language teaching. The Routledge handbook of
cognitive linguistics. Basingstoke: Palgrave MacMillan.
Kleiber, G. (1993). Prototypensemantik: Eine Einführung. Tübingen: Narr.
Kramsch, C. (2021). Language as symbolic power. Cambridge: Cambridge University Press.
Lakoff, G. (1987). Women, fire, and dangerous things. What categories reveal about the
mind. Chicago: University of Chicago Press.
Lehrer, A. (1990). Prototype theory and its implications for lexical analysis. In S. L.
Tsohatzidis (Ed.), Meanings and prototypes. Studies in linguistic categorization (pp. 368-
381). London, New York: Routledge.
Littlemore, J. (2009). Applying cognitive linguistics to second language learning and
teaching. London: Palgrave Macmillan.
Mangasser-Wahl, M. (2000a). Von der Prototypentheorie zur empirischen Semantik.
Dargestellt am Beispiel von Frauenkategorisierungen. Frankfurt a.M.: Peter Lang.
Mangasser-Wahl, M. (2000b). Roschs Prototypentheorie – Eine Entwicklung in drei Phasen.
In M. Mangasser-Wahl (Ed.), Prototypentheorie in der Linguistik: Anwendungsbeispiele
– Methodenreflexion – Perspektiven (pp. 15-32). Tübingen: Stauffenburg.
Müller, H. M., & Weiss, S. (2000). Prototypen und Kategorisierung aus neurobiologischer
Sicht. In M. Mangasser-Wahl (Ed.), Prototypentheorie in der Linguistik:
Anwendungsbeispiele – Methodenreflexion – Perspektiven. Tübingen: Stauffenburg.
Organisation for Economic Co-Operation and Development [OECD]. (2010). Educating
teacher for diversity: meeting the challenge. Paris: OECD. doi: 10.1787/20769679.
Roelcke, T. (2002). Kommunikative Effizienz. Eine Modellskizze. Heidelberg: Winter.
Rosch, E. (1978). Principles of categorization. In E. Rosch & B. B. Lloyd (Eds.), Cognition
and categorization (pp. 28-49). Hillsdale, New Jersey: Lawrence Erlbaum Associates.
Schmid, H.-J. (2000). Methodik der Prototypentheorie. In M. Mangasser-Wahl (Ed.),
Prototypentheorie in der Linguistik: Anwendungsbeispiele – Methodenreflexion –
Perspektiven (pp. 33-54). Tübingen: Stauffenburg.
Tyler, A. (2012). Cognitive linguistics and second language learning. Theoretical basics and
experimental evidence. New York & Oxon: Routledge.
Windeck, A. (2019). Kategorisierungsmodelle und Prototypizität bei Abstrakta. Eine empirische
Untersuchung zur Kategorienkohärenz von Nomina Abstrakta. Berlin: Logos Verlag.
Wierzbicka, A. (1990). “Prototypes save”: on the uses and abuses of the notion of “prototype”
in linguistics and related fields. In S. L. Tsohatzidis (Ed.), Meanings and prototypes.
Studies in linguistic categorization (pp. 347-367). London, New York: Routledge.
1088
Enhancing language self-efficacy of EFL university students

through experiential learning: a study of the learning league
project
Center for Language Education, The Hong Kong University of Science and Technology,
Hong Kong.
Abstract
The Learning League project aims to study the improvement of language self-
efficacy among English as a foreign language (EFL) university students in
Hong Kong after delivery English language classes to secondary school
students in a 12-week programme. The project adopts a blended learning
approach, utilizing the learning-by-teaching pedagogy and gamification to
enhance students' motivation and English speaking skills. 13 university
students were recruited as student tutors. They received teaching training
before teaching English to 63 secondary school students. English speaking-
related weekly tasks and selected materials were provided to the student tutors
to complete and read before they taught. The language self-efficacy of
university students were measured using language self-efficacy scale (Wong,
2005). The programme evaluation showed positive results for the university
students’ English speaking skills and language self-efficacy.
Keywords: Experiential learning; English proficiency; language self-efficacy;
tertiary education; gamification; blended learning.

Enhancing language self-efficacy of EFL university students through experiential learning
1. Teaching and Learning English as a Second Language (ESL) Under the

Pandemic
1.1. Impact of the Pandemic on ESL Teaching and Learning

The Hong Kong education sector has been facing significant challenges under the COVID-
19 pandemic. Despite the government's support for funding e-learning initiatives since the
1990s by funding such as Teaching Development and Language Enhancement Grant
(University Grants Committee, 2022), Quality Education Fund (Hong Kong SAR
Government, 1997), Quality Enhancement Grant Scheme (Legislative Council, 2008),
schools and universities were unable to fully utilize these resources during the pandemic. The
transition to online teaching and learning was difficult as existing resources were often
inadequate and led to extra preparation work for educators (Tang, 2020). The learning of
foreign language speaking skills by Hong Kong students, especially English which is highly
valued in both education context and workplace, was hindered (Ching, 2020) because e-
learning has been assistive. The switch from face-to-face to online classes is beyond change
of mode of lesson delivery. Both in-class and out-of-class learning opportunities need to be
redesigned. It was challenging for university and secondary schools to create contexts for
their students to use the language beyond regular classroom when students stayed home for
lessons (Mok, Xiong, and Bin Aedy Rahman, 2021) where even opportunities for English
corridor talks were gone. The English speaking assessment of university entry examination
(Hong Kong Diploma of Secondary Education Examination, HKDSE) was cancelled for
three consecutive years due the pandemic (Tsang & Yiu, 2022). There were less opportunities
and motivation for both university students and secondary school students to learn and
practice speaking English. There is a need for continued effort to motivate and create
opportunities for students to speak the language beyond the classroom.
1.2. Secondary and University Students’ Motivation Learning EFL Speaking Online
The motivation for learning English speaking among university and secondary school
students declined when learning happened behind the camera during the pandemic. Social-
economic issue had the most impact on students’ learning motivation. Yau, Yeung and Lee
(2022) suggested that camera use concerning teachers’ and students’ presence is an important
factor influencing students’ online experience possibly due to the fact that most Hong Kong
students lived in crowded space with family members. It was even more difficult for students
to concentrate during real-time online teaching (Yeung & Yau, 2022) when students
encounter IT problems such as unstable network connection and device problem. Students
also reported that interaction with other students was rather limited and hence it was difficult
to connect with their peers socially and academically. All these issues in online teaching
discouraged students to improve their English speaking skills.
1090
Since the English speaking assessment for the university entry examination (HKDSE) was
cancelled for 3 consecutive years due to the pandemic, the secondary school teachers spent
more time on training other English skills than speaking in online classrooms. Opportunities
for using English outside classroom, which excite some students, was forgone. According to
the survey and interview conducted by Hong Kong Association for Academic and Teaching
Exchange (Lam, 2022), out of 510 secondary school respondents, 70% of students were most
anxious about English. One of the interviewees said, “When I was in Secondary two, our
school would take us to Stanley to interview foreigners. But the activity was cancelled due
to the pandemic…Since I mostly speak Chinese in daily life, my English pronunciation has
gotten worse." The situation for the underprivileged ones was even worse for schools was
probably the only channel for most of them to expose to English. While the development of
speaking proficiency is related cognitively to other proficiencies of the English language, the
insufficient support has a negative impact on underprivileged students’ English language
public examination results and possibly the chance of entering university.
The lack of meaningful opportunities to speak English, low motivation and insufficient
support exacerbate the problem of low spoken English proficiency among university students
and secondary school students, making it difficult for them to achieve their full potential in
education and beyond.
1.3. Project Needs

The Learning League project aims to address the challenges of lack of opportunities and low
motivation in practising English speaking. The learning-by-teaching program utilized the
skills and knowledge of university (higher EFL proficiency) students to support secondary
school (lower EFL proficiency) students. Through completing online tasks, reading materials
about learning ESL speaking and teaching speaking to secondary school students, the
university students could realise their language gap and improve their English speaking skills.
At the same time, the secondary school students benefited from additional support to improve
their proficiency through practising with the university students. The project has given both
groups an opportunity to use the language in a supportive and empowering environment.
2. Project Design
13 university students were tasked to teach 63 students from three local secondary schools,
ranging from secondary 1 to 5. Each student tutor was assigned 4-6 secondary students to
teach, using materials designed by the project team. The university students delivered 12
face-to-face lessons of 1.5 hours each over a 4-month period.
1091
2.1. University Students’ Profile

13 students from the Hong Kong University of Science and Technology (with IELTS overall
score of at least 6.5) were recruited as student tutors. Among the 13 students, 5 students were
undergraduate students while 8 were postgraduate students. The project had been promoted
to them as a service-learning project which they signed up as voluntary teachers.
2.2. Language Teaching Support for University Students

Prior to their teaching, the university students received training from three University
Lecturers specialising in English Language (the project team) on contrastive linguistics of
Chinese and English and teaching pedagogy. The training enriched the university students’
English language knowledge, helping them to understand rationale of the teaching materials
design for effective lesson planning and delivery. During the programme, the project team
and their colleagues (also University Lecturers of English Language) observed university
students’ classes at least twice during the teaching period and provided them with feedback
on English language knowledge and teaching technique.
In addition to in-person coaching, the university students acquired English language
knowledge and additional teaching skills through completing tasks and reading selected
resources on an online platform at their own pace. They then applied teaching skills and
English language knowledge they had learnt in their weekly teaching. The learning-by-
teaching pedagogy applied by the project allowed university students to consolidate their
English language knowledge while providing academic support to secondary school students
and improved their language self-efficacy.
Such design aims to provide university students with a well-rounded educational experience,
both as English language learners and as educators.
2.3. Design of Learning Materials for Secondary School Students

The design of learning materials for secondary school students included gamification
elements to enhance their motivation and engagement. Gaming was the theme of the course
learning materials. This theme is popular among teenagers and it is easy for both secondary
school and university students to relate and motivate them to participate actively in lesson
activities. Selected newspaper articles and TED talks were included in lesson materials as
content input so both the secondary school students and university students read and listened
to authentic English materials. The activities for lessons incorporated gamification elements
such as rewards for completing tasks, process tracking, and narratives for higher learning
motivation and engagement in class.
1092
2.4. Data Collection

The university students were required to complete a self-efficacy survey before and after the
12 lessons. The project team studied changes in language self-efficacy of the university
students by comparing pre-teaching and post-teaching surveys.
3. Implementation
The table below summarises the key implementation details of the project.
Table 1. Key Implementation Details.
Month / Year Content / Activity / Event
Feb – Jun 2022 Reconfirm with the participating local secondary schools and recruit
other possible schools.
Confirm logistics and programme timeline with partnering schools.
Jun - Aug 2022 Design materials for student tutors’ small-class teaching training,
peer teaching and other project-related tasks.
Sept 2022 - Jan 2023 Recruit university students as student tutors.
Project team offers teaching training for student tutors.
Conduct orienteering activities for student tutors and secondary

school students.
Feb – May 2023 Student tutors teach in secondary schools.
Project team observes student tutors’ classes regularly, write log for
record and discussion with student tutors.
Project team updates teaching materials from the student tutors’

teaching experience and class observation.
Jun – Jul 2023 Hold evaluation meetings among secondary school teachers and
student tutors.
Conduct survey for teaching and learning effectiveness among

student tutors and other secondary school students.
Aug – Sept 2023 Programme evaluation.
1093
4. Discussion
As mentioned in 1.3, the Learning League project aimed to improve language self-efficacy
of EFL university students after teaching English language to secondary school students. This
section discusses how the strategies implemented address challenges related to the
improvement of students’ speaking competency and low motivation in learning English
speaking.
4.1. English Language Proficiency of University Students for Lesson Delivery

One of the biggest challenges was to ensure that the university students had sufficient spoken
English competency to effectively teach the secondary school students. To address this issue,
the project team had introduced topics in contrastive linguistics in teaching training and
designed English language knowledge materials relevant to lessons they taught to enrich their
knowledge. In addition, the blended learning approach adopted in the project design proved
to be successful in providing a comprehensive learning experience for both the university
and secondary school students. It has several advantages, including increased student
engagement and motivation, improved access to resources, and personalized learning
experiences. The online resources allowed university students to access learning materials at
their own pace, they can look for further information about lesson content and related lesson
content before teaching in classroom. Some university students highlighted they had learnt
new language knowledge from the blended learning materials and also new vocabulary
during lesson preparation.
4.2. Low Motivation Among EFL Secondary School Students

With the cancellation of English speaking assessment in university entry examination, the
secondary school students were less motivated to practise English speaking skills. The project
team prepared interactive English language activities in the orienteering day and gamified
tasks in the weekly teaching materials to raise EFL secondary school students’ interest in
speaking the language. The secondary school students even gave speeches to their peers
which was new experience to some and had boosted their confidence in using English for
communication.
5. Conclusion, limitation, and future direction

The Learning League project was successful in addressing challenges of improving language
self-efficacy of university students and the low motivation of secondary school students to
learn English. The blended learning approach, which combined online resources and in-
person sessions, provided a personalized and engaging learning-by-teaching experience for
university and students. Additionally, incorporating fun and gamified activities, service-
1094
learning and a learning-by-teaching pedagogy helped to increase motivation and improve the
language skills of all participants.
5.1. Project Limitations

The project has certain limitations that could impact the validity and generalizability of the
results. One limitation is the sample size, with only 13 student tutors and 63 secondary school
students, it may not be possible to generalize the findings to a larger population. Additionally,
the 12-week study period may not be long enough to fully gauge the effectiveness of the
teaching and learning of the programme. Furthermore, the study is limited to EFL context in
Hong Kong and may not be applicable to other language acquisition setting.
5.2. Future Directions

While the project is a step towards promoting education equity and improving English
proficiency, there is still room for improvement and future directions. For example, the
project could be expanded to include more universities and secondary schools in Hong Kong
for a larger sample size, providing a more comprehensive picture of the programme's
effectiveness. Additionally, the project can be extended to other foreign languages and
cultures, allowing a wider reach and impact. Aligned with the use of technology in education,
the programme could be modified to incorporate additional forms of technology, such as
virtual and augmented reality in teaching training, to enhance the university students’
learning experience. These modifications will contribute to the further development and
refinement of the project and its potential impact on EFL learning in Hong Kong higher
education.
References
Ching, S.L. 程尚連 (2020, February 14). Tingke bu tingxue wangluo jiaoxue ge shi ge fa停
課不停學網絡教學各師各法. Sing Tao Daily.
Hong Kong SAR. (1997). Quality Education. Retrieved from The Chief Executive's 1997
Policy Address: https://www.policyaddress.gov.hk/pa97/english/paindex.htm
Lam, E. (2022, November 28). English exam worries due to lack of practice.
https://www.thestandard.com.hk/section-news/section/4/247705/English-exam-worries-
due-to-lack-of-practice. The Standard.
Legislative Council Panel on Education. (2022, July 5). Amendments to the Post Secondary
Colleges Ordinance (Cap. 320). Retrieved from Legislative Council:
https://www.legco.gov.hk/yr2022/english/panels/ed/papers/ed20220705cb4-603-1-e.pdf
Mok, K.H., Xiong W., Bin Aedy Rahman, H.N. (2021). COVID-19 pandemic's disruption
on university teaching and learning and competence cultivation: Student evaluation of
online learning experiences in Hong Kong. International Journal of Chinese Education
January–April 2021, 1–20.
1095
Pintrich, P. R., & De Groot, E.V. (1990). Motivational and self-regulated learning
components of classroom academic performance. Journal of Education Psychology,
82(1), 33-40. http://doi.org/10.1037/0022-0663.82.1.33
Tang, W. 鄧穎琳 (2020b, February 8). Yueke youzhiyuan eryue xuefei jianban jiaoshi xu
fang wuxinjia huo shujia bu hui 約克幼稚園二月學費減半教師需放無薪假或暑假
補回. HK01.
Tsang, J., Yiu, W. (2022, January 28). DSE: Hong Kong’s exam authority cancels English-
speaking assessment amid Omicron outbreak. https://www.scmp.com/news/hong-
kong/education/article/3165102/dse-hong-kongs-exam-authorities-cancel-english-
speaking. South China Morning Post.
University Grants Committee. (2022, October 14). Teaching Development and Language
Enhancement Grant. Retrieved from University Grants Committee:
https://www.ugc.edu.hk/eng/ugc/activity/teach_learn/tdg.html
Wong, M. (2005). Language Learning Strategies and Language Self-Efficacy. Regional
Language Centre Journal, 36(3), 245-269.
Yau, A. H. Y., Yeung, M. W. L., & Lee, C. Y. P. (2022). A co-orientation analysis of
teachers’ and students’ perceptions of online teaching and learning in Hong Kong higher
education during the COVID-19 pandemic. Studies in Educational Evaluation.
Yeung, M. W. L., & Yau, A. H. Y. (2021). A thematic analysis of higher education students’
perceptions of online learning in Hong Kong under COVID-19: challenges, strategies and
support. Education and Information Technologies.
1096
Data-driven project-based learning in specialized translation

classes – the case of comparable corpora
Katrin Herget1, Teresa Alegre2
1
Department of Languages and Cultures / Centre for Languages, Literatures and Cultures,
University of Aveiro, Portugal, 2Department of Languages and Cultures / Centre for
Languages, Literatures and Cultures, University of Aveiro, Portugal.
Abstract
The aim of this study is to contribute to data-driven project-based learning in
Translation Studies in the context of Higher Education. Based on the
implementation of a corpus-based project, a group of second year Master’s
students in Specialized Translation was actively involved in the compilation
and exploration of comparable text corpora in vulgarization of medical
science. Comparable corpora are becoming increasingly significant in the
field of Language Technology and represent an essential basis for the
extraction of terminological and phraseological units. The study falls within
the scope of medical eponyms in English and Portuguese corpora. In this
context, special attention was paid to the importance of corpus building for
the dissemination of medical information to the general public. Against this
backdrop, the project enhanced proactive learning by familiarizing students
with corpus methodologies, textual conventions, style and register. By
compiling and analyzing comparable corpora, students became autonomous
researchers and developed skills and competencies for their future
professional practice.
Keywords: Data-driven project-based learning; specialized translation;
comparable corpora; medical eponyms; Portuguese-English.

Data-driven project-based learning in specialized translation classes
1. Introduction
The aim of this study is to present a data-driven project-based learning approach based on
the implementation of a classroom project in a higher education setting. The idea of
adopting a project-based learning (PBL) approach is to provide students with real-life work
environment, introducing them to a more dynamic and motivating learning setting. We
explore how PBL can contribute to students’ professional preparation and show how data-
driven work can add value to language learning.
Data-driven learning has been widely investigated in the field of Language Pedagogy and
Learning. The term is commonly used to describe “tools and techniques of corpus
linguistics for pedagogical purposes” (Gilquin & Granger, 2010, p. 359). According to the
authors, data-driven learning provides several advantages, such as confronting the learner
with authentic language, as well as giving them access to "a large number of authentic
instances of a particular linguistic item" (p. 359). In this respect, Lenko-Szymanska &
Boulton (2015) claim that
learners can benefit tremendously from the direct use of corpora. They gain
access to authentic language, which they can query in a variety of ways for
the information which is interesting and relevant to them at a particular
moment and which allows them to refine their understanding of how
language really behaves (p. 3).
The present study aims at applying a data-driven learning approach within the scope of a
translation project on the construction and exploration of comparable corpora, and intends
to contribute to the implementation of innovative learning resources in Higher Education.
2. Project-based learning
Over the last decades, there has been an increasing interest in the study on project-based
learning (PBL) in higher education settings. According to Thomas (2000, p. 1) "Project-
based learning (PBL) is a model that organizes learning around projects" that focus on
"questions or problems that 'drive' students to encounter [...] the central concepts and
principles of a discipline" (p. 3). This model came up as a response to the need of engaging
students in authentic activities, reflecting real-world challenges and needs. Higher
education institutions must be aware of these changes and come up with adequate solutions
in course design. With respect to this, Uden & Beaumont (2006) believe that “[u]niversity
education should, ideally, provide students with the necessary skills, values, and attitudes
that are essential to cope with the dynamic complexities of the modern world. […] there is a
lack of deep learning about the complex issues and problems that graduates have to face in
1098
the real world” (p. 26). Due to a terminological fuzziness, PBL is herein used an umbrella
term for innovative teaching activities that center on learning through projects. In the
present study, data-driven learning is implemented in the broader context of PBL and
provides a specific focus on corpus-guided decision-making / construction and exploration.
3. Comparable corpora in the translation classroom

Due to the importance of Specialized Translation in today’s globalized world, translation
classes have to respond to a diversified professional reality, preparing students to actively
deal with different communicative situations. According to López-Rodríguez & Tercedor
Sánchez (2008), “getting familiar with corpora and annotation” ranks among learner-
centered activities that help to develop and promote the learners’ autonomy. In Applied
Linguistics, the collection of textual data has long been fundamental for the study of
specific lexico-grammatical phenomena and patterns, in order to obtain a better
understanding of different language registers and varieties. In the field of Translation
Studies, Baker (1993) already predicted, in her seminal work on Corpus Linguistics, "that
the availability of corpora and of corpus-driven methodology will soon provide valuable
insights in the applied branch of translation studies" (p. 242). Since then, research on
corpora in Translation Studies has developed at a fast pace and has become fundamental
both in theoretical and in applied studies. Molés-Cases & Oster (2015, p. 204) present a
detailed overview of practical fields of application of corpus-driven work in translation
training. According to Krüger (2012, pp. 507-508) corpora “allow for a better
contextualisation and control of the texts to be investigated and provide a higher
representativeness [sic], generalisability [sic] and replicability of the findings”. Depending
on the research field and objectives, different types of corpora are used. Translation Studies
mainly distinguish between comparable and parallel corpora. Due to advances in Machine
Translation, the importance of collecting comparable data is getting more and more
significant. Comparable corpora are an essential basis for the extraction of bilingual
dictionaries, because, unlike parallel corpora, there is no influence from the source text
structure. According to McEnery & Hardie (2012, p. 20), a comparable corpus contains
“components that are collected using the same sampling method, e.g. the same proportions
of the texts of the same genres in the same domains in a range of different
languages in the same sampling period” [emphasis in the original]. In this sense, Mikhailov
& Cooper (2016, p. 217) also define comparable corpora as text collections that were
compiled “on the same principles (size of the collections, size of the samples, topics
covered, chronological period, etc.) in different languages, or different variants of the same
language: e.g. texts on atomic energy in French and Spanish, or texts in the German of
Germany, Austria and Switzerland”. According to Bernardini, Stewart & Zanettin (2003, p.
6) comparable bilingual corpora are important for translation students in that they provide
1099
them with an understanding of both target and source texts, “allowing them to compare
terminology, phraseology and textual conventions across languages and cultures”. In the
field of Specialized Translation comparable corpora assume a fundamental role as they
provide evidence of specific lexico-grammatical structures, as well as information on the
frequency of linguistic patterns in specialized domain texts.
4. Motivation for the Study

In times of unlimited access to information through the internet in form of online
dictionaries, databases, or machine translation systems, translation students need to be
aware of the challenges arising from the amount of available data. In order to choose
suitable texts for the compilation of ad-hoc corpora in the field of medical science
vulgarization, students are required to assess the quality of online-texts according to the
purpose of the task and to apply a set of criteria to ensure the usability and reliability of the
respective data. Against this backdrop students were familiarized with the advantages of
corpus compilation and exploration for their future professional work. Another motivation
for the study results from the fact that the source language of a text available on the internet
is not always obvious, due to a growing amount of machine-translated texts. Therefore, it is
a fundamental requirement to make sure that the source text is authentic and that it is not
the result of a translation.
5. Method and Study Design

The aim of this study is to actively involve a group of 12 MA students of Specialized
Translation in the compilation and exploration of comparable text corpora in the field of
vulgarization of medical science in Portuguese and English. The corpus analysis was
carried out with the help of the concordance program AntConc (Anthony, 2022). The
project was subdivided into three stages, which will be presented in the following.
Stage 1 - Project preparation: Contextualization and pre-corpus building stage
In this preliminary stage, students learn to search and select websites from the field of
medical sciences in Portuguese and English by using a specific set of criteria: i) websites
belong to the same domain; ii) texts belong to the same genre; iii) texts aim to address a
general public on health issues; iv) texts are written by health specialists or technical
journalists; v) texts belong to the same sampling period, and vi) texts are written in English
/ European Portuguese.
1.1 Corpora in specialized translation
a) Identifying different types of corpora (monolingual, bilingual, multilingual, parallel, and
comparable) with the main focus being on comparable corpora;
1100
b) Reflecting on the advantages of comparable corpora (e.g. comparable corpora give

evidence of terminology and textual conventions in a particular language context);
c) Importance of ad-hoc corpora for Specialized Translation (e.g. compilation of texts for
the analysis of specific lexico-grammatical patterns, language varieties, communication
levels, etc.);
d) How comparable are comparable corpora? (comparability in terms of dimension, field
area, subject content, target audience, sender-receiver relationship, time period, textual
genres, mode, etc.).
1.2 Medical eponyms and their use in popularization texts
a) Identification of medical eponyms (e.g. presentation of Wordcloud, Figure 1, as an
inductive learning approach, etc.);
Figure 1: Wordcloud representing the frequency of eponyms in the Portuguese corpus.
b) Variability of eponyms in medical vulgarization texts (synonyms, concurrent

designations, etc.);
c) Identification of specific translation problems related to medical eponyms (e.g. an
eponym (Paget) denotes two different diseases: bones versus breast cancer).
Stage 2 - Project execution
2.1 Compilation of ad-hoc corpora
a) Selection of adequate websites (vulgarization of medical science);
b) Discussion on the adequacy and reliability of the findings.
2.2 Corpus compilation and concordance analysis with AntConc
a) Compilation of reliable texts according to criteria such as: domain, authorship, target-
audience, language variety, chronological period, etc.;
b) Concordance search and extraction of medical eponyms (Figure 2);
1101
Figure 2: Results of concordance search on Epstein-Barr in AntConc.
c) Results discussion and group reflection on the variation of eponyms in both corpora.
Stage 3 - Evaluation
a) Evaluation of the project by means of a portfolio, including results from the eponym
concordance search, extraction of relevant text segments and respective analysis, as well as
reflection on the importance of comparable corpora in Specialized Translation;
b) Online questionnaire (ongoing) on students' perceptions regarding the data-driven project
work.

This paper explored a data-driven project-based learning approach in Specialized
Translation and aimed at familiarizing a group of 12 MA students with the compilation and
exploration of comparable corpora in the field of medical science vulgarization. Data-
driven learning based on projects actively involved students in corpus-based research
related to eponyms in medical science vulgarization texts for a large audience. The
enormous amount of available texts on the internet bears several challenges for translation
students, who are required to apply a set of specific criteria to ensure the usability and
reliability of the respective data. One of these challenges consisted in identifying a specific
communicative situation and defining the target audience for corpus building. Another
challenge involved the validation of the selected texts as original, non-translated items. The
project consisted of three stages that aimed at developing students' research and analytical
skills by involving them actively in the process of searching and compiling comparable
1102
corpora. The first stage consisted of project preparation, giving students an overview of
context and pre-corpus building requisites. The second stage (project execution) was
dedicated to the compilation of ad-hoc corpora, as well as concordance search and
extraction of eponyms. At the end of this stage, students were actively engaged in
discussion and group reflection on the challenges and opportunities of comparable corpora.
The third and last stage entailed evaluation by means of a portfolio and a submission of an
online questionnaire (ongoing) to assess students' perceptions. The project enhanced
proactive learning by familiarizing students with corpus methodologies, textual
conventions, style and register. By compiling and analyzing comparable corpora, students
became autonomous researchers and developed skills and competencies for their future
professional practice. The results of the survey questionnaire will help to identify possible
limitations and make subsequent adjustments with regard to the design and implementation
of future projects.
Acknowledgments
The authors would like to thank the MA students for their participation in this project.
References
Anthony, L. (2022). AntConc (Version 4.2.0) [Computer Software]. Tokyo, Japan: Waseda
University. Available from https://www.laurenceanthony.net/software.
Aronson, J. K. (2014). Medical eponyms: taxonomies, natural history, and the evidence.
BMJ 2014;349:g7586 doi: 10.1136/bmj.g7586.
Baker, M. (1993). Corpus Linguistics and Translation Studies – Implications and
Applications. In M. Baker, G. Francis & E. Tognini-Bonelli (Eds.), Text and
Technology. In Honour of John Sinclair, (pp. 233-252). Amsterdam: John Benjamins.
Bernardini, S., Stewart, D., & Zanettin, F. (2003). Corpora in Translator Education: An
Introduction. In F. Zanettin, S. Bernardini & D. Stewart (Eds.), Corpora in Translator
Education (pp. 1-13). Manchester: St. Jerome Publishing.
Gilquin, G., & Granger, S. (2010). How can DDL be used in language teaching? In A.
O’Keeffe & M. J. McCarthy (Eds.), The Routledge Handbook of Corpus Linguistics
(pp. 359-370). Oxon, New York: Routledge.
Krüger, R. (2012). Working with Corpora in the Translation Classroom. Studies in Second
Language Learning and Teaching Department. SSLLT 2 (4). 505-525.
Laviosa, S. (2002) Corpus‐based Translation Studies: Theory, Findings, Applications.
Amsterdam, New York: Rodopi.
Lenko-Szymanska, A., & Boulton, A. (2015). Data-driven learning in language pedagogy.
In A. Lenko-Szymanska & A. Boulton (Eds.), Multiple Affordances of Language
Corpora for Data-driven Learning (pp. 1-14). Amsterdam: John Benjamins.
1103
López Rodríguez, C.I. (2016). Using corpora in scientific and technical translation training:
resources to identify conventionality and promote creativity. Cad. Trad., Florianópolis,
v. 36, nº especial 1, p. 88-120. http://dx.doi.org/10.5007/2175-7968.2016v36nesp1p88.
McEnery, T., & Hardie, A. (2012). Corpus Linguistics: Method, Theory and Practice.
Mikhailov, M. & Cooper, R. (2016). Corpus Linguistics for Translation and Contrastive
Studies. A guide for research. London, New York: Routledge
Molés-Cases, T., & Oster, U. (2015). Webquests in Translator Training: Introducing
Corpus-based Tasks. In A. Lénko-Szymanska, & A. Boulton (Eds.), Multiple
Affordances of Language Corpora for Data-driven Learning (pp. 199-224).
Amsterdam: John Benjamins.
Thomas, J. W. (2000). A Review of Research on Project-Based Learning. Autodesk
Foundation.
[https://my.pblworks.org/resource/document/a_review_of_research_on_project_based_l
earning].
Uden, L. & Beaumont, C. (2006). Technology and problem-based learning. Hershey, PA,
USA: Information Science Publishing.
1104
Cultural heritage and its dissemination through linguistic and

technical learning in Higher Education
Daniela Gil-Salom1, Eliseo Marzal-Calatayud², Damián López-Rodríguez²
1
Department of Applied Linguistics, Universitat Politècnica de València, Spain, 2Department
of Computer Systems and Computation, Universitat Politècnica de València, Spain.
Abstract
This paper presents the first results of the innovation and educational
improvement project FUSIONA, which combines the transversal learning of
German as a foreign language (GFL), the interdisciplinary learning, the
cultural and scientific-technical heritage protection and its social
dissemination. The principal objectives of this service-learning project are, on
one side, to redesign a practical task of the course of GFL and, on the other
side, to spread the cultural heritage of the four museums of Universitat
Politècnica de València (UPV). Transversal skills, like effective
communication, teamwork or social and environmental commitment are
enhanced, and collaborative learning is put into practice by the design and
edition process of informational videos. Thereby, GFL and Audiovisual Media
students and teachers work together in order to create and evaluate the
outputs. The currently quantitative and qualitative results show that
interdisciplinarity is also feasible when language learning, media technique
and culture join together.
Keywords: Interdisciplinarity; German as a foreign language; scientific-
technical heritage; transversal skills; collaborative learning; service-
learning.

Cultural heritage and its dissemination through linguistic and technical learning in Higher Education
1. Introduction
One of the high-impact educational practices (Kuh, 2001a; 2001b; Kilgo et al. 2015) put forth
and endorsed by the Association of American Colleges and Universities (AAC&U) is
service-learning (SL) (Tapia, 2007; Batlle, 2011). The SL project presented, combines the
transversal learning of German as a foreign language (GFL), the contents and techniques of
the course Audiovisual Media (AM) and the interdisciplinary learning methodology, with the
aim of detecting the cultural and scientific-technical heritage protection and fostering its
social dissemination. The main goal of the project, named FUSIONA (fusion of art, science,
technology, society and German language) is to develop effective oral communication in
GFL, thus enhancing the interaction of the students who carry out an interdisciplinary task.
The task consists in the design and edition of an informational video about the cultural
heritage of their university.
In the last years, the idea of diversifying the practical tasks of the course Academic and
Professional German A1 (from now on, German A1) attending to new contents and formats,
such as video recordings had grown up among some course teachers. At the same time, we
got the perception that the UPV cultural heritage and its four museums are really unknown
by the society and by the students themselves. This fact has been demonstrated with the
responses of a questionnaire, which was passed before the experience. During two hours 144
students from different bachelor and master degrees at the UPV answered to the question:
Do you know about the museums at the UPV? 72,9% negatively, and 27,1% positively. These
figures were our starting point.
The paper is structured as follows: in Section 2 we describe the interdisciplinary innovation
approach to teaching; in Section 3 we describe the methodology followed in the experience;
in Section 4, we summarize the results; and some conclusions are arisen in the last Section.
2. Interdisciplinary innovation
The Project involves two aspects. On one hand, it is necessary to remake the format and
content of the practical tasks, which are fundamental for students’ performance. They prepare
them not only for the academic tests that evaluate the different linguistic skills (speaking,
listening, writing and reading), but also for their future professional lives. However, these
tasks should also help students to become social responsible citizens, as the University Social
Responsibility (USR) points out (Ali, Mustapha, Osman, & Hassan (2021) and is stated in
the UNESCO with its call for pursuing the 17 Sustainable Development Goals (SDGs) of the
2030 Agenda.
Traditionally, all the four practical tasks have been oral presentations. According to the
opinion of students and teachers, other outputs could be introduced, as for instance, videos
1106
or audios. This is the reason why we started the project together with other lectures of the
subject Audiovisual Media, in which the quality of these formats is taught, contents that are
impossible to be dealt with in GFL.The teachers of Audiovisual Media transmit some basic
theoretical knowledge to the students of GFL and the students of Audiovisual Media evaluate
the videos.
The project involves the development of three of the skills our students are required.
Graduates of the Polytechnic University of València official degrees must show that they are
capable of: (1) Demonstrating proficiency in digital communication using a variety of
support media adapted to the situation and audience (effective communication); (2) Helping
find solutions to challenges or projects, showing empathy and assertiveness when sharing
ideas, reflections and arguments within collaborative work (Teamwork and leadership); and
(3) Contributing to the design, development and implementation of solutions that meet social
demands, taking the Sustainable Development Goals as a reference (Social and
Environmental commitment).
On the other hand, we want to enrich the learning of the foreign language together with that
of the artistic, scientific and technological heritage of our university. The professional work
of the museums management teams aims to pursue the SDG 11, as they protect the cultural
heritage, the education for the sustainable development supporting research and cultural
participation.
The course German A1 is transversal for all the Polytechnic University of Valencia degrees,
that is, students of different degrees and grades meet together for this course lectures. In the
project 90 students of four German A1 groups (two in the Autumn semester and two in the
Spring semester), and 60 students of Audiovisual Media (course of the Design and Creative
Technologies Degree) participated during the first year of the experience.
3. Methodology
As a previous step to this experience, the team conducted a study about the knowledge of the
museums among the students in the campus. The team carried out a random poll in the leisure
area of the campus to stablish a starting point for the project. The results, as mentioned before
in the first Section, demonstrate that the majority of the students at the campus did not know
about the existence of the museums. Those who did know about the museums, informed
about which of them they knew (Q3), and which of them they did visit (Q4). The results are
summarized in Figures 1 and 2. The answers show that the most known museums are the
Informatics and Telecommunications ones, despite the fact that the Sculpture museum is an
outdoor museum placed in the same leisure area where the poll was passed.
1107
Figure 1. Previous questionnaire Q3: Which museums do you know?
Figure 2. Previous questionnaire Q4 Which ones have you visited?
In order to fulfill the general goal, three specific objectives are outlined:
a) To design a new practical task integrating information about the location and one example
of the pieces exhibited in the (name of the university) museums.
b) To enhance the collaborative (Johnson y Johnson, 1991; Kagan, 1994; Morales, 2007;
Prieto, 2007) and interdisciplinary (Fernández et al., 2017) work, both among teachers and
students.
c) To disseminate the best outputs on social media and in the (name of the University)
museums webs, only once after a selection process composed by museums staff, students and
teachers.
1108
Table 1 shows the different linguistic structures and possible vocabulary that is expected to
be used for the museum description included.
Table 1. Linguistic resources
Introduction Hallo, heute besuchen wir das _______ (Name vom Museum) / Hallo,
wir präsentieren heute das ________ (Name vom Museum).
Location Das __________ (Name) Museum liegt im Gebäude (Gebäude), an der

______ (Hochschule). / Die Skulpturensammlung liegt in den Gärten
von der UPV, es ist eine Open-Air-Galerie.
Area / sections (Name vom Museum)_______ ist _____ m 2 groß und hat
_____Sektionen.
Historia y piezas Das Museum gibt es seit ________ . Hier findet

man ______ (Geräte / Spielzeuge, Apparate / Skulpturen, usw.).
Example piece Zum Beispiel, das ist ein/-e ________ (Nomen). Das ist kein/-e
________ (Nomen). Es ist _____ groß / breit / etc.
Cierre Wir finden das _________ (Museum) _________ sehr

interessant! Komm und besuch es!
Schedule Das _________ (Name Museum) ist von _____ Uhr bis ____ Uhr
geöffnet. Am ________ (Wochentag) ist es geschlossen.
Entrada Der Eintritt ist frei.
The document handed to the students, which explain the new practical task, describes also
the steps to follow, and the necessary linguistic elements of the German language to
accomplish it. The five steps are: (1) Choose a museum (better if it is not the one of your
faculty); (2) Write a brief description of the museum (name, location); (3) Choose a piece in
order to present it; (4) Write a brief description of the piece; (5) Express in an oral way the
written description and record it in video format.
Two or three students at most of a group have to present contents orally and are also video
recorded. If they are three, more contents have to be prepared. Linguistic structures and
resources needed for oral discourse are shown in Table 1.
The highest grade to be obtained in this combined task is 0,75 points (video recording: 0,25
+ oral expression: 0,5) from a total of 10 points of the final grade of the subject.
1109
The new rubric for assessment has include oral expression, pronunciation, vocabulary used
and grammar correction. The teachers of GFL design and implement this rubric.
To complete the assessment of the practical tasks of the Project, the teachers of Audiovisual
Media devise a rubric related to technical aspects. This rubric serves as a tool and a guide to
record videos, and an assessment tool for the students of the Audiovisual Media subject, who
will be in charge of evaluating this aspect.
In order to select the best products to be disseminated on social media and webs, those people
representing the museums of the UPV will meet the teachers of the two subjects.
In addition, once the Project is finished, each person participating in the team will be capable
of assessing the process followed, so as to improve the implementation of each phase. This
discussion among the agents of the project is crucial for its quality. Perception of students,
teachers and service beneficiaries –museums in our case- are crucial. This evaluation
happened the day of the final presentations, after their assessment and selection in front of a
public audience, as is suitable for a project of ApS. In this event, certificates from museums
and teachers were given to students participating in both subjects; they certify the
participation in an ApS project, including two subjects and four UPV museums.
4. Results
Table 2 summarizes the museums selected by the different groups. It can be noted that
Informatics and Toys museum were only selected by four and eight groups, while
Telecommunications and Sculpture museums were selected by twelve and ten groups
respectively. This does not fit the distribution of general knowledge of the museums
previously analyzed (Figures 1 and 2). Hence, possibly, the practice has improved some
general knowledge of museums in the campus.
Tabla 2. Museums selections
Museum Group A1A1 Group A1A2 Group A1A4 Total
MUCAES 2 2 6 10
TELECOM. 4 6 2 12
JUGUETE 3 3 2 8
INFORMÁTICA 1 1 2 4
1110
Additionally, students took a questionnaire after the delivery of the practice. The answers
show that the experience is positive, and highlights the possibility of changing the room by
an outdoor activity. The answers also show that students appreciate the opportunity of
improving their knowledge of the cultural heritage exhibited in the University museums.
5. Conclusions
The implementation of this project started during the first semester of the academic year
2022-23, so it is not yet concluded. A second experience is needed to compare and better
evaluate if the collaboration has been profitable for as many students as possible. The
objectives and the methodology of this project could be transferred to other GFL levels, like
A2, B1 or B2 also offered at the UPV. Given the transversal kind of the course, that is, a
foreign language, it could be also implemented in other languages like French or Italian. We
do not mention here English or Catalan, because the museums already offer their information
in these language on their webs.
It could be also interesting that other courses, different to those of foreign languages would
incorporate information about the cultural heritage in order to enrich the students education.
The history of the diverse disciplines is essential to understand the current achievements; a
historical point of view is necessary to better understand the present, in all fields, even in the
artistic, scientific and technological ones. This kind of transdisciplinary task opens the
problem to set the balance in the assessment of the different skills, it is a challenge to develop
a transdisciplinary output performing all the skills with similar proficiency.
Acknowledgements
We want to thank the ICE (Institute of Education Sciences) of the UPV for supporting this
Project.
References
Ali, M., Mustapha, I., Osman, S., & Hassan, U. (2021). University social responsibility: A
review of conceptual evolution and its thematic analysis. Journal of Cleaner Production,
286, 124931.
Batlle, R. (2011). Voluntariado y aprendizaje-servicio. Una metodología educativa que
consiste en aprender haciendo un servicio a la comunidad. Son de paz, 1, 3-6.
Batlle, R. (2018). Guía práctica de aprendizaje-servicio. Proyecto Social, 4-34.
Fernández, M. A. M., Urbano, E. R., Ordoñez, M. B. Q., Díaz, V. M., Urbano, I. G., &
Esteban, B. P. (2017). Quién soy. Proyecto de coordinación horizontal interdisciplinar
para el desarrollo de competencias. Revista de Innovación y Buenas Prácticas Docentes,
3(3), 29-38.
1111
Johnson, D. y Johnson, R. (1991). Learning together and alone. Cooperative, competitive

and individualistic learning. Needham Heights, Allyn and Bacon.
Kagan, S. (1994). Cooperative Learning. San Clemente, CA: Kagan.
Kilgo, C. A., Ezell Sheets, J. K., & Pascarella, E. T. (2015). The link between high-impact
practices and student learning: Some longitudinal evidence. Higher Education, 69, 509-
525.
Kuh, G. D. (2001a). Assessing What Really Matters to Student Learning: Inside the National
Survey of Student Engagement. Change, 33(3), 10-17, 66.
Kuh, G. D. (2001b). The National Survey of Student Engagement: Conceptual Framework
and Overview of Psychometric Properties. Bloomington: Indiana University Center for
Postsecondary Research. Retrieved from
http://nsse.indiana.edu/2004_annual_report/pdf/2004_conceptual_framework.pdf.
Morales, P. (2007). Aprender a trabajar en equipo evaluando el proceso. En L. Prieto
(Coord.), La enseñanza universitaria centrada en el aprendizaje (133-151) Octaedro.
Prieto, L. (2007). El aprendizaje cooperativo. PPC.
Salam, M., Awang Iskandar, D. N., Ibrahim, D. H. A., & Farooq, M. S. (2019). Service
learning in higher education: A systematic literature review. Asia Pacific Education
Review, 20, 573-593. https://doi.org/10.1007/s12564-019-09580-6
Tapia, M. N. (2007). El aprendizaje-Servicio en las OSC en AAVV. Aprendizaje y servicio
solidario en las organizaciones de la sociedad civil. Buenos Aires.
1112
Institutional influencers and support for tutoring in a South

African higher education institution
Department of Finance and Investment Management, University of Johannesburg, South
Africa.
Abstract
Tutoring contributes to student performance. However, the institutional
factors that affect tutoring effectiveness have been explored to a limited extent.
This study assessed institutional factors affecting tutors’ effectiveness and
support strategies to improve their function. Interview data among twenty
tutors in the Business and Economics faculty in a higher education institution
in South Africa was analysed using inductive thematic analysis to output
themes emerging from the data. Findings revealed that technical issues,
unclear instructions, inadequate resources and training influenced tutors’
performance. Regular engagement/communication, tutor workshops, training
tailored to specific tutors’ needs and challenges, timely provision of tutorial
materials, incentives and supporting infrastructure could improve tutors’
effectiveness. The findings are beneficial to higher education stakeholders in
developing measures to ensure effective tutoring for students. There is scope
for future studies on the same topic to elicit views when tutoring is conducted
face-to-face as this study was impacted by the covid-19 pandemic.
Keywords: South Africa; student performance; institutional factors, tutoring,
university.

Institutional influencers and support for tutoring in a South African higher education institution
1. Introduction
Poor student performance in terms of dropout and failure rates is a problem that policymakers
are constantly addressing (Sibanda et al., 2015). The overall dropout rate for undergraduate
college students in the United States is 40%, with over 30% of college freshmen dropping
out before their sophomore year (Hanson, 2021). This percentage is now hovering at 10.3%
in the European Union (Rita, 2020). About 50-60% of first year university students in South
Africa drop out (Dyomfana, 2022). Low educational accomplishment results in poverty,
unemployment, and inequality (Mlachila & Moeletsi, 2019). Therefore, emphasis should be
placed on designing effective means and providing the necessary resources and infrastructure
to assist students who require further assistance (Fluke et al., 2014). Extra help and support
are some of the methods utilized to help pupils enhance their academic performance in higher
education. An efficient way to improve completion rates and academic performance of
students is through tutoring (Guerra-Martin et al., 2017; Karpur, 2018).
Tutoring of different forms, including one-on-one, peer, group, cross-age, and online tutoring
via email or discussion forums, entails assistance and support by non-professional teachers
in an engaged, intentional, and methodical manner (Mckay, 2016). Traditional tutoring
involves a more experienced student working with less experienced students in a planned and
organized manner, which could be face-to-face or online; however, the quality/expertise may
be lower than a professional teacher’s (Mlika et al., 2022). Hence, the efficiency of the
strategies utilized in the tutorial process needs to be continually revisited to ensure that the
needs of all involved are met. McKay (2016) investigated the relationship between tutor
effectiveness and student performance using a pre- and post-test and found that students who
attended tutorials improved their grades by 20%. Students’ academic competence, conceptual
knowledge and understanding are increased with tutors’ active facilitative and guiding role
(Mlika et al., 2022). Guerra-Martin et al. (2017) demonstrated the effectiveness of formal
tutoring to improve the academic performance of at-risk nursing students in Spain using a
pre and post-test.
Although studies exist on tutoring effectiveness, limited studies have focused on tutoring in
business and economics. Extant studies focused on tutoring effectiveness in nursing (Guerra-
Martin et al., 2017), and engineering (Mckay, 2016). Finance-related modules are technical,
and several students find them challenging at an academic and professional level of study
(Olokoyo & Oyewo, 2014). Therefore, there is a need to evaluate the factors that contribute
towards the effectiveness of tutoring in the context of the social sciences. Further, limited
studies have investigated institutional influencers from tutors’ perspectives. Furthermore, this
study was undertaken during the pandemic in 2020; thus, some findings were focused on the
experience of tutors during this period. Therefore, the study’s objectives were to explore the
institutional factors that affect tutoring effectiveness and support strategies to improve
1114
effectiveness. Using thematic analysis of qualitative data, the critical factors and suggested
support strategies were identified to aid in improving the learning experience of students.
2. Tutoring Effectiveness
Guerra-Martin et al. (2017) indicated that the effectiveness of a tutoring program could be
established by improved student performance. Koehler (2018) also identified the
preparedness of the tutor as a measure of effectiveness. Tutoring effectiveness can also be
measured by comparison with the desired outcomes and expectations of the lecturer and
tutorial program (Bixby et al., 2011). If the desired outcomes of the tutoring program are
communicated, the tutor can perform as expected. On this basis, this study reflects on how
tutors performed in their role of assisting lecturers and students to improve teaching and
learning.
Mlika et al. (2023) evaluated the effect of tutors’ expertise and active guidance on students’
learning using meta-analysis. Holiday (2012) assessed the test scores of students before and
after tutoring; however, focusing on peer tutoring. McKay’s (2016) study assessed the impact
of tutors on first-year academic performance in Geography at a South African university.
Therefore, although research on the effectiveness of tutoring has been undertaken, it is not as
widespread as the use of tutoring (Holiday, 2012). This research is therefore important
because strategies can be developed to improve tutoring effectiveness and in turn student
performance.
3. Institutional Factors affecting Tutoring Effectiveness

Factors affecting the effectiveness of tutors include the materials and time available for
preparation (Koehler, 2018). The more time a tutor has to prepare for a tutorial and having
access to the material required for preparation all contribute to the effectiveness of a tutor. In
addition, a supportive environment through providing institutional infrastructure and support
in various forms such as training and communication helps tutors to excel in their function
(Joubert & Snyman, 2018). Thorburn (2021) identified that the online learning environment
was fraught with connectivity issues and indicated that the COVID-19 pandemic has exposed
significant connectivity issues in South Africa and worldwide. Joubert and Snyman (2018)
stated that restricted access to online platforms, administrative problems, limited or no
interaction between lecturers and tutors, inability to make use of online platforms and
insufficient training are some of the institutional factors that restrict the effectiveness of tutors
in an online environment. Yusuf and Ahmad (2020) added that a lack of necessary tools and
equipment for learning, and inadequate infrastructure in terms of computers affect tutors.
Further, tutoring effectiveness may be hampered by the tutorial delivery, what materials or
medium used and subject technicality (Currey et al., 2015).
1115
4. Research Methods
A qualitative interpretive approach, where a social phenomenon is understood from the eyes
of the participants rather than the researcher, was used (Cohen et al., 2007). Interviews were
used to obtain in-depth information about tutors’ opinions, beliefs and experiences (Rahman,
2016). Semi-structured interviews were used as they allowed flexibility and augmentation
with follow-up questions, probes and comments (DeJonckheere & Vaughn, 2019). The
research was undertaken in 2021 using virtual engagements on Microsoft Teams and
Whatsapp, after institutional ethical clearance. Convenience sampling was used to identify
tutors from four departments including Finance, Economics, Management, and Accounting
(Barratt, 2009). These were initially identified and approached through the educators they
worked with. Willing and available participants were then interviewed. Twenty tutors
comprising twelve females and eight males participated. Most (eight) of them were in their
second year of tutoring; 11 were students at the Honours (fourth) level of study.
Data analysis was undertaken using inductive thematic analysis to detect emerging patterns
and analyze, organize, characterize, and report identified themes (Braun & Clarke, 2006).
The process was as follows: familiarisation and developing potential themes, initial coding,
searching and reviewing potential themes, categorization, consensus-seeking and writing.
5. Findings and Discussion
5.1. Factors affecting Tutoring Effectiveness

The following factors were identified as institutional factors influencing the performance of
tutors during the period investigated:
Materials available for preparation - The availability of materials for preparation is critical in
the tutor being effective in performing their role. This includes getting access to memos and
explanations, textbooks and other materials to facilitate preparation before the tutorial
sessions. Tutors explained: “They made sure that we had all the resources that we needed to
conduct the tutorials; discussing the work chart at the beginning of the week; the materials
that we use for tutorials and practicals, getting papers, the markers; The memo should be
clear; I think they should give us the memos and explanations”.
Tutor training - Tutor training is a vital function provided by the university to enlighten them
on tutoring strategies, handling challenges, and treating students. “They once provided some
training and provide proper support; tutor training, because you learn how to improve; The
training at the beginning of the year is worthwhile; effective; sometimes, you think you’re
being professional when you are being unprofessional, that training helps”.
Communication from lecturers and head tutors - Poor communication was identified by tutors
as a factor that affects them in performing their duties, for example, unclear and untimely
1116
instructions and lack of communication of expectations. As a result, delays in marking

assignments were experienced: “… clearer instructions on tasks upfront; not being able to
communicate with the lecturer to know that this is what he or she taught.” On the contrary,
weekly communications and meetings with head tutors were productive and helpful: “… with
the head tutor, it was really good, we had meetings on Mondays which helped us prepare;
discuss what we will do that week and if you have any questions, you ask.”
Technical and infrastructure support - Technical issues were faced by tutors as they conduct
their tutorials, for example, data, connectivity, and difficulty with the learning platforms.
Data was allocated by the institution; however, there were delays in the process, which
sometimes affected how tutorials were done. “Data allocation, it began timely. When the
lockdown started it was delayed, but eventually, it came through”. Some tutors also reported
lacking suitable devices to conduct the tutorials; they used tablets and phones, which do not
have the screen share function. Yusuf and Ahmad (2020) also identified factors such as lack
of necessary tools and equipment for learning, and inadequate infrastructure in terms of
computers as factors affecting tutoring effectiveness. Others were located in areas where
connectivity was poor: “WIFI, network, by the time you come back you find that some
students are gone”. This aligned with Thorburn’s (2021) findings that the pandemic exposed
significant connectivity issues in South Africa and worldwide. Further, some tutors were
unable to effectively use online platforms and applications, for example, Blackboard
collaborate and Zoom. “… others do not know how to share screen.” These challenges were
also identified by Joubert and Snyman (2018) who amongst other factors, indicated that the
training that tutors received on how to navigate online platforms was limited and thus should
be improved. In addition, the lengthy administrative process of appointments for
remuneration was highlighted as a problem. This deviates time for learning as tutors spend
time filling out forms and following up on appointments.
5.2. Towards improving Tutors’ Effectiveness

The strategies recommended were mainly related to the online learning environment in which
the tutors were operating during the lockdown. These include:
More frequent communication - Information flow from the lecturer is critical to the
effectiveness of a tutor as they clarify what the tutor is supposed to do. As reported, “I would
suggest that the department is more involved with tutors”. Improving tutor-lecturer
relationships through meetings and frequent engagements was suggested, a view supported
by Joubert and Snyman (2018), who found that tutors’ feelings of isolation, and limited or
no interaction between educators and tutors were some of the factors that affected the
effectiveness of tutors.
More support with the relevant infrastructure and resources - The provision of more materials
could assist in the preparation for tutorial sessions: “Possibly provide some devices; or
1117
another application that allows us to use the screens of our phones because Blackboard only
supports the use of laptops or tablets; I think lecturers can just try to offer more support to
tutors with the material”. Further, the provision of devices to facilitate the use of these
platforms, and recording tutorial sessions in case of network issues were suggested, as
supported by Yusuf and Ahmad (2020). Additionally, availing materials (timeously) for
preparation was highlighted as they need materials like textbooks, memorandum of activities,
class notes, and slides, as supported by Olugbenga (2020); they have to be provided with
high-quality instruction material to assist in preparing for executing tutorial sessions in the
appropriate language of instruction.
Training and workshops/seminars - Receiving more training so that the tutors can get the best
out of the various platforms was highlighted: “Tutors should be taught more on how to use
Blackboard, create the whiteboard, and navigate that, so that we know how to share a screen,
and attach documents.” Tutors should undergo continuous training, as supported by Joubert
and Snyman (2018). Further, workshops for tutors would be beneficial. Holding meetings
and briefings more frequently can keep tutors updated on the developments within the
tutoring landscape. “What should be done is to have workshops; I think it is time to start
incorporating more of the courses and training that you have to do for online learning
because if you look critically, at the courses we did, they were based on the in-person basis
of learning and looking at the knowledge now, it is outdated; training that has to do with this
[covid-19] change”.
Incentives for tutor role - Tutors can perform their role more effectively if they are provided
with incentives that contribute to their motivation. Tutors at the university are remunerated
for the services they render. One tutor noted: “… incentives like money, that matters; tutors
may be more willing to work”. This view was shared by Jayachandran (2014), which opined
that one of the reasons for low quality of education in developing countries is that teachers
lack strong performance incentives. Therefore, remuneration can incentivise tutors to
perform well.
6. Conclusion
The study sought to identify institutional factors influencing tutorial effectiveness within
higher education in South Africa. Results from interviews revealed that the online
environment was not easy to navigate. Inadequate communication and support with material
resources impeded tutors’ progress. Regular engagement/communication of the lecturers’
intentions, workshops and training, timely provision of tutorial materials, and supporting
infrastructure could improve tutors’ effectiveness. The study adds to the existing literature
on tutoring in higher education, and in particular, factors affecting tutoring effectiveness. The
findings will be beneficial to higher education policymakers and authorities in designing
1118
effective tutorial and infrastructure structures to support the function within institutions. The
structure of the tutorial system should be remodelled to ensure tutors work in the most
conducive environment. Further studies could include more diverse representation with more
faculties to see if different views will be elicited. Other studies could also be conducted to
reflect tutors’ experiences during face-to-face learning as these might yield different results
from the current one conducted during the covid-19 pandemic; thus, influenced by the online
learning environment. Further, studies could consider other aspects of tutoring such as
personal attributes, and the effectiveness of various tutoring strategies and techniques.
References
Barratt, H. (2009). Methods of sampling from a population.
https://www.healthknowledge.org.uk/public-health-textbook/research-methods/1a-
epidemiology/methods-of-sampling-population. (accessed: 15 September 2021).
Bixby, K., E., Gordon, E. E., Gozali-Lee, E., Gray Akyea, S., & Larson Nippolt, P. (2011).
Best practices for tutoring programs: A guide to quality. Saint Paul, Minnesota: Saint
Paul Public Schools Foundation.
Boud, D., & Molloy, E. (2013). What is the problem with feedback? Feedback in higher and
professional education. London: Routledge Taylor & Francis Group.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research
in Psychology, 3(2), 77–101. Doi: 10.1191/1478088706qp063oa
Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education 6th edition.
New York: Routledge.
Currey, J., Eustace, P., Oldland, E., Glanville, D., & Story, I. (2015). Developing professional
attributes in critical care nurses using Team-Based Learning. Nurse Education in
Practice, 15(3): 232-238. DOI: 10.1016/j.nepr.2015.01.011
DeJonckheere, M., & Vaughn, L. (2019). Semistructured interviewing in primary care
research: a balance of relationship and rigour. Family medicine and community health,
7(2), 1-7. Doi: 10.1136/fmch-2018-000057
Dyomfana, B. (2022). Half of university students drop out in the first year.
https://www.careersportal.co.za/news/half-of-university-students-drop-out-in-first-year
(accessed 25 April 2023).
Fluke, S., O’Connor, A., Hoff, N., & Peterson, R., L. (2014, September). Academic Support,
Strategy Brief. Lincoln, NE: Student Engagement Project, University of Nebraska-
Lincoln and the Nebraska Department of Education.
http://k12engagement.unl.edu/academic-supports. ( accessed 18 August 2021).
Guerra-Martin, M., D., lima-serrano, M., Salvador, & Rodríguez. (2017). Effectiveness of
Tutoring to Improve Academic Performance in Nursing Students at the University of
Seville. Journal of New Approaches in Educational Research, 6(2), 93-102.
10.7821/naer.2017.7.201
Hanson, M. (2021). College Dropout Rates. EducationData.org.
https://educationdata.org/college-dropout-rate (accessed I June 2021).
1119
Holiday, T. (2012). Evaluating the Effectiveness of Tutoring: An Easier Way. Learning

Assistance Review, 17(2), 21-32. https://eric.ed.gov/?id=EJ1002282
Jayachandran, S. (2014). Incentives to teach badly: After-school tutoring in developing
countries. Journal of Development Economics, 108, 190-205.
Joubert, Y., T. & Snyman, A. (2018). Challenges experienced with online tutoring in an ODL
institution. Progression South African Journal for Open and Distance Learning Practice,
39(1), 126-145. 10.25159/0256-8853/2139
Karpur, R. (2018). Factors Influencing the Students Academic Performance in Secondary
Schools in India. University of Delhi, India.
Koehler, J., K. (2018). Tutor Effectiveness of Student-Athletes at a Division I University.
Masters of Education in Human Movement, Sport, and Leisure Studies Graduate
Projects, Bowling Green State University, Ohio.
Mckay, M., T. (2016). Do tutors matter? Assessing the impact of tutors on first year academic
performance at a South African university. Journal of Student Affairs in Africa, 4(1), 53-
64. Doi: 10.14426/jsaa.v4i1.144
Mlachila, M., & Moeletsi, T. (2019). Struggling to Make the Grade: A Review of the Causes
and Consequences of the Weak Outcomes of South Africa's Education System, IMF
Working Papers. https://www.elibrary.imf.org/view/journals/001/2019/047/article-
A001-en.xml (accessed 1 October 2021).
Mlika, M., Zorgati, M. M., & Mezni, F. (2022). The impact of tutor expertise on the students’
scores in active learning methods: a meta-analysis. Journal of Advances in Medical
Education and Professionalism, 10(4): 235–245.
Olokoyo, F., & Oyewo, B. (2014). Making a case for new teaching methods in financial.
Management using empirical evidences from Nigeria. International Conference on
Education and New Learning Technologies, 76(9), 1-19. 10.13140/2.1.4834.7524
Olugbenga, M. (2020). The role of principals in the effective utilisation of instructional
materials among selected schools in Kaduna state Nigeria. Kolkata, India: Exceller Open.
Omona, J. (2013). Sampling in Qualitative Research: Improving the Quality of Research
Outcomes in Higher Education. Makerere Journal of Higher Education, 4(2), 169-185.
10.4314/majohe.v4i2.4
Rahman, M., S. (2016). The Advantages and Disadvantages of Using Qualitative and
Quantitative Approaches and Methods in Language “Testing and Assessment” Research:
A Literature Review. Journal of Education and Learning, 6(1), 102- 112.
10.5539/jel.v6n1p102
Rita, M., F. (2020). School dropouts: Croatia among the lowest rates, far ahead of the EU
and Italy. European Data Journalism Network.
https://www.europeandatajournalism.eu/eng/News/Data-news/School-dropouts-Croatia-
among-the-lowest-rates-far-ahead-of-the-EU-and-Italy (accessed 1 July 2021)
Sibanda, L., Iwu, C., & Benedict, O., H. (2015). Factors Influencing Academic Performance
of University Students. Demography and social economy, 2(24), 103-115.
DOI: 10.15407/dse2015.02.103
1120
Thorburn, B. (2021). New study reveals ongoing connectivity issues in remote learning.
https://www.boltontechnical.co.za/blogs/news/ongoing-connectivity-issues-in-remote-
learning (accessed 6 July 2021).
Yusuf, B. N. M., & Ahmad, J. (2020). Are we prepared enough? A case study of challenges
in online learning in a private higher learning institution during the COVID-19
outbreaks. Advances in Social Sciences Research Journal, 7(5), 205-212. Doi:
10.14738/assrj.75.8211
1121
Globalisation vs diversity in national languages in HE context:

case of Estonia
Peep Nemvalts1, Triin Roosalu2, Eve-Liis Roosmaa2, Helena Lemendik3
1
School of Humanities Centre for Academic Estonian, Tallinn University, Estonia, 2 School
of Governance, Law and Society Institute for International and Social Studies, Tallinn
University, Estonia, 3School of Humanities Centre for Academic Estonian, Tallinn
University, Estonia & Institute of Estonian and General Linguistics, University of Tartu,
Estonia.
Abstract
Given the rapid globalisation since the 1990s and the rising support of a
culture of internationalisation in higher education across Europe, the position
of national languages in academia deserves more attention. The “Strategy for
the Internationalization of Estonian Higher Education 2006–2015” has
diminished the role of Estonian by removing the requirement of the existence
of Estonian-medium education from doctoral education.
In 2011–2012 we surveyed Estonian doctoral students’ opinions and stances
about their usage of academic languages. Only a minority of doctoral students
claimed to be always unrestricted in their language choice, and many veered
towards using primarily English.
Recent strategies in education, research and language aim to advance
Estonian-medium HE, and in 2022 we conducted a re-survey to see how the
situation has changed. This paper presents some results and implications of
our two studies, discussing the current academic language situation and
policies.
Keywords: Globalisation; intercultural strategies; linguistic diversity;
academic Estonian; internationalisation of HE.

Globalisation vs diversity in national languages in HE context: case of Estonia
1. Introduction
Rapid globalisation since the 1990s has significantly impacted national languages used by
scholars, and the language situation in academia has been somewhat contradictory. In the
realm of education, the institutional strive towards internationalisation of higher education
has been supported across the EU, either with specific mobility supporting schemes at the
European level, with national strategies that aim to increase competitiveness in the global
market of education, and with institutional policies that set the goal to benefit from exporting
education by increasing international students has resulted in a culture of internationalisation
within higher education (see discussion in Abdulai et al. 2021). This has increased the
demand for instruction in the English language in higher education institutions across
European countries. Meanwhile, the parallel trends of measuring academic excellence in
citations have privileged publishing in English-language academic journals and rendered
academic writing in any national language with lesser potential readership somewhat less
relevant for aspiring academics.
Those trends are visible in all national contexts, especially the smaller countries. At the same
time, language diversity is a well-grounded principle in the EU: “Article 165(2) of the Treaty
on the Functioning of the European Union (TFEU) emphasises that ‘Union action shall be
aimed at developing the European dimension in education, particularly through the teaching
and dissemination of the languages of the Member States’, while fully respecting cultural and
linguistic diversity (Article 165(1) TFEU).” (EU Language policy)
This paper explores the case of higher education in Estonia, a country with a population of
1.3 million. The Development Plan of the Estonian Language 2011–2017 stated:
““The Estonian Higher Education Strategy 2006–2015” aims to ensure the existence of
Estonian-medium education on all educational levels in all fields of study, thus creating a
possibility of using foreign languages for the teaching of most specialities within the same
field of study. The “Strategy for the Internationalization of Estonian Higher Education 2006–
2015” diminishes the role of Estonian-medium education further by removing the
requirement of the existence of Estonian-medium education from doctoral education.”
Choosing a language is not so straightforward as academics, and students indeed feel
increasing pressure to use English instead of Estonian, given the culture of
internationalisation of higher education. In 2011–2012 our team of linguists and sociologists
from Tallinn University accomplished the study “Estonian as a language of higher education
and academic research” supported by the EU. As a part of this, in 2012, the team surveyed
Estonian doctoral students’ opinions and stances about the usage of academic languages
(Roosmaa et al. 2014).
1124
Given that recent education, research and language strategies for 2021–2035 again aim to
advance Estonian-medium higher education, we conducted a re-survey in 2022. This paper
presents two studies’ key results and implications, discussing the current academic language
situation and policies.
2. Research design and material of the study
2.1. Research methods

In designing the research project, our primary research questions were:
● How do doctoral students in Estonia experience the issue of language choice?
● What kind of difficulties have they experienced in academic work − incl. reading
and writing − both in Estonian and English?
● How much is the Estonian language used for doctoral dissertations in Estonia?
We applied the concept of intercultural strategies introduced by Berry (1997) as an extension
of his earlier concept of acculturation modes.
Doctoral students, like any academicians, live in an intercultural environment, being at the
same time part of their domestic research culture, tradition, and language, as well as of the
international academic arena – the latter most often colonised by English-language research,
some of which published by speakers (or, rather – writers) of English as a second language.
From two fundamental issues – Cultural Maintenance and Contact-Participation – the four
strategies have been derived, facing all peoples living interculturally: integration,
assimilation, separation/segregation and marginalisation. Cultural Maintenance: is it
considered valuable to maintain one’s identity and characteristics? Contact-Participation: is
it considered to be of value to maintain relationships with larger society?
These issues are based on the distinction between orientations towards one’s group and those
towards others (Berry 1997). This distinction is rendered as (i) a relative preference for
maintaining one’s heritage culture and identity, and (ii) a relative preference for having
contact with and participating in the larger society along with other ethnocultural groups. It
has now been well demonstrated that these two dimensions are independent (see overview in
Kruusvall et al. 2009). In this model, the English-language (and ESL) academic culture can
be viewed as the larger society, and national academic languages as ethnocultural groups,
while different ethnocultural groups develop their strategies. However, in terms of
intercultural studies, it is maintained that the best-working strategy relies on both preferences
being strong – that means we assume that the most fruitful integration to the academic world
occurs only when academic thinking is possible, and happening, in the Estonian language
just as well as in the dominant language.
1125
In the broad and representative surveys of doctoral students in Estonian universities, we were
interested about what are their experiences and motives concerning language choice.
2.2. Material of the study

The amount of doctoral students has diminished. In 2012 we had 2,926 doctoral curricula in
Estonian, while this year, there are 1,775 Estonian and Estonian-English curricula in total.
The sample description in table 1 accounts for self-selection – respondents are those who
care about the issue. A vast majority have Estonian as their first language.
Table 1. The sample of 2012 & 2022 surveys.
Sample description 2012 2022
No. of universities participated 7 6
Respondents N 240 101
Gender, % of females 70% 70%
Average age (min/max) 33 (23/64) 37 (24/57)
Estonian as the first language 92% 92%
Main activity: doctoral student 40% 36%
Main activity: employed 54% 49%
Studies are related to work, yes % 80% 75%

Studying social sciences, culture, humanities 49% 61%
1st–2nd year of studies 33% 32%
3rd–4th year of studies 51% 43%

Source: own analysis
We categorised the respondents into two subgroups: doctoral students of social sciences and
culture, incl. humanities (SCH), and those of natural sciences and technical disciplines (NT).
We asked if they have participated in five types of academic activities and in which language:
(1) writing academic research texts to publish,

(2) writing popular science texts,
(3) reading academic research publications,
(4) reading popular science publications,
(5) participating in academic conferences.
1126
SCH doctoral students appear to participate in research or scholarly communication in the

Estonian language in more academic activities than NT doctoral students do. Among both
research classes, the variety in types of scholarly work communication in the Estonian
language has increased from 2012 to 2022, especially among the NT students and particularly
regarding publication writing. In 2022, we observed a significantly higher rate of SCH
doctoral students, 92%, participating in academic conferences held in the Estonian language,
compared to 82% in 2012. Among NT students, there appear to be more writers of academic
(65%) and popular science publications (70%) in 2022, compared to 52% and 50%,
respectively, in 2012.
Comparison over time shows relatively stable results for engaging in academic
communication in English. Overall, differences between the two research classes are minor,
yet interestingly, among NT students, we observe a substantial rise in writing popular science
publications, from 23% in 2012 to 62% in 2022. However, in both research classes, students
write popular science publications in English to a lesser degree compared to other types of
participating in scholarly communication. More SCH doctoral students have mentioned
writing academic publications in 2022 (93%), compared to 78% in 2012.
3. Doctoral students’ opinions on language use
We were interested in what aspects of language have bothered doctoral students when using
one or another language for their academic activities. Results are presented in Figure 1.
Figure 1. Aspects of language use which have bothered doctoral students when reading a text
in one’s study field in Estonian. Source: own analysis
1127
It appeared that when reading a scientific or study text in one’s field in Estonian, doctoral
students of all domains are mainly bothered by terms used and sentence constructions. There
are no notable changes between the two surveys. However, among SCH doctoral students, a
rise in feeling that something is strange occurs, but they cannot name what exactly is strange.
When writing in Estonian, NT doctoral students encounter more difficulties, particularly in
the usage of terms: 58% in 2012 and even more of them, 73% in 2022. The percentage of
those who have no difficulties has further diminished in 10 years, from 17% to 12%, while
more SCH students in 2022 (38%) experience no problems in Estonian writing, compared to
32% in 2012.
3.1. Difficulties in finding appropriate terms

As a part of the study, we focused on two aspects of the usage of terms by doctoral students:
• How do they assess their possibilities to find appropriate terms in the current
academic environment?
• How do they perceive the appropriateness of terms they use in their research?
Firstly, there are two main tendencies in doctoral students’ opinions on terms usage:
1. The share of SCH doctoral students who often cannot find a suitable term in
Estonian has diminished in 10 years from 57% to 48%, while the percentage of NT
postgraduate students who often cannot find a proper term in Estonian has increased
from 71% to 77%. The share of those who never have difficulties finding an
appropriate Estonian term has increased among SCH students. Still, the low (3%) in
natural and technical sciences is unchanged.
2. The share of SCH doctoral students often experiencing difficulties with English
terms has increased; however, the percentage of such NT students has diminished.
Furthermore, the share of those who never have difficulties finding an appropriate
English term has been smaller among SCH students in 2022 (11%) but larger among
NT students (28%).
Secondly, how do doctoral students perceive the appropriateness of terms they use in their
research? Some felt it was best to adopt English terms into Estonian. Twenty-five respondents
argued that a uniform and univocal specialised vocabulary existed in English: "I know
English terms best because I mostly read texts in the same language, and my working
language is English." Among others, 36 students considered Estonian terminology deficient,
hard to find, and lacking uniformity. On the other hand, such term adoption was criticised as
well, explicitly by seven students: "There are many anglicisms instead of correct terms in
Estonian." Doctoral dissertation was suggested to be the means to introduce new Estonian
terms.
1128
4. Conclusions and discussion

The analysis of the data from this survey led to the conclusion that it is easy for newcomers
to academic writing to assimilate into the mainstream host culture of English and give up
"boring" Estonian writing, especially since
• the norms and behaviour patterns of others support this,

• the regulatory practices or research evaluation and standards support this,
• the organisational space where professors and graduate students meet – guidelines
– support this by overlooking the potential incentives to use Estonian.
Estonian doctoral students of every field have difficulties finding appropriate terms and
sentence constructions in 2022, unchanged from 2012. It may indicate that more systematic
courses in academic Estonian and terminology are needed at all higher education levels to
advance clarity of the codification of knowledge and specialised communication. Improved
skills in expressing academic ideas in Estonian may positively impact the educational system.
Hence, the motivation to publish in Estonian should be improved. For now, these possibilities
are few, almost lacking in natural and technical sciences.
The Estonian Language Strategy 2021–2035 does have the general objective to ensure the
vitality and function of the Estonian language as a primary language in every sphere of life
in Estonia, to guarantee everyone the right and opportunity to use the Estonian language, to
preserve and strengthen the status and reputation of the Estonian language and Estonian
cultural and information space, and to value knowledge of other languages. One of the
strategic objectives of this plan includes coordinated terminology planning as a part of
research on the Estonian language to achieve up-to-date specialised vocabulary in every field
of scientific research and every area of life. Another strategic objective for education aims,
among the rest, in collaboration with universities to apply language use and
internationalisation principles in higher education which will grant the functioning of the
Estonian language for higher education and academic research. These objectives are in
harmony with the goals of Education Strategy 2021–2035 to promote the development of
Estonian as a language of higher education and research, including developing and
introducing Estonian terminology; support the maintenance and development of Estonian-
language higher education curricula; reinforce the learning of Estonian by international
students and academic staff. It is also a prerequisite for increasing the knowledge transfer
capacity of research institutions and higher education institutions, set as an objective by the
Research and Development, Innovation and Entrepreneurship Strategy 2021–2035 (RDIE
Strategy).
While perceived as not in line with the culture of internationalisation in higher education, we
claim that implementing these plans will advance language diversity in the globalised world.
Parallel use of national and international academic languages is inevitably needed, advancing
1129
diversity of thinking. Sharing knowledge internationally is also possible in other broadly

spoken languages than English, e.g. Spanish, French, German, and Portuguese – not to forget
regional languages like Swedish in the Nordic countries.
Likewise, a clear academic national language positively impacts general education through
HE, especially with the help of teacher training. Any advanced language promotes the proper
functioning of the society where this language is mainly used.
References
Abdulai, M., Roosalu, T., & Wagoner, B. (2021). International students mobility and value
added through internationalization of higher education in Estonia and Denmark.
Language and Intercultural Communication, 21(5), 602-617. doi:
10.1080/14708477.2021.1900214.
Berry, J. W. (1997). Immigration, acculturation and adaptation. Applied Psychology: An
International Review, 46, 5-68.
Development Plan of the Estonian Language 2011–2017. Estonian Language Foundation.
https://www.digar.ee/viewer/en/nlib-digar:272452/247242/page/1
Education Strategy 2021–2035 = Estonian Language Strategy. Republic of Estonia, Ministry
of Education and Research. https://www.hm.ee/en/media/2919/download
EU Language policy = Language policy | Fact Sheets on the European Union | European
Parliament (europa.eu)
Kruusvall, J., Vetik, R., & Berry, J. W. (2009). The Strategies of Inter-Ethnic Adaptation of
Estonian Russians. Studies for Transition States and Societies, 1(1), 3-24.
Language Strategy 2021–2035 = Estonian Language Strategy. Republic of Estonia, Ministry
of Education and Research. https://www.hm.ee/en/media/2830/download
RDIE Strategy 2021–2035 = Estonian Research and Development, Innovation and
Entrepreneurship Strategy 2021–2035. Republic of Estonia, Ministry of Education and
Research, Ministry of Economic Affairs and Communications.
https://www.hm.ee/en/media/1614/download
Roosmaa, E-L., Roosalu, T., & Nemvalts, P. (2014). Doktorantide teadustöö keele valikutest.
Ülikool ja keelevahetus. Tartu Ülikooli ajaloo küsimusi XLII, 37-52.
https://ojs.utlib.ee/index.php/TYAK/article/view/11876
1130
Bottom-up curriculum innovation through grants for lecturers

Miriam Ossevoort1, Robert Inklaar2, Jan Riezebos3
1
Department Educational Quality, 2Department Global Economics & Management,
3
Department Operations, Faculty of Economics and Business, University of Groningen, The
Netherlands.
Abstract
The aim of this case study is to evaluate the bottom-up approach of curriculum
innovation in higher education through the implementation of Teaching
Innovation Grants (TIGs). Through the TIGs, lecturers were granted hours to
innovate their course. Of the 81 applications, 52 were granted. The TIGs were
implemented in BSc and MSc courses on topics in economics and business.
Each grant touched upon one or more of the educational themes - (1)
internationalization, diversity & inclusion, (2) ethics, responsibility &
sustainability, (3) digital business & data science, (4) employability, and (5)
active & blended learning – the faculty board has formulated as ambitions for
improving and renewing what and how we teach in our degree programs.
Overall, the TIGs has facilitated lecturers in innovating their teaching. They
perceived the workload as manageable, but intensive. Support by teaching
assistants and/or educationalist was an important factor for a successful
implementation.
Keywords: Curriculum innovation; higher education; teaching grants;
bottom-up approach.

Bottom-up curriculum innovation by grants for lecturers
1. Introduction
A sound curriculum - set of courses - of a degree program ensures that learners gain
knowledge and skills they need to succeed in their future career. Currently, the tremendous
advancement in science and technology and the economic and social changes, necessitates
timely revision of a curriculum to avoid learners to gain obsolete knowledge and skills.
Consequently, the curriculum has to change from time to time to achieve the current and
future needs of a sector.
Curriculum innovation is a complex and multi-faceted endeavor. Several processes and
procedures involved and issues of concern in managing curriculum innovation are known
(Law, 2022). Furthermore, several internal and external factors cultivate innovations (Bajada
et al., 2019). Combining a top-down strategy (e.g. emphasizing the operational aspects of the
strategy, effective communication and consultation at all levels) with a bottom-up approach
(e.g. initiatives led by lecturers) will likely lead to success (Lisewski, 2004; Zhu & Engels,
2014). As part of the bottom-up approach, for lecturers (1) involvement from the start of the
curriculum innovation project, (2) time allocated for curriculum innovation, and (3) support
by educationalists seem to be key factors for success (Cooper, 2017; Hurlimann et al., 2013;
Kirkgöz, 2009; Law, 2022).
One bottom-up approach to stimulate and facilitate lecturers in developing and improving
their teaching in higher education is by allocating time and creating support through teaching
innovation grants (TIGs). Namely, each lecturer is an expert in their course and is in the best
position to design and implement change to teaching and assessment. By providing a clear
top-down strategy by the faculty board on its ambition for improving what and how to teach,
lecturers are able to tailor-made this ambition for their own teaching practice.
The aim of this case study is to evaluate the effectivity of the bottom-up approach to foster
educational innovation by using TIGs for lecturers. The procedure of the TIGs is described
in section 2. The questions we want to answer are:
1. How many of the TIGs were received, approved, and implemented?
2. What are the characteristics of the implemented TIGs?
3. What is the outcome of the implemented curriculum innovation perceived by the
lecturers?
2. Setting-up the Teaching Innovation Grants
2.1. Context
The curriculum innovation procedure took place at the Faculty of Economics and Business
(FEB) of the University of Groningen, the Netherlands. FEB’s educational portfolio consists
1132
of four broad bachelor’s degree programs, twelve specialized master’s degree programs, and
one research master program. Yearly, the total number of students is around 8000. In FEB’s
Strategic Plan 2021-2026, the faculty board has formulated ambitions for improving and
renewing what and how we teach. They want to implement or strengthen the following
educational themes in the degree programs: (1) internationalization, diversity & inclusion
(ID&I), (2) ethics, responsibility & sustainability (ERS), (3) digital business & data science
(DB&DS), (4) employability, and (5) active & blended learning. A projectteam of academic
staff and educationalists was installed to manage this project. The project is financed by funds
from the government. The outline of the project was communicated to all staff. For each
degree program a curriculum working group (CWG) consisting of the program management
and lecturers of that program, was installed to innovate the curriculum of their degree
program and implement the above-mentioned themes according to the model described by
Wolf (2007). Besides this top-down approach, bottom-up individual lectures could apply for
a teaching grant if they wanted to innovate their course.
2.2. Teaching Innovation Grants: procedure

The call for proposal for a Teaching Innovation Grant (TIG) was published on January 6th,
2021 on the website of FEB and send by email to all program directors. The proposal should
(1) consist of a concrete improvement in teaching or assessment of a course; the
improvements should relate to changes in teaching and assessment methods, or topics
prioritised in the Strategic Plan (see above), (2) be for innovative ideas beyond regular course
maintenance and updates, that require a more substantive time investment than a change such
as introducing a new textbook, (3) introduce an improvement that can be sustained after the
initial time investment, and (4) envisage concrete improvements in learning outcomes that
will be evaluated. A requirement for submitting a proposal was that it had been discussed
with the relevant program director and that the applicant would evaluate and report on the
implementation of the proposed change. The number of hours allocated for an approved
proposal was 50 hours for the applicant (with the possibility for more if necessary). The
deadline for submission of the proposals was March 1, 2021. The decision was published
before March 31st 2021, so that approved grants could be implemented in the hours allocated
for regular teaching for the academic year 2021-2022.
The evaluation process of the proposals consisted of the following steps: (1) Two project
members, one staff member and one educationalist, read and evaluated each proposal
individually. The evaluation consisted of pass or not pass on the four criteria (a concrete
improvement, a substantive time investment, a sustainable plan, and a change that can be
evaluated), its link with the Strategic Plan’s topics, and the overall quality. All submitted
proposals were divided between three staff members and two educationalists; (2) the final
recommendation to award a grant was made together by one staff member and one
educationalist; they checked all proposal assessments to guard consistency; (3) the steering
1133
group of the curriculum innovation project had to approve the recommendations; (4) as a
final step, all applicants were notified about the final decision. During the evaluation process,
the project team concluded that there was substantial variation in the substantiveness, with
some proposals requiring considerably greater time investment than others, but several of the
smaller proposals would still be very worthwhile. As a consequence, the final
recommendations could be a) award of a full grant (50 hours or more), b) award of a small
grant (25 hours), c) revise or d) decline.
During the implementation of the TIGs in the academic year 2021-2022 (as of September
2021), support was offered by educationalists. An evaluation form was sent out to the
lecturers who received a TIG after the implementation of the innovation (see section 3).
3. Methodology
This paper presents the results of the TIGs implemented in the academic year 2021-2022
from the call of proposals to the evaluation of implemented granted TIGs as a case study. To
describe the characteristics of the approved TIGs, all information, such as in which degree
program the innovation took place, which educational themes of the strategic plan were
addressed, and amount of granted hours, were analyzed. The lecturers filled out an evaluation
form consisting of open questions: questions (1) whether they were successful in
implementing the change or whether they need to make adjustments, (2) what the impact of
the improvement was, (3) whether the workload was manageable, and (4) what they have
learned from potentials and pitfalls of the innovation, whether they will keep it next year, and
what advice they could provide to colleagues. These qualitative data were analyzed and
repeated elements in answers of different questions were grouped and extracted from the
data. The presented results are grouped by educational theme.
4. Teaching Innovation Grants
4.1. Call for proposals

A total of 81 proposals were received. On March 15 all individuals who submitted one or
more proposals were notified about the recommendation including a brief motivation by the
project team. Those who we recommended for a small grant were asked whether they would
carry out their project with the smaller number of hours and all agreed. Those who we
declined were offered the opportunity for further communications on their proposal. Of those
81, the project group proposed granting 52 proposals for a total of 2345 hours. This is about
10% of all courses and 0.3% of the total number of FTE of academic staff at FEB. The granted
hours per TIG was 25 (n=13), 50 (n=37), 80 (n=1), and 90 (n=1) hours. The steering group
agreed to this proposal.
1134
4.2. Characteristics of the granted TIGs.

Of the 52 TIGs that were granted, 25 concerned BSc courses and 27 concerned MSc courses
(Table 1). All the BSc degree programs contained courses that were granted a TIG, of the
MSc degree programs, only the research master and one regular economic-related master
degree pogram did not have courses that were granted a TIG. The BSc Business
Administration had the most TIGs (n=12). The implementation of the TIGs took place in
semester 1 (n=33) and semester 2 (n=19).
Table 1. Level and topics (business or economics) of the degree programs the courses of the
TIGs were implemented.
level Economics Business Total

(n) (n) (n)
Bachelor 10 15 25
Master 8 19 27
The distribution of the TIGs among the five educational themes are listed in Table 2. Multiple
themes could be addressed in one TIG. In case of addressing multiple educational themes in
one TIG, it mostly was a combination of active & blended learning with another theme.
Table 2. The distribution of the implemented TIGs among the five educational themes.
Educational themes n
Internationalization, diversity & inclusion (ID&I) 5

Ethics, responsibility & sustainability (ERS) 4
Digital business & data science (DB&DS) 6
Employability 9
Active & blended learning 30
4.3. Evaluation
Of the 52 grants, two grants were returned by the applicant and five grants were delayed due
to organizational or personal reasons.
Five TIGs on internationalization, diversity & inclusion (ID&I) were implemented, except
for one. One of the implemented TIGs combined this topic with employability. In two of the
TIGs, the tool CATME was implemented to support group formation. Both lecturers stated
that the initial investment to get to know CATME is considerable. One lecturer plans to use
the tool in the future, because CATME offered very useful insights about the interaction
between the team members and team satisfaction. In the other course, the lecturer is still in
1135
doubt of using the tool again, since it is extra work, and the advantage is hard to measure. In
another course an assignment on deep democracy was implemented. The lecturer learned that
students do have an interest to learn about diversity and inclusion. As the lecturer has left and
the costs of execution are high, the assignment will not be adopted structurally. In the fourth
TIG, the implementation was more focussed on using cases to bring ‘practice’ closer to
students in a group assignment.
Only four TIGs included the theme Ethics, responsibility & sustainability (ERS). Three of
the TIGs combined this with active & blended learning, and one was combined with the
themes employability and DB&DS. Looking at the evaluation of these TIGs, actually only
one of these TIGs really focused on developing content on sustainability within a course.
This course was changed considerably adding new video content and a practice-oriented
‘free’ assignment (output could be e.g. a podcast, video, report, etc). The staff member
involved was very pleased with the results indicating that sustainability was integrated in
much greater depth.
TIGs in the theme Digital business & data science (DB&DS) implement one or both of the
following types of innovations: (1) providing students with more interactive tools, to enhance
their ability to practice and learn material outside the classroom, e.g. use SOWISO to
generate practice material for mathematics and statistics (all TIGs), and (2) have students
engage with new tools to strengthen their digital competencies, e.g. use Python to do
optimizations (three TIGs). In terms of implementation, a common theme is the setup costs
of many of these tools, e.g. developing tips for every question in a course or developing new
quizzes and in some cases, the technical aspects were a barrier.
Nine TIGs related to employability. These TIGs address: (1) introduction of tools and training
for employability skills, and (2) development of material to bring “business life” into the
classroom. Both types of projects benefit from the use of videos, for example as a way to
provide information about different skills, to enable peer review of presentation skills, or as
a way to show how processes work in a business. The evaluations of the TIGs relating to
employability skills are positive about the TEL-tools that are used. Due to changes in
available budgets, one tool will not be used further, for another tool technical problems need
to be resolved for using it next edition of the course. Interestingly, two TIGs that were
implemented in courses of the same program intentionally addressing a different skill,
resulted in a optimal program-level alignment.
Of the 30 executed TIGs related to the theme active & blended learning, 22 were on active
learning only, the other combined active learning with employability (3), ERS (3), or
DB&DS (2). 23 of the 30 TIGs were implemented as planned, seven TIGs were partly
implemented due to COVID measures (3) or changing insights (4). The plans to implement
active and blended learning improvements were very diverse, e.g. using technology-
1136
enhanced learning (TEL) tools, flipped classrooms with knowledge clips, team assignments,
weekly quizzes, and interactive tutorials. In all TIGs students were more engaged (e.g.,
watched video lectures to learn content before plenary session, SOWISO), active (e.g., asked
more questions, weekly quizzes), and interactive (e.g., giving peer feedback using
FeedbackFruits, working in teams on cases or assignments, discussions in Q&A
sessions/lectures).
Overall, most lecturers stated that the newly developed tools/assignments/teaching methods
will be kept and improved if applicable. The workload of the innovations varied from
manageable to manageable due to support of colleagues, teaching assistants or
educationalists.

The aim of this paper is to evaluate the bottom-up curriculum innovation through the
implementation of Teaching Innovation Grants (TIGs). The TIG scheme has stimulated and
facilitated lecturers in developing and improving their teaching and implementing the faculty
board’s ambitions for improving and renewing what and how we teach through five
educational themes. Of the 81 applications, 52 were granted, but due to organizational or
personal circumstances two grants were not executed and five were delayed. The
implementation took place in BSc as well as MSc courses of different degree programs. Each
grant touched upon one or more of the educational themes. Lecturers perceived the workload
as manageable but intensive. Support by teaching assistants and/or educationalist was an
important factor for success. Both factors have also previously be shown to contribute to the
success of curriculum innovation (Cooper, 2017; Hurlimann et al., 2013).
Due to the success of this first round and to further implement the educational themes in our
degree programs, we continued with granting TIGs in the following academic year. Again,
allocating time for lecturers and support by educationalist and/or teaching assistants will be
offered to contribute to its success. Based on the experiences in the first round, applicants
will be advised about the good practices as well as potential pitfalls through a repository that
will be developed of the first TIGs to share experiences. This repository will contain
information about e.g. the type of curriculum innovation, the link to the educational themes,
and how it was implemented. This will allow other lecturers to find specific types of newly-
developed teaching practices to be used in their own courses.
1137
Acknowledgements
Authors thank Richard Jong A Pin, Maryse Brands and Giulia Witcombe for their input in
the selection procedure and Lisanne de Jong for providing information for writing this
manuscript.
References
Bajada, C., Kandlbinder, P., & Trayler, R. (2019). A general framework for cultivating
innovations in higher education curriculum. Higher Education Research & Development,
38(3), 465-478. doi: 10.1080/07294360.2019.1572715.
Cooper, T. (2017). Curriculum renewal: barriers to successful curriculum change and
suggestions for improvement. Journal of Education and Training Studies, 5(11), 115-
128. doi: 10.11114/jets.v5i11.2737.
Hurlimann, A., March, A., & Robins, J. (2013) University curriculum development – stuck
in a process and how to break free. Journal of Higher Education Policy and Management,
35(6), 639-651. doi: 10.1080/1360080X.2013.844665.
Kirkgöz, Y. (2009). The challenge of developing and maintaining curriculum innovation at
higher education. Procedia Social and Behavioral Sciences, 1, 73-78.
Law, M. Y. (2022). A review of curriculum change and innovation for higher education.
Journal of Education and Training Studies, 10(2), 16-23. Doi:10.11114/jets.v10i2.5448.
Lisewski, B. (2004). Implementing a learning technology strategy: top-down strategy meets
bottom-up culture. Research in Learning Technology, 12(2), 175-188. doi:
10.1080/0968776042000216228.
Wolf, P. (2007). New directions for teaching and learning. Chapter 2: A model for
facilitating curriculum development in higher education: a faculty-driven, data-informed,
and educational developer-supported approach. Online Wiley Periodicals, Inc. doi:
10.1002/tl.294.
Zhu, C., & Engels, N. (2014). Organizational culture and instructional innovations in higher
education: perceptions and reactions of teachers and students. Educational Mangement
Adminstraction & Leadership, 42(1), 136-158. doi:10.1177/1741143212499253.
1138
Designing a Metaverse for an Immersive Learning Experience

Mark Frydenberg1, Shivam Ohri2
1
Computer Information Systems Department, Bentley University, Waltham, MA, USA,
2
MBA and MS/Human Factors and Information Design Candidate, Bentley University,
Waltham, MA, USA.
Abstract
This paper shares preliminary findings of a pilot project to study the potential
of the metaverse as an option for course delivery in an immersive learning
environment. As part of a first-year seminar course on the metaverse, the
authors identified and applied design principles for building an effective
immersive learning space to create a virtual space that mirrored a popular
campus technology learning center. After learning about the metaverse
throughout the course, students attended a class session online to experience
this virtual environment, and then reflected on their own interest in and
perceived value of immersive learning spaces.
Results from a student survey suggest that while students are skeptical about
their own continued involvement with the metaverse after the course is over,
many recognize potential of the metaverse as a social platform that fosters
engagement and collaboration.
Keywords: virtual reality; metaverse; immersive learning spaces;
collaboration; online learning; emerging educational technologies.

1. Introduction
In the years since Neal Stephenson introduced the concept of the metaverse in his novel Snow
Crash (Stephenson 2003) in 1992, technology has advanced to turn his science fiction vision
of living in virtual worlds into reality with the development of virtual reality (VR). The
novel’s characters visited an alternate world while wearing goggles and experienced it
through the eyes of their avatars, just as people engage with the metaverse of today.
A single definition of the metaverse does not exist, and the use of the term often differs based
on context. The metaverse is not one unique virtual space; today the metaverse is a collection
of decentralized virtual spaces, or worlds, such as Meta Horizon Worlds, Roblox,
Decentraland, and the Sandbox, where people gather synchronously online to conduct
business meetings, play video games, attend social events, purchase real estate, and visit
destinations they may not be able to get to in real life. Many metaverses have their own digital
economies. Avatars or digital twins form a person’s online identity across metaverses.
(Kshetri 2022). At its most basic level, the metaverse is the next iteration or future of the
Internet (Ramesh et al. 2022) as a Web3 application that brings value to the user generated
content that characterizes Web 2.0 (Suderman 2022).
The advent of the COVID-19 pandemic accelerated the introduction of immersive virtual
environments for educational purposes (Duan et al. 2021). Three-dimensional, immersive
virtual learning spaces often reflect physical campus locations creating a stronger connection
to the university campus; participants interact with each other through their avatars, causing
students to consider issues of identity and how they wish to be perceived in the metaverse.
They may also interact with elements of the virtual learning spaces, such as writing on virtual
whiteboards or sharing websites on virtual browsers. They can communicate in real time
through voice or chat, or share a webcam, as in most web-based video conferencing
applications. Such spaces and their content persist from one visit to the next, and as such,
foster the development of virtual communities (Kye et al. 2021), and some educational
metaverses may offer the purchase of digital goods.
This paper describes preliminary results and the experiences of incoming first-year students
enrolled in a discovery seminar studying the metaverse from multi-disciplinary perspectives.
As part of the course, the authors designed and created an immersive virtual learning space
modeled after a campus learning space. During the last class meeting, students visited this
virtual space using VR headsets or web browsers to experience the metaverse’s potential as
a viable course delivery platform. Questions guiding this research are:
RQ1. What principles should be considered when creating an immersive virtual
learning space designed to resemble a campus physical learning space?
1140
RQ2. After experiencing a class in an immersive learning space, what potential

values do students see for learning in such an environment?
2. Immersive Educational Environments in the Metaverse

Many educational institutions are exploring virtual immersive environments (Hassanzadeh
2022; Hedrick et al. 2022) yet “few … have taken steps to offer courses in [a] metaverse
(Hassanzadeh 2022:10).” Such environments can improve learning, empower students, and
create a sense of presence among students (Hassanzadeh 2022). Virtual reality enables users
to experience immersive spaces that cannot exist in reality (Kye et al. 2021). Several digital
tools, including Alakazam, AltSpace, Frame, Gathertown, and Zepeto, facilitate the process
of designing immersive virtual spaces.
Duan (2021) created a model of the Chinese University of Hong Kong, Shenzhen (CUHKSZ)
as a metaverse for social good, “a mixed environment where students’ actions in the real
world could correspondingly affect the virtual world, and vice-versa.” Location based sensors
allow students in physical locations to interact with students in equivalent locations in the
corresponding virtual world. Such worlds require significant usability testing to be effective
(Lee and Gu 2022).
The use of immersive classrooms also requires significant onboarding and training. When
using Meta’s Horizon Workrooms as a virtual classroom, Hedrick (2022) found that
gathering in a virtual space complete with desks, a whiteboard and a projector had its
limitations in group size, ease of setup, and lack of documentation for a beta product. Still,
they found that collaboration was possible and successful in such a virtual environment after
the group acclimated to the user interface.
A goal for creating an educational metaverse is to provide a sense of community and link
back to a familiar classroom or physical learning space. How one joins an educational
metaverse has an impact on the user experience and participant retention. While a fully
immersive experience with or without VR headsets may be preferred, “whether it satisfies
students who want to experience learning that does occur in the same place or at the same
time with other peers, is still unanswered (Lee and Hwang 2022)”. Students sitting at
computer screen or wearing VR headsets with pre-designed digital content “are less likely to
experience the psychological sensation of being in a communal space (Lee and Hwang
2022)”.
3. First-Year Discovery Seminar: Living in The Metaverse

This study, conducted at the end of the Fall 2022 semester, involved students enrolled in two
sections of a first-year discovery seminar (FDS) course. These sections were taught by two
1141
different instructors who followed the same schedule, assignments, topics, and readings. FDS
is intended to help students acclimatize to their new surroundings in college and learn about
research by exploring complex problems and themes from multidisciplinary perspectives.
The theme of the two sections discussed in this paper is “Living in the Metaverse.”
The course presented an overview of the metaverse from business, social, and technology
perspectives, and provided several hands-on opportunities to explore various metaverses both
in a browser and wearing VR headsets. Students attended events in AltSpace, designed virtual
worlds and games in Roblox, created avatars in ReadyPlayer.Me, and investigated metaverse
features in Decentraland. Their research papers focused on issues from governance to the
impact of the metaverse on mental health.
During the final two-hour class session, students met in a virtual learning space modeled after
the university’s technology learning center. Before the class, students added slides
summarizing their research papers, to a common PowerPoint Online slide deck. The
instructor showed the slide deck on one of the floor-to-ceiling displays, as students
maneuvered their avatars around the virtual space to see it. They listened to their classmates
present their work in real time.
Many students participated in the metaverse activity using both their laptops and wearing VR
headsets. Students joined on their laptops, positioning themselves in hallways and classrooms
in the building where the class was held or wearing earbuds to avoid audio interference and
feedback from students nearby. At 15-to-20-minute intervals, students proceeded in assigned
groups of five to the nearby technology learning center, where they wore Quest 2 headsets to
join the class for an immersive VR experience, until the next group arrived for their turns.
This process allowed students to experience the immersive learning space in two different
modalities. Students then gathered back in person in the physical classroom to debrief about
their experiences for the last 15 minutes.
4. Designing an Immersive Metaverse Learning Space

Frame (http://framevr.io) is the development tool chosen for the design and implementation
of a three-dimensional immersive social virtual meeting space replicates the university’s
technology learning center. Visitors can explore spaces created in Frame using mobile
devices, laptops, or desktop computers through a browser, or by wearing a VR headset (such
as Oculus Quest), making Frame a popular choice for prototyping, or developing virtual
spaces. Frame metaverses support accessories such as floor-to-ceiling displays, live web
browsers, and interactive whiteboards. Frame is persistent in that content added remains in
place from one visit to the next. “Frame is widely used as an adaptive learning space suitable
for educational purposes as it allows teachers and students in the same place to freely upload
a wide range of content to deliver curriculum” (Lee and Hwang 2022). This proof-of-concept
1142
immersive metaverse learning space was designed to stimulate the sensory input of the
participants. The process was implemented in three phases: creating a shared experience,
extending realism, and incorporating gamification, each abiding by principles of human-
computer interaction. (Lee and Gu 2022; Terblanche 2014).
4.1. Phase 1. Creating a Shared Experience

Phase 1 emphasizes creating a shared experience and maintaining a sense of individualism.
Every participant in the metaverse can fully-customize their 3D avatar and choose their
clothing, accessories, body color, hairstyle from a collection of preexisting assets in Frame.
The virtual space also enables participants to create digital identities by designing
personalized avatars or linking their ReadyPlayer.Me avatars, created by generating 3D
models from 2D photos captured through a webcam. This enables students to maintain their
individual identity in the mirrored digital classroom.
4.2. Phase 2. Conveying Realism

Phase 2 extends the concept of realism as cues to educate the participants, thereby setting the
FTUE (first-time user experience) for the metaverse. The essence of maintaining these real-
world cues for a student participant goes beyond humans to virtual environments and objects.
The color theme of the walls in the room, placement of smart screens, tables, monitors, and
keyboards, to the most vital element of the present time – sanitizer bottles on a table, all
resonate to create a sense of attachment which delivers an exceptional, easy-to-adapt human-
centered FTUE. Figure 1 compares the mirrored virtual metaverse (top) with the physical
space of the campus learning center (bottom).
Users entering the virtual classroom for the first time rely on these cues to navigate a virtual
space that resembles the physical world. One significant difference between physical and
virtual spaces is the absence of seating in the virtual environment. To create a more open
space, moveable chairs are not included in this design because avatars do not need to sit.
4.3. Phase 3. Gamification

Phase 3 targets practical features by gamifying the new age of online learning enabled by
gathering in the metaverse. Adding gaming elements into non-gaming sectors intrigues users
to take part and perform (Groh 2012). “Gamified elements used in the Metaverse allow
immersive and engaging ambiance, positively influencing the user mindset for social
interaction (Tayal, Rajagopal, and Mahajan 2022:1598).” This immersive space features a
floor-to-ceiling fully functional web browser displaying actual online content, a whiteboard
to sketch, and an ultrawide floor-to-ceiling LED screen that everyone within the shared
environment can view. These elements provide engaging, interactive, intuitive learning tools
1143
within the immersive learning space. Figure 2 is a snapshot of a live session conducted with
students and shows the use of a metaverse platform as a shared learning environment.
Figure 1. Top: Virtual metaverse; Bottom: Physical campus learning center
Figure 2. Live class session
5. Results
The authors administered a survey to 44 students in two synchronized sections of FDS, taught
by two different instructors, on the last day of each class meeting. Two students were under
18 years of age, so their responses were not included. The remaining 21 students from each
section were between 18 and 21 years old. All had used Quest headsets while completing two
assignments earlier in the course. Responses showed that 27 of the 42 eligible students agreed
or strongly agreed that they would like to explore VR more with headsets, 20 agreed or
strongly agreed that VR environments offer a sense of place and connection to a physical
1144
environment. Still, they were divided on their future level of participation in the metaverse.
Only 12 agreed or strongly agreed that they could see themselves involved in a virtual
community, and only 13 agreed or strongly agreed that they could see themselves attending
an event in the metaverse. See Figure 3.
Figure 3. Survey data of participants
These results suggest that learning about the metaverse was fun and engaging, but many
students are still skeptical about their personal involvement for extended periods. Students
realized how the metaverse can provide new opportunities for doing business and interacting
socially, but they also commented on social concerns related to increased engagement with
the metaverse (abundance of virtual meetings, lack of social skills and lack of motivation).
Said one student, “People may not be as willing to interact and speak with one another,
whether online or in-person. The possibility of harassment [increases] because of the lack of
authority in the metaverse… With no bounds to hold them accountable for their actions,
people might freely bully and act harshly in the metaverse.”
6. Conclusion
The ability to create immersive and interactive learning environments that simulate real-life
experiences could significantly enhance student engagement and comprehension.
Preliminary results suggest that students find the idea of learning in the metaverse to be
compelling, providing access to educational resources and experiences that may not be
feasible in traditional classroom settings.
Students experienced that meeting in the metaverse may change the future of instruction, and
these interactions can benefit them in the real world. Exploring an immersive learning space
was new for most students and provided a tangible way to relate to the conceptual knowledge
gained by studying aspects of the metaverse. After the course, several students reflected that
1145
they would have liked fewer readings, more activities using VR headsets, and more
opportunities to explore and design virtual worlds. The need for technical infrastructure and
digital literacy among educators and students are potential challenges. As technology
advances, increased accessibility of hardware and the influx of human-centered interfaces
will influence the design of a new kind of digital classroom.
References
Duan, Haihan, Jiaye Li, Sizheng Fan, Zhonghao Lin, Xiao Wu, and Wei Cai. 2021.
“Metaverse for Social Good: A University Campus Prototype.” Pp. 153–61 in
Proceedings of the 29th ACM International Conference on Multimedia.
Groh, Fabian. 2012. “Gamification: State of the Art Definition and Utilization.” Proceedings
of the 4th Seminar on Research Trends in Media Informatics.
Hassanzadeh, Mohammad. 2022. “Metaverse, Metaversity, and the Future of Higher
Education.” Sciences and Techniques of Information Management 8(2):7–22. doi:
10.22091/stim.2022.2243.
Hedrick, Emily, Michael Harper, Eric Oliver, and Daniel Hatch. 2022. “Teaching & Learning
in Virtual Reality: Metaverse Classroom Exploration.” Pp. 1–5 in 2022 Intermountain
Engineering, Technology and Computing (IETC).
Kshetri, Nir. 2022. “A Typology of Metaverses.” Computer 55(12):150–55. doi:
10.1109/MC.2022.3204978.
Kye, Bokyung, Nara Han, Eunji Kim, Yeonjeong Park, Soyoung Jo, and Sun Huh. 2021.
“Educational Applications of Metaverse: Possibilities and Limitations.” Jeehp 18(0):32–
0. doi: 10.3352/jeehp.2021.18.32.
Lee, Han Jin, and Hyun Hee Gu. 2022. “Empirical Research on the Metaverse User
Experience of Digital Natives.” Sustainability 14(22):14747. doi: 10.3390/su142214747.
Lee, HyeJin, and Yohan Hwang. 2022. “Technology-Enhanced Education through VR-
Making and Metaverse-Linking to Foster Teacher Readiness and Sustainable Learning.”
Sustainability 14(8):4786. doi: 10.3390/su14084786.
Ramesh, U. V., A. Harini, Ch. Sri Divya Gowri, K. Viyshnavi Durga, P. Druvitha, and K.
Siva Kumar. 2022. “Metaverse: Future of the Internet.” International Journal of Research
Publication and Reviews 3(2):93–97.
Stephenson, Neal. 2003. Snow Crash. Random House Worlds.
Suderman, Peter. 2022. “The Metaverse Is Already Here.” Reason, July, 46–53.
Tayal, Swati, K. Rajagopal, and Vaishali Mahajan. 2022. “Virtual Reality Based Metaverse
of Gamification.” Pp. 1597–1604 in 2022 6th International Conference on Computing
Methodologies and Communication (ICCMC).
Terblanche, Juanita T. 2014. “Using HCI Principles to Design Interactive Learning
Material.” Mediterranean Journal of Social Sciences 5(21):377.
1146
Technology-enhanced learning: Cloud Computing to implement

cooperation among schools
Paolo Musmarra1, Francesco Saverio Tortoriello2, Ilaria Veronesi3
1
Department of Computer Science, University of Salerno, Italy, 2Department of Mathematics,
University of Salerno, Italy, 3Department of Mathematics, University of Salerno, Italy.
Abstract
In this article we describe an interdisciplinary laboratory implemented thanks
to the technology designed as part of the activities elaborated in the
Mathematical High School Project by the research group of the Department
of Mathematics of the University of Salerno. Four classes of different schools,
led by their teachers and coordinated by university researchers, have produced
an educational computer game by sharing the various statuses of the work in
the cloud. The teaching activity was aimed at deepening literary texts
concerning mathematics transformed into a game thanks to the skills acquired
by the students in computer science in which the advancement in the various
levels required the resolution of enigmas and mathematical riddles. In this
work, we will also present the topics that lead us to evaluate in an extremely
positive way the use of cloud computing in curricular teaching in a wide vision
of knowledge.
Keywords: Interdisciplinarity; cloud computing; education; technology;
mathematical language.

Technology-enhanced learning: Cloud Computing to implement cooperation among schools
1. Introduction
Contemporary society is defined as "global" because every action, every cultural, economic,
social, and political choice has repercussions that are independent of the context in which it
took place and determine effects and interrelationships in much broader contexts. The school
must therefore encourage students to carry out collaborative activities both in the form of
group work also through the structuring of paths that involve wider networks, for example
between geographically distant schools. In this way, in addition to broadening the cultural
horizon of the participants, a sharing of good educational practices is developed. Technology
is certainly an extremely effective tool to encourage these interactions even at a distance
because, in economic terms, it allows the meeting, exchange and share processing through
computer tools and programs. This has therefore led to the design of experimental laboratory
paths oriented in this educational direction. The use of technology is natural to high school
students who are part of that category called "digital natives", therefore the choice to exploit
tools close to the daily life of youngers leads them to feel like main actors, the protagonists
of the formation of their knowledge with an active role in the construction of their skills.
Consequently, it is important to observe and analyze the educational effects of activities that
are developed in collaboration among different schools and research institutes, sometimes
very far both for the distance and for the educational paths, thanks to virtual platforms.
In the “Sustainable Development Goal 4 - Ensure inclusive and equitable quality education
and promote lifelong learning opportunities for all” of the 2030 Agenda for Sustainable
Development, that is guided by the purposes and principles of the Charter of the United
Nations, we read: “The spread of information and communications technology and global
interconnectedness has great potential to accelerate human progress, to bridge the digital
divide and to develop knowledge societies… By 2030, substantially increase the number of
youth and adults who have relevant skills, …substantially increase the supply of qualified
teachers, including through international cooperation for teacher training in developing
countries,…” (Agenda 2030).
Sharing and collaboration become more difficult to manage when institutions are placed in
distant locations. In these cases, the technologies are powerful and effective tools for
cooperative work and the Cloud is a widespread methodology for managing the
administration of educational institutions: the electronic register, which is also accessible to
students and parents, database files and material sharing in the various departments are all in
cloud mode. The use of these powerful tools for sharing in educational activities is much less
widespread even if it could become extremely powerful. In fact, with Cloud Computing,
teachers and students may implement collaborative learning thanks to the use of archiving,
exchange, and online document editing (Alabbadi,2011), (Dmitriev et al., 2012), (Sultan,
2010). Generic software, like Google Apps for Education or Microsoft Office, offers online
productivity applications such as word processing, spreadsheets and functions that can be
1148
Paolo Musmarra, Francesco Saverio Tortoriello, Ilaria Veronesi
used in the classroom (Bennett, Pence, 2011), (Bonham, 2011), (Nevin, 2009) (Rienzo, Han,
2009). This provides to work and share progresses or results in real-time. Cloud services
become a powerful architecture for performing complex large-scale computing tasks and a
wide range of IT functions, from archiving to computing, database and application services.
The cloud is also convenient for teachers as it does not require complex software operations,
it is enough to connect to a web browser and learn simple instructions to access shared
platforms. It is even economical because last-generation software is used without having to
buy it. The schools are very sensitive to this theme because of the scarce economic resources
available, so the business model of sharing through the cloud may offer excellent insights
into the world of school and, even if it has only been talked about for a few years, it is not a
recent idea (Baldassarre, 2018), (Katiyar, 2018).
In our research-work, we choose the Cloud platform as a privileged tool for the development
of a teaching model that is oriented to a cooperative-metacognitive approach that encourages
students to improve study and learning strategies, the management of variables involved in
the construction of their knowledge, the social mediation and positive interdependence. The
elaboration of students' responses to the experiential stimuli favored by teachers, highlights
the production of skills in individual cognitive potentialities that are particularly enriched by
the exchange in different school contexts. As Vygotskij (Vygotskij, 1934) argues, cognitive
development theory focuses on the important contributions made by society to individual
development. This theory emphasizes the interaction between developing individuals and the
culture in which they live and considers human learning as a social process. Therefore, social
interaction between students is effective for developing skills and strategies and the
interaction between students and teachers is fundamental for cognitive development since
teachers are more competent people in different contexts.
The choice of the theme for the laboratory activities is linked to the desire to develop the
entire relationship between language and thought in the various areas of knowledge and with
different communication channels. The narrative book with mathematical content has been
deepened, analyzed in the various literary and scientific fields, dissected and subsequently
reconstructed to be reproduced in the context of a video game (with various narrative
moments in the form of a short cartoon). The whole process was developed through a careful
analysis of the different languages to be used in different contexts. The contents of the book,
as cultural mediators, have also been developed through the use of various languages and this
led students to enhance critical thinking through processes of analysis and synthesis at a high
level of skills.
The decision to remotely share the materials on the platform led students to improve digital
skills with a specific language; the need to discuss via the web about the choices to be
implemented for the activities, required the commitment of the students to pay particular
attention to reciprocal communications to avoid misunderstandings in the activities to be
1149
carried out. The development of computer works close to the habits and lifestyles of youths
led them to acquire transversal skills and experiences without a specific added effort.
2. Cloud Computing, a powerful didactical tool to develop transversal skills
2.1. The background: the project “Mathematical High School”

The "Mathematical High School Project" (MHS) is a project developed by the Department
of Mathematics of the University of Salerno in 2015 in collaboration with several
Departments of the same University. Currently, the project has expanded nationwide and
involves 26 universities and over 160 high schools. The activities are developed during
extracurricular hours and are presented in collaboration between university researchers and
teachers of the schools involved. The MHS activities aim at building interdisciplinary skills
that go beyond the fragmentation of knowledge into the various school subjects and make
use of the full potential of technology in the educational domain by exploring innovative
experimental paths (Tortoriello, Veronesi, 2021), (Bimonte et al., 2022). Mathematics
becomes the universal language that is declined in various areas and is the link between the
two cultures, the humanistic and the scientific ones (Rogora, Tortoriello, 2021). The digital
platforms that facilitate the sharing of materials have opened up new avenues for fostering
collaborative networks between universities and schools, thereby enabling a more
comprehensive and inclusive approach to collaboration. This has resulted in the creation of
diverse pathways in MHS schools that emphasize the use of innovative technologies to
promote an interdisciplinary approach underpinned by constructivist pedagogy.
2.2. Our research – a “digital storytelling” experience

In this paragraph, we report the experience of an activity developed within the MHS project
starting from the school year 2017/18. In that year the project envisaged the participation of
university researchers, school teachers of Italian literature, Math and Computer Science and
four 10th-grade classes from MHS schools located in different cities in South Italy. About
eighty students participated in the activities.
The aim of the Laboratory was to develop an interdisciplinary path between literature and
mathematics through the reading of famous books that dealt with mathematical ideas and
contents in narrative form, to transform the plots into a serious game with enigmas and
mathematical riddles to overcome the levels and continue the narration of the stories. In this
way, mathematical literary contents and computer skills become both the object of the study
and the development tool of a path that is certainly fascinating and captivating for students.
The researchers divided the activities to be carried out among schools to develop the video
game with the Skretch software, choosing for each school the best area in which to produce
materials consistent with what has been done in the activities of the MHS. In particular, a
1150
group of students choose within the plot of the book which moments to develop in the
activity, another group of students developed the backgrounds and the play environments,
and the last group of students built the actions on the characters designed by students from
other groups. Upstream of computer programming activities, students with their Italian
literature professors read books that had become a curricular subject of teaching activities. In
the various passages with references to geometry and mathematics, mathematics teachers
joined colleagues realizing activities in didactic cooperation designed by the teaching team
through online meetings and sharing of material and exchange of ideas on Cloud platforms.
The books, in the form of a fictionalized adventure, deal with many maths topics that students
face in their school curriculum of the upper two-year course and which have been the starting
point for developing collaborative didactic laboratory activities and for designing skills tests
to advance in the educational game path. In a meeting on an online platform, the researchers
and teachers of the schools shared a development plan for the entire course. Firstly, the
cultural moments of the educational path were outlined: the reading of the book in class, the
discussion of the mathematical themes addressed and the use of the language used by the
authors, the synthesis of the contents of the book reworked as the plot of a video game.
Subsequently, the scanning of the contents to be developed for each group of students and
the realization times of the various components of the videogame were defined.
Students worked under the supervision of their teachers, who provided them with help and
support following the schema:
Step 1: brainstorming - The first phase of the activity was dedicated to reading books in
Italian, including "What is the name of this book?" (Raymond Smullyan), "The Parrot
Theorem" (Denis Guedj), "A Tangled Tale" (Lewis Carroll). During the teaching activity in
the classroom, the teachers created working groups in which the students deepened the
analysis of the texts, focusing on the characterizing passages and subsequently the reduction
of the plot in the form of video game levels
Step 2: creating scenes - In this step, whose duration is 2 weeks, the students worked on the
direction and screenplay: they wrote the storyboard, built the sequences and described the
contents and scenarios of the various levels. The activities have been organized as follows:
during the curricular time, in the classes, teachers helped their students to arrange their ideas,
write the storyboard and organize the story sequences; at home, students collaborated in
writing scenes using cloud resourcing and storing them in a repository on a shared drive.
Periodically teachers organized online briefings with students to verify the work in progress.
The skills that the students had to put in place in this didactic moment consisted precisely of
the use of language. They had to read books and narrative texts with numerous mathematical
references recalled with rigorous sectorial vocabulary. They then had to synthesize the plot,
break it down into sequences and define the main ones to be selected. For the design and
1151
development of the video game, they had to rework the texts transforming them into short
dialogues in a theatrical readjustment. All this had to happen without losing the plot of the
book, thus keeping the product rich in content but fluid in its playability to stimulate interest
in the mathematical contents of the game. Obviously, the answers to questions, riddles and
route choices can be found right in the book mentioned so players are encouraged to read or
re-read the book in the various steps to look for key clues that allow progress in the game
levels. We chose to use narrative text because it is widely used in literature as a powerful tool
to implement the achievement of mathematical objectives, now it is common practice to
contextualize mathematical problems in contexts of reality that are perceived as less
"abstract" and closer to the fields experience and knowledge of the students and thus
diminishes the fear of trial and evaluation. Students also demonstrate that coordinated
logical-mathematical thinking and narrative thinking cooperate more effectively through the
narrative tool because intuition and creativity make it possible to convey arguments that are
subsequently structured with the formal rigor of the scientific sphere through a rational
analytical development.
Step 3: creating the digital story - In this phase, lasting about a month, under the guidance of
teachers and researchers developed the video game with the Scratch software using the
materials that had been produced in advance and saved in Drive, then created the scripts of
the various phases and the various levels of the game, processed the backgrounds with the
scenes, built the characters adding the music and videos. They developed the activity so that
the player is the protagonist that advances in the game through challenges to be solved with
different skills: cultural and scientific skills, mathematical logical skills, speed and dexterity,
visual and spatial memory exercises and logical reasoning.
Figure 1. Two scenes from the digital story
Step 4: presentation - The presentation of the work took place at two different times. First, in
each school involved in the project, the teachers of the classes organized a seminar in which
the students presented the video game they had produced to other boys and girls of their
Institute. Subsequently, during the National Seminar of Mathematical High Schools at the
University of Salerno a workshop was organized in which students presented to their peers
(with a “Peer-to-peer” education model) and subsequently to the teachers who attended the
conference (with a “flipped classroom” approach) the activities they had carried out and the
result they had obtained, that is the videogame.
1152
3. Conclusions
In our educational path, we have chosen to develop a constructivist learning model which
through the use of technologies has intended to develop interdisciplinary paths with close
interconnections between the mathematical area and the humanistic area in which students
have been made protagonists of their process of learning both as videogame directors and
developers and as gamers. We also wanted to develop a collaboration path between students
from different schools and cities through cloud computing platforms to create virtuous
connections and circuits for sharing good educational practices. since this path was started
two years before the covid forced us to carry out distance learning, we believe that it was a
far-sighted starting point for the development of the mathematical high school.
In this paper, after a careful examination of the various definitions of cloud computing and
an introduction to the implementation of cloud computing in education, the digital
storytelling activity carried out with crowdsourcing and cloud techniques is presented. the
use of a constructivist approach enhanced with technologies has shown that the participation
of students and the quality of the video game as the final product of the course were
qualitatively excellent thanks to the use of the cloud and the sharing of results step by step,
above all, the students' skills on the topics addressed are superior to those obtainable with
traditional teaching. According to teachers' support, we can conclude that this model allows
students to learn by doing, enriching the path with active participation.
The experimental laboratory offered teachers and researchers the opportunity to observe the
didactic impact on students. Various aspects emerged that deserve emphasis:
• The path develops in the literary, mathematical and computer fields and allowed
each participant to deepen the topics covered by choosing the cultural environment
most congenial to their study preferences, so each student has always felt involved
and interested having been able to evaluate how to work in the project;
• The students worked in groups within their class and compared with other classes
interacting thanks to the cloud for sharing materials and remote online meetings.
This allowed them to develop soft skills such as communication, respect for other
points of view, team collaboration, designing with teamwork through the sharing of
skills and expertise;
• The choice to center the entire project around the construction of a Scratch
videogame allowed to convey information and knowledge of less attractive
disciplines such as mathematics and literature through the semiotic mediation of the
artifact in the interaction of tools, methods and languages that enabled the
strengthening of students' skills in an interdisciplinary key.
The activities did not include a formalized assessment mechanism to stimulate students’
participation. The reports generated by the students at the completion of the project indicate
1153
a profound sense of contentment with the activities undertaken, evincing a strong inclination
towards future participation in similar ventures.
References
Agenda 2030 https://sdgs.un.org/2030agenda
Alabbadi, M. M., Cloud computing for education and learning: Education and learning as a
service (ELaaS), 2011 14th International Conference on Interactive Collaborative
Learning, Piestany, 2011, pp. 589-594
Baldassarre, M. T., Caivano, D., Dimauro, G., Gentile, E., Visaggio, G. (2018) "Cloud
Computing for Education: A Systematic Mapping Study," in IEEE Transactions on
Education, vol. 61, no. 3, pp. 234-244, doi: 10.1109/TE.2018.2796558.
Bennett, J., & Pence, H. E. (2011). Managing laboratory data using cloud computing as an
organizational tool. Journal of Chemical Education, 88(6), 761e763
Bimonte G., Tortoriello S.F., Veronesi I., “Game Theory Lab: A Gamification Laboratory
for High School Students”, Handbook of Research on International Approaches and
Practices for Gamifying Mathematics, 2022
Bonham, S. (2011). Whole class laboratories with Google docs. The Physics Teacher, 49(1),
22e23.
Boss, G., Malladi, P., Quan, D., Legregni, L., Hall, H.. Cloud computing, 2009, p.4
Dmitriev, S., Kononov, A., Shiriaev, M., & Malozemov, S. (2012). Cloud computing for
education in state technical University of Nizhny Novgorod. In Proceedings of the 9th
IFAC Symposium on Advances in Control Education (ACE '12) (pp. 418e420)
Katiyar, N.S., Bhujade, R., (2018) "A Survey : Adoption of Cloud Computing in Education
Sector". International Journal of Computer Trends and Technology (IJCTT) V60(1):15-
25 June 2018. ISSN:2231-2803. www.ijcttjournal.org. Published by Seventh Sense
Research Group.
Nevin, R. (2009). Supporting 21st Century learning through Google apps. Teacher Librarian,
37(2), 35e38
Rienzo, T., Han, B. (2009). Microsoft or Google web 2.0 tools for course management.
Journal of Information Systems Education, 20(2), 123-127
Rogora, E., Tortoriello, F.,S. (2021) "Interdisciplinarity for learning/teaching mathematics."
Bolema: Boletim de Educacao Matematica,
Sultan, N., Cloud computing for education: A new dawn?. International Journal of
Information Management,Volume 30, Issue 2,2010,Pages 109-116
Tortoriello F.S., Veronesi I. (2021) “Internet of things to protect the environment: a
technological transdisciplinary project to develop mathematics with ethical effects”,
2021, Soft Computing, https://doi.org/10.1007/s00500-021-05737-x
Tortoriello, F.S., Veronesi, I. (2021) Historical path in mathematical games: a didactic
laboratory activity dedicated to upper secondary school students, Proceedings WMSCI
Vygotskij L.S., Pensiero e linguaggio, (1934), Firenze, Giunti-Barbera, 1966, trad Adele
Fara Costa,Maria Pia Gatti, Maria Serena Veggetti.
1154
Inverting the classroom using on-demand lightboard micro

lecture films (learning glass)
Anja Pfennig
School of engineering and life science, HTW Berlin, University of Applied Sciences, Berlin,
Germany.
Abstract
Critical thinking and self-directed learning are corner stones to prepare first
year mechanical engineering students for the demands of the employment
market. Creating a positive learning environment presupposes solutions to
mechanical engineering problems related to material science. The blended
learning environment embeds inverted classroom teaching scenarios as the
opportunity to gain higher learning outcome and self-reliant study skills.
Inverting the classroom is a method to let the students study the basic scientific
background of engineering materials on their own and then work practically
during face-to-face time. Because during digital self-study periods (especially
during the Covid-19 pandemic) students often feel lonely lighboard videos
were provided in addition to traditional lecture material. Short lightboard
videos (<10 min) are produced easily on student demand on a glass panel
(learning glass). Here, the lecturer always keeps eye contact with the audience
and students feel addressed in person. As students were able to manage the
input the learning outcome is directly related to the teaching material.
Keywords: Light board, inverted classroom, flipped classroom, blended
learning, first year students.

Inverting the classroom using on-demand lightboard micro lecture films
1. Introduction
Nowadays information is readily accessible and not centered in one expert or lecturer.
Decentralization and globalization expects students to acquire autonomy and to appropriate
their own learning process: Huguet et al. (2017). Problem-solving and communication skills
are already important for first year students, because collaborating, thinking innovatively and
knowing how to solve problems are considered to be 21st century essential skills: Efstratia
(2014). One constructive teaching approach is the inverted classroom teaching model: Berret
(2012), Brame (2015) having positive effect on self-efficacy beliefs and intrinsic motivation:
Thai et al. (2017). The flipped classroom constitutes a role change for instructors, who give
up their front-of-the-class position in favor of a more collaborative and cooperative
contribution to the teaching process: Educause (2016) (Figure 1). Students are given more
of the responsibility for the learning process and therefore engage in critical thinking: Kaya
et al. (2014), Pfennig (2017). Students take over active parts in hands-on work and
communication and become devoted to their learning and even determined activators of the
face-to-face time. Deeper learning outcomes: Simon et al. (2010) are produced. What the flip
does particularly well is to bring about a distinctive shift in priorities— from merely covering
material to working toward mastery of it: Educause (2016). However, Setren et al. (2019)
assign success only for MINT courses and show no evident success of the method for
economic related teaching.
Figure 1. The principle of inverting the classroom. Instructors give up their front-of-the-class position in favour of
a more collaborative and cooperative contribution to the teaching process
Inverting the classroom comprises of the scientific input during defined self-studying phases.
During face-to-face time more time is given to explore science, to raise questions and discuss
details as well as to communicate in an appropriate scientific pathway in equal measure:
Pfennig (2016), Pfennig (2019), Pfennig (2022), Setren et al. (2019), Berret (2012), Brame
(2015). Lively taught science as means of medium to catch and/or raise students` attention
may result in good grades. This paper discusses the advantages of lightboard (learning glass)
videos as means of teaching material in inverted classroom teaching scenarios over lecture
capturing: Crooka and Shofield (2017) or even lecture videos: Pfennig (2019).
1156
Anja Pfennig
2. Setting of inverted classroom scenarios in first year materials science

Material science for first year mechanical engineering students at HTW Berlin is taught via
the “design-led” teaching approach: Ashby et al. (2013) that has been explained in detail by:
Pfennig (2016), Pfennig (2022). Because blended learning reveals a higher learning
performance compared to the e-learning setting: Thai et al. (2017) the best teaching
environment was provided implementing a blended learning setting. Micro-projects were
combined with inverted classroom teaching scenarios and were included in the blended
learning materials science course: Pfennig (2016), Pfennig (2019), Pfennig (2022) described
the course structure and the alternative grading system based on a portfolio in detail.
The inverted classroom teaching method requires and at the same time enables individual
self-directed, location-independent and asynchronous studying. To most of the students the
important advantage are the teaching materials provided online and the possibility to study
the science on their own independent from place and time and according to their individual
tempo in self-chosen study groups. Simple methods such as “Think-pair-share” or “peer
instruction”: Simon et al. (2010) and (digitalized) classroom response offer a quick overview
of students` state of knowledge enabling the lecturer to respond easily to the needs of the
class. During face-to-face time students and lecturer discuss difficulties and thoughts, answer
questions and work on engineering related problems (Figure 2).
Figure 2. Change of roles: Students learning session and presentation in class after preparing scientific
backgrounds at home.
3. The importance of teaching material

Teaching first year students challenges lecturer, because diversity of the class is high. Besides
the different educational and family backgrounds offering various level of understanding
some students ignore the significance of the scientific basics. To motivate students and guide
their learning progress seems to be one of the main outcomes of a first year class. Students
motivation, performance and learning outcome is directly related to the teaching material
provided Pfennig (2019), Pfennig (2022), Kaya et al. (2014). This unconditionally has to be
congruent with the class progress and learning outcome. Otherwise students do not relate the
study material to their personal success and knowledge.
1157
Teaching material was partly generated by higher class students (peer-to-peer approach):
Pfennig (2019). Generally, it is provided online via Moodle to offer all students the same
preconditions. The teaching material addresses many learning styles, all focusing on the same
learning outcome. However, different students show manifold learning preferences that have
to be met to succeed in a blended learning setting. Figure 3 visualizes the teaching materials
and activities offered in the first year material science course. This Figure emphasizes the
strength of visualization – that can be of high use in a first year class. Learning outcome of
certain themes is much higher when difficult scientific explanations, especially models, are
visualized and assigned to distinct symbols and proceedings.
Figure 3. Teaching material
4. Advantage of lightboard videos

Micro lectures with self-guiding tests: Pfennig (2018) offer high quality self-study material.
However, most inverted classroom lectures will be accompanied by lecture videos, such as
lecture capturing or demonstrating videos: Pfennig (2019). Capturing lectures seems to be a
low-threshold method to provide lectures and content to students although recordings of
lectures comprise at least five different techniques: Crooka and Shofield (2017). Lecture
videos provide an audio and visual stimulus covering different learning methodologies:
Gulley and Jackson (2016). Presupposed the video included is analogous to the desired
learning outcomes of the lecture: Al-Jandan, Farooq and Khan (2015) lecture videos are
definitely a reinforcement, rather than a replacement for lectures: Havergal (2015).
However, the biggest disadvantage is that the lecturer generally turns her or his back towards
the students. From earlier surveys it is known that students prefer lecture videos where human
1158
Anja Pfennig
contact is made: Pfennig (2019). Therefore, lightboard teaching videos offer a successful
solution to teach short sequences online and face students directly at the same time (Figure
4). The lightboard at HTW Berlin was successfully designed and developed in a mechanical
engineering student peer-to-peer project.
How-to use Heat treatment of steel
Figure 4. Examples of lightboard videos at HTW Berlin.
A lightboard consists of a high quality optiglass panel which is surrounded by LED lights.
Extra strong lights enable for high quality contrasting and clear colors of the video.
Fluorescent pens highlight the writing that is done in light absorbing surrounding. A camera
is positioned properly (centered on the entire glass board) before the lecturer. The video is
inverted after filming either using camera software or post production software. The
production of lightboard videos is of low threshold. The contents of the lecture should fit on
one 16:9 screen – just as the black board offers in ordinary class rooms. They are easy to
make, because a lightboard video is a replica of a scientific panel painting such as usually
developed analogously in class. The equipment is easy to use. However, there are a couple
of recommendations to be met:
• Plan on less than 10 minutes for one video, best is 4-5 minutes. Otherwise divide in
2 parts
• Face forward as much as you can
• Smile as much as you can
• Leave 5 cm spacing from the edge to ensure your writing fits the screen
• Use proper spacing, colors and distinct headings
• Do not erase errors (it appears human) and students feel addressed personally
• Do not plan on 2 panel screens. One should do to cover your contents
• Speak slowly and clearly
• Introduce the video and finish with a “good bye”
• Wear solid and middle dark colors (green, dark red), no patterns
1159
4. First evaluation of implementing lightboard videos

Material science has a high work load for first year students and a very theoretical subject.
However, most of the students rate the different learning materials beneficial for their
learning progress. The lightboard videos were considered as bridge between the self-study
period at home and contact time in class. Joy of studying is enhanced when students feel
addressed in person, because the lecturer always faces the audience and “keeps eye-contact”
with students – especially during the Covid-19 pandemic (Figure 5). Also, the restriction to
one blackboard panel picture and time limit less than 10 minutes have a lasting effect in terms
of facilitation and personal learning progress. Students feel secure with the quantity of the
material as well as the quality because delivered directly by the lecturer similar to in-class
lectures. Therefore, most of the students studied widely using the ca. 50 lightboard videos
even suggesting themes and questions they came along during preparation time. Because
more students were engaged in the learning procedure the class was more homogeneous. The
most important advantages mentioned by the students (besides eye-contact) is that the
lightboard videos (along with other digital teaching materials) are independently reusable.
Another advantage is that students may post technical terms or sequences of the lecture they
considered difficult and a lightboard video can be made right away without any preparation
and very little post production. Lightboard videos may be produced on-demand and are
therefore considered very powerful and successful teaching material. Note, that after the
pandemic students did not relate to the lightboard videos as important because they
experienced face-to-face time again.
70
ME 1.Sem. SS 2017, N=33
ME 1.Sem. SS 2018, N=40
60
Number of students in %
ME 1.Sem. SS 2019, N=37 50

ME 1.Sem. SS 2020, N=17
ME 1.Sem. SS 2021, N=32 40
ME 1.Sem. SS 2022, N=11
30
20
10
0
on ique t al. awn oard ture tion tion H5P
ati e r c a a
nim echn ribe ndd ackb o le oder nim ideo
t a et sc h a bl i de m ard a ve v
i n ip o n, n, v e
if bo
rp
o
Sw Vide otio otio nl t cti
we m wm atio ligh intera
Po w m
Slo Slo an
i
Lecture video technique
Figure 5. Preference of video technique. Lightboard videos were established first in SS2019. SS2020 and SS2021
were taught fully online.
1160
Anja Pfennig
During plenary sessions students acted lively and were eager to dispose and share their
knowledge, collaborate and learn more of the details. During face-to-face time a strong
structure of the problem based work (e.g. templates, clear work order) and always transparent
and steady course demands are essential regarding learning outcome and better grades:
Efstratia (2014). Drawback of this modern teaching method is the very high preparation time.
Venturing into an alternative method may discourage lecturer because they are expected to
manage additional activities, demands and collaboratitive student work: Efstratia (2014).
There is always the risk of losing students who are not willing to study at home. Implementing
lightboard videos as teaching source however showed that more students were prepared and
also positively willing to work with other material provided online.
5. Conclusion
Inverting the classroom involves students to take over the responsibility for their own
learning process. This modern teaching method is assessed as beneficial in terms of student
grades, learning output and understanding of scientific background, concentration and
attentiveness as well as both joy of studying and teaching. However, the success of the
method is directly related to the teaching material. During self-study periods students need
strong guidance, transparency of the grading system and to overcome the feeling of
loneliness. Therefore, lightboad teaching videos are a good alternative to conventional lecture
capturing videos because the lecturer always keeps eye contact with the audience and
students feel addressed more personally. Lightboard videos bridge self-study periods at home
and contact-time in class. Provided directly by the lecturer and restricted to one blackboard
panel of less than 10 minutes teaching students feel secure with the quantity of the material
as well as the quality.
Most important advantages comprise of 1. the low threshold – everyone can start right away,
2. lecturers “keeping eye-contact” with students, 3. offering good pace for student learning
(short in-put videos that are less than 10 minutes long), 4. being immediately producible on
student demand, 5. only little post production time being necessary. Overall lightboard videos
are attributed to a lasting effect in terms of facilitation and personal learning progress.
References
Al-Jandan, B.A., Farooq, I. and Khan, S.Q. (2015). Students’ perspectives on the relevance
of internet-based educational videos in dental education. Journal of Taibah University
Medical Sciences, 10(3), 288-292.
Ashby, M., Shercliff, H., Cebon, D. (2013). Materials Engineering, Science, Processing and
Design, 2013: ISBN-13: 978-0080994345, Pub date: Oct 02 (3rd edition).
Berrett, D. (2012). How ‘flipping’ the classroom can improve the traditional lecture. The
1161
Chronicle of Higher Education. http://chronicle.com/article/How-Flipping-the-

Classroom/130857/, 08.04.2017.
Brame, C.J. (2015). Flipping the Classroom, http://cft.vanderbilt.edu/files/Flipping-the-
classroom.pdf, call: 09/2015.
Crooka,C., Schofield, L. (2017). The video lecture. The Internet and Higher Edu., 34, 56–64
Educause Learning initiative (2016). THINGS YOU SHOULD KNOW ABOUT™ Flipped
Classrooms, https://net.educause.edu/ir/library/pdf/eli7081.pdf, call: 01/2016.
Efstratia, D. (2014). Experiential education through project based learning, ERPA 2014,
Procedia - Social and Behavioral Sciences, 152, 1256 – 1260.
Gulley, O.D., Jackson, A.L. (2016). A case study on using instructor-recorded videos in an
upper level economics course. International Review of Economics Education, 23, 28–33.
Havergal, C. (1015). Videoing lectures ‘has no impact’ on attendance, says study. Retrieved
from: https://www.timeshighereducation.com/news/videoing-lectures-has-no-impact-
attendance-says-study.
Huguet, C.; Pearse, J.; Noè, Leslie, F.; Castillo, R.N.; Diego, M.; Jimenez, H.A.; Patiño A.,
Mónica, A. (2017). Use of emerging technologies in flipped classes, 3rd International
Conference on Higher Education Advances, HEAd´17, 23-25 June 2017, 891-898.
Kaya, H., Bodura, G. (2014). Nursing students' opinions regarding project based learning.
ERPA 2014. Procedia - Social and Behavioral Sciences, 152, 379 – 385.
Pfennig, A. (2016) Inverting the Classroom in an Introductory Material Science Course,
Procedia - Social and Behavioral Sciences, 228, 32-38.
doi:10.1016/j.sbspro.2016.07.005.
Pfennig, A. (2019). "Making it Work" - Practice Examples of Preparation and Embedding
Peer-to-Peer Lecture Films in Successful Inverted Classroom Scenarios. PEOPLE:
International Journal of Social Sciences, 5(1), 640-655.
Pfennig, A. (2022). Successfully Inverting the Classroom - Recommendations to Teach (fully
online). Proceedings 6th Int. Conf. Ed. & Multimedia Techn. (ICEMT 2022), 324-330.
Setren, E., Greenberg, K., Moore, O., Yankovich, M. (2019). Effects of the Flipped
Classroom: Evidence from a Randomized Trial, SEII Discussion Paper #2019.07, MIT
Department of Economics, National Bureau of Economic Research.
Simon, B., Kohanfars, M., Lee, J., Tamayo, K, & Cutts, Q. (2010). Experience report: peer
instruction in introductory computing. Proceedings of the 41st ACM Technical
Symposium on Computer Science Education, 341-345.
Thai, N.T.T., De Wever, B., Valcke, M. (2017). The impact of a flipped classroom design on
learning performance in higher education: Looking for the best “blend” of lectures and
guiding questions with feedback. Computers & Education, 107, 113-126.
1162
Let’s experience learning in the metaverse

Eman AbuKhousa
Faculty of Computer Information System, Higher Colleges of Technology, United Arab
Emirates.
Abstract
The metaverse and learning experience design (LXD) are shaping the future
of learning and teaching in higher education. The metaverse is a virtual
extension of the physical world that frees learning from time and space
constraints while LXD is a learn-centric approach to design technology-
mediated learning. LXD focuses on cognitive and affective engagement of
learners with the learning experience; and the metaverse promises the
adaptive infrastructure and flexible design space to realize digital learning as
a journey of a series of ongoing, intensively personalized, immersive, adaptive,
and competency-based experiences. In this paper, we discuss and reflect on
how the potentials of metaverse can expand the effectiveness of LXD to address
certain aspects of learning experience such as the influence of prior
experiences, emotions, motivation, and attention. The metaverse presents
unique opportunities to manage these aspects and to offer more attractive
environments for learning activities.
Keywords: The metaverse; learning experience; learning experience design;
empathy mapping; cognitive engagement; affective engagement.

1. Introduction
The metaverse is the new virtual universe for us to connect, socialize, play, share, experiment,
create and trade. It is an online, collective, immersive, and interactive space that represents a
digital extension to everything we experience, including learning. While it is still not fully
developed, metaverse is constantly evolving with more potential applications to improve
education and learning (Hwang & Chien 2022). The creation of learning experience (LX) in
the metaverse is discussed in previous work (AbuKhousa et al., 2023) as the process that
involves the development of 3D virtual learning environments and the creating of virtual
representations of learners (avatars). It also includes designing and implementing learning
activities which users can access and interact with through immersive interfaces and controls.
While more frameworks are emerged to describe the technological aspects of metaverse
education (e.g. Dahan et al., 2022; Cui et al., 2023); still little attention is paid to address
cognitive and affective aspects that drive individuals to access the learning experience on
metaverse to learn what they need verse what they want and to keep them engaged in the
learning process. On the other hand, learning experience design (LXD) is an emerging area
of learning design and technology (LDT) that integrates modern methods and processes such
as design thinking, user experience (UX) and human-computer interaction (HCI) to focus on
all the aspects of the learner’s interaction with the learning tasks during technology-mediated
learning. This includes the cognitive engagement as well as the affective response and
subsequent engagement (i.e., level of involvement and participation) with the learning
context (Tawfik et al., 2022).
The objective of this article is to explore how principles of LXD can inform the creation of
learning experience on metaverse; and how the adaptation capabilities of metaverse may
contribute into expanding the LXD to create multiple, equally effective learning experiences
to support diverse and emerging cognitive and emotional needs of the learners. What faculty
and learners in high education want in their virtual learning experience is not only a
personalization of the content but also an adaptive competency-based learning process with
more personalized assessment of the outcomes (Shearer et al., 2020). The autonomy and
individualism focus should not be limited to the “pace” learners need to complete the learning
tasks or to achieve certain level of learning outcomes. Learning journey should be designed
as a set of ongoing experiences until desired competence is achieved (Clark, 2022;
AbuKhousa et al., 2023).
In the following sections, we present a discussion of design thinking process in LXD and the
potentials of the metaverse to enhance the creation of effective learning and growth-
production experiences.
1164
Eman AbuKhousa
2. Learning Experience Design (LXD)

The dictionary by Merriam-Webster defines experience as “act or process of
directly perceiving events or reality.” Learning is also a process during which learners
develop perceptions and responses as result of their presence and interacting with the learning
environment. They notice, interact, and interpret events in the physical and social
surroundings and make changes in accordance with their intents and resources available for
them in this process (Boud, 2008). It is the never-end situated sensing and doing activity with
outcomes that remain in the memory and affect our behavior. Understanding the influence of
the affective context on learning leads to designing good learning experiences that encourage
and sustain learning (Clark, 2022). On the other hand, and according to common elements
theory, learning happens through a series of experiences that engage learners in inferential
reasoning and metacognitive activities to produce adaptive responses to problems in their
environment (Butterfield & Nelson, 1989). Investment on these cognitive efforts is
influenced by personal mental frameworks and associated with dispositional individual
differences in self-control and goal-directed behavior (Kührt et al., 2021). For example,
learners high in cognitive motivation, i.e., who have a high tendency to engage in and enjoy
effortful cognitive tasks, will require integration of additional motivational attractions into
their learning experience to avoid cognitive boredom. Other learners are subjected to the risk
of cognitive overload if they are required a high interaction to complete the learning task
(Wang et al., 2014). Thus, designing effective learning experiences requires understanding
not only of learners, their needs, and the context in which they are embedded, but also the
concepts and theories of learning, attention, memory, cognitive efforts, engagement; then
making empirical evidenced decisions on how to create learning experiences that stick.
Based on a qualitative content analysis of 15 book chapters, Schmidt and Huang (2022)
introduced a definition of LXD that is “a human-centric, theoretically grounded, and socio-
culturally sensitive approach to learning design, intended to propel learners towards
identified learning goals, and informed by UXD methods.” The authors also proposed a
conceptual model of LXD that presents the complex nature of LXD as a convergence of
design, disciplinary, theoretical, and methodological perspectives, with a central focus on
learning. Another present LXD as a confluence of two major constructs: (a) interaction with
the learning environment focusing on UX elements and learner’s utility of technology such
as interface customization, content placement on the interface and functionality of presented
components; and (b) interaction with learning space focusing on engagement elements and
how learners perceives the interface items and interact with the content (Tawfik et al., 2022).
Learning is already a challenging process and LXD is all about making the efforts to consider
the socio-cultural, design and technological influences and factors to encourage effortful and
long-term learning; or as Clark (2022) states “it is a matter of design for both head and hear.”
1165
In their open-access book “Learner and User Experience Research,” Schmidt et al., (2020)
put significant efforts into presenting theories, methods, and design models of LXD as well
as examples of applied LXD processes to create digital learning experiences. Such work
serves as an essential guide for the learning designers on how to design and operationalize
learning experiences for a variety of contexts.
3. Design Thinking Process in (LXD)

Design thinking emphasizes empathy, i.e., develop an immersive understanding of the users
for whom the products, services, or learning experience are being created, and combines it
with rapid prototyping, and adductive reasoning to deliver the best possible evidence-based
solution (Siricharoen, 2021). The design thinking process in LXD goes as follows:
1. Empathize: This stage involves understanding the learners, their needs and
emotions, and the problem space. Learners are not approached as a population but
as “personas” i.e., specific segments of learners who share goals, motivations,
behavior patterns and challenges. Empathy involves research, observation, and
interaction with learners to experience their feelings and understand what it is like
to have their learning needs or challenges.
2. Define: In this stage, the learning problem is defined based on the insights gathered
during the empathize stage. This involves understanding the opportunities and
constraints presented by analyzing the learner, learning context, and learning task.
3. Ideate: The ideation stage involves generating potential solutions or experiences to
the learning problem.
4. Prototype: The prototype stage involves creating virtual representations of the most
promising ideas for the learning experience generated during the ideation stage.
5. Test: In this stage, prototypes are tested with real learners to validate or invalidate
assumptions and to gather feedback. Based on the results, the design process may
loop back to any of the previous stages or move forward to the implementation stage.
Schnepp and Rogers (2022) described the above design thinking framework as an effective
and accessible method to produce effective learning experiences; and suggested a practical
straight-forward steps on how to implement it for LXD in higher education. However, it is a
more complicated matter as the design thinking with its sensitivity to learners and their needs
may potentially undermine the perspectives of learning theories and how learning occurs
(Clark, 2022). Plus, the testing part will involve assessment of many technical aspects and
their influence on cognitive efforts, affective engagement and learning outcomes
achievement. Not to mention the learners’ desire to be able to negotiate assessment
expectations and to incorporate their input in the design, evaluation and evolving of their
1166
Eman AbuKhousa
learning experiences. Professionals in higher education will be required to develop a wider

set of knowledge, skills, and competencies across a range of disciplines to be able to
incorporate LXD into their practices (MacLean and Scott, 2011; Olney, 2023).
4. LXD in the Metaverse: Expanding the LX

The metaverse provides an adaptive infrastructure and a flexible design space that enables
creation of interactive and customizable profiles (i.e., avatars), and opportunities for
considering not only sociocultural factors but also the physical biometric markers and track
their influence on cognitive performance; all of which are required to meet the demands for
learning experiences in the digital age (Shearer et al., 2020). In the adaptive learning
environment, the learning context and elements are dynamically adjusted according to the
outcomes of learner’s behaviors and actions while providing instant feedback and support.
The artificial intelligence (AI) component of metaverse provide capabilities first to gather
insights about the learners; second to create the problem space (i.e., simulating a real-life
situations and learning tasks) including the generation of the non-player characters (NPCs)
to interact with learner and augment the learning experience; and third to track and analyze
the learner ’s behaviors and actions while solving the problem (living the experience and
engaging with the space objects); and finally to adjust the learning space and experience
according to the outcomes of learner’s solution to the problem (Hare & Tang, 2022;
AbuKhousa et al. 2023).
Clark (2022) states that the art and science of LXD lies between three words: emotion,
motivation, and attention. The empathy mapping in LXD focuses on understanding and
articulating the thoughts, feelings, and behaviors of a particular persona of learners. The goal
is to emotionally motivate learners to engage actively in the experience. Boud (2008) argues
that for maximizing learning from any experience, the new experience should be based on
personal foundation of learner’s feelings and prior learning experiences. The challenge
remains for online learning experience is how to use learners’ prior offline experiences to
design the new experiences and how to link these new experiences to the learners’ emotional
drives. Moreover, how to analyze the emotions and feelings presented during and after the
experience to seize the opportunities for further learning or understand enhancements for the
experience growth-production. Prior experiences also influence attention and the attempts of
LXD to sustain learner’s attention. Attention of learners is mostly affected by their
personality characteristics (Van Calster, 2018). The efficacy of external attention technique
(e.g., sound cue, visual cue, etc.) may increase the individual’s focus and stay on the task,
but it does not indicate the learning happens.
In the metaverse, the virtual world and physical world will interact at the highest level of
independence, i.e., changes in one of world will be reflected in the other world. The real time
1167
synchronization between the two worlds will enable the achievement of experience-duality,
where learners use avatars to experience situations and find solutions in virtual spaces that
are closely paired-up with their real-world counterparts. On the other hand, and as in real life,
they will experience events in virtual space that are out of control and where learning happens
unintentionally as they experiment solutions to handle these events. Lifelong learning is that
exact informal learning that occurs through interests, practice, and self-development (Clark,
2022). Learning experience design in the metaverse is envisioned to deliver learning as a
situated practice within environments prepared to present the real practice space and the
networks of actions and interactions among human and non-human objects. Besides the BIG
data and AI capabilities to profile learners (i.e., develop empathy based on online/offline
repository of life-time data) and to produce learning scenarios that serve the unique, multiple
and the continuously changing individual needs and interests, learners will be driven to
immerse themselves in the learning experience as part of living their life! The emotion,
motivation and attention aspects are addressed by the reciprocal intentions in the space, i.e.,
understanding the actions of other players and the ability to interpret surrounding signals and
manage unforeseen events as they arise from the context. The component of the metaverse
where learners generate and trade digital content or assets; and where they can gain tangible
(e.g., private space, premium avatar customization, virtual merchandise) and intangible
rewards (e.g., social status in the virtual community, unique experiences, access to exclusive
virtual spaces or the ability to interact with NPCs in new ways) can drive the engagement
and interaction with the learning experience environment.
5. Concluding Remarks
The emerging LXD approach adopts empathy mapping technique to understand the
perspectives, needs, and motivations of learners; and to identify challenges, and opportunities
for creating learning experiences that meet their needs. The challenge lies in the complex
relationships between the individual learner, learning tasks, technology and learning context;
and how the elements of these constructs are influencing the cognitive and affective aspects
of the learning. To maximize learning, more emphasis should be put on motivation, attention,
engagement, and practice. In the metaverse, the experience is a series of immersive real-time
social activities where learners with their unique identities in the virtual world connect,
communicate, practice, exchange, trade, and share contents within a context that is governed
by rules and norms, just as in real life communities. In this essence, the metaverse offers
unique opportunities for LXD to generate more engaging learning experiences that can be
customized to meet the needs of individual learners. Also, it offers opportunities to manage
attention such as the ability to switch/teleport between different environments and activities
in real-time. It is expected that the metaverse will continue to evolve and to offer more
realistic and immersive virtual spaces that is more to be a more attractive alternative to
1168
Eman AbuKhousa
physical environments for many activities including learning. It is time for higher education
to consider living and teaching in the metaverse.
References
AbuKhousa, E., El-Tahawy, M. S., & Atif, Y. (2023). Envisioning Architecture of Metaverse
Intensive Learning Experience (MiLEx): Career Readiness in the 21st Century and
Collective Intelligence Development Scenario. Future Internet, 15(2), 53.
Boud, D. (2005). Locating Immersive Experience in Experiential Learning. In Conference
program; University of Surrey: Guildford, UK, 2008; p. 9.
Butterfield, E. C., & Nelson, G. D. (1989). Theory and practice of teaching for
transfer. Educational Technology Research and Development, 37(3), 5-38.
Clark, D. (2021). Learning experience design: How to create effective learning that works.
Kogan Page Publishers.
Cui, L., Zhu, C., Hare, R., & Ying, T. (2023). MetaEdu: A New Framework for Future
Education.
Dahan, N. A., Al-Razgan, M., Al-Laith, A., Alsoufi, M. A., Al-Asaly, M. S., & Alfakih, T.
(2022). Metaverse framework: A case study on E-learning environment
(ELEM). Electronics, 11(10), 1616.
Hare, R., & Tang, Y. (2022). Hierarchical deep reinforcement learning with experience
sharing for metaverse in education. IEEE Transactions on Systems, Man, and
Cybernetics: Systems.
Hwang, G. J., & Chien, S. Y. (2022). Definition, roles, and potential research issues of the
metaverse in education: An artificial intelligence perspective. Computers and Education:
Artificial Intelligence, 3, 100082.
Kührt, C., Pannasch, S., Kiebel, S. J., & Strobel, A. (2021). Dispositional individual
differences in cognitive effort investment: establishing the core construct. BMC
psychology, 9, 1-14.
MacLean, P., & Scott, B. (2011). Competencies for learning design: A review of the literature
and a proposed framework. British Journal of Educational Technology, 42(4), 557-572.
Olney, T., Rienties, B., Chang, D., & Banks, D. (2023). The Learning Design & Course
Creation Workshop: Impact of a Professional Development Model for Training Designers
and Creators of Online and Distance Learning. Technology, Knowledge and Learning, 1-
19.
Schmidt, M., & Huang, R. (2022). Defining learning experience design: Voices from the field
of learning design & technology. TechTrends, 66(2), 141-158.
Schmidt, M., Tawfik, A. A., Jahnke, I., & Earnshaw, Y. (2020). Learner and User
Experience Research: An Introduction for the Field of Learning Design & Technology.
EdTech Books. https://edtechbooks.org/ux
Schnepp, J., & Rogers, C. (2022). A Practical Approach to Learner Experience
Design. International Journal of Teaching and Learning in Higher Education, 34(1), 161-
169.
1169
Shearer, R. L., Aldemir, T., Hitchcock, J., Resig, J., Driver, J., & Kohler, M. (2020). What
students want: A vision of a future online learning experience grounded in distance
education theory. American Journal of Distance Education, 34(1), 36-52.
Siricharoen, W. V. (2021). Using empathy mapping in design thinking process for personas
discovering. In Context-Aware Systems and Applications, and Nature of Computation
and Communication: 9th EAI International Conference, ICCASA 2020, and 6th EAI
International Conference, ICTCC 2020, Thai Nguyen, Vietnam, November 26–27, 2020,
Proceedings 9 (pp. 182-191). Springer International Publishing.
Tawfik, A. A., Gatewood, J., Gish-Lieberman, J. J., & Hampton, A. J. (2022). Toward a
definition of learning experience design. Technology, Knowledge and Learning, 1-26.
Van Calster, L., D'Argembeau, A., & Majerus, S. (2018). Measuring individual differences
in internal versus external attention: The attentional style questionnaire. Personality and
Individual Differences, 128, 25-32.
Wang, Q., Yang, S., Liu, M., Cao, Z., & Ma, Q. (2014). An eye-tracking study of website
complexity from cognitive load perspective. Decision support systems, 62, 1-10.
1170
Innovation in education by design thinking

Institute of Educational Leadership, Ludwigsburg University of Education, Germany.
Abstract
The dynamic changes currently caused by various megatrends challenge the
educational sector. Prospective strategies are required to cope with these
issues. This is where the innovation methodology of Design Thinking comes
in: it can be described as creative thinking in heterogeneous teams to develop
creative solution ideas for complex challenges. This approach helps to initiate
new or adjusted strategies for special target groups in the context of education.
In this article, essential aspects of Design Thinking are introduced, and
references to innovations in the field of education are presented. Two examples
from the higher education sector are then used to illustrate the practical
implementation.
Keywords: Design thinking; innovation; megatrends; higher education.

1. Introduction
Megatrends "describe extremely complex dynamics of change and are a model for the
transformation of the world" (Zukunftsinstitut, 2021). According to the Zukunftsinstitut, 12
megatrends can be identified, e.g., the megatrends neo ecology, new work and connectivity
(ibid.).
Megatrends ensure that a variety of innovations will also be necessary in the field of higher
education, such as the derivation of future skills according to Ehlers (2020), the examination
of new topics in terms of content, the optimisation of digital teaching-learning scenarios, the
development of new target groups and new types of programme designs.
2. Design Thinking
While solutions to manageable questions can be explored and solved with known problem-
solving strategies, the complex challenges described above cannot be mastered in this way
(Rittel & Webber, 2013). Buchanan sees the opportunity to develop new solutions through
Design Thinking (Buchanan, 1992).
Today's understanding of Design Thinking as a methodology for initiating target group-
oriented innovations can be traced back significantly to the d.school founded in 2005 at
Stanford University (Meinel et al., 2015). Since 2007, Design Thinking has also been taught
at the d.school of the Hasso Plattner Institute in Potsdam. Both institutions cooperate closely
and conduct joint research programmes in the field of Design Thinking (Meinel & Leifer,
2011). In the meantime, Design Thinking has established itself internationally at other
universities and colleges in teaching and research.
According to Meinel and Leifer, Design Thinking can be characterised as follows: "Its
human-centric methodology integrates expertise from design, social sciences, engineering,
and business. It blends an end-user focus with multidisciplinary collaboration and iterative
improvement to produce innovative products, systems, and services'' (Meinel&Leifer, 2011,
p. 8). These essential characteristics of Design Thinking can be similarly found in other
definitions (Uebernickel et al., 2015, p. 16; Lockwood, 2009, p. xii; Plattner et al., 2009, p.
59; Brown, 2008, p. 86).
Design Thinking has already proven its value in the development of product and service
innovations in well-known business enterprises. Also in Education Design Thinking is
applied in various contexts and several universities, such as the Massachusetts Institute of
Technology (MIT), Stanford University, the University of Potsdam and the University of St.
Gallen, have made Design Thinking an integral part of their management training and
research (Eppler & Hoffmann, 2012). Furthermore, Ehlers (2020) underlines the importance
of Design Thinking by including it among the 17 Future Skills.
1172
2.1 Key Elements

The essential key elements of Design Thinking are collaboration in a multidisciplinary team,
a variable workspace and an iterative process flow (Plattner et al., 2009).
Multidisciplinary teams can draw on different professional qualifications. Through this
broad range of expertise on the one hand and the profession-related different approaches on
the other, a greater diversity of perspectives can be achieved, which expands the solution
horizon (Page, 2017). "The evolution from design to Design Thinking is the story of the
evolution from the creation of products to the relationship between people and products, and
from there to the relationship between people and people" (Brown, 2019, p. 47).
A workspace that can be flexibly designed opens atypical uses and stimulates creativity.
Variable furnishings such as high tables, stools and furniture on wheels are suitable for this
purpose. In addition, pinboards and flip charts should be available for knowledge
communication and documentation and prototyping materials are also needed (Plattner et al.,
2009). With the help of sticky notes, the writable surfaces can additionally be extended to
windows and cupboard doors (Lewrick et al., 2018).
The Design Thinking process comprises several phases that build on each other. The six-
steps according to the HPI School of Design Thinking (2023) are presented in figure 1:
Figure 1: Problem space and solution space in the iterative Design Thinking process (Schmidberger &
Wippermann, 2022, p. 40).
The Design Thinking process aims at finding a solution for a so-called Design Challenge.
The Design Challenge presents a target group-oriented question for a complex problem and
1173
is the starting point of the Design Thinking process. A unique feature of the process is the
iterative, nonlinear approach, which is a result of its exploratory nature. In this way, the
results obtained can be reflected upon, and questions that arise can lead to the further
development of the solution idea (Lewrick et al., 2018).
The Design Thinking process moves between two fictional spaces. First, the focus is on
exploring the problem space. Only then is the focus on developing new solution ideas for the
design challenge. This separation between problem space and solution space is called the
double diamond and represents an important aspect of Design Thinking (Design Council,
2007).
In this way, a detailed consideration of the problem is made possible without hastily trying
to find solutions. This detailed clarification of the actual problem creates the basis for the
development of target group-oriented solutions to complex challenges. At the beginning of
the process, the focus is on divergent thinking. This means that the problem is approached as
openly and unbiasedly as possible. To find out for which concrete challenge the target group
needs a solution, different perspectives of the target group are included, and the Design
Thinking team's own view is critically reflected on.
When evaluating and classifying the target group observation, the focus is then increasingly
on convergent thinking. The results of this exploration phase enable the definition of a
common point of view. This is crucial for a transition into the solution space. Here, divergent
thinking is called for again, creating new solution impulses and enabling initial prototypes to
be developed. When it comes to concretising the solution ideas, making decisions, and
putting them into practice, convergent thinking is increasingly required. Throughout the
process, the connection to the problem space is always maintained. This iterative procedure
is intended to ensure that the solution finding corresponds to the actual needs of the target
group (Design Council, 2007).
But tools, methods and processes alone do not fully unlock the potential of Design Thinking.
The team’s mindset is also important. It determines whether a team can successfully
collaborate across disciplines, move through complexity, or inspire others to learn
experimentally in order to think out of the box to create extraordinary solutions. Individuals
with a human-centered mindset are open and non-judgmental towards people with different
backgrounds. They are able to empathize with the emotions and needs of others and they feel
comfortable taking on other perspectives, even if they don’t correspond with their own
experiences. These team members discover the problem space with curiosity and take every
opportunity to improve, adapt or refine their own understanding, even if that means deviating
from the original plan or redefining the problem to solve (Graves & Fuchs, 2022).
1174
3. Innovation in Education by Design Thinking

To illustrate how innovation in education can be fostered by Design Thinking, two examples
of the Ludwigsburg University of Education in Germany are described.
3.1 Seminar on the topic "Education for Sustainable Development (ESD)“

In the summer semester of 2022, students from various master’s degree programmes (Early
Childhood Education, Adult Education and Cultural Education) took part in the seminar
"Education for Sustainable Development" at the Ludwigsburg University of Education,
Germany. The seminar aimed at developing solutions for challenges in the field of education
for sustainable development through Design Thinking. It started with a kick-off event that
allowed the students to get to know each other personally and introduced them to the basics
of Design Thinking. Furthermore, the students were given access to various learning media
such as videos, study texts and literature on the topics of Design Thinking and ESD via the
online learning platform of the University. During this asynchronous self-learning phase, the
students dealt with in-depth impulse questions and used the online learning platform as an
exchange forum. Subsequently, the acquired learning content was discussed and critically
reflected upon during synchronous classroom sessions. An essential aspect was the gathering
of practical experience in the application of Design Thinking in the context of ESD. For this
purpose, the students formed four multidisciplinary teams (each consisting of approx. 10
participants), each of whom worked on a design challenge, both synchronously in classroom
sessions and asynchronously (e.g., in the interview phases) throughout the semester. Due to
the importance of the Design Thinking mindset, several aspects of it could be experienced
by the students, e.g., by Impro Theater exercises. Special attention was paid to adopting a
positive attitude towards mistakes in the sense of failing forward to include them as a
necessary part of explorative learning.
During the Design Thinking process, two teams were accompanied by a lecturer who took
on the role of the Design Thinking coach. The Design Challenges were developed in
cooperation with the Office for University Didactics and the Office for Sustainability and
Mobility at the Ludwigsburg University of Education, Germany. The Design Challenges
were as follows:
How could we make learning even more attractive for students in the future, considering the
quality criteria of ESD?
How could we motivate students to support sustainability issues at our university?
Based on the six-step Design Thinking process presented in this paper, the students first
explored the problem space. For this purpose, interviews were conducted with the target
group, and the results were evaluated using an empathy map. Then, with the help of creative
techniques, solution ideas were developed, and the first prototypes were created. The
1175
reflection at the end of the semester showed that the students had dealt intensively with the
learning content. Through the methodology of Design Thinking, a creative approach to the
Design Challenges in the field of ESD was supported, and innovative prototypes were created
that can be tested in practice in the next step.
3.2 Development of a new profile for a master’s degree programme

Since 2003, the Ludwigsburg University of Education in Germany has offered a master’s
degree programme in Educational Leadership that prepares educational leaders for their
complex tasks against the background mentioned above (Iberer & Müller, 2012). The
participants from all fields of education learn with and from each other. Against the backdrop
of massive changes in the educational landscape, Design Thinking was used to develop a
blueprint for the future curriculum and to find concrete levers. There was a particular focus
on the composition of the study content, the design of the methodology and didactics, and an
even more effective approach to the target group. The design challenge was: "How can the
curriculum (objectives, content, methodology) of the master’s degree programme in
Educational Management be even more closely aligned with the needs of the participants?“
The interdisciplinary teams consisted of members of the institute, current students, alumni
and cooperation partners. During a one-day workshop, various prototypes (e.g., as video)
were designed and concrete impulses for further development in terms of content,
methodology and processes were collected.
4. Conclusion and critical reflection

Design Thinking might be a powerful methodology to develop extraordinary solutions if the
required resources are matched, e.g., time slots for coaching and for iteration phases. It is
also necessary to explore the problem space, e.g., by conducting interviews with the target
group as well as testing the prototypes with the user in the solution space.
To fully unlock the potential of the methodology, an additional crucial resource is the
person’s mindset. Therefore, the development and support of a Design Thinking mindset
needs to be addressed because not all persons may feel comfortable creating solutions in the
described way. Nevertheless, according to our experiences, the structured design process is
helpful for most team members.
The Design Thinking concept itself is not new and it describes the typical design process.
But as soon as all described resources are met, Design Thinking can be used to tackle wicked
problems and complex challenges. However, the strength of this methodology is its user-
centeredness which supports creating innovations that are tailored to the target group’s needs.
According to our expertise, the initiation and implementation of innovative ideas can be
supported by Design Thinking in Education (Schmidberger & Wippermann, 2022;
1176
Schmidberger & Wippermann, 2018). In this article, two examples show how solutions to
complex problems in higher education can be developed with Design Thinking. Furthermore,
Design Thinking is applied in various international contexts in Education and is among the
17 Future Skills. However, there is a lack of evidence regarding the impacts of Design
Thinking on Education on the one hand and of systematic approaches to evaluate it at a large
scale across sectors on the other hand. Therefore, further research on Design Thinking could
focus on these aspects.
References
Brown, T. (2008). Design Thinking. In: Harvard Business Review. 06/2008, S. 84 – 92.
Brown, T. (2019). Change by Design. How Design Thinking Transforms Organizations and
Inspires Innovation. Revised and updated Edition. New York: Harper.
Buchanan, R. (1992). Wicked Problems in Design Thinking. In: Design Issues. Vol. 8, No.
2, pp. 5 – 21.
Design Council (2007). Eleven lessons: managing design in eleven global companies. Desk
research report. London: Design Council.
Ehlers, U.-D. (2020). Future Skills: The Future of Learning and Higher Education. Books
on Demand.
Eppler, M. J.; Hoffmann, F. (2012). Design Thinking im Management. Zur Einführung in die
Vielfalt einer Methode. In: OrganisationsEntwicklung. Zeitschrift für
Unternehmensentwicklung und Change Management. Kreativität im Change
Management. Mit Design Thinking Organisationen verändern. 02/2012, S. 4 – 7.
Graves, M.; Fuchs, J. (2022). Teaching the Design Thinking Mindset: A Practitioner´s
Perspective. In: Schmidberger, I.; Wippermann, S.; Stricker, T.; Müller, U. (Hg.), Design
Thinking im Bildungsmanagement. Innovationen in Bildungskontexten erfolgreich
entwickeln und umsetzen. Wiesbaden: Springer VS. doi: 10.1007/978-3-658-36951-4
HPI School of Design Thinking (2023). Design Thinking. Online: https://hpi.de/school-of-
design-thinking.html (20.04.23).
Iberer U.; Müller, U. (2012). „Lernen am Unterschied ® “: das Modell der PH Ludwigsburg.
In: S. G. Huber (Hg.), Handbuch Führungskräfteentwicklung. Grundlagen und
Handreichungen zur Qualifizierung und Personalentwicklung im Schulsystem (S. 993-
1000). Köln: Link.
Lewrick, M. & Link, P. v Leifer, L. (2018). Das Design Thinking Playbook. Mit
traditionellen, aktuellen und zukünftigen Erfolgsfaktoren. München: Vahlen.
Lockwood, T. (2009). Design Thinking. In: Lockwood, Thomas (Editor): Design Thinking.
Integrating Innovation, Customer Experience, and Brand Value. 3. Edition. New York:
Allworth Press, pp. vii – xvii.
Meinel, C.; Leifer, L. (2011). Design Thinking Research. In: Plattner, H.; Meinel, C.; Leifer,
L. (Hg.): Design Thinking. Understand – Improve – Apply. Berlin und Heidelberg:
Springer, pp. xiii – xxi.
1177
Meinel, C.; Weinberg, U.; Krohn T. (2015). Design Thinking Live – Eine Einfühung. In:
Meinel, Christoph; Weinberg, Ulrich; Krohn Timm (Hg.): Design Thinking Live.
Hamburg: Murmann, S. 11 – 23.
Page, S. E. (2017). The Diversity Bonus. How great Teams pay off in the knowledge economy.
New Jersey: Princeton University Press.
Plattner, H.; Meinel, C.; Weinberg, U. (2009). Design Thinking. Innovationen lernen –
Ideenwelten öffnen. München: FinanzBuch.
Rittel, H. W. J.; Webber, M. M. (2013). Dilemmas in einer allgemeinen Theorie der Planung.
Deutsche Übersetzung. In: Reuter, W. D.; Jonas, W. (Hg.): Thinking Design. Basel:
Birkhäuser, S. 20 – 38.
Schmidberger, I.; Wippermann, S. (2018). Design Thinking in der Erwachsenenbildung. In:
Deutsches Institut für Erwachsenenbildung (DIE) (Hg.): weiter bilden. DIE Zeitschrift
für Erwachsenenbildung, Heft 3/2018, S. 53–56.
Schmidberger, I.; Wippermann, S. (2022). Die Innovationsmethodologie Design Thinking.
In: Schmidberger, I.; Wippermann, S.; Stricker, T.; Müller, U. (Hg.), Design Thinking im
Bildungsmanagement. Innovationen in Bildungskontexten erfolgreich entwickeln und
umsetzen. Wiesbaden: Springer VS. doi: 10.1007/978-3-658-36951-4
Uebernickel, F., Brenner, W., Pukall, B., Naef, T. & Schindlholzer, B. (2015). Design
Thinking – Das Handbuch. Frankfurt a.M.: Frankfurter Societäts-Medien.
Zukunftsinstitut (2021). Wie um die Megatrends gesellschaftlicher Wandel entsteht. Online:
https://www.zukunftsinstitut.de (14.04.2023).
1178
Combining integrated curriculum and project-based learning: a

short film case study from media and communication students
Hibai López-González1, Maria-Jose Masanet1, Maddalena Fedele1, Sandra Rius1,
Jaume E. Vilaseca1, Lydia Sánchez1, Sergio Villanueva Baselga1, Adrien Faure-
Carvallo1, Anna Marquès1, Jorge Franganillo1, Maria Ángeles García Asensio2
1
Department of Library, Information Science, and Audiovisual Communication, Universitat
de Barcelona, Spain, 2Department of Hispanic Studies, Literary Theory, and Communication,
Universitat de Barcelona, Spain.
Abstract
Undergraduate students in media-related fields expect their courses to be
focused on real-world work environments, ultimately producing media
products that can be proudly shared on social media and become part of their
CV. The present paper presents a combination of integrated curriculum (IC)
and project-based learning (PBL) to design a year-long activity that traverses
various courses through an academic year to produce a quality media
outcome. It utilizes face-to-face group interviews with the students and the
instructors involved in the experience, survey questionnaires, as well as a
comparison of the grades before and after the implemention of the project. The
results show that the quality of the media products rised, that overall
satisfaction of instructors was high, but students had mixed feelings. The lack
of self-assessment tools and the emergence of group conflicts considering the
length of the project were cited as the main limitations.
Keywords: Media; project-based learning; integrated curriculum; film
studies.

Combining integrated curriculum and project-based learning
1. Introduction
Higher education is in constant search of new learning methods that can capture and maintain
the attention of students, maximizing class attendance and student adherence to the programs,
while minimizing dropout rates (Aina et al., 2022; de Oliveira et al., 2021). Among the
various ways to achieve such goals, in recent years some pedagogical approaches have gained
momentum, including integrated curriculum models (IC) and project-based learning (PBL).
IC is an all-encompassing umbrella term that, generally speaking, views courses and modules
from a university degree not as independent entities but as lego pieces which converse
between them to build a common strategy (Drake & Reid, 2018). IC aims at dissolving the
artificial separation between subjects, setting up a framework that brings together the
contents, abilities, and fields shared by all of them. Therefore, fusion and interdisciplinarity
are frequent buzzwords in IC studies. Although IC is not exclusive of any developmental
stage of children, adolescents, or adult education, it has been more often applied in contexts
of secondary rather than higher education (Alonso-Sáez & Berasategi-Sancho, 2017).
On the other hand, PBL, although originally conceived in the 1960s (Graaff et al., 2007), has
become a well-established learning method that is based on the idea that students pursue their
own interests towards the completion of a project that requires learnings from different fields
(Krauss & Boss, 2018). PBL also relies on the assumption that as students engage with their
own projects, their involvement and overall satisfaction will consequently be higher, and they
will feel empowered (Guo et al., 2020). A meta-analytic review of the available evidence
about the true impact of PBL on academic achievement showed that PBL usually works, with
a moderate-to-large effect size, especially in social science contexts, and in Western
educational systems (Chen & Yang, 2019).
Although IC and PBL have been widely used in separate, few previous experiences blending
IC and PBL have been reported in the scientific literature. Scholars from Lapland University
of Applied Sciences in Finland report a blended learning experience using IC and PBL in
combination, indicating that the vast majority of students found it pedagogically motivating
(Mielikäinen, 2021).
The present paper reports the design and some preliminary results from a learning innovation
project blending IC and PBL (referred to as “IC+PBL” in the following pages), carried out
within an undergraduate program in media studies at a higher education institution. Students
from media-related areas often complain about the little connection between course contents
and real-world work environments, and also expect to produce high quality media materials
that can be shared on social media and become part of their portfolio.The aim of the project
was to develop a learning activity that will require media students to apply knowledges from
various fields and during two consecutive semesters in order to come up with a finished, well-
rounded media product.
1180
Hibai López-González et al.
2. Method
2.1. Participants and aim

The project was carried out with first-year undergraduate students enrolled in a BA in media
and communication studies at a large public university in Spain. The BA comprises four
academic years and each year consists of two semester of 30 European Credits Transfer
System (ECTS). In total, 230 first-year students distributed along three years have taken part
in this IC+PBL.
Students from this BA had typically complained about the lack of connection with reality of
the offered courses. Specifically, they had felt unprepared to develop large scale media
projects once they graduated. An additional source of dissatisfaction had to do with a general
lack of motivation to engage in long-term projects.
In order to tackle these issues, the IC+PBL was designed in a way that could boost motivation
by giving the students the chance to design their own long-term project, voted by all
classmates. The IC+PBL was also structured so that it could realistically replicate the
environment of a real-world media production, including all phases from creation, pre-
production, production, postproduction, and communication.
2.2. Design and procedure

The IC+PBL was defined as a year-long group project starting in September and ending in
May. Teams of about 8 members were randomly allocated at the beginning of the year. All
teams had to make a short film about a topic previously decided by the whole class. Each
team pitched different proposal, everybody voted, and the most voted proposal was the
compulsory topic for every team.
The IC+PBL needed three years to be fully developed. In year one (2019-20), the IC+PBL
was in pilot version, and only lasted one semester. In year two (2020-21), the IC+PBL
traversed two semesters, with one course from each semester. In year three (2021-22),
students had one course from the first semester and two courses from the second semester
partially or fully devoted to the IC+PBL. For 2022-23, it is scheduled that the IC+PBL will
involve a total of four courses.
The workload was distributed along the courses involved. Typically, first semester courses
were used to select the proposals, to look for similar successful projects (i.e., benchmarking),
to coordinate between teams in order to produce content and elements that needed to be used
by every team (e.g., opening credits, visual style, typography fonts…). The first version of
the IC+PBL product was shown in class at the end of the first semester, and the final version
at the end of the academic year.
1181
2.3. Assessment tools

The assessment of the development and efficacy of the IC+PBL was based on four criteria:
• A face-to-face group interview with the students to examine their opinions. These
interviews were conducted twice in the process, between the first and second
semester, and at the end of the academic course.
• A face-to-face group interview with the professors involved in the teaching of the
courses that comprised the project. This took place once at the end of the academic
course.
• An analysis comparing the students’ grades of the main module from the previous
academic year and the years after the project was implemented.
• A survey questionnaire shared with the students six months after the completion of
the project. The questionnaire asked students about their (i) overall satisfaction with
the experience, (ii) the in-group conflicts that the project caused, (iii) how often they
shared the media materials that resulted from the projects on social media, (iv)
whether they liked that a project traversed two consecutive semesters, (v) if they
were proud of the final result.
2.4. Ethics
The project obtained permission from the universities’ centre for learning innovation (REF
2021PID-UB/013). Two revisions of the original project draft were submitted until final
permission was obtained. As per university legal requirement, the students involved in the
project had the right to abandon it during the first month of each semester and be evaluated
by means of a traditional written exam. Students received no compensation.

The qualitative data from face-to-face group interviews showed an overall satisfaction with
the PBL. Students appreciated engaging with a project that lasted longer than usual and
allowed them to put into practice knowledge and skills obtained from various disciplines and
courses. The real-world connection of the IC+PBL was among its most valued
characteristics. The students found attractive the idea that the IC+PBLs were discussed in
class and that the common topic for everybody was decided by means of a vote.
Students showed criticism towards two main aspects of the IC+PBL. First, they mentioned
the lack of self-assessment methods as a limitation of the project. Self-assessment tools were
important for some students because, in their opinion, they allowed them to root out students
with little implication with the project. This was particularly relevant for teams in which
conflicts abounded. Self-assessment tools, as explained by students, would have increased
the overall perception that the IC+PBL fairly evaluated each member group, and that the
1182
group as a whole was not penalised for including unmotivated members. Second, the random
allocation to groups worked fine during the first semester each year, but by the time the
second semester started, some teams began to show disruptions. Students felt that it was ok
for first year students to see their teams picked by the professors, as they have not acquainted
with anybody yet. However, as the course progressed, they missed some kind of mechanisms
to change teams or to stop working with specific individuals.
From the instructors’ point of view, the IC+PBL was also satisfactory, but with a number of
limitations. The duration of the IC+PBL (one academic year) added an extra difficulty in
terms of coordination. The professors whose courses started on the second semester were not
as familiarised with the project as those professors with courses in the first semester, and this
resulted in some confusion about the progress of the projects, and the assessment.
Concerning the adequacy of the IC+PBL to achieve the aims promised in the syllabus, the
results are promising. The students outperformed the grades of similar students from the
previous editions of the same course. Table 1 shows a comparison between the grades
obtained by students pre and post IC+PBL implementation. The average mark showed a
significant increase from M = 7.93 – 8.25 (in the pre-IC+PBL period two different
assignments are considered) to 8.41 in the year 2021-22. Arguably, the interpretation of these
results is that the quality of the deliverables increased after the implementation of the
IC+PBL. No correlation analysis was carried out because the assignments that comprised the
final mark were not comparable. Repeated means procedures were not suitable as these were
independent samples. No causal relationhip can be established from average grades and
IC+PBL implementation.
Table 1. Grade comparison for the course “Audiovisual Language” before and after the
implementation of the IC+PBL
Pre-IC+PBL period IC+PBL period à
Academic year 2019-2020 2020-21 2021-22
Assignment Short film Script PBL PBL
Assessment weighting 20% 20% 60% 60%
Average grade 7.93 8.25 8.56 8.41

Source: Own data. Notes: Assessment weighting refers to the percentage for the final grade that the IC+PBL
represents. Average grades computes only grades for the specific assignments listed above on a scale of 1 to 10.
However, the results from the survey questionnaires somehow contradicted the overall
satisfaction of the students expressed in the face-to-face interviews. Table 2 shows the items
asked. Considering the score of 3 as the threshold of neutrality, students indicated relatively
negative emotions towards their final product, as seen in the fact that they were not especially
proud of it (M = 2.67, SD = 1.50), and that they very rarely shared it on social media (M = 2,
1183
SD = 1.48). This item is very important because it showed the lowest scores from the survey
and it does not only capture a perception but a behavioral dimension. Opinions about the
IC+PBL lasting two semesters instead of one also attracted relatively negative scores. On the
other hand, the overall assessment of the experience was relatively positive (M = 3.25, SD =
0.87), and students appreciated the fact that the same project traversed different courses (M
= 3.58, SD = 1.16). These results are preliminary and they only cover the opinion of 12
students. Not only the small number of respondents but the potential bias in self-selection
(Bethlehem, 2010) for responding (i.e., did the students who had a worse experience self-
select for participating in the questionnaire?) call for further assessments of the experience
in the future.
Table 2. Students’ assessment of the IC+PBL experience (N = 12)
Item Mean (SD)
Fights in my team were frequent 3 (1.04)

I am proud of the final product delivered by my team 2.67 (1.50)
I liked the fact that the project lasted two semesters and not just one 2.75 (1.29)
I have shared on social media my team’s work so others could see it 2 (1.48)
I liked the fact that different courses were part of the same project 3.58 (1.16)
Overall assessment of the experience* 3.25 (0.87)
Source: Own data. Notes. Items were assessed on a 1 to 5 likert scale (1=totally disagree; 5=totally agree). SD =
Standard deviation. *This question asked for a 1 to 5 overall assessment of the IC+PBL experience (1=very
negative; 5=very positive).
4. Conclusion
The present paper introduces a combination of an integrated curriculum (IC) and a project-
based learning (PBL) experience with undergraduate students from a BA in media and
communication. The experience was designed to increase motivation by creating a project
that connected real-world tasks in a media production environment with class activities. As
an IC experience, it integrated skills and abilities from different courses in a way that students
were able to produce more sophisticated products than in one course, one semester situations.
Students and faculty staff alike expressed positive opinions about the IC+PBL, while
acknowledging a series of limitations such as the lack of self-assessment tools and the
difficulty to form groups that stay in good terms and motivated for the duration of an entire
academic year. Survey questionnaires indicated that some students had more mixed opinions
about the overall experience.
1184
References
Aina, C., Baici, E., Casalone, G., & Pastore, F. (2022). The determinants of university
dropout: A review of the socio-economic literature. Socio-Economic Planning Sciences,
79, 101102. https://doi.org/https://doi.org/10.1016/j.seps.2021.101102
Alonso-Sáez, I., & Berasategi-Sancho, N. (2017). The integrated curriculum, university
teacher identity and teaching culture: The effects of an interdisciplinary activity. Journal
of New Approaches in Educational Research, 6(2), 127–134.
https://doi.org/10.7821/naer.2017.7.235
Bethlehem, J. (2010). Selection Bias in Web Surveys. International Statistical Review /
Revue Internationale de Statistique, 78(2), 161–188.
http://www.jstor.org/stable/27919830
Chen, C.-H., & Yang, Y.-C. (2019). Revisiting the effects of project-based learning on
students’ academic achievement: A meta-analysis investigating moderators. Educational
Research Review, 26, 71–81.
https://doi.org/https://doi.org/10.1016/j.edurev.2018.11.001
de Oliveira, C. F., Sobral, S. R., Ferreira, M. J., & Moreira, F. (2021). How Does Learning
Analytics Contribute to Prevent Students’ Dropout in Higher Education: A Systematic
Literature Review. Big Data and Cognitive Computing, 5(4), 64.
https://doi.org/10.3390/bdcc5040064
Drake, S., & Reid, J. (2018). Integrated curriculum as an effective way to teach 21st century
capabilities. Asia Pacific Journal of Educational Research, 1(1), 31–50.
https://doi.org/10.30777/apjer.2018.1.1.03
Graaff, E. de, Kolmos, A., de Graaff, E., & Kolmos, A. (2007). History of Problem-Based
and Project-Based Learning. Brill.
https://doi.org/https://doi.org/10.1163/9789087900922_002
Guo, P., Saab, N., Post, L. S., & Admiraal, W. (2020). A review of project-based learning in
higher education: Student outcomes and measures. International Journal of Educational
Research, 102, 101586. https://doi.org/https://doi.org/10.1016/j.ijer.2020.101586
Krauss, J., & Boss, S. (2018). Reinventing project-based learning: Your field guide to real-
world projects in the digital age (3rd editio). International Society for Technology in
Education.
Mielikäinen, M. (2021). Towards blended learning: Stakeholders’ perspectives on a project-
based integrated curriculum in ICT engineering education. Industry and Higher
Education, 36(1), 74–85. https://doi.org/10.1177/0950422221994471
1185
Teamwork and student engagement during practical sessions in

laboratories
Chalak Omar1, Sarah Plumb2
1
Multidisciplinary Engineering Education, The University of Sheffield, UK, 2The Elevate
Hub, The University of Sheffield, UK.
Abstract
Teamwork is an important pedagogical approach which equips students with
the necessary skills required for their learning and develops their
employability. This research focuses on teamwork in practical sessions, and it
investigates the factors that facilitate students’ engagement during practical
sessions, the optimum group size and how teamwork contributes to student
learning. Students were asked to complete a questionnaire after a team
working activity in the laboratory. The responses were quantitatively and
qualitatively analysed, and it was found that communication is the most
important element which enhanced the team’s performance and engagement.
The collaboration, discussion and interaction between the members were also
found to be important to student engagement. The optimum group size was
dependent on the complexity and the nature of the activity. It was found that
effective teamwork contributes significantly to the enhancement of student
learning and to developing required employability skills.
Keywords: Teamwork, soft skills, engagement, practical education.

Teamwork and student’s engagement during practical sessions in laboratories
1. Introduction
Active learning and effective engagement are important, interrelated factors in any higher
education institute, requiring students to be engaged in an activity related to a learning
outcome to develop their understanding of any given topic. The active learning approach is
known to be more effective at engaging students in their learning compared to traditional
lectures (Prince, 2004). Although active learning sometimes refers to the instructional
activities that are happening inside a classroom, it also includes practical sessions in
laboratories since students are actively engaged and thinking about what they are doing.
To ensure that the learning outcomes are delivered, and students gain the required skills in
the practical sessions, students need to be engaged in the session. Some articles have reported
on the performance of students during practical sessions, however, there is limited research
on the extent of engagement of students during those same activities.
Teamwork has many advantages including creating an active and positive learning
environment if used effectively. However, there are also some disadvantages of group or
teamwork. It is vital to understand the advantages and disadvantages of group work before
designing any activity to maximize the benefits and create an effective learning environment.
Beebe and Masterson (2003) summarised the advantages of group work saying that groups
have more knowledge and members are actively involved in discussion and contribution to
decisions and therefore more interaction with others. The disadvantages of this approach
include pressure from one group member on the others to agree on a decision to avoid
conflict. This might lead to a bad decision and therefore affect the outcome. The dominance
of a single individual in the discussion can also lead to dissatisfaction from other members
as they feel isolated from the discussion and final decisions. This may also result in other
group members relying heavily on the active members and acting as free riders either through
choice, dwindling motivation or lack of confidence to put themselves forward.
Despite the disadvantages, group work remains a popular approach in higher education
settings to facilitate student application of knowledge. Group work activities result in
excellent learning outcomes if there is effective student participation, and the activity is
properly designed by the teacher to ensure a positive learning environment is in place.
Group size should be selected carefully based on the complexity of the activity and the time
available. According to Beebe and Masterson (2003), a small group is considered to have
three members or two. A group size of four and more is large and difficult to coordinate.
Research articles reported a different optimum group size, some suggested 3-4 members be
the optimum size (Csernica et al., 2002) and others suggested 4-5 to work best (Davis, 1993).
The optimum group size depends on the nature and complexity of the activity and the
available time. It was reported that a smaller group size works best when the time available
is short (Cooper, 1990; Johnson, 1991). However, research suggests that it is important to
1188
create a group with mixed ability levels of skills and experience. This means that the groups
are more effective when formed by the teacher or randomly allocated than groups formed by
students' self-selection (Felder & Brent, 2001).
Some articles in Engineering education have reported that students liked working in teams
and agreed on the benefits of teamwork. Freeman and Greenacre (2011) found that 78% of
students agreed that teamwork helped them to learn and gain skills from others. The result
(Williams, 2011) also suggested that students learnt more when working on group tasks
compared to the traditional tutor presentation. Hammar Chiriac (2014) attempted to
investigate group work in higher education settings. They found that 97% of students from a
total of 210 students who participated in the study from two different universities, agreed that
working in groups facilitated their learning by improving both their knowledge and their
collaborative ability.
However, Freeman and Greenacre (2011) have also reported that a significant part of students
also voted against the benefits of teamwork due to the existence of what is called the free
riders who rely on others to get marks and rewards. The free rider issue can be overcome
through self and peer assessment where each member assesses themselves and other members
in the group which then will be considered when determining the final individual mark.
Multidisciplinary Engineering Education (MEE) is a dedicated department in the Faculty of
Engineering at the University of Sheffield which is responsible for delivering all practical
activities to undergraduate and postgraduate students from all departments at scale. To
provide the experience and the required skills to large cohorts, students often work in teams
to perform the activities in the laboratories. However, the approach needs to maintain the
required level of engagement from the team members to ensure the learning outcomes are
met. Some works in the literature investigated teamwork in different fields and others work
on group sizes and the advantages and disadvantages of working in teams, however, there is
no explicit research on which elements of the teamwork approach help and encourage
students to be engaged in the session. This research aims to investigate the factors and
elements of teamwork that facilitate student engagement during practical sessions in
laboratories.
2. Participants and Methods

Participants in this study were students from Chemical and Biological Engineering at the
University of Sheffield. It included students from the foundation year (13.8%), postgraduate
taught PGT (25.9%), and 4 undergraduate cohorts UG (60.3%). In these courses, practical
activities in laboratories are essential and students must work in the lab to collect data which
is then used to complete the post-lab activities. In all the practical activities used in the study,
students worked in groups of different sizes (ranging between 2-5 students per group)
1189
depending on the complexity of the activity and the capacity of the laboratory. In the majority
of these activities, students self-selected their groups from their peers in the laboratory. These
groups only functioned during the lab session and all post-lab activities were based on
individual student work. However, students from 2nd year were assigned to groups randomly
by the teacher and these groups were functional in the lab and outside the lab throughout the
semester. The members of each group met before the practical session to plan the activity,
then executed the experiment in the laboratory together, and finally analysed the data.
In this research, data from students was collected using a structured questionnaire, which was
ethically approved following the research ethics approval procedure at the University of
Sheffield. The questionnaire focused on students’ experience while working in laboratories
and not outside the lab. It included multiple choice questions about the format of the group,
year group and other background information as well as open-ended questions concerning
the students’ experience and their perceptions of group work. In total, 58 students from
different cohorts responded to the questionnaire.

The questionnaire and the data analysis were conducted based on the following research
questions:
1. What is the optimum group size for a practical session in a laboratory?
2. What factors of teamwork facilitate students’ engagement in the lab?
3. What positive experiences did students gain while working in groups? And how
does this contribute to their learning and future?
Students were asked about their views of working in teams in the laboratory. Most students
(86%) agreed that working in groups is better and more efficient to complete the tasks in the
laboratory than working alone. Only 4% of students suggested that working alone in the
laboratory would be more efficient. However, 10 % of students said that this depends on the
type and complexity of the activity.
These answers indicate that students value teamwork during practical activities as it is more
efficient, bringing more ideas and better chances to solve problems, as evidenced by this
student’s comments “Team, more ideas and solutions to potential issues”. This agrees with
the findings of others, where teamwork was found to be a good approach to gaining skills
and facilitating students’ learning and improving their knowledge and collaborative ability
(Freeman & Greenacre, 2011; Hammar Chiriac, 2014; Williams, 2011). This is because
teams have more than one way to solve a problem due to different experiences and knowledge
of various team members, which results in improving students’ performance in the session.
1190
Some students (a very small percentage) preferred working alone, however, looking into the
details of their comments, it looks like they wanted to be active all the time and to be more
efficient. This indicates that the number of students in the group might have been more than
what was required to complete the tasks in the experiment. When investigating this further,
it was found that one of the responses came from a PGT student who was experimenting with
a team of 3. The reason for their response might be that PGT students are more experienced
than UG students and therefore they might be able to carry out more tasks in a shorter time
frame.
One of the aims of this work was to gather students’ views on the group size and identify the
optimum size of a group. A majority of 83% confirmed that the group size (between 2 and 5
for different activities) was appropriate to complete the task in the laboratories. However, the
remaining 17% suggested that the group size was not appropriate for the practical activity.
The engagement in the laboratories was investigated, by asking students in each group to rate
their engagements between 1-5, 1 being “poor” and 5 being “Excellent Engagement”. The
majority of 95% have rated this between 4 and 5, which means that their engagement was
good or excellent (Figure 1). The average engagement rate was 4.4/5 for the self-selected
groups and 4.1/5 for the teacher-selected groups, which could suggest that students are
slightly better engaged in groups of their choice. It is believed that better engagement results
in better student performance in the session and therefore better learning experience (Crown,
2007).
60%
50%
40%
Percentage
30%
20%
10%
0%
1 2 3 4 5
Level
Figure 1: Percentage of students rating their engagement between 1-5, 1 being “poor” and 5 is “Excellent”.
The group size can affect the student’s engagement during the activity. Some comments
suggested reducing the size for the simple experiment to keep everyone engaged and other
comments suggested increasing the size for the complex activities. As mentioned earlier, the
group size in this study was between 2-5 in each group and 83% confirming that the set group
size was appropriate with a higher percentage of them being engaged, indicating that, from
1191
the students’ perspective the optimum group size is around 2 to 5 students. This agrees with
the findings from others in the literature who reported that the optimum group size is 4-5
(Davis, 1993) and 3-4 (Csernica et al., 2002). However, the definite group size is difficult to
determine as it depends on the complexity of the activity and tasks the group must complete
during the session.
To explore the factors that enhance students’ engagement in the laboratories, students were
asked to rank the importance of different elements of teamwork to their engagement based
on their experience in the laboratories. The average responses for each element were
calculated and are shown in Figure 2. The element that scored 4 and below on average was
important, between 4-5 was considered not to be significant and above 5 is not important.
From the results from the survey, it can be concluded that communication, collaboration,
interaction, and discussion are found to be important factors/elements of teamwork which
enhance students’ engagement. On the other hand, punctuality and motivation from peers
were found not to be important. There were no clear trends for the categories of feedback
from peers, suggestion solutions, and enthusiasm and participation from other team members
toward enhancing the engagements within a practical session.
4
Ave. Ranking
0
n
tio
d…
n
tio
ica
an
ity
ra
k
un
l
n
ac
bo
ua
ns
d…
tio
m
b
lla
io
t
ed
nc
an
m
rs
ut
Co
ra
Co
Pu
Fe
e
sm
ol
te
pe
gs
In
ia
m
tin
us
fro
th
s
ge
En
ns
g
tio
Su
i va
ot
M
Figure 2: Average ranking of the importance of each of the different elements in teamwork on a 1-8 scale (1 is
most important and 8 is least important).
Communication was found to be the most important factor in engaging students in practical
sessions. The success of a group in achieving quality work depends on the way the group
members communicate, their personalities and how motivated they are in contributing to the
1192
group work. There is clear evidence in the literature that effective communication improves
the performance (Beebe & Masterson, 2003). Teams with good communication and clear
aims are known to perform better compared to those without clear aims (Crown, 2007).
In order to explore whether teamwork facilitates student learning and helps them gain
transferable skills, the questionnaire in this research also included a question on how
teamwork contributes to student learning. The responses to this question were also
qualitatively analysed and were divided into two categories. As expected, most students
confirmed that working in teams during the practical sessions helped them develop
employability skills (communication, leadership, time management and problem-solving).
There were also a lot of responses which confirmed that teamwork during these sessions
improved their understanding and enhanced their knowledge of the subject. Enhancing
student learning and helping them develop these soft skills are the reasons why a teamwork
approach is used during practical sessions in laboratories.
When the responses were analysed, it was found that 62.5% of the responses were related to
employability and other soft skills. Students confirmed that teamwork helped them develop
the skills required for their life after university. Students appreciated that teamwork tasks at
university address a requirement for this type of skill from industry amongst other contexts
they are likely to encounter. The remaining responses (37.5%) were related to enhancing the
knowledge of students. Students benefited from the ideas of different members to enhance
and reinforce their knowledge. They also valued the diversity in the groups and appreciated
the fact that the diversity came from different cultural backgrounds which benefited all team
members. A comment about diversity can be seen in one student's comments below:
“Different people come from different backgrounds and have different values. Coming
together with all of these traits gives us more diversity. People generally like to invest in
something different that would benefit them.”
Gaining employability skills is important for graduates. Skills such as communication,
networking, collaboration, and leadership are important attributes which the activities in
laboratories provide an opportunity for students to develop. The comments and the feedback
received from students suggest that students are aware of the importance of communication,
discussion, interaction, and collaboration and how these help enhance their learning.
The fact that this study included participants from different year who were doing different
experiments at different levels of difficulty, makes the findings easily applicable to other
laboratories. In terms of group size, a size between 2-5 was found to be good (83% agreed
on their group size). For example, 2 students per group will be optimum for activities which
involve one setup where students only measure a variable. For activities which involve using
more than one piece of equipment and students have limited time, then 3 students per group
will be optimum to complete the tasks. However, when a pilot scale rig (or industrial scale
1193
equipment) is used in an activity where different unit operations are working simultaneously,
then at least 4-5 students are required to complete such an activity. It is also recommended
to ensure elements such as communication, collaboration and discussion are properly utilised
during the activity to maximize students’ engagement. This can be done by incorporating
some activities, such as asking questions to encourage team discussion and interactions.
4. Conclusion
Teamwork is an important approach used in universities to enhance student learning and
provide the required skills. However, students must be engaged to maximise their learning.
The research in this paper investigated the factors in teamwork that will facilitate student
engagement and how teamwork contributes to their learning. It was found that
communication between the team members is the most important factor to keep students
engaged in the session. Good communication between team members enhances team
performance and therefore their engagement. Collaboration, discussion, and interaction
between team members were also found to be important for student engagement. The
optimum group size was also investigated, and it was found that a size between 2-5 students
can be considered optimum, however, the exact group size will always depend on the nature
of the activity and its complexity. Finally, teamwork makes a significant contribution to
student learning, and it helps develop employability skills.
References
Beebe, S. A., & Masterson, J. T. (2003). Communicating in small groups. Boston, MA.
Cooper, J. (1990). Cooperative learning and college teaching: tips from the trenches.
Teaching Professor, 4 (5), 1-2.
Crown, D. F. (2007). The Use of Group and Groupcentric Individual Goals for Culturally
Heterogeneous and Homogeneous Task Groups:An Assessment of European Work
Teams. Small Group Research, 38(4), 489-508. DOI: 10.1177/1046496407300486
Csernica, J., Hanyak, M., Hyde, D., Shooter, S., Toole, M., & Vigeant, M. (2002). Practical
guide to teamwork. Lewisberg, PA: College of Engineering, Bucknell University.
Davis, B. G. (1993). Preparing or revising a course. Tools for teaching.
Felder, R. M., & Brent, R. (2001). Effective strategies for cooperative learning. Journal of
Cooperation & Collaboration in College Teaching, 10(2), 69-75.
Freeman, L., & Greenacre, L. (2011). An examination of socially destructive behaviors in
group work. Journal of Marketing Education, 33(1), 5-17.
Hammar Chiriac, E. (2014). Group work as an incentive for learning–students’ experiences
of group work. Frontiers in psychology, 5, 558.
Johnson, D. W. (1991). Cooperative Learning: Increasing College Faculty Instructional
Productivity. ASHE-ERIC Higher Education Report No. 4, 1991. ERIC.
1194
Prince, M. (2004). Does active learning work? A review of the research. Journal of
Engineering Education, 93(3), 223-231.
Williams, S. (2011). Engaging and Informing Students through Group Work. Psychology
Teaching Review, 17(1), 24-34.
1195
The impact of practical training on student understanding of

plagiarism
Mairéad Hogan
J.E. Cairnes School of Business, University of Galway, Ireland.
Abstract
In order to develop an understading of plagiarism and how best to avoid it, a
workshop was conducted with masters’ students. They took part in a 4 hour
interactive session that covered types of plagiarism, appropriate
acknowledgement of sources, paraphrasing and contract cheating. They also
completed a literature review afterwards for which they received two rounds
of feedback on plagiarism issues. Students completed a survey assessing their
understanding of plagiarism pre and post the workshop. A paired samples t
test showed a significant improvement in student understanding of plagiarism
as a result of the intervention. It is planned to roll out the initiative to other
student groups in the coming academic year.
Keywords: Plagiarism; academic integrity; student understanding; plagiarism
prevention.

The impact of practical training on student understanding of plagiarism
1. Introduction
During the height of the Covid-19 pandemic, educational institutions around the world
pivoted to online teaching and assessment without warning or preparation. Despite this, most
universities managed to quickly adapt, albeit in many cases in an ad hoc manner. In the
University of Galway, the institution in which this study is based, teaching staff moved to a
combination of live and recorded lectures via Zoom or similar. The experience and support
of the university’s Centre for Excellence in Learning and Teaching meant the many queries
of the teaching staff regarding technology and teaching approaches were quickly answered,
resulting in a different, but largly educationally successful, experience for staff and students.
A more difficult issue to resolve was that of assessment. Unlike physical exams, online
evaluation and submitted assignments provide significant opportunities for students to breach
academic integrity (Amzalag et al., 2022). Students can easily access the internet, consult
with classmates and quickly copy material from a variety of sources (Mbhiza, 2021;
Peytcheva-Forsyth et al., 2018; Sarwar et al., 2018). By necessity, written assignments and
timed online exams took the place of formal written exams. However, in common with many
other institutions (Meccawy et al., 2021; Mokdad & Aljunaidi, 2020), this did result in
increased reports of plagiarism within the School of Business in which this study took place.
With the relaxation of Covid-19 restrictions, in-class teaching and physical exams returned.
However, as is happening elsewhere (Jena, 2020; Mbhiza, 2021; Pettit et al., 2021), a
discussion on retaining the benefits of alternative assessment methods, whilst minimising
academic integrity issues, is ongoing. In person exams, the traditional approach to assessing
student knowledge, have the benefit of being administered in a secure environment where
students do not have access to materials other than those specifically allowed. However, this
approach can result in passive, rather than active, learning on the part of the student (Altay,
2013) with students simply memorizing the material for the exam (Flores et al., 2014), and
frequently forgetting what they have learned shortly afterwards (Rawson et al., 2013). In
constrast, approaches such as authentic assessment which increase realism are shown to
promote active learning by providing cognitive challenge and requiring judgement and
analysis, this improving learning outcomes for students (Hogan, 2020; Villarroel et al., 2020).
Based on academic integrity issues with students over the years, and in common with findings
elsewhere (Aasheim et al., 2012; Ali et al., 2012; Dawson & Overfield, 2015; Marshall &
Garry, 2005), it is clear that many students have a poor understanding of the ways in which
a student submission can be considered to be plagiarized. Thus, it was decided to take a
proactive approach to educating the students with the aim of reducing plagiarism incidents
and facilitating assessment integrity outside the exam hall.
1198
Mairéad Hogan
2. What is Plagiarism?
According to Cambridge Dictionary plagiarism is “the process or practice of using another
person’s ideas or work and pretending that it is your own”. On initial reading, it would appear
to be a relatively simple, clear-cut concept. Howeves, it is not as clearcut as it may initially
appear (Aasheim et al., 2012; Ali et al., 2012; Dawson & Overfield, 2015; Marshall & Garry,
2005). Students tend to have a clear understanding that cutting and pasting without
acknowledgement or purchasing an assignment, constitute plagiarism (Ali et al., 2012;
Marshall & Garry, 2005). However, actions such as lack of paraphrasing or citing sources
they have not read, are less likely to be considered to be plagiarism by students (Marshall &
Garry, 2005).
2.1. Education on Plagiarism

There is much academic discussion on how best to teach students about academic integrity.
Löfström et al. (2014) stress the importance of developing an understanding of the concepts,
rather than just informing them of the rules. Bertram Gallant (2017) discusses the benefits of
education rather than relying on sanctions. Other approaches include that of Brown and
Janssen (2017) who developed a workshop which facilated students exploring the concepts
relating to plagiarism and the development of a joint integrity code. They noted a decrease in
plagiarism cases consequent to the workshop. Fenster (2016) reported positive results after
their students took part in a one hour workshop focused on paraphrasing. However, not all
education is equal. Holt et al. (2014) found no improvement in students’ knowledge
following an online training course. They did, however, find a significant improvement in
those who completed a relatively high stakes homework plagiarism assignment.
3. Design of the Training

Given the documented success of various workshops on plagiarism knowledge, it was
decided to develop a workshop that covered academic integrity from a number of
perspectives. The workshop consisted of the following elements:
Student discussion on their understanding of plagiarism. A ‘Think-Discuss-
Share’ approach was used where students spent some time making note of what they
believed was meant by plagiarism. The then discussed their thoughts in groups of 3.
Finally, they shared the group consensus. Students were also presented with a series
of scenarios, ranging from clear and obvious plagiarism to no plagiarism (Carroll,
2022, p. 52) and asked to identify where the line is for plagiarism.
Discussion on types of plagiarism. The many ways in which students can plagiarise
were presented and discussed in groups.
1199
The difference between collusion and collaboration. As groupwork is so

prevalent, the differences between collaboration and collusion were discussed.
Students examined various scenarios in small groups and were tasked with
identifying the line between collaboration and collusion (Carroll, 2022, p. 19).
How and why we credit sources. The reasons we credit our sources were
discussedas were the rules for doing so.
Paraphrasing. Poor paraphrasing is a major cause of student plagiarism. Strategies
for effective parphrasing were discussed and students practiced paraphrasing with
feedback.
Contract cheating. The final section of the workshop covered contract cheating and
involved the students in discussions on various scenarios relating to contract
cheating.
Following on from the workshop, the students wrote a 2-page literature review on a topic
relevant to their major project. They received feedback on any plagiarism issues, such as poor
paraphrasing. They corrected and re-submitted the document and received futher feedback.
4. Methodology
Students completed a survey (see Table 1) prior to participating in the plagiarism workshop.
This survey, adapted from those of Marshall and Garry (2005), Clarke et al. (2022) and
Kokkinaki et al. (2015), was created to gauge student understanding of what constitutes
plagiarism. They were asked how capable, on a scale of 1-5, they believe themselves to be
of avoiding plagiarism in assignments. They were then asked to examine a number of
scenarios and answer yes, no or don’t know as to whether they believed them to be
plagiarism. After completing the workshop and the literature review practice, the students
completed the survey a second time to see had their understanding improved. A total of 60
students took part in the workshop, with 32 completing both the pre and post workshop
surveys. They were all students on a Masters in Information Systems Management. Of the
32 students, 14 were female and 18 were male. Of those, 5 had English as a first language,
with the majority of the remaining students being from India (17) or China (9).

Students were initially asked to indicate their confidence levels in their ability to avoid
plagiarism on a scale of 1-5, with 1 being very unconfident and 5 being very confident.
Students were confident going in, with a mean value of 4.09. Post workshop, that confidence
increased to 4.16. However, there was no significant difference in the values. That confidence
1200
Mairéad Hogan
may have been misplaced in some cases as the number of correct identifications of plagiarism
pre-workshop had a mean of 11.06 correct and ranged from 1 to 14 correct.
The individual who only got 1 correct describing themselves as ‘neither confident non
unconfident’. That individual rated themselves as ‘fairly confident’ after the workshop and
got all 16 correct in the post-workshop survey. For that individual at least, the workshop
appears to have greatly improved their knowledge of what constitutes plagiarism.
The descriptions of potential plagiarism can be seen in Table 1. The mean pre-workshop
value was 11.06 correct and improved to a mean of 13.28 correct post workshop. Paired
samples t-test was conducted. The results indicated a significant difference in the number of
correctly identified examples of plagiarism pre (M=11.06, SD = 2.663) and post (M=13.28,
SD=1.631) workshop; t(31) = -3.738, p <=0.001. As can be seen in Table 1, the number
correctly identifying plagiarism increased post-workshop for each scenario presented.
The lowest number of correct responses both pre and post was Q.3. This suggests, and is
supported by most cases of plagiarism encountered with the School of Business, many
students struggle to understand what paraphrasing means. While they did get some practice,
along with examples and discussion, the time devoted to it was insufficient. They also got
feedback on their paraphrasing in the submitted literature review. In future workshops, more
focused time will be spent on paraphrasing. There is, however, also the possibility that, given
the number of students whose first language is not English, the subtleties of the difference
between “changing several words” and paraphrasing may have been missed.
Quite a number of students believed there was plagiarism in Q2, Q5 and Q6. It is not clear
why this is the case but the phrases “copying”, “same theme as an existing one” and “someone
else’s work” may have triggered a gut feeling that it was plagiarism. The numbers
recognizing that these are not plagiarism post-workshop increased significantly.
Ultimately, the workshop seems to have improved students awareness of what constitutes
plagiarism. They had an opportunity to practice academic writing and received feedback on
errors in paraphrasing and appropriate acknowledgement of sources. Verbal feedback from
the students was positive, with many stating the workshop and subsequent literature review
helped to clarify their thoughts on how best to refer to other people’s work in their writing.
1201
Table 1. Is this plagiarism?
Correct
Pre Post
1. Translating information from a source in a foreign language without 25 30

appropriate acknowledgement of the source (Y)
2. Copying exact words from another source but placing them within quotation 18 28
marks and with appropriate acknowledgement of the source (N)
3. Taking a section of text from another source, changing several words in it and 9 10
acknowledging the source (Y)
4. Rewriting a piece of text from another source in your own words and only 13 25
acknowledging the source in a reference list at the end of your paper (i.e. no
in-text citation) (Y)
5. Creating a new piece of work on the same theme as an existing one but in a 22 27
new context and without copying the existing one. (N)
6. Rewriting a short section from someone else’s work in your own words and 17 25
including appropriate acknowledgement of the source (N)
7. Copying short sentences (less than 50 words) from another source without 29 32
appropriate acknowledgement of the source (Y)
8. Paying someone to write part or all of a piece of work that you then submit as 31 32
your own work (Y)
9. Working with other students on an individual assignment and submitting it as 27 29

your own work (Y)
10. Using another piece of work to identify useful secondary sources that you cite 18 20
in your own work, but without reading the secondary sources (Y)
11. Copying exact words from another source with appropriate acknowledgement 15 19
of the source (Y)
12. Resubmitting an assignment (or part of an assignment) previously submitted in 24 30

one module for assessment in another module (Y)
13. Copying exact words from another source without appropriate 30 31

acknowledgement of the source (Y)
14. Copying a web site and putting your own words and name into the content part 27 27
of the pages. (Y)
15. Copying the ideas from another piece of work and writing about them in your 26 30
own words, without appropriate acknowledgement of the source (Y)
16. Using pictures from the internet without appropriate acknowledgement of the 23 30
source (Y)
1202
Mairéad Hogan
6. Conclusion
The students on a masters’ course participated in a half-day workshop on plagiarism. The
workshop allowed practical opportunites to discuss the nuances of plagiarism and to practice
the skills necessary to avoid it. This intervention improved student understanding of
plagiarism. While it can be deemed a success in that regard, the ultimate test will be its impact
on the number of plagiarism cases this academic year. They are being tracked and will be
compared with previous years. To date, there have been 3 cases, all of which involved the
use of parphrasing software by students whose first language is not English. While this was
discussed during the workshop as being plagiarism, perhaps there was insufficient focus on
the issue. This will be addressed in future iterations.
Overall, the students were extremely positive about the experience and actively participated.
Perhaps the best feedback relating to the workshop was being approached by a student in a
different programme asking why they did not also get this opportunity. If plagiarism numbers
are reduced, the workshop will be rolled out to other programmes on an ongoing basis.
References
Aasheim, C. L., Rutner, P. S., Li, L., & Williams, S. R. (2012). Plagiarism and programming:
A survey of student attitudes. Journal of Information Systems Education, 23(3), 297-313.
Ali, W. Z. W., Ismail, H., & Cheat, T. T. (2012). Plagiarism: To what extent it is understood?
Procedia - Social and Behavioral Sciences, 59, 604-611.
doi:10.1016/j.sbspro.2012.09.320
Altay, B. (2013). User-centered design through learner-centered instruction. Teaching in
Higher Education, 19(2), 138-155. doi:10.1080/13562517.2013.827646
Amzalag, M., Shapira, N., & Dolev, N. (2022). Two sides of the coin: Lack of academic
integrity in exams during the Corona pandemic, Students' and lecturers' perceptions.
Journal of Academic Ethics, 20(2), 243-263. doi:10.1007/s10805-021-09413-5
Bertram Gallant, T. (2017). Academic integrity as a teaching & learning issue: From theory
to practice. Theory Into Practice, 56(2), 88-94. doi:10.1080/00405841.2017.1308173
Brown, N., & Janssen, R. (2017). Preventing plagiarism and fostering academic integrity: A
practical approach. Journal of Perspectives in Applied Academic Practice, 5(3), 102-109.
Cambridge Dictionary. The Cambridge Dictionary. Retrieved from
https://dictionary.cambridge.org/dictionary/english/plagiarism
Carroll, J. (2022). A handbook for deterring plagiarism in higher education: Oxford Centre
for Staff and Learning Development.
Clarke, O., Chan, W. Y. D., Bukuru, S., Logan, J., & Wong, R. (2022). Assessing knowledge
of and attitudes towards plagiarism and ability to recognize plagiaristic writing among
university students in Rwanda. Higher Education, 85(2), 247-263. doi:10.1007/s10734-
022-00830-y
1203
Dawson, M. M., & Overfield, J. A. (2015). Plagiarism: Do students know what it is?
Bioscience Education, s(1), 1-15. doi:10.3108/beej.8.1
Fenster, J. (2016). Teaching note - evaluation of an avoiding plagiarism workshop for social
work students. Journal of Social Work Education, 52(2), 242-248.
doi:10.1080/10437797.2016.1151278
Flores, M. A., Veiga Simão, A. M., Barros, A., & Pereira, D. (2014). Perceptions of
effectiveness, fairness and feedback of assessment methods: a study in higher education.
Studies in Higher Education, 40(9), 1523-1534. doi:10.1080/03075079.2014.881348
Hogan, M. (2020). From Times Square to Eyre Square: Hackathons as authentic learning for
Information Systems students. Paper presented at the 6th International Conference on
Higher Education Advances (HEAd’20). , Valencia, Spain.
Holt, E. A., Fagerheim, B., & Durham, S. (2014). Online plagiarism training falls short in
biology classrooms. CBE Life Sciences Education, 13(1), 83-89. doi:10.1187/cbe.13-08-
0146
Jena, P. K. (2020). Impact of Covid-19 on higher education in India. International Journal
of Advanced Education and Research, 5(3), 77-81.
Kokkinaki, A. I., Demoliou, C., & Iakovidou, M. (2015). Students’ perceptions of plagiarism
and relevant policies in Cyprus. International Journal for Educational Integrity, 11(1).
doi:10.1007/s40979-015-0001-7
Löfström, E., Trotman, T., Furnari, M., & Shephard, K. (2014). Who teaches academic
integrity and how do they teach it? Higher Education, 69(3), 435-448.
doi:10.1007/s10734-014-9784-3
Marshall, S., & Garry, M. (2005). How well do students really understand plagiarism? Paper
presented at the 22nd annual conference of the Australasian Society for Computers in
Learning in Tertiary Education (ASCILITE).
Mbhiza, H. W. (2021). Shifting paradigms: Rethinking education during and post-COVID-
19 pandemic. Research in Social Sciences and Technology, 6(2), 279-289.
doi:10.46303/ressat.2021.31
Meccawy, Z., Meccawy, M., & Alsobhi, A. (2021). Assessment in ‘survival mode’: student
and faculty perceptions of online assessment practices in HE during Covid-19 pandemic.
International Journal for Educational Integrity, 17(1). doi:10.1007/s40979-021-00083-9
Mokdad, M., & Aljunaidi, S. (2020). Whither plagiarism in distance learning academic
assessment during COVID-19? Paper presented at the 2020 Sixth International
Conference on e-Learning (econf).
Pettit, M., Shukla, S., Zhang, J., Sunil Kumar, K. H., & Khanduja, V. (2021). Virtual exams:
has COVID-19 provided the impetus to change assessment methods in medicine? Bone
and Joint Open, 2(2), 111-118. doi:10.1302/2633-1462.22.BJO-2020-0142.R1
Peytcheva-Forsyth, R., Aleksieva, L., & Yovkova, B. (2018). The impact of technology on
cheating and plagiarism in the assessment – The teachers’ and students’ perspectives. In
AIP Conference Proceediungs, 2048(1).
Rawson, K. A., Dunlosky, J., & Sciartelli, S. M. (2013). The power of successive relearning:
Improving performance on course exams and long-term retention. Educational
Psychology Review, 25(4), 523-548. doi:10.1007/s10648-013-9240-4
1204
Mairéad Hogan
Sarwar, S., Idris, Z. M., & Ali, S. M. (2018). Paid academic writing services: A perceptional
study of business students. International Journal of Experiential Learning & Case
Studies, 3(1), 73-83.
Villarroel, V., Boud, D., Bloxham, S., Bruna, D., & Bruna, C. (2020). Using principles of
authentic assessment to redesign written examinations and tests. Innovations in Education
and Teaching International, 57(1), 38-49. doi:10.1080/14703297.2018.1564882
1205
Evolution of a continuous assessment and feedback concept in a

computer science 101 course
Antje Neve, Dieter Pawelczak
Department of Electrical and Computer Technology and Communication Technology,
University of the Bundeswehr Munich, Germany.
Abstract
Teaching first semester students in fundamentals of computer science is
challenging due to the heterogeneous group. For the last three years we
explored different methods of continuous assessment in the course. One main
goal for this group of students is to encourage participation in the classroom
while also being able to continuously monitor the learning outcomes of the
most recent topics. All three methods: repetitive oral assessment, partial exam
and online quizzes received good evaluation results from the students, showed
an increased engagement and better examination results. However, continuous
assessment also increases the workload for both groups: the students and the
teachers. The combination of a partial exam (midterm) and continual online
quizzes showed a good compromise with respect to the effort on both sides and
reveals good examination results. This work describes a fine granular
technical and didactical workflow of the lecture suitable for first semester
students and discusses the evaluation results.
Keywords: Computer science education; continuous assessment; online quiz.

Evolution of a continuous assessment and feedback concept in a computer science 101 course
1. Introduction
Teaching first semester students is a special joy and challenge because they start a new chap-
ter in their life and teachers have the chance to offer new methods of learning. For the last
three years a continuous assessment and feedback approach with different methods has been
used to encourage participation in the classroom and to increase learning outcome in a foun-
dational computer science 101 course. Continuous assessment (CA) works for both groups
in the class room – the students must engage with the materials longer and on a regular basis
and teachers are able to monitor the process and can guide the learning process in a timely
manner (Sun, 2018). However, continuous assessment also increases the workload of the
lecturers (Serrano et al., 2018). Therefore, this work shows how a combination of methods
leads to more engagement and class room attendance on the students side resulting in better
exam grades and describes a fine granular technical and didactical workflow to achieve the
above mentioned goals. Furthermore, the feedback of students’ course evaluation reveals
indicators how the teaching concept can be adapted to other courses.
According to da Silva Burke (2022) CA is an integral part of the Bologna process with the
focus on increasing students’ engagement and distributing the workflow from a single exam-
ination phase to a continuous learning and assessment phase. Different methods of CA have
evolved and been established in higher education, like a portfolio of homework assignments,
partial exams, exercise assignments, online quizzes, presentations, compare e.g. Day et al.
(2018), de Sande and Murphy (2014). In STEM education, traditionally lab courses request
a continuous engagement of students and, as Wolf (2010) proposes, modern concepts offer
virtual labs for training and/or additional assessment. Without doubt, CA increases the en-
gagement of students, but there is no general agreement in research, that CA is beneficial for
all types of curricula and students. Several studies show, that CA is especially supportive for
better performing students, while less good performing students might be overwhelmed by
the workload during the semester compare e.g. Day et al (2018), Merello and Zorio-Grima
(2018). A CA based approach therefore shall be adjusted to find the right balance of work
load throughout the semester. An important factor for a successful implementation of CA is
the feedback given to the students. While the correction of written homework assignments
and partial exams might be time consuming and leading to a delayed feedback, the use of an
automated test and feedback system helps to report immediate individual feedback and to
lower the workload of the lecturer. As Baleni (2015) emphasizes, such tests can be performed
in a much shorter period allowing an immediate monitoring of the students’ achievments.
We examined different methods of CA, starting with continuous oral assessment at the be-
ginning of the course and now using a mixture of real-time formative assessment through
online quizzes and assessment by a partial exam (midterm) with a focus on the feedback to
the students.
1208
2. Lecture Setup
2.1. General Setup

Every year about 70 students start their studies of electrical engineering and computer science
at our faculty. The students are divided into two groups and taught a foundational computer
science course two times per week for 90 minutes. The lecture setup is as follows: Twelve
different topics are covered during the semester, for example, introduction to computer arith-
metics, data structures, basics of computer networking, operating systems and computer
architectures. Additionally five lab exercises are given.
2.2. Course Assessment

Each lecture starts with revision questions covering the subjects of the last lecture. During
the Covid Pandemic in 2020 and 2021 these revision questions were given directly by the
lecturer and oral participation and correct answers were awarded. For exam preparation a list
of all questions without solutions were given to the students. Similar questions are part of the
final exam. The differences in the course setup over the years are shown in Table 1.
Table 1. Different methods for continuous assessment from 2020 to 2022.
Year 2020 2021 2022
List of Quiz Questions Orally discussed in Orally discussed in Converted to online

classroom classroom quizzes
Quiz Conduction ca. 15 min. ca. 15 min. ca. 5 min. online plus
(at beginning of lecture) oral discussion oral discussion 2-5 min. discussion
Assessment of Quiz Answers graded Answers not graded Answers not graded.
Questions
Written Midterm None 30 min. Graded 30 min. Graded

Exam
Feedback Directly by the Directly by the Direct quiz feedback

lecturer during oral lecturer during oral via LMS and graded
quiz quiz and graded midterm
midterm
Due to the smaller groups because of hybrid teaching during the pandemic it was feasible to
orally discuss and grade these revision questions in a timely and fair manner. This proved to
be more difficult with a full classroom in 2021, therefore answers were not graded then. In
2022 the questions were converted into more structured online quizzes, which will be
described in section 2.3. The students can access the quizzes with a mobile device, tablet or
1209
laptop via the learning management system (LMS). The online quizzes take about five
minutes each and are not repeatable but the questions and (correct) answers remain visible
for the students throughout the semester. The students and the lecturer receive the results
instantly and can use the results for a short discussion about knowledge gaps or miscon-
ceptions.
After half of the semester a voluntary midterm exam is conducted, enabling the students to
collect points towards the final exam. Within a week the students receive individual feedback
of their midterm results.
PDF
Question pool in TeX
tex_to_qti.py tex_to_tex.py
XML
QTI PDF
Online Quiz in LMS MC Question in written Exam
Figure 1: Technical flow – Conversion of questions from LaTeX via QTI import into LMS as online quiz, as
standard PDF or as a MC question for the exam.
2.3. Technical Setup

Figure 1 shows the technical flow for creating the different document types for the questions
and quizzes. An important goal was the independence of the underlying LMS achieved by
the following setup: Starting point for all quizzes is a pool of questions written in LaTeX.
This document has two display options: a students’ version with a space left to write their
solutions and a teacher’s version with a sample solution. Converted to pdf format the stu-
dents’ version can be applied to discuss questions in classroom and solutions can be added
by the teacher using e.g. a tablet connected to the video projector. For the online quizzes we
1210
defined a fixed format for the sample solution in order to convert the questions automatically
into a corresponding XML file necessary for the IMS Question and Test Interoperability
(QTI) format, which is required for the import into the LMS. For single or multiple choice
(SC/ MC) questions, the proper answers are prefixed with a (T), whereas wrong answers are
marked as a LaTeX comment starting with %(F). Points or negative points can optionally be
given in rounded brackets. The question itself is marked with a comment defining the
question type. A python script tex_to_qti.py converts the questions from LaTeX to QTI.
Other supported questions types are: cloze questions, numerical answers, short answers, text
matching, sequencing/ordering. Another python script tex_to_tex.py generates a proper
LaTeX output that can be included in the final exam. Although questions can be created and
maintained directly in the LMS, setting up questions and maintaining the questions in LaTeX
significantly simplifies the work flow, especially as the same questions can be used in
different formats and use cases. Furthermore, the LaTeX documents can be managed and
maintained in a version control system.
3. Evaluation
3.1. Course Evaluation Methodology

In addition to a standardized online course evaluation in week 10 of the semester, an online
evaluation of the course’s assessment (midterm and online quizzes) has been conducted in
the last lecture using the LMS. The evaluation period for both evaluations was scheduled for
a week, but for a better response rate both online evaluations were conducted during the
lecture. The response rate were about 60 % and 58 % respectively.
3.2. Evaluation Results

An important result of the standardized online evaluation is that 74 % of the students perceive
the workload of the course adequate as defined in the module description. Only 5 % state a
too high workload, which is comparable to other course evaluations in the first semester.
With respect to the evaluation of midterm and online quizzes, all participants of the evalu-
ation agreed or fully agreed that the voluntary midterm exam should be graded and account
to the final grades as well in the future. “Midterm is a valuable concept and helps to assess
current learning outcome” is one of the open feedback results in the evaluation.
Figure 2 shows an excerpt of the students evaluation with respect to the online quizzes. The
opinions differ with respect to grading the online quizzes. While about 50 % would like the
possibility to voluntarily collect points for the final exam (Q1), 50 % disagree on a mandatory
grading of the quizzes (Q2). The majority (60 %) agree to continue with the quizzes mainly
as a self-monitoring tool (Q3). Some students even stated in the open feedback question, that
they would not like the quizzes to be graded as the cloze questions a hard to answer properly.
1211
Q1. Would you also like to collect points for the exam with the Q4. Does the quiz at the beginning of the lecture make you
quizzes in the future? deal more intensively with the material? 3%
23% 25% 25% 18% 10% 8% 18% 44% 28%
fully agree fully disagree fully agree fully disagree
Q2. Should the quiz results be included in the exam grade in Q5. Do you find the time at the beginning, which is used for
the future? the quizzes, useful for your learning success? 3%
8% 15% 28% 35% 15% 53% 45%
Q3. Should the quizzes continue to be used for voluntary self- Q6. Would you rather have a review from the lecturer than
monitoring in the future? the quizzes?
60% 23% 18% 8% 30% 50% 13%
Figure 2: Students’ evaluation results for the appliance of quizzes in the course (5 point Likert scale).
In opposite to our expectation only 26 % of the students stated that they prepared more for
the lecture due to the tests (Q4). About 98 % of the students agreed or fully agreed that the
time used for the online quizzes are useful for their learning success (Q5). Only a small group
of students (8 %) would rather prefer a review by the lecturer instead of doing the online
quizzes (Q6). Quotes from the evaluation confirm this impression. Several students wrote in
the free form of the evaluation that doing online quizzes was fun and a good repetition.
Critical responses came with respect to technical improvements of the LMS necessary to run
the tests on some mobile operating systems and partly, that questions were too easy.
4. Discussion
Figure 3a) depicts the participants in the class over the course of the semester in 2021 (blue
dots) when the tests were given orally and discussed directly in class. The orange crosses
indicate the actual participants of the online quizzes at the beginning of the lectures in 2022.
Both curves show a decline in the number of participants. The slope of the curve in 2022 is
actually flatter than in 2021 and for both curves we see a decline after the midterm exam.
With the online quizzes in 2022 more students attended the lecture and the quizzes compared
to the year before where only a minority of the students actually participated in the classroom
discussion. So it can be concluded that higher involvement in the class, as well as more
continuous and intense work with the course materials can be achieved through continuously
conducting online quizzes. In Figure 3b) we can see a generally high level of the actual quiz
results with an overall mean value of 74 %. Although we tried to provide a proper distribution
of questions types and consistent difficulty level for each quiz, the average results vary from
quiz to quiz. One reason is certainly that on some mobile devices the text matching questions
did not work properly, and another, that cloze questions and short answers allowed only for
1212
small spelling mistakes. Interestingly there is no rising trend during the course and towards
the exam but rather a constant moving average.
Figure 3: a) Participation vs. Class Attendance b) Average Points in Quizzes c) Exam results per sub task A1-A5
and overall exam pass rate.
Table 2 Overview of the received average grades and pass rates of the last four years.
Year 2018 2020 2021 2022
Final Grade [1.0 = best, 5.0 = failed] 3.7 3.4 3.3 2.4
Final Grade with Midterm - 3.4 3.1 2.2
Average Grade of Midterm Exam - 3.0 3.1 2.2
Percentage of Students passing the Exam 64% 76% 82% 97%
Table 2 shows the development of the exam and the midterm exams and the percentage of
students passing the exam since 2018 while Figure 3c) details the average results of the final
exams in 2021 and 2022. Throughout the years the structure of the exams and its grading
scheme has been identical. The only difference between the last two exams (comp. Fig. 3c))
was that we used two multiple choice questions in the first sub assignment which were similar
to the questions the students saw during their quizzes. This accounted for an average of two
more points for this sub assignment which is only 3 percent of the total reachable number of
points in the exam. Notably, every exam sub task in 2022 showed better results than the year
before. It also has to be considered that there are always differences between the years re-
garding the level of achievement of the particular group of students. We see that over the
course of three years the average grade of the midterm and final exam improve by nearly one
grade. Additionally, the pass rate of the final exam increases with the introduction of CA.
5. Conclusion
On a more personal level, after two years of online and hybrid teaching setups, both students
and the lecturers were happy to be back in the classroom. With the evolution of the course
1213
through the online quizzes enabled by the technical workflow shown in Figure 1 more inter-
action and engagement with more students were possible. The technical workflow enabled
an independence from the current learning management system while keeping maintenance
of the pool of questions on low administrative level and putting the focus on the content of
the quizzes. The concept proves to be valid for first semester students, who are entering the
university system and can be guided into the learning methods that are necessary here, com-
pare Jimaa (2011). We conclude that continuous assessment through online quizzes and the
midterm exam supports developing continuous studying habits, which is a major achieve-
ment for first semester students. The positive development of the exam results over the years
indicate improved understanding of the topics by the students.
References
Baleni, Z. (2015). Online formative assessment in higher education: Its pros and cons. The
Electronic Journal of e-Learning Volume 13 (4) 2015, pp. 228-236
da Silva Burke, T.S. (2022). Continuous assessment in a large first-year engineering
mechanics course: the effect of participation and performance in compulsory and
voluntary assessments on final grades. 2022 IEEE IFEES World Engineering Education
Forum - Global Engineering Deans Council (WEEF-GEDC), Cape Town, South Africa,
2022, pp. 1-5, doi: 10.1109/WEEF-GEDC54384.2022.9996208.
Day, Indira N. Z., van Blankenstein, Floris M., Westenberg, P. Michiel and Admiraal,
Wilfried. F. (2018) Explaining individual student success using continuous assessment
types and student characteristics, Higher Education Research & Development, 37:5, 937-
951, DOI: 10.1080/07294360.2018.1466868
de Sande, J. C. G. and Murthy, A. (2014). Including peer and self-assessment in a continuous
assessment scheme in electrical and electronics engineering courses. 2014 IEEE Frontiers
in Education Conference (FIE) Proceedings, Madrid, Spain, 2014, pp. 1-5, doi:
10.1109/FIE.2014.7044284.
Jimaa, S. (2011). The impact of assessment on students learning. Social and behavioral
sciences, vol. 28, no. 2011, pp. 718-721.
Merello, Paloma and Zorio-Grima, Ana (2018). Explanatory factors of student performance
in online tests for the continuous assessment: Is attendance really important? 4th Int. Conf.
on Higher Education Advances (HEAd’18), València, p. 969-977. DOI:
Serrano, N.; Blanco, C.; Carias, F. and Reina, E. (2018). Information from Automated
Evaluation in an Engineering School. 4th Int. Conf. on Higher Education Advances
(HEAd’18), València, p. 987-994. DOI: http://dx.doi.org/10.4995/HEAd18.2018.8132
Sun, Y. (2018). Data analysis of traditional summative assessment and continuous assess-
ment-Engineering Science Departments case study. 2018 IEEE Int. Conf. of Safety
Produce Informatization (IICSPI), Chongqing, China, 2018, pp. 531-535, doi:
10.1109/IICSPI.2018.8690451.
Wolf, T. (2010). Assessing Student Learning in a Virtual Laboratory Environment. IEEE
Trans. on Education, Vol. 53 (2), May 2010
1214
Black student achievement plan evaluation and assessment

Tr’Vel Lyons, Mabel Hernandez, Sy Doan, Shafiqa Ahmadi, Darnell Cole
University of Southern California, RAND Corporation, United States.
Abstract
The study analyzes focus groups and individual interviews with 179 adult
participants and 116 student participants during the 2022-23 academic year.
Adult participants included principals, administrative coordinators of
instruction, teachers, staff members and parents of students within a large
school district in California. All of these adults and students were active
benefactors and staff members of a Black Student Achievement Plan (BSAP),
an initiative created to foster educational equity and college preparedness for
schools with over 200 Black students or a document history of academic
underperformance for Black students. We examine the implementation of the
Black student achievement plan i.e., the adoption of culturally sustaining
curricula and instruction, the allocations of funds to hire necessary staff,
funding of culturally enriching activity and experiences, etc. The findings
suggest that BSAP hires are serving as invaluable support systems for many
students and often serve as respite for students experiencing adverse
conditions. Findings suggest that BSAP is a crucial part in increasing
students’ college readiness and provides students with supports and
educational experience e.g., field trips that increase their likelihood for
attending college. This research informs districts and college readiness
programs about the necessary supports to provide to Black and historically
underrepresented students.
Keywords: Assessment; educational equity; learning outcomes; curriculum
alignment; postsecondary completion.

1. Introduction
In February 2021, a California School Board committed to significant investments in Black
students' education and emotional wellbeing by establishing the Black Student Achievement
Plan (BSAP). BSAP addresses Black students' unique needs related to historic and ongoing
inequitable educational opportunities, particularly as it is paired with engaging “the current
landscape of local and national advocacy for racial equity have served as the inspiration to
act now" (Black Student Achievement Plan, 2021, p.1; Ladson-Billings, 2006). A-G
completion, California’s required courses for high school college entry, sits among the top
of the 16- success metrics identified by the district. BSAP intends to remove barriers to
postsecondary completion by championing "high academic performance, social-emotional
awareness and management, and a positive cultural identity" for Black students. (Black
Student Achievement Plan – Board Amended, 2021).
In January 2022, the school board approved an evaluation proposal submitted by USC’s
Center for Education, Identity, and Social Justice and the RAND Corporation. The core
objective of the BSAP evaluation is to examine implementation of BSAP across school sites
to determine the extent to which the BSAP levers improve higher education for career success
among Black students. We designed this research to provide actionable formative feedback
to inform operations of and implementation changes to the ongoing plan. Our evaluation is
driven by the following research questions:
1. How does BSAP improve or hinder Black students’ academic success, particularly
considering access to diverse representation, high-quality curricula, and culturally-
relevant teaching?
2. How does BSAP improve or hinder students’ access to a supportive school climate and
mental and social-emotional supports?
3. How does BSAP improve levels of engagement between parents/guardians and schools,
Black students’ engagement with extracurricular activities, and the presence of
community organizations on campus?
2. Methodology
2.1. Logic Model

The logic model highlights three key levers of inputs: (1) curriculum and instruction, (2)
school climate and wellness, and (3) community partnerships. Within each of these levers,
we outline more detailed supports designed to palliate the systemic barriers to the pursuit of
postsecondary certificates and degrees. Within our research team, we used the model to
inform development of our interview and focus group protocols and to indicate which forms
of secondary data we would need to collect in order to track trends in outcomes over time.
1216
Our evaluation strategies and research tools are sensitive to how identities such as race,
ethnicity, gender, sexual orientation, religion, and disability can highlight systemic issues,
inform solutions that create equity, and foster shared values and democratic ideals.
Figure 1. BSAP logic model
2.2. Case Study Method

While we do not outline the full design of the BSAP evaluation plan in this short paper, we
explain methods as conducted thus far. We collected data from different groups of high
schools for the two primary sides of the evaluation–implementation and impact. We initially
selected 15 schools from the 53 Group 1 schools for more in-depth qualitative data collection;
we are calling these 15 our case study schools. Case studies utilize a wide range of data types,
and allow us to narrow in on how features of an intervention might operate differently in
different school environments (Yin, 2018; p. 3). Case study methodology is particularly
useful for evaluating educational program implementation, as it allows us to obtain rich data
that capture teachers’ and students’ perspectives on the processes of teaching and learning.
Such data can allow for unexpected insights to emerge, and inform future refinement of the
program.
1217
2.3. Case Study Site Selection

The research/evaluation team selected case study schools based on various factors, and in
consultation with the BSAP team. For the proposes of this short paper, we analyzed data from
5 high schools from our 15-school sample to resemble the larger group of 53 Group 1 BSAP
schools. We used criteria including: racial/ethnic composition of student body, percent of
students qualifying for free and reduced-price lunch, graduation rates and percent of students
fulfilling A-G requirements, and school model (e.g., traditional, magnet, charter). We
matched these criteria across our case study schools and the full sample of 53 BSAP schools,
aiming for a sample that would closely mirror the larger group of schools.

The 15 case study schools are receiving a greater degree of qualitative attention than the
remainder of the BSAP Group 1 sites. In September and October of 2022, our research team
visited each of the case study school campuses for a full day, during which time we conducted
multiple interviews, focus groups, observations, and gathered documents. Through these
various data collection methods, our team spoke to principals, ACIs, BSAP staff, teachers,
students, parents, and community organizations. Approximately 54% of the adult participants
identified as African American/Black, 10% identified as Latinx/Hispanic, 2% identified as
Mexican American/Chicano, 3% identified as Other Hispanic, 2% identified as Asian, and
13% identified as White. Nearly 71% of the adult participants identified as women, 28% as
men, and 1% marked Other. All participants have also been assigned a pseudonym to protect
their identity. All interviews and focus groups were audio recorded and transcribed. The table
below presents all the data collection tasks completed at the sample school sites during Fall
2022.
3. Findings
3.1. Lever 1: Curriculum and Instruction

Implementation Strengths
3.2. Diverse representation in learning materials

One dimension of BSAP Lever 1, as illustrated in Figure 1, is to build greater diversity of
representation in curricula that Black students are exposed to. Across 11/15 sites, adult or
student participants shared that there had been a noticeable increase in culturally relevant
books, pedagogy, and educational experiences. Students and parents especially observed the
presence of historical Black figures in hallways and classrooms. The reading areas were often
filled with books that featured Black protagonists and included a diversity of plot lines that
students might relate to. A father of an elementary school child noted that his children were
1218
responding well to seeing themselves in the lessons, “They can see themselves in the lesson,
they can see themselves on the board, they can see themselves in the books to actually help
them and encourage them to actually read and be involved in the classroom.”
Like other parents at the sites, this father found that the reading material in the classrooms
and posters in the hallways had more of a positive impact on the cultural identity of his
children than the negative stereotypes they were used to seeing in the media.
One high school principal echoed the importance of providing Black students with positive,
diverse representation in their reading material. Describing his approach to aligning his
school’s curriculum with the goals of BSAP, he said:
One of my goals right now is to ensure that every English class…has a culturally relevant
novel that every grade level will read…So, what they do is they read the book with a
facilitator from [community-based organization], who is our partner, and they go over to their
[center], which is right next to our campus, and they perform either a little skit or a vignette
or a monologue. So they'll actually act out a scene that really resonated with them.
By bringing in a facilitator from a community-based organization and giving students an
opportunity to perform a monologue, this principal is providing a wide range of ways students
can engage and make sense of diverse representation in their reading material.
A notable strength of BSAP is its initiative to increase the diverse representation in books,
posters, and curriculum that students are exposed to. When the curriculum is implemented,
it can significantly correlate with students’ sense of belonging on campus and their academic
engagement in the classroom. Moreover, other cultures stand to learn and benefit from
curricula that discusses Black experiences and history. Another elementary teacher described
the enriching nature of culturally relevant curriculum for all students when he said, “It not
only impacted my African-American students, but also some of my other students, my
Hispanic students…I had one of the Hispanic students who was like, my favorite thing was
learning about all the different African-American people this year.”
Critical opportunities for improvement
3.3. Adopt culturally relevant pedagogy in classrooms

In the previous section, we unpacked diverse representation and culturally relevant
curriculum as a strength. In the 11/15 instances where sites adapted and taught with the new,
Black-centered reading material, it certainly is. However, we found that across 4/15 schools,
there still is a lot of progress to be made toward establishing compliance with BSAP’s
mission to incorporate culturally resonating and sustaining pedagogy in all schools. For
instance, multiple teachers at one school felt that the initial rollout of BSAP was rushed and
1219
did not allow teachers to incorporate culturally relevant units in an impactful way. A science
teacher described her frustration with adding cultural relevance to her lesson, saying:
Even though the idea of the unit plan was to integrate it deeply, because of the timing, the
roll out and the people, it became something tacked on top. And so as a science teacher, the
example I always use is, it's like the Black astronaut, the Black astronaut lesson where you're
like, "Okay, be culturally relevant. Here's a Black astronaut. They did this. Isn't that cool?"
Move on to the next thing. And that is not an authentic way of empowering our students to
feel like they belong in an underrepresented field to understand the history…
This teacher expressed discontent with taking a previously constructed lesson and
incorporating race as an afterthought. She would rather make the lesson relevant to students’
environment in a meaningful way than spotlight the race of the subject or author as a garnish
and hope that the students will relate. Teachers shared that the rollout simply did not permit
them enough time to produce a curriculum that would be culturally relevant and impactful.
One parent of an elementary student shared that she felt that a lack of Black-centered
instruction was derived from non-Black teachers’ preferences to center their own respective
histories. When asked if she believed the curriculum was relevant to her child, she responded:
No, absolutely not. No sir…But sometimes I think that the teachers, they still kind of
intertwine to tweak whatever that they want to teach their own kind. So it's basically like the
black kids are going to get left out because now you're over here tweaking the curriculum for
the other children, but not tweaking it for the black children because they can teach about
they [sic] heritage.
This parent believed that teachers might have been asked to teach a certain curriculum, but
they may have refused to do so because they were reluctant to prioritize the stories or needs
of Black students. As a result, Black children’s history remained tangential to many lessons.
3.4. Culturally resonant math education

Math education is another critical improvement opportunity. A high school counselor
presented site-specific data that spoke to the need for improvement in math education for
Black students when she stated, “Last year 23% of our students were either proficient or
above in math. That means that 77% were not…only 7% of our Black students met that goal,
and zero of them exceeded the standard.” She conveyed disbelief that only 7% of Black
students were sufficient in math, and suggested providing professional development that was
more focused on students’ varying learning styles.
In the achievement plan, a local district states that they are committed to providing students
with learning opportunities that “validate and affirm who they are” honoring the “cultural
heritages, languages, and histories” of its students and their families. The district also lists
1220
“dismantling racism in mathematics” as a part of its instructional foci. We found, though,

most case study schools (8/15) have not started or are resistant to BSAP’s mission to make
math curricula more culturally resonant for Black students. One elementary principal shared,
“typically the math problems…may not reflect some of the cultures that we have
represented.” Concerning the implementation of culturally resonant math, a high school
principal stated “I think it was a lot to ask… [teachers] were having a hard time fitting in
where that made sense.” Further, another principal echoed the sentiment of time being a
limitation when he shared, “I think it's still a work in progress. It's been a challenge to find
time during the school day to do it.”
Increasing proficiency by way of ensuring equity in mathematics instruction is listed as a key
initiative in the district’s achievement plan. We found that several schools encountered
limitations when attempting to adopt the curriculum. Therefore, we assert that supporting
teachers and schools to make math instruction more culturally resonant is a critical
improvcment opportunity.
3.5. Lever 2: School Climate and Wellness

Implementation Strengths
3.6. BSAP staff as support system and advocates for Black students
Across all 15 case study sites, there was evidence that BSAP staff played a vital role in Black
students’ lives, learning, and experience in school. BSAP staff also demonstrated care and
familiarity with students and their lives outside school, allowing them to advocate for
students in ways other educators could not. BSAP staff were often like parental figures to
students and were very aware of life circumstances that could impact students’ experience in
school.
Students at one middle school described their relationship with one BSAP staff member—
we will call him Mr. Nathan Blue1—whom they spoke with daily. One of the students said
he was his “brother, uncle, nephew.” Another student responded, “Mr. Nathan Blue is my
dad.” A third clarified, saying, “He's just like a father figure.” One student reassured, “That's
my dad. You know, when I call him my dad, he calls me his son. Why? Because me and him
have a great, strong relationship and we always share things with each other.” Mr. Nathan
Blue was not related to any of the students at this school but had built such a strong
relationship with students that he became like family to them.
Parents at an elementary school felt similarly about Ms. Williams, one of the BSAP staff
members at their campus. Parents described Ms. Williams as “the school” and “part of the
1
Any names used are pseudonyms, selected by participants.
1221
village” because she knew all the students and everything about the school to support students
and their families. Parents reassured us that she had a high level of care for students and that
they knew if their children were with Ms. Williams, they would be comfortable, safe, and
happy. Ms. Williams cared so deeply about students that she fostered children that attended
the school, and they referred to her as their other mom.
Critical Improvement Opportunities
3.7. Language that minimizes advocacy for racial equity

Some ACIs, principals, teachers, and BSAP staff utilized language that communicated that
BSAP was a program for only Black students, or a student club, alienating students from the
broader campus community. The use of this language further reinforced the idea that BSAP
could go away after a period of implementation, and may have contributed to a lack of
investment by some teachers. One high school ACI expressed frustration at how schools were
adopting BSAP, saying:
One of the biggest disdains I have is schools and folks identifying students as BSAP students.
I hate it. And I feel like some of us ACI's are guilty of it because we almost made it like a
club, like it's a frat, like it's a sorority…We have a shirt that says BSAP. But they like wearing
it like all the BSAP kids got the BSAP T-shirt. So it's like a club. So now I'm finding and this
is my big disdain. Do not call these students BSAP students. Don't do it. We don't call them
GATE-ers and we don't call them ELL-ers and we don't call them SPED-ers because they are
being supported by a plan. So why are we call them BSAP students? We should not be
identifying students, calling them out or categorizing them by race, language acquisition or
level of intellect or ability.
Addressing students by the plan that supports them rather than as students, further minimizes
BSAP’s advocacy for racial equity and fosters and environment that otherizes students. This
is equivalent to defining students with disabilities by their disability, calling them disabled
students. Such that when teachers, school staff, and leaders reference students only within
the context of their disability, not only is that a reductionist view of who they are and a slight
or insult, but also it minimizes how and why an institution should respond to the full humanity
of these students. In this case, an educational institution is responsible for addressing more
than a student's disability. A parallel construction exists for ESL (English as a Second
Language) students, who are now referred to as ELL (English language learners). We suggest
referring to students with BSAP support might be a more useful reference. Additionally,
using such language indicated that these students were BSAP’s responsibility, and may have
decreasing some staff members’ sense of responsibility in supporting them.
1222
3.8. Districtwide and schoolwide hiring and staffing challenges

Non-BSAP leaders and staff at 9/15 schools struggled to fully understand the roles of BSAP
staff members and how to best leverage their support. The lack of clarity at times created
tension between BSAP staff members and other staff members at the school sites. To
complicate matters further, several schools had unfilled BSAP positions and incomplete
teams to carry out the plan with limited support.
Although the plan includes several school-level roles with specific goals and duties at Group
1 schools—i.e., Administrative Coordinator of Instruction, Pupil Services & Attendance,
Psychiatric Social Worker, and Restorative Justice Teacher—once on site, BSAP staff found
themselves performing additional job duties that were not part of the original plan. Several
BSAP staff members shared that principals and other school leaders often asked them to
perform unrelated tasks and duties. BSAP staff members obliged because they cared about
Black students’ well-being, even when these tasks were beyond the scope of their jobs. Two
BSAP staff members at a high school explained how difficult it was to get time for planning
programs and initiatives when administrators consistently called them to perform other
duties. One explained their challenges, saying:
We get pulled in two different directions. I think a lot of that is definitely tied
to the sense that overall, it's not like they would look at us as working on “just
BSAP”, like this is what they do. It's more so like you're [a restorative justice
teacher], you're basically the “mini dean”. You're school climate, you're
campus aid. We need you to go get these kids from these classes and bring
them over to the mini dean. And that's not really what we're supposed to be
doing.
BSAP staff members at other campuses also felt like the disciplinarians at their schools; some
wore yellow “campus aide” vests and walked through the hallways making sure students got
to class on time.
The lack of understanding of the plan and BSAP staff members’ roles and responsibilities
prevented many of them from providing better support for Black students. Additionally, it
created additional work for ACIs to support their BSAP staff members and communicate
with principals and school leadership regarding how they should utilize their BSAP staff
members at their schools. Tina, the ACI at Promise Elementary, explained:
I spent more time this year pouring into my team, making sure my teams knew what their
roles were, making sure they understood our 16 indicators, our success indicators, and how
those relate to their jobs. And how you know, every day, this is what this is your focus, you
know, as it relates to the black students and with the principals as well, making sure that they
understand that their PSA isn't just a PSA.
1223
Similarly, BSAP staff at several sites mentioned that school leadership often did not
recognize that some staff members were specifically designated for BSAP and not for the
rest of the school community. Several BSAP staff members performed duties far beyond their
scope, so ACIs needed to continually refocus them to ensure that the plan's goals were
understood and met.
4. Conclusion and Recommendations

While the above findings are preliminary, and will continue to develop over the course of our
research, they do point to some specific and actionable recommendations for LAUSD
leadership, school leaders, and school-level BSAP staff members.
We provide these recommendations not only to evaluate the program and make suggestions
that will better serve Black students, but also to bring awareness to the legal liability
associated with a racially hostile environment. Above all other recommendations below, we
want to highlight a critical first step for school and district leaders: to better understand what
constitutes a racially hostile educational environment and how it may open the individual,
the school, as well as the district to legal liability. We included this early in the report to
demonstrate its urgency. School leaders would benefit from assessing ways that their own
schools may have hostile environments for Black students, and working with other staff
members to address these concerns.
References
Ladson-Billings, G. (2006). From the achievement gap to the education debt: Understanding
achievement in US schools. Educational researcher, 35(7), 3-12.
Sue, D. W., Capodilupo, C. M., Torino, G. C., Bucceri, J. M., Holder, A., Nadal, K. L., &
Esquilin, M. (2007). Racial microaggressions in everyday life: implications for clinical
practice. American psychologist, 62(4), 271.
Yin, R. K. (2018). Case Study Research and Applications: Design and Methods (6th ed.).
Thousand Oaks, CA: Sage.
1224
Reflective practice ePortfolios: a digital teaching tool to enhance

third year BA culinary and gastronomic science students’
professional learning experiences
Clare Gilsenan1, Marie English2
1
Galway International Hotel School, Atlantic Technological University, Galway, Ireland.
2
School of Business, Atlantic Technological University, Galway, Ireland.
Abstract
The aim of this research was to explore the use of reflective practice eportfolios
as a digital tool to enhance BA Culinary and Gastronomic Science students’
professional learning experiences. A deductive approach was applied to the
data where the participants reflective practice entries were coded using Braun
and Clarke’s framework for thematic analysis. In line with Vygotsky’s (1978)
social constructivist theory, the results depicted the participants were able to
learn through social interactions with their lecturer and peers, who assisted
with scaffolding knowledge and skills, until the participants were more
confident in their learning. By using the reflective practice eportfolio meant
that the participants were able to track their improvement and growth related
to their reflection process. The results are far reaching in the context of
culinary arts education and offer significant insights into embedding a
reflective practice eportfolio into the culinary curriculum.
Keywords: Social constructivism; reflective practice; eportfolios; culinary
arts education; professionalism.

Reflective practice ePortfolios to enhance culinary students’ professional learning experiences
1. Introduction
Professionalism is critically important for the culinary industry, however the difficulty with
attracting and retaining highly professional personnel makes it problematic to implement.
Higher education is expected to prepare graduates for the professional workforce, yet Ryan
(2018) argues that higher education institutions are failing to adequately prepare graduates
for employment, particularly, in professional (soft) skill development. In a study by
Lazíková, Takáĉ, Rumanovská, Michalička, and Palko (2022) a significant gap between the
skills that graduates acquire and those that employers require was reported.
Reflective practice has been defined as “a process for the learner to integrate the
understanding gained in one’s experience in order to enable the better choices or actions in
the future” (Rogers, 2001, p.41). Heymann et al. (2022) insists that central to becoming a
professional is the ability to be able to reflect. Embedding reflective practice into culinary
arts may enable the student to think about and reflect upon their studies, identity, and futures,
thus integrating theory into practice, to the point that students recognise the theoretical
constructs underpinning professional development in a culinary arts context. The Reflective
Cycle by Gibbs (1988) has widely influenced the broad field of education. Gibb’s model
presents six stages of reflection, from the description of the problem to action planning. It
presents a framework for examining experiences. Adie and Tangen (2015) assert such
professional skills can be enhanced using self-reflection, and that reflective practice is
essential in higher education for effectively preparing graduates for their future professional
lives.
According to Abrami and Barrett (2015) the utilisation of reflective practice aligns to social
constructivism. Social constructivism is a learning theory developed by educational
psychologist Lev Vygotsky. Social constructivism theory contends that a student’s cognitive
ability is gained through collaborative social guidance and construction. One of the core
constructs of Vygotsky’s theory of social constructivism is the zone of proximal development
(ZPD). Vygotsky (1978, p.86) defined ZPD as “the distance between the actual
developmental level as determined by independent problem solving and the level of potential
development as determined through problem solving under adult guidance or in collaboration
with more capable peers”. In line with Vygotsky’s (1978) notion of providing instruction
within the ZPD, it was expected that scaffolding would decrease over time, as the students’
understanding increased. This research focuses on culinary arts education from a holistic
perspective, meaning education is not only a process of transmission, but that students learn
knowledge and skills through different pedagogical methods in a social setting. Language
also plays a significant role in learning development and how individuals view the world.
This means that learning is transmitted via language and interpreted by experiences in a social
setting as argued by Vygotsky (1978). Part of the preparation for a career in the food product
development sector involves an understanding of sensory analysis. Specific language and
1226
concepts such as olfactory and gustatory senses, chemesthesis, discrimination and hedonic
sensory testing are of critical importance in the lexicon of terminology needed by sensory
professionals working in the food product development sector. Without an understanding of
such a lexicon, the subject has no value or application for the students. Furthermore,
Vygotsky (1986, p.148) stated “Scientific concepts develop earlier than spontaneous
concepts because they benefit from the systematicity of instruction and cooperation”. This
suggests that learning is best accomplished through co-operative instruction between the
lecturer and more experienced peers. Thus, assisting a student’s movement through the ZPD
from novice to expert with the use of tools such as language-based tools (i.e., ePortfolio) and
physical tools such as Compusense.
An eportfolio is a digital tool that has been successfully used to promote employability skills
and self-regulated learning in Ireland (Farrell, Buckley, Donaldson, and Farrelly, 2021). A
portfolio has been defined “as a collection of documents that provide tangible evidence of
the wide range of performances, essential knowledge, critical dispositions and experiences
that you possess as a growing professional” (Campbell, Cignetti, Melenyzer, Nettles and
Wyman, 2014, p.3). By using a reflective practice eportfolio, the student is afforded the
opportunity to document their thoughts and actions, whilst reflecting on the link between
what they learn in the classroom and what they do in practice. Through interactions with the
lecturer and the students’ peers, the student acquires specific knowledge and skills.
The aim of this research was to explore the usefulness of a reflective practice eportfolio as a
digital tool to enhance BA Culinary and Gastronomic Science students’ professional learning
experiences. This study also focused on supporting the student in reflective practice, so that
the theory they learned in their lectures would be reinforced through the direct application of
practice.
2. Methodology
Social constructivism was deemed to be the most suitable research philosophy as the research
question and aim aligned with the key principles of the social constructivist paradigm. The
choice of this research philosophy allowed for the use of data collection methods that
facilitated the achievement of the research objectives to explore perspectives of third year
BA Culinary and Gastronomic Science students’ professional learning experiences of using
a reflective practice eportfolio. A case study methodology was chosen as the methodological
framework for the study. Atlantic Technological University (ATU), in Galway, Ireland was
chosen as the research site and third year BA Culinary and Gastronomic Science students,
studying the module Introduction to Sensory and Consumer Science were selected as the
research participants for this research. The module was chosen as it is used to develop the
learners’ knowledge of, and ability to use, sensory science and consumer, preparing the
1227
students for internship in the food product development sector. Eleven BA Culinary and
Gastronomic Science students volunteered to take part in the study.
The participants in the study were required to undertake a team sensory analysis report and a
reflective diary, of which the outputs were displayed in their eportfolio. Participants
submitted three reflections in their eportfolio as specified in the module. The data collection
methods for this case study included analysis of the participants’ reflective practice entries
in their eportfolios using Braun and Clarke’s (2022) framework for thematic analysis. This
approach allowed the researcher to interpret the participants’ learning experiences through
the data. Ethical issues in terms of privacy and general data protection regulations (GDPR)
were acknowledged. To protect the identities of the eleven participants who had volunteered
for the study, pseudo-names were used to identify the participants. In addition, power
implications that could have existed between the researcher and the participants were
considered.

This research paper presents selected findings of the thematic analysis of the participants’
reflective eportfolio entries. For this research paper, only one of the core themes will be
discussed, namely Development of Awareness of Self.
The subthemes that emerged from the core theme of Development of Awareness of Self are
depicted in Table 1.
The results highlight eighteen percent of participants in this study mention in their reflective
practice eportfolio having learned from their past experiences. Those participants indicate
that they were able to reflect on their past experiences of working in teams. They were able
to critically appraise what they had experienced, and this suggests that they were able to
improve their experience this time around. As asserted by Vygotsky “The child’s memory
not only makes fragments of the past more available, but also results in a new method of
uniting the elements of past experience with the present” (Vygotsky, 1978, p.36).
1228
Table 1. Core theme, sub-themes, and key quotations from participants on their learning
experience of the usefulness of a reflective practice eportfolio (n=11)
Core Theme Sub-Theme Key Quote
Development Learning from Past “In the past, I have been in groups where there was
of Experiences disunity…working with Emma, the work was always equally
Awareness divided. In addition to this, we communicated frequently
of Self during the week (twice minimum) to track our progress and
to ensure that any difficulties were resolved quickly.” (Ella)
“I have fallen short in the past in terms of third level
education.” (Elijah)
Acknowledging “I felt confused when it came to writing the research report
Struggles as our course had been so practical heavy for the first two
years.” (Noah)
“I did feel better and confident using Pebble-pad once I
contacted my lecturer and they helped me out further.”
(Emma).
Celebrating “I now have a greater understanding of how to design a
Successes digital questionnaire, and move data seamlessly through the
Microsoft suite, which in turn improved my digital
competences. Additionally, this lent to my problem-solving
abilities.” (Ella)
“The collaborative work this semester helped me work in a
team with another student and allowed a different
perspective into my work and helped me think differently.”
(Levi)
Taking “I felt I had to take a leadership role to ensure the project got
Responsibility for done in time and everything was correct.” (Charlotte)
Learning
Motivation for the “It is now a new skill set that I look forward to utilising in
Future the future.” (Emma)
Regarding the theme Acknowledging Struggles, a little under a third of the participants (27%)
said they struggled with the academic writing that was expected from them and twenty-seven
percent of participants acknowledge having difficulty using the eportfolio software
Pebblepad. Culinary arts students often find the theoretical aspects of the curriculum
challenging. This can be problematic as students, when on internships, may be working in
product development, so knowledge of terms and processes are required. By using
Vygotsky’s theory of social constructivism, the lecturer-researcher developed a lexicon of
terms appropriate to sensory analysis. The lecturer-researcher instructed and assisted the
student so that their movement through the ZPD from novice to expert was supported. It was
evident over time that the participants had internalised this new sensory language, which
became their internal speech. Vygotsky (1978) highlights the importance of speech for one’s
cognitive development and states “Sometimes speech becomes of such vital importance that,
1229
if not permitted to use it, young children cannot accomplish the given task” (Vygotsky, 1978,
p. 26).
Nighty-one percent of participants celebrated the successful collaborative relationships they
developed with their team, learning new digital skills, specifically relating to the eportfolio
digital tool, independently planning, delivering, and analysing the results of sensory testing.
This is in line with Vygotsky’s social constructivist approach. The participants demonstrated
in-depth knowledge of their market sector and showed they could critically reflect on sensory
concepts and theories, thus improving their ability to communicate which would prove
important for internships in the product development sector.
By acknowledging the need for self-improvement, the participants were taking responsibility
for their own learning and their professional development. Six participants (55%) in this
study also provided examples of them taking responsibility for their own and others learning
throughout the study. This reflects their moving from novice to expert learner as articulated
in Vygotsky’s metaphor of the ZPD. Using reflective practice eportfolios means that the
participants would now be able to track their improvement and growth related to their
reflection process. This is critically important for the participants of this study to continue
using their reflective practice eportfolio as it will improve their learning but may also enable
them as graduates to progress quickly in employment in the future. It could be argued that by
using the reflective practice eportfolio, the participants were better able to link theory and
practice effectively. The participants were able to learn through social interactions with their
lecturer and their peers, who assisted with scaffolding the knowledge and skills, until the
participants were more confident in their learning. This is consistent with the social
constructivist theory. Finally, the findings from the research revealed sixty-four percent of
participants believe that self-reflection enabled them to learn from experience and apply that
knowledge to future experiences such as being able to effectively design, deliver and analysis
the results of consumer sensory panels. Participants communicated how these skills would
benefit them in the future. Moreover, the findings of this research concur with Farrell et al.
(2021) that it is reasonable to suggest that reflective practice eportfolios should be viewed as
an important teaching and learning digital tool in higher education in Ireland.
4. Limitations
This was the BA in Culinary and Gastronomic Science 3rd year students’ first engagement
with reflective practice eportfolios and, although the sample size was small, it reflected the
class group. It was not the intention to generalise the results to the wider culinary student
population, rather to trial a new type of assessment, with a view to embedding it in future
years.
1230
5. Conclusions
The findings of this research showed how the participants learned to use language and skills
gained from social interactions for sensory analysis through co-operative instruction between
the lecturer and peers. The participants’ movement through the ZPD from novice to expert
was also reflected in the results. By using a reflective practice eportfolio, the participants are
afforded the opportunity to document their thoughts and actions, whilst reflecting on the link
between what they learn in the classroom and what they do in practice.
6. Implications of this Research, from a Pedagogical Perspective

The implications of this research indicate that the application of reflective practice eportfolios
in a culinary arts context may assist students to construct meaning from their theoretical,
practical, and individual experiences in a social setting. The learning experience provided by
internship for the student is highly significant, hence, the language and skills used in the
classroom needs to be mirrored in the food product development industry. By taking a social
constructivist approach, the culinary arts lecturer and more experienced peers will help
scaffold the knowledge and skills for the student so that their movement through the ZPD
from novice to expert will be supported.
References
Abrami, P., & Barrett, H. (2005). Directions for research and development on electronic
portfolios. Canadian Journal of Learning and Technology /La revue canadienne de
l’apprentissage et de la technologie, 31(3). https://doi.org/10.21432/T2RK5K
Adie, L. & Tangen, D. (2015). “The use of multimodal technologies to enhance reflective
writing in teacher education,” in Teaching Reflective Learning in Higher Education, ed.
M. E. Ryan (New York, NY: Springer), 127–138. doi: 10.1007/978-3-319-09271-3_9
Braun, V. & Clarke, V. (2022). Thematic analysis: A practical guide. SAGE.
Campbell, D., Cignetti, P., Melenyzer, B., Nettles, D. & Wyman, R., Jr. (2010). How to
develop a professional portfolio: A manual for teachers (5th ed.). Pearson.
Farrell, O., Buckley, K., Donaldson, D. & Farrelly, T. (2021). Eportfolio in Ireland: A
landscape snapshot of the current practice. Irish Journal of Technology Enhanced
Learning, 6(1) https://doras.dcu.ie/26539/1/99-Article%20Text-445-1-10-20211212.pdf
Gibbs, G. (1988). Learning by doing: A guide to teaching and learning methods. Further
Education Unit Oxford Polytechnic.
Heymann, P., Bastiaens, E., Jansen, A., van Rosmalen, P. and Beausaert, S. (2022). A
conceptual model of students’ reflective practice for the development of employability
competences, supported by an online learning platform. Education and Training, 64(3),
380-397.https://www.emerald.com/insight/0040-0912.htm
1231
Lazíková, J., Takáč, I., Rumanovská, Ľ., Michalička, T., & Palko, M. (2022). Which Skills
Are the Most Absent among University Graduates in the Labour Market? Evidence from
Slovakia. Social Sciences, 11(10), 438. MDPI AG. Retrieved from
http://dx.doi.org/10.3390/socsci11100438
Rogers, R., R. (2001). Reflection in higher education: a concept analysis. Innovative Higher
Education, 26(1), 37-57.
Ryan, Ann (2018) Graduate Employability; Bridging the gap from Higher Education to
Employment. (Master’s thesis, Griffith College). http://go.griffith.ie/id/eprint/179
Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes.
Vygotsky, L. S. (1986). Thought and Language. Massachusetts: The MIT Press.
1232
Job demands and resources in the work of university teachers in

central and eastern Europe
Klára Kovács1, Beáta Dobay2, Szabolcs Halasi3, Tamás Pinczés4
1
Cenrter for Higher Education Research and Development Hungary, University of Debrecen,
Hungary, 2Department of Physical Education and Sports, J. Selye University, Slovakia,
3
Hungarian Language Teacher Training Faculty, University of Novi Sad, Serbia,
4
Department of Physical Education and Sport, Ferenc Kölcsey Teacher Training Institute,
Debrecen Reformed Theological University, Hungary.
Abstract
The situation of higher education as a workplace as well as that of the
academics has changed significantly in recent decades. Earlier, compared to
working in other fields, teaching at a university was considered to be a
predictable, calm, relatively stress-free, flexible, autonomous, and socially
recognized profession. Teachers and colleagues were protected from many
sources of workplace stress, such as uncertainty, work-related ambiguity, or
low job control, but today this is no longer the typical case (Kinman, 2014).
The expectations (or needs) related to these roles and activities put strong
pressure on the teachers, they are serious sources of stress that have a negative
impact on their health, well-being (KIinman et al., 2006; Kinman, 2014).
Based on the job demands-resources theory (JD-R theory) this paper
investigates the workplace factors affecting the health and well-being of
Central and Eastern European (CEE) university teachers.
Keywords: Job demands and resources; university teachers; central and
eastern Europe.

Job Demands and Resources in the Work of University Teachers in Central and Eastern Europe
1. Introduction
The situation of higher education as a workplace as well as that of the instructors has changed
significantly in recent decades. Earlier, compared to working in other fields, teaching at a
university was considered to be a predictable, calm, relatively stress-free, flexible,
autonomous, and socially recognized profession. Instructors and colleagues were protected
from many sources of workplace stress, such as uncertainty, work-related ambiguity, or low
job control, but today this is no longer the typical case (Kinman, 2014). The workplace
characteristics that used to protect the university workforce from certain sources of
workplace-related stress seem to have disappeared by now, following the expansion of
university education (Malik et al., 2017). As a result of the expansion, diversification of
higher education, and the shift in the direction of being regarded as a service provider,
university teachers have to perform multiple tasks and roles at the same time: lecturers,
researchers, mentors, and sometimes they even have to fulfill the function of pastors (Bell et
al. 2012; Kinman, 2014;). The expectations (or needs) related to these roles and activities put
strong pressure on the university treachers, they are serious sources of stress that have a
negative impact on their health, well-being (Kinman et al., 2006; Kinman, 2014).
In our study, we examine the workplace factors affecting the health and well-being of East-
Central European university teachers. Our main question is what difficulties the teachers of
the region have to deal with during their work, and what are the resources that contribute to
their well-being. We used the job demands-resources theory (JD-R theory) as a theoretical
background to examine this issue. The JD-R theory is a theoretical framework that helps to
explain and understand the relationship between workplace characteristics and employee
performance and well-being. It classifies the workplace characteristics into two categories
that are negatively correlated: (negative) work requirements and work-related resources
which have a direct impact on the employee’s stress level, motivation, health problems, and
numerous organizational outcomes. Workplace resources are the characteristics of the
workplace that are necessary to achieve the work goals set, contribute to work enjoyment,
motivation, and engagement (Bakker and Demerouti, 2014). Based on the JD-R theory we
also examined the job demands and the workplace researches among university teachers in
CEE. Earlier research primarily took a psychological approach and examined psychological
stress sources and resources, and showed their correlation with regard to work performance,
workplace and psychological well-being, or even burnout. But in our exploratory research,
we examine difficulties and resources related to the work of the instructors from a higher
education research perspective and using a qualitative method (focus group interviews).
Previous empirical works that utilized this theory applied the correlations of stress and
resources primarily for different aspects of well-being; as the first step of our study, we would
like to identify these factors based on the experiences of the teachers in the region under
investigation. The geographical area (Central and Eastern Europe) can be considered a
1234
novelty, as it is a region that contains several countries that have similar cultural roots and
historical antecedents and are in a similar social-economic situation but at the same time,
they have many peculiarities. The applied method is suitable for getting to know the
characteristics and challenges of teaching work in the countries and institutions examined in
a close-up, valid way, over a broad basis, based on the opinions and experiences of those
affected, laying the foundation for further quantitative research that reveal correlations.
A portion of the sources of stress are related to workplace requirements (long working hours,
administrative burdens, providing academic and mentoring support, carrying out and
fulfilling tasks required for quality assurance, pressure for grants and publications, managing
a large number of e-mails, etc.), another portion to the factors limiting efficiency (ineffective
management, the lack of administrative and technical support, poor communication, rushed
work pace, frequent interruption of work, conflicting roles and limited opportunities to
prepare for teaching, research, and professional development), and other problems (such as
lack of respect, harassment, interpersonal conflicts, or job insecurity). The high workload
also significantly affects the health of the instructors, which is related to and can be
manifested in symptoms of depression, aggressiveness, impatience, rejection, and
procrastination (Iyaji et al., 2020).
1.1. Health-behavior, physical activity among the instructors

Several studies draw attention to the unhealthy lifestyle of university teachers, employees
(inactivity, unhealthy diet, high stress, etc.) which also leads to loss of life among them in
many cases (Iyaji et al., 2020). Still, only a few studies deal with the physical activity of the
academics, the problem of inactivity, while compared to the average population, they are
more likely to be familiar with physical activity’s importance and its beneficial effects on
various dimensions of health, especially those who work and do research on this field. But
this knowledge is not always followed by the activity (Kwiecień-Jaguś et al., 2021), while
physical ativity plays an essential positive role in health promotion and wellbeing (Cooper &
Barton, 2016; Lotan et al., 2005). Practicing organized physical activity contributes to the
treatment and prevention of mental illnesses and disorders, and increases the level of physical
and mental well-being among both the mentally ill and the general population (Fox, 1999).
Research conducted at a South African university came up with devastating results about the
health status of the teachers: the majority of the respondents are overweight or obese,
generally, the colleagues are not healthy, a great portion of them feel that they are under a lot
of pressure, so they are exposed to a lot of stress at work, though they are looking for
opportunities that would support their health and well-being. 63% of women and 48% of men
did less than the recommended 150 minutes of moderate-intensity exercise, so they can be
characterized as having an inactive lifestyle. Negative behavioral patterns can also be
observed among them when it comes to nutrition: low fruit and vegetable consumption and
high consumption of sugary soft drinks. Overall, the respondents feel that they are
1235
undervalued by their colleagues, they are undermotivated at work and they are dissatisfied
with the management, the opportunities, and the wellness initiatives at their faculty (Koen et
al., 2018).
In a comparative study, one fifth of Australian and only 18% of British academics
participated in at least 150 minutes of moderate-intensity physical activity, and a similar
proportion did at least 90 minutes of more vigorous exercise. The vast majority of them
(77.4% of the Australians, 88.2% of the British) did not participate in the health-oriented
programs and events organized by their university. In both countries, approximately 10%
indicated that their institution does not even provide such opportunities. At the same time,
we are well aware that an organization has to provide its employees with access to wellness
programs that encourage them to participate. Behavioral changes brought about as a result of
these programs can be useful for the development of institutional culture, reduce absenteeism,
medical expenses, and contribute to the well-being of the employees. (Koen et al., 2018).
Among Cameroonian academics, a positive correlation was found between inactive lifestyle
and burnout, while the opposite, i.e. a physically active lifestyle, especially sports, function
as a protective factor against numerous diseases (cardiovascular diseases, diabetes, obesity,
cancer, metabolic syndrome, stress, depression, aging, dementia, etc.). (Ngalagou et al.,
2019). In a Nigerian study, it was found that although physical activities and recreation are
positively correlated with quality of life, unfortunately, university teachers do not make use
of the sports facilities and infrastructure provided by their institution, the majority of
respondents spend their free time with passive activities, mostly due to work requirements
(Omolawon et al., 2011).
2. Methodology
In our research, we used a qualitative method, focus group interviews, to reveal the workplace
stress and resources affecting the well-being and health of Central and Eastern European
university teachers. For the analyses, seven focus group interviews were conducted with
academics from nine higher education institutions in five countries, online between July and
October 2022. The length of the interview material is about eight hours, which was
transcribed into a total of 168 pages of written text. The interview outline broadly examined
the work and situation of the academics along the following dimensions: introduction,
dimensions of teaching effectiveness, challenges of higher education pedagogy, the impact
of teaching work on well-being and health, institutional culture, leisure time and cultural
consumption, experiences and attitudes towards disabled students, religiosity, language and
ethnic diversity in higher education, the role of the institution in preserving and developing
the health of the university teachers.
1236
The number of participants in the focus group interviews was between four and eight from
the following institutions (41 people in total): University of Debrecen (UD), Reformed
Theological University of Debrecen (RTUD), University of Nyíregyháza (UNY) (Hungary);
Sapientia Hungarian University of Transylvania (branch Miercurea Ciuc, SAPI, Romania),
Partium Christian University (PCU, Romania); Ferenc Rákóczi II. Transcarpathian
Hungarian Institute (THU, Ukraine), J. Selye University (JSU, Slovakia), University of Novi
Sad (UNS), Subotica Tech - College of Applied Sciences (STCAS) (Serbia). The Hungarian
universities are institutions from the North-Eastern region of the country, where the
proportion of disadvantaged students is overrepresented compared to other regions, so the
teachers have to deal with special challenges. From the other countries, we have picked the
higher education institutions of the Hungarian minority, so the teachers and students are also
in a special situation, due to being members of a minority. The interviewees were selected in
such a way as to obtain heterogeneous groups according to gender, age, and position, as well
as field of study (faculty). In terms of position, there were participants from all levels, from
PhD students who are part-time teachers to full professors, several interviewees also holding
senior positions (department head, institute head, doctoral program president, dean, assistant
dean, assistant rector). Based on the age known, the youngest respondent was 31 and the
oldest was 65. 18 men and 23 women participated in the interviews.
During the analysis, we used deductive and inductive coding: we examined the difficulties
and resources related to work in connection with the JD-R theory (deductive), and we looked
for other sources of stress and supporting factors inductively. Based on the answers, we
performed a type analysis to typify the difficulties and resources, and a thematic analysis was
performed regarding the coping strategies used to overcome stress, sports and other activities
related to health behaviors, institutional contributions, and expectations. The codes are: 1.
work-related difficulty, stress, and 2. resources; 3. individual protective factor, coping
strategy; 4. institutional supporting factor and 5. expectations, that could promote the well-
being of academics.

Similarly to research conducted among university teachers in Britain, Australia (Kinman et
al. 2006; Bell et al., 2012; Fetherston et al., 2021), Pakistan (Aziz and Quraishi, 2017), Saudi
Arabia (Alqarni, 2021), China (Han et al., 2020) and Nigeria (Iyaji et al., 2020), the most
important workplace difficulties, challenges, and resources are related to teaching roles,
interpersonal relationships, support by the management, and infrastructural conditions.
Difficulties that came up with regards to teaching include that it is undervalued compared to
research performance, even though it takes a lot of work and energy to prepare, be up to date
in the course material, to organize the classes, and pass on practical, but always professional
knowledge. However, student feedback is not always adequate in this regard, especially
1237
during the time of COVID, when there was no personal contact, so university teachers got
feedback neither from the students, nor from the institution in general about how well they
were able to deliver the study material to the students. Hungarian institutions of higher
education do have a mostly voluntary feedback system, as well as a completely informal
website (markmyprofessor), where the work of a given academic can be evaluated
anonymously, but these are not necessarily objective, as students are influenced by the
difficulty and importance of the course material, and of course the grade received. This way
the measurement of the quality of education itself can also be problematic. They deal with
the workplace stress by playing sports, physical activity and quality, active time spent with
the family in order to preserve their health and well-being. At the same time, in accordance
with earlier studies (Kinman, 2014; Koen et al., 2018; Iyaji et al., 2020), the vast majority of
subjects do not play sports or perform regular physical activity (only 13 people do).
Significant differences can be observed between the institutions (countries) examined;
among the subjects in Ukraine, no one mentioned physical activity or sports, while when it
comes of gender, there was an equal proportion of active teachers.
The most important message of our study is that the institutional environment is key to both
the effective work and well-being of the instructors. This requires well-functioning
technology and suitable infrastructure (PCE), financial renumeration independent of external
grants for existential security. Meanwhile, it is even more important to define predictable,
transparent institutional goals, strategy, and jointly agreed values, to create a predictable and
controlled system of evaluation and rewards, which do not change according to the whims of
the current leadership or with the change of leadership (in Serbia and Romania). For this
reason, constant and effective communication between the leadership and the smaller
subunits is very important. Although a stable, predictable system of evaluation already exists
in several countries (Romania, Ukraine), and compliance with it is another source of stress,
but it has only been introduced into the operation of the institutions examined in Hungary
during the recent changes in higher education. Thus, its effectiveness is not yet known, but
at the same time it promotes predictability and transparency, which go hand in hand with the
differentiation of wages. This could result in a kind of competitive situation within the
teaching community, which can negatively affect interpersonal relationships. Among the
resources related to work, a good community or team within the faculty or with colleagues
from other institutes stands out, so it is extremely important for the management of the
institution to support these common, grassroots teambuilding programs. In addition,
institutions can contribute to the well-being and health of the instructors primarily by offering
free or at least discounted participation in sports, cultural and leisure events, as well as mental
health counseling, but it is important for these not to be self-serving (the colleagues from the
university should not participate in the programs to make up for the missing audiences of the
otherwise overfunded organizations of the institution) and haphazard: without a targeted
health strategy, these are not sufficiently effective. These team-building informal programs
1238
were also greatly affected by COVID: they were either forced back into the online space, or
they were completely stopped, or they are being revived currently, so in this case, too there
appears the need to support grassroots initiatives (department excursions, playing football,
theater, etc.).
Acknowledgement
This publication was supported by the project “Investigating the role of sport and physical
activity for a healthy and safe society in the individual and social sustainability of work
ability and quality of work and life (multidisciplinary research umbrella program).
References
Alqarni, N. A. (2021). Well-being and the Perception of Stress among EFL University
Teachers in Saudi Arabia. Journal of Language & Education Volume, 7(3), 8–22.
Aziz, F., & Quraishi, U. (2017). Empowerment and mental health of university teachers: A
case from Pakistan. Pakistan Journal of Social and Clinical Psychology, 15(2), 23–28.
Bakker, A. B., & Demerouti, E. (2014). Job demands–resources theory. In Chen, P. Y & C.
L.Cooper (Eds.), Work and Wellbeing: Wellbeing: A complete reference guide (pp. 1–
28). John Wiley & Sons, Ltd.
https://www.isonderhouden.nl/doc/pdf/arnoldbakker/articles/articles_arnold_bakker_34
4.pdf
Bell, A. S., Rajendran, D., & Theiler, S. (2012). Job stress, wellbeing, work-life balance and
work-life conflict among Australian academics. E-Journal of Applied Psychology, 8(1),
25–37.
Cooper, K., & Barton, G. C. (2016). An exploration of physical activity and wellbeing in
university employees. Perspectives in Public Health, 136(3), 152-160.
Fetherston, C., Fetherston, A., Batt, S., Sully, M., & Wei, R. (2021). Wellbeing and work-
life merge in Australian and UK academics. Studies in Higher Education, 46(12), 2774–
2788. https://doi.org/10.1080/03075079.2020.1828326
Fox, K. R. (1999). Influence of Physical Activity on Mental Well-being. Public Health
Nutrition, 2(3), 411-418.
Han, J., Yin, H., Wang, J., & Zhang, J. (2020). Job demands and resources as antecedents of
university teachers’ exhaustion, engagement and job satisfaction. Educational
Psychology, 40(3), 318–335.
Iyaji, T. O., Eyam, S. O., & Agashi, V. A. (2020). An Assessment of Some Stress Factors in
Academic Profession and Their Health Implications among Academics in Tertiary
Institutions in Cross River State Nigeria. International Journal of Innovative Research
and Advanced Studies, 7(12), 53–59.
Kinman, G. (2014). Doing More with Less? Work and Wellbeing in Academics.
Somatechnics, 4(2), 219–235. https://doi.org/10.3366/soma.2014.0129
1239
Kinman, G., Jones, F., & Kinman, R. (2006). The Well-being of the UK Academy, 1998–
2004. Quality in Higher Education, 12(1), 15–27.
Koen, N., Philips, L., Potgieter, S., Smit, Y., van, N. E., Nel, D. G., & Visser, J. (2018). Staff
and student health and wellness at the Faculty of Medicine and Health Sciences,
Stellenbosch University: Current status and needs assessment. South African Family
Practice, 60(3), 84–90. https://doi.org/10.1080/20786190.2017.1396788
Kwiecień-Jaguś, K., Mędrzycka-Dąbrowska, W., Kopeć, M., Piotrkowska, R., Czyż-
Szypenbejl, K., Hansdorfer-Korzon, R., ... & Jarzynkowski, P. (2021). Level and factors
associated with physical activity among university teacher: an exploratory analysis. BMC
Sports Science, Medicine and Rehabilitation, 13(1), 1-12.
Lotan, M., Merrick, J., & Carmeli, E. (2005). A review of physical activity and well-
being. International Journal of Adolescent Medicine and Health, 17(1), 23-32.
Malik, N. A. A., Björkqvist, K., & Österman, K. (2017). Factors associated with occupational
stress among university teachers in Pakistan and Finland. Journal of Educational, Health
and Community Psychology, 6(2), 1–14.
Ngalagou, P. M., Assomo-Ndemba, P. B., Manga, L. O., Ebolo, H. O., Ayina, C. A., Tanga,
M.-Y. L., Guessogo, W. R., Ndongo, J. M., Temfemo, A., & Mandengue, S. H. (2019).
Burnout syndrome and associated factors among university teaching staff in Cameroon:
Effect of the practice of sport and physical activities and leisures. L’Encéphale, 45, 101–
106.
Omolawon, K. O., Ibraheem, T. O., & Omolawon, K. O. (2011). Social Factors Predicting
Recreational Sports Participation Among Academic Staff Of Tertiary Institutions In
Kwara And Kogi States, Nigeria. International Journal of Sport Management, Recreation
& Tourism, 7, 30-43.
1240
From training practice in the professional world to university:

characterization of Vocational Training students to retain them in
university studies
Ana Munárriz, Yeray Rodríguez, María Isabel Goicoechea, María Jesús Campión
Department of Statistics, Computer Science and Mathematics, Public University of Navarre.
Abstract
This paper analyses the Vocational Training students who enroll at the School
of Economics and Business Administration of the Public University of Navarre.
For this purpose, a descriptive study of this admission profile has been carried
out. In addition, their educational strengths and weaknesses have been studied,
as well as whether accessing a Degree after having studied vocational training
affects their performance and dropout.
According to the results, students mostly choose the Bachelor's Degree in
Business Administration and the Bachelor's Degree in Economics, with a
better performance in the subjects that are similar to those previously studied
in Vocational Training. At the same time, they show deficiencies in the
quantitative area, being the access route a determining factor in the result.
Keywords: Mathematics; economics; business; vocational training.

From training practice in the professional world to university: characterization of VET students
1. Introduction
The Lisbon Strategy (European Council, 2000), revised in 2004, was set with the intention
of generating Europe as the most competitive and dynamic knowledge-based economy
environment in the world, based on a sustainable growth model. To do so, modernisation had
to be achieved through active investment in education and training systems; i.e., educational
programmes had to be designed to ensure the employability, productivity and income-
earnings of the target groups. Subsequently, the European Credit System for Vocational
Education and Training (Official Journal of the European Union, 2009) was regulated,
formalising the recognition and accumulation of learning outcomes in order to generate a
qualification and, therefore, the inclusion of skills in the labour market.
The promotion of Vocational Education and Training (VET) means that, in the coming years,
a greater number of Higher VET graduates will be able to access University through this type
of non-university Higher Education. For this reason, facilitating the transition between VET
and University is one of the goals on the agendas of educational institutions.
From an institutional perspective, efforts are currently being focused on the development of
a regulatory framework that promotes the recognition of credits between both programs.
Thus, the Public University of Navarre (UPNA) and, in particular, the School of Economics
and Business Administration (FCEE), already have agreements binding both Higher VET
programs. While this is an excellent measure to avoid duplicities and shorten the time in
which students who have previously studied VET obtain a university Degree, it is necessary
to complement these measures with others that facilitate the process of integration of the VET
student into the university environment. (García et al., 2017; Panera et al., 2004).
To achieve this goal, it is necessary to evaluate what type of initiatives can be implemented
by universities. The design of specific tools to strengthen the weaknesses of this less frequent
student profile can help these students on starting university with a similar level of knowledge
as their classmates who have completed their Baccalaureate studies. In this way, they will be
able to achieve the potential student profile set out in the White Paper on Undergraduate
Degrees in Economics and Business (2005). Educational innovation projects promoted by
universities and other institutions can be decisive in achieving this goal.
In order to know where these kinds of measures should be considered, it is necessary to know
which is the profile of students who enroll in university after completing these previous
studies and which are their strengths and weaknesses regarding the Degree they are studying.
This is precisely the objective of this work: analyze the profile of the students of the FCEE
of the UPNA who access through VET. With this objective, the aim is to answer four
fundamental questions: What is the profile of FCEE students who enroll in UPNA through
VET? Which is their performance in the subjects throughout the Degree? Is admission
1242
through VET a determining factor in passing the subjects of the first semester? Does having
enrolled in the university through VET influence the dropout rate of students?
These questions will be answered through descriptive statistics and econometric relationships
applied to the data provided by the Integrated University Information System for this purpose.
2. Methodology
Firstly, a quantitative analysis of the data provided by the Integrated University Information
System (SIIU) and the internal data collected by the UPNA on academic performance per
subject and Degree is done, on a sample of 3662 students.
For this purpose, there is one database with general information on students who have studied
Bachelor's Degrees at the FCEE and another database on student performance in the subjects
of the offered Degrees. Both databases have information from the academic year 2010/11 to
2019/20. For data analysis, R-Commander, Gretl and Excel were used.
2.1. Descriptive analysis of student profile

Univariate descriptive statistics are used to analyze individual variables such as gender, age
or vocational group. In addition, frequencies and percentages of the students that enter
university through the VET route of each Degree are calculated and it is determined if this
type of student is distributed homogeneously among those Degrees or if it is concentrated
particularly in some of them.
2.2. Performance analysis by subject

The analysis of performance by subject; that is to say, the pass rate, the success rate and the
no-show rate are evaluated for the Bachelor's Degree in Business Administration (BBA) and
the Bachelor's Degree in Economics (BEcon).
Subjects with success rates lower than 60%, pass rates lower than 50% or no-show rates
higher than 35% will be reviewed. In addition, those subjects in which students coming from
VET obtain better success rates than those coming from Baccalaureate are also reviewed.
2.3. Analysis of the relationship between VET access way and the passing of subjects.
The five subjects evaluated are those corresponding to the first semester of the Bachelor's
Degree, since they represent the first challenge of their university academic career and which
will determine the continuation of their studies. An econometric model with a limited
dependent variable is proposed to determine the probability of failing each subject:
1
𝑃𝑟 (𝑌! = 1) =
1 + 𝑒 "($!%$"×'()%$#×*+,-./0%$$×123%$%×45567789/:%$&×;+-"<9==3>(*)?)
1243
In this model, 𝑃𝑟 (𝑌! = 1) appears as the dependent variable, which is interpreted as the
probability of failing the subject 𝑌! (one model per subject), and the regressors are dummy
variables ("VET" and "Country") and continuous qualitative variables ("Age", "Access
Mark" and "NP ECTS"). VET is the variable interest and the remaining regressors are control
variables that have a significant impact in the model, so their exclusion could invalidate the
results. The model will be estimated by Maximum Likelihood.
2.4. Analysis of VET access way as a determining factor in dropout.

The following econometric model with a limited dependent variable is used to explain
students’ dropout from the BBA and the BEcon, according to the access route:
1
𝑃𝑟 (𝐷𝑟𝑜𝑝𝑜𝑢𝑡 = 1) =
1+ 𝑒 "($!%$"×'()%$#A;+-"<9==3>(*)?)
In this case, 𝑃𝑟 (𝐷𝑟𝑜𝑝𝑜𝑢𝑡 = 1), is interpreted as the probability of dropping out from the
studies by students while “VET” is the variable interest and “NP ECTS” is the control one of
the model. The model is also estimated by Maximum Likelihood.
3. Results
3.1. Descriptive analysis of student profile

Since 2010/11 and up to 2019/20 academic year, 422 people have accessed the School
through a VET Degree. This is the second most frequent route, after the Baccalaureate (3097
people). About 60% of these students are women and the average age at which they access
the studies is 22 years old, above the Baccalaureate students, who access between 18 and 19
years old. This profile is mostly enrolled in the BBA and the BEcon, with 79.62% and 18.48%
of the total number of VET students, respectively. In contrast, this profile is quite infrequent
in double Degrees and international programs. This may be explained, among other factors,
by the fact that the admission grade for both Degrees is lower than either in double Degrees
or in international programs.
The Vocational Groups with the highest frequency among VET students are Administration
and Management (167 students) and Commerce and Marketing (61 students). These fields
are the most similar to the Degree studies offered by the FCEE and those with the highest
number of credits recognized between both Higher VET programs.
A total of 160 students, out of the 422 who entered the FCEE through VET access, have
completed their Degree and 146 have dropped out, which represents a 34% of the students
who are admitted by this route. This rate is significantly high and ten percentage points
greater than the dropout rate for Baccalaureate students, who have a 24%. Regarding the data
1244
by Degree, the dropout rate among VET students is 31.55% for Business Administration and
44% for Economics.
Among VET students, the average time to get the title is 4.77 years, which is similar to the
average time of students with a Baccalaureate Degree. However, it is common for VET
students to have ECTS credits recognised at the beginning of their studies, so it can be
concluded that VET students find it more difficult to complete their Degree. The rates of
graduates among VET students are better in the case of Business Administration than in
Economics, with a percentage of 41.37% and 24.36% of graduates, respectively. In both
cases, the rates are lower than those obtained by students who enter with a Baccalaureate
Degree, whose graduation rates are 44.54% and 37%, respectively.
3.2. Performance analysis by subject

Graduates coming from VET show a notably good performance in accounting subjects in
both Degrees, with 9 percentage points higher pass rates than those coming from
Baccalaureate. The same results are obtained in the case of the subject of Financial
Operations for the BBA, although the difference is more tenuous, around 5 percentage points.
However, the performance of those whose access is through VET is clearly lower in the
subjects of Mathematics in the BBA and Mathematics I in the BEcon, with lower pass rates
than those coming from the Baccalaureate (-14% in both cases), as well as higher no-show
rates (about 13% in both cases).
Particularly, in the BEcon, negative results are transferred to subjects such as Introduction to
Economics, Microeconomics, Mathematics II and Finance, because of the mathematical
background they have.
3.3. Analysis of the relationship between VET access way and the passing of subjects.
From the five models proposed for each grade in Section 2.3, Table 1 shows only those in
which the VET variable is statistically significant at a significance level of 5%. As it is a logit
model, the slope column must be studied, due to the fact that it shows the marginal effects
when the rest of the variables take their mean value.
Table 1. Results of the analysis of the subjects.
Vocational Training Access Students

Degree Coef. Signif. (5%) Slope R2 McF
Mathematics I BEcon 1 True 0.287 0.589

Mathematics BBA 0,630 True 0.131 0.596
Financial Operations BBA -0,787 True -0.155 0.559
1245
In the BBA case, the analysed subjects are financial Operations and Mathematics, where the
signs of the coefficients are as expected: in the case of Financial Operations, the sign is
negative, which implies that having studied VET previously increases the probability of
passing, a result consistent with reality, due to the training context of the VET Degrees from
which the students come. Specifically, coming from VET makes the student 15.45% more
likely to pass this subject than a student who enters university through other studies, keeping
the rest of the variables at their averages.
In the case of Mathematics, the effect of accessing university through VET has the opposite
effect. The sign of the coefficient is positive, so that, in accordance with the slope, entering
university through VET increases the probability of failing the subject by 13.07%, if the
remaining variables of the model keep at their mean values.
The same effect happens in the subject of Mathematics in the BEcon; i.e., the sign of the
coefficient is positive, since accessing university through VET causes an increase on the
slope of 28.75% of failing this subject, compared to those who come from another
educational pathway, with all remaining variables constant.
3.4. Analysis of VET access way as a determining factor in dropout.

In the BBA, the coefficients of the two regressors of the Section 2.4 model are statistically
significant at a significance level of 5%. Therefore, accessing university through VET studies
influences the probability of the student dropping out; in this sense, given that the sign of the
coefficient of the variable "VET" is positive, access through VET increases the probability
of dropping out by 4%, keeping the rest of the variables constant.
For the BEcon, the variable VET is not statistically significant at 5% nor 10%, so it cannot
be said that it is different from 0. This indicates that the VET access way has no effect on the
probability of dropping out.
4. Discussion
Firstly, according to the descriptive analysis in Section 3.1 of the students’ profile, at least
one out of ten students who entered the FCEE between 2010/11 and 19/20 did so with a VET
qualification. Their minority position in mainly Baccalaureate formed groups means that their
potential and shortcomings often go unnoticed. The analysis of the provided information
builds an image of the students who access the UPNA's FCEE through this pathway, as well
as identifies their educational strengths and weaknesses. In this way, the areas of knowledge
in which actions can be carried out so that VET students can integrate into the university in
an appropriate way and develop the syllabus satisfactorily have been discovered. This student
profile essentially opts for the BBA, followed, distantly, by the BEcon.
1246
In relation to their performance, these students present good results in subjects close to the
modules taken in the Higher VET programs of the Vocational Groups of Administration and
Management and Commerce and Marketing, the most common Groups of those who enter
the Faculty. As can be seen in Section 3.2, they stand out in accounting subjects, presenting
better results than their counterparts in the Baccalaureate. In this sense, similar results are
obtained to those presented by García et al. (2017) in another branch of knowledge: students
who come from Higher VET programs obtain good grades in specific subjects related to
technical Degrees and lower grades in theoretical subjects.
In this line, the Section 3.2 shows that VET students have greater difficulty in subjects related
to the quantitative area, in which a high level of mathematical skills is necessary to pass them
satisfactorily. Specifically, Mathematics subjects in the BBA and in the BEcon are those with
the lowest pass rates and high failure rates. Comparing the data with those obtained from
students who access the Degree through the Baccalaureate, it is shown that, although the
results of the second one are not good either, they are substantially better, which is in line
with the literature (Fernández et al. 2001, Gil et al. 2014, Pérez et al. 2013).
This deficit may be because some of the VET students have not taken Mathematics subjects
since the 4th year of Secondary Education, either because they entered the following year to
an Intermediate VET program or because they did not choose Mathematics in the
Baccalaureate (Gil et al. 2014). Such shortcomings may condition performance in other
subjects, as pointed out in the White Paper on the Bachelor's Degree in Economics and
Business (2005). This is evident among VET students studying Economics, who have greater
difficulties in subjects such as Introduction to Economics or Microeconomics, where
functions and other mathematical tools are constantly used to build models.
The same conclusions are obtained in Section 3.3 by studying the influence of the entry route
on the probability of passing or failing a given subject. Coming from VET makes the
probability of not passing the Mathematics subject significantly higher than coming from the
Baccalaureate access way. This difference may be due to the fact that in Economics the
admission marks are historically lower and, in addition, it has fewer similarities than Business
Administration with Higher VET programs, so students who have enrolled in this Degree
have done so with lower marks and, probably, it has not been the preferred option.
For this reason, it is considered that the main point of improvement should be to put on the
same level the mathematical competences as those of their counterparts who entered by the
Baccalaureate pathway, thus, covering the training deficit prior to the start of the Bachelor's
Degree and, in this way, avoiding the deficiencies in the rest of the subjects.
For that purpose, an educational innovation project has been developed with the aim of
creating a virtual platform to cover individualized training needs through gamification. The
project Digital Gamification to address pre-university mathematical training deficiencies in
1247
students of Economics and Business Degrees of the UPNA will be promoted during the
following two years. The improvement that will be entailed by the implementation of the
virtual platform may help to reduce the drop-out rate of these profiles, evidenced in Section
3.4. This is particularly of paramount importance in a context where greater accessibility to
Higher VET programs and the expansion of their training offer will lead to an increase in the
critical mass of students with Higher VET Degrees who enter university through this route.
Finally, note that the study was carried out at a single university and, therefore, the sample
size in some cases did not provide conclusive results. There are also two future lines of
research, one related to the replicability of the study in other universities and the other to the
analysis of the impact of the virtual platform as a way of alleviating detected deficiencies.
References
Agencia Nacional de Evaluación de la Calidad y Acreditación, ANECA. (2005). Libro
blanco: Título de Grado en Economía y en Empresa [White Paper: Degree in Economics
and Business Studies]. Madrid.
Council of the European Union. (2000). The Lisbon European Council: Presidency
Conclusions, 23rd-24th March 2000.
Fernández, P., Gómez D., Masero, I., Zapata, A. (2001). Reflexiones sobre la adecuación de
los conocimientos previos de los alumnos al temario de matemáticas de la Escuela de
Empresariales [Thoughts on the adequacy of students' prior knowledge for the
mathematics syllabus of the Business School]. Revista Electrónica de Comunicaciones y
Trabajos de ASEPUMA, 9, 1-13.
García, A. Lías, A. I., Martínez, M. (2017). Influencia de la Formación Previa en el Proceso
de Integración en la Universidad [The Influence of Prior Learning on the University
Integration Process]. Revista Iberoamericana de Tecnología en Educación y Educación
en Tecnología, 19, 41-49.
Gil, M. C., Carrillo-Fernández, M., Cruz D. I., González-Concepción C., Pestano-Gabino C,
Sosa, D. (2014). Análisis de la formación matemática preuniversitaria en el periodo 2001-
2012 y consecuencias en los estudios de Economía y Empresa: El caso de la Universidad
de La Laguna [Analysis of pre-university mathematics education in the period 2001-2012
and its consequences for economics and business studies: The case of the University of
La Laguna.]. Revista Internacional de Estudios Organizacionales, 3(1), 59-82.
Official Journal of the European Union. (2009). Recommendation of the European
Parliament and of the Council on the establishment of a European Credit System for
Vocational Educaation and Training (ECVET), C(155), 11.
Panera, F. Davalillo, A. Lorente, E. (2004). El proceso formativo inicial como un proceso de
adaptación del estudiante a la universidad. Factores explicativos del rendimiento
académico en el primer año de estancia en la universidad [The initial training process as
a process of adaptation of the student to the university. Factors explaining academic
performance in the first year at university]. Pedagogía universitaria, hacia un espacio de
1248
aprendizaje compartido: III Simposio Iberoamericano de Docencia Universitaria.

Universidad de Deusto, 1, 495-508.
Pérez, F., Gómez, T. Caballero, R. (2013) Evolución de la adaptación de los alumnos del
Grado de Economía en la asignatura de Matemáticas a través de sus éxitos [Evolution of
the adaptation of undergraduate students of Economics in the subject of Mathematics
through their successes]. Anales de ASEPUMA, 21 .
1249
An examination of significant factors influencing college student

employment cognition
Qiaoguan Ye1, Ningjie Wu1, Sandy Chen2,Yinggui Qiu1, Hongjie Zhang1, Xuan Zhang1
1
School management, Hubei University of Chinese Medicine,China,2College of Education,
Ohio Uninversity, United States.
Abstract
The increasingly severe employment problem of college students has become
a public concern in China. The literature review suggested that college
students’ employment cognition play a major role in this issue. To improve
students’ employment cognition thus becomes important. The paper argued
that the first step in the improvement of students’ employment cognition be
knowing the key factors that might influence one’s employment cognition.
Through literature review, this paper identified three key influential factors,
i.e., professional training, academic satisfaction, and word-of-mouth.
Empirical data were then collected to test their relationships with employment
cognition. SEM analysis revealed that these factors were significantly and
positively related to employment cognition. These findings provide Chinese
educators and potential employers with insights on how to help college
graduates broaden their job search scope and match their majors with ideal
jobs. In a broad sense, these findings could significantly contribute to the
wellbeing of the Chinese society.
Keywords: Academic satisfaction; employment cognition; professional
training; unemployment.

An examination of significant factors influencing college student employment cognition
1. Introduction
With the expansion of colleges and universities in China in the recent years, the number of
students who successfully complete their higher education has greatly increased. This is good
news for the Chinese society on one way. On the other hand, it has simultaneously brought
some “negative issues” to the attention of some educators and researchers.
One of these is high pressure on the newly graduated students since competition for job
opportunities seems to have been mounted and many of them struggle to find jobs. The high
competition in the job market is argued to also come from the economic situation in China.
Reuter (Polland, 2022) revealed that China’s economy has been weakened by the Covid-10
Pandemic and the youth unemployment, as of June 2022, is already more than three times
China's overall joblessness rate, at a record 18.4%.
Another “negative” observation of the job market is the mismatch between the college
student major and actual employment. Anecdotes and news reports show that many graduates
have entered professions unrelated to their majors. To seek job security, many graduates aim
for or accept governmental positions, regardless of their majors (Polland, 2022).
It looks like that the “golden employment age” for college graduates in China has been over.
However, the recent call by President Xi Jinping on the nation's youth to seek out new
opportunities in “the west of the country” reveals deeper insights into the “college graduate
unemployment” issue (Shan, Zhang, & Hu, 2022). Relevant literature reveals that most
college graduates desire for positions in large companies located in cities and governmental
agencies, particularly those large cities located in relatively developed east regions. It also
casts doubts on Chinese college students’ overall employment cognition, which greatly
influences college students’ career choices (Lent, et al, 2000; Gushue, et al, 2006). Chinese
students’ employment cognition seems to be low. Studies show that many students are not
clear about their future career paths; they do not have their own opinions, but blindly follow
the advice of their parents, friends, or strangers (e.g., Glewwe, et al, 2011; Munir, et al. 2022).
This paper supports the Chinese government’s viewpoint regarding the “real cause” of
“college graduate unemployment” and “job mismatch” issues and argues that improvement
of student employment cognition be an effective solution to these issues. To get started, this
study aims to investigate key factors that might significantly influence college student
employment cognition.
2. Conceptual Model
A survey of the relevant literature reveals that college student employment cognition might
be influenced by multiple factors, including professional training, academic satisfaction, and
word of mouth (WOM), and others. This study focuses on testing and reporting the direct
1252
Qiaoguan Ye, Ningjie Wu, Sandy Chen,Yingggui Qiu, Hongjie Zhang, Xuan Zhang
relationship of student employment cognition (as a dependent variable) and three

independent variables, professional training, academic satisfaction, and WOM, as figure 1
shows. Although the authors did investigate the interrelated relationships among the
independent variable, this paper only reports the direct relationships as shown in the
framework. Figure 1 serves as a guide for literature review and data collection.
Figure 1. The Proposed Theoretical Conceptual Framework.
2.1. Professional Training and Employment Cognition

College student professional training is found to influence student employment cognition by
a number of research studies. For instance, Monteiro, et al. (2020) found that career advice
services provided by colleges and universities were positively correlated with student
employment cognition. Álvarez-González, et al. (2017) contended that skill development,
academic performance, personal environment and contacts, and cognition of the labor market
increase students’ self-confidence, which is positively reflected in how they perceive their
employment cognitive ability. This view is supported by Tymon, et al. (2020).Through a
sample of employed and unemployed job seekers, Zikic, et al. (2009) found that people who
spent more time exploring careers and participating in more training programs reported
higher job search clarity and self-efficacy.
In addition, mentors play an important role in professional training and development. Using
a web-based descriptive survey design, Adeniran R.K et al. (2013) detected significant
differences in the mentoring role and certain professional development and career
advancement measures between colleges and universities, and concluded that mentors are
role models, mentoring relationships are important for professional growth, and standardized
1253
professional development structures are needed to facilitate professional growth. Based on

the evidence in the literature, the following hypothesis is thus proposed:
H1: Professional training has a positive impact on Chinese college student employment
cognition.
2.2. Satisfaction of College Students

A number of studies show that students’ academic satisfaction seems to has positive effects
on student employment cognition. Yun, et al. (2019) surveyed more than 120 Chinese college
students and identified significant relationships between academic studies, and student
satisfaction and employability. Liu, et al (2021) found that compared students who know
their employment status to those who do not know their employment status, the latter have
lower academic satisfaction. An empirical study done by So, et al (2014) indicated that
student academic satisfaction has positive effects on goal selection, job information and
future plan, which is helpful for college students to establish correct employment cognition.
Thus, this study proposes the following hypothesis:
H2: College students' satisfaction has an impact on their employment cognition.
2.3. Word of Mouth

Huete-Alcocer (2017) found after reviewing relevant literature that word-of-mouth is the
oldest medium for sharing opinions on products or services, and it is also the medium most
likely to influence consumer behavior because of the high reliability and credibility through
family and friends' transmission. Yim, et al. (2010) found that customers' online and offline
word-of-mouth had a statistically significant impact on their evaluation of a restaurant's brand
equity. Their results also show that customers' evaluation of the three dimensions of brand
equity has a significant impact on their return intention, and the online and offline word-of-
mouth of stable brand equity for family restaurants. Similar to the situation that consumers
face when choosing which products to buy, college graduates are also influenced by the
information spread by the surrounding people. College graduates will pay attention to the
information passed on by others, analyzed and judge the quality of their employment situation
according to the information obtained, and then make their own employment choice. This
study thus proposes the following hypothesis:
H3: Word of mouth plays a role in establishing correct employment cognition.
3. Methodology
Research publications that systematically investigate the key factors that significantly
influence Chinese student employment cognition are very limited. Therefore, this study
collects its own data through a survey to test the proposed hypotheses. This questionnaire is
1254
divided into two parts, the first part is the basic characteristics of the respondents, the second
part is to measure the variables of the conceptual model, and the variables are measured
through questions derived from the literature. The dependent variable of the questionnaire is
employment cognition, the independent variable is college students' academic satisfaction,
professional training, and WOM. We used a set of 5-point Likert scale to state the questions,
with “1” = “strongly disagree” and “5” = “strongly agree”.
Before the formal survey, this paper first conducted a pilot study and used Cronbach α
coefficient and factor analysis to test the reliability and validity of the questionnaire structure,
which was found to be good. Through principal component analysis, it is found that the
individual questions of college students' satisfaction and employment cognition in the rotated
component matrix have unclear attribution, so the questionnaire is modified and improved
for these questions. In the pilot study, a total of 102 questionnaires were distributed, and 97
valid questionnaires were screened out, with an effective rate of 95.1%. Formal
questionnaires were distributed to about 350 college students.
3.1. Participants
An online survey was conducted among Chinese college students across the country. The
basic information of demographic characteristics included gender, age, education level, type
of school, living area,, future employment, and whether they would choose a job in line with
their major. The questionnaire was released on April 24, 2022, and closed on August 6.
A total of 358 questionnaires were collected, and the invalid questionnaires such as too short
time and illogic were screened out. A total of 307 valid questionnaires were retained, with an
effective rate of 85.76%. In this formal survey, 68.70% respondents were female and 31.3%
male. Most of the participants were between 18 and 22 years old, accounting for 82.08%.
3.2. Data Analysis Procedure

The data analysis was carried out in four steps, similar to the previous research. First,
descriptive statistical analysis was carried out to calculate variable mean value, standard
deviation and correlations with the help of SPSS26.0 statistical tool. Second, Reliability and
validity test was tested with the help of SPSS26.0 and AMOS23.0 statistical tools. Reliability
analysis is measured by Cronbach α. Third, Structural Equation Model (SEM) analysis was
conducted. Measurement error control and model fitness test were included to predict the
influence of professional training on establishing correct employment cognition. AMOS23.0
statistical tool was used in this part. The was followed a test by using the Bootstrap Method
to see if college students' satisfaction has any effect in the process of establishing correct
employment cognition. These analyses identified many significant relationships among
independent variables, which will be reported in a follow up study.
1255
4. Results
4.1. Preliminary Analysis

The absolute values of skewness (<2) and kurtosis (<4) of all observed variables were normal
distribution. The mean value, standard deviation and mutual coefficient matrices of the
variables are shown in Table 1.
Table 1. Basic statistics of variables.
Variables MD SD PT AS EC WOM
PT 3.65 .66 —
AS 3.92 .57 .568 * * —
EC 4.01 .56 .512 * * .602 * * —

WOM 3.70 .57 .475 * * .447 * * .568 * * —
Note: PT = professional training; AS = academic satisfaction; EC = employment cognition; WOM = word-of -
mouth.
Table 1 shows that the overall employment cognition of Chinese college students is above
the average, but not super. Their professional training is only 3.65, the lowest score among
all the variables examined. This indicates that word of mouth communication and academic
satisfaction do have greater influence on student employment cognition than their
professional training. In addition, the results show that all the variables are significantly
correlated, which confirmed the literature information.
4.2. SEM Analysis

The identification index of the model (see Table 2) and the identification model were
obtained as follows. Table 2 shows that the index fit is very good, indicating that the model
has a high fitting degree. The path coefficients of the structural equation model are shown in
Table 3.
It is inferred that professional training significantly but weakly influences employment
cognition (r=0.187, P< 0.05); academic satisfaction significantly and strongly influences
employment cognition (r=0.598, p<0.000). The significant and strong relationship between
professional training and academic satisfaction is apparent (r=0.658, p <0.00). This is Further
identified by Bootstrap method with 5000 sampling times, the results are shown in Table 4.
Since the direct effect of professional training on the establishment of correct employment
cognition is relatively small, but the total effects are big, it seems to indicate that college
students' satisfaction has mediating effect between PT and EC. Hierarchical moderated
regression was used to verify whether word of mouth plays a role in establishing correct
1256
employment cognition. The results confirmed that WOM has significant and positive
influence over Employment Cognition.
Table 2. Goodness-of-fit indices summary of the SEM.
Fit Indices Baseline Estimated value
c! /𝑑𝑓 < 3.00 1.712
GFI > 0.90 (good) 0.933
> 0.80 (reasonable)
AGFI > 0.90 (good) 0.909
> 0.80 (reasonable)
RMSEA < 0.1 (reasonable) 0.048
0.05 or less (good)

RMR < 0.05 (good) 0.022
< 0.1 (reasonable)
NFI > 0.90 0.935
TLI > 0.90 0.966
CFI > 0.90 0.972
PNFI > 0.50 0.787
PGFI > 0.50 0.693
Table 3. The path coefficient values of the SEM.
Unstandardized path Standardized path
Path coefficients coefficients S.E. CR. P
PT→AS 0.638 0.658 0.071 9.027 ***
PT→EC 0.165 0.187 0.067 2.458 0.014
AS→EC 0.546 0.598 0.082 6.633 ***
1257
Table 4. Total effects.
Bootstrapping
Model Point Bias-corrected Percentile 95%

effects
Pathways estimation 95%
Lower Upper Lower Upper
Total effects 0.514 0.346 0.696 0.352 0.704
PT→EC Direct effects 0.165 0.020 0.322 0.021 0.323

Indirect effects 0.349 0.201 0.536 0.201 0.536
Table 5. Effects of WOM.
Unstandardized standardized
coefficients coefficients
Model Variables T SIG.
B S.E. Beta.
Constant .981 .185 5.313 .000
1 AS .367 .046 .374 8.036 .000
WOM .427 .046 .434 9.337 .000
Constant .937 .182 5.154 .000

AS .374 .045 .382 8.338 .000
2
WOM .438 .045 .445 9.721 .000
AS*WOM -.185 .054 -.142 -3.452 .000
This test reveals that word of mouth significantly influence one’s employment cognition as
r=0.343, p < 0.000. At the same time, it significantly but negatively interacts with academic
performance, indicating that the higher of a student’s academic satisfaction, the less likely
the student uses “word of mouth” as a career information source.
5. Conclusions
The key findings of this study are that each one of the independent variables, i.e., professional
training, academic satisfaction, and word-of-mouth, has a significant and positive effect on
college students' correct employment cognition. However, academic satisfaction has the
1258
strongest impact on employment cognition, followed by word-of-mouth. It is interesting to

see that the independent variables are significantly correlated with each other. For instance,
word-of-mouth mediates the relationship between academic satisfaction and employment
cognition negatively, indicating students with higher academic satisfaction are less likely to
turn to word-of-mouth for career information. These findings provide the Chinese educators
and potential employers with insights on how to help college graduates broaden their job
search scope and match their majors with ideal jobs. Colleges and universities should
strengthen professional training, improve student academic satisfaction, and effectively
manage word-of-mouth channels. This study has two major limitations. One is the nature of
an online survey, which might automatically filter out those college students who do not go
online often. Another is that only three independent variables were tested and direct
relationship were reported in this study. The immediate future study of this research is to
identify more independent variables and examine how they are correlated with each other or
if any mediating relationships exist between them.
References
Glewwe,P., Huang, Q., & Park, A. (2011). Cognitive skills, non-cognitive skills, and the
employment and wages of young adults in rural China. Conference paper retrieved from
URL: https://ageconsearch.umn.edu/record/103407/
Gushue, G. V., & Whitson, M. L. (2006). The relationship among support, ethnic identity,
career decision self-efficacy, and outcome expectations in African American high school
students: Applying social cognitive career theory. Journal of Career Development, 33(2),
112-124. doi:10.1177/0894845306293416
Lent, R. W., Brown, S. D., & Hackett, G. (2000). Contextual supports and barriers to career
choice: A social cognitive analysis. Journal of Counseling Psychology, 47, 36-49.
doi:10.1037/0022-0167.47.1.36
Munir, H., Ramzan, S., Wang, M., Rasool, Y., Sumbal, M. S., & Iqbal, A. (2022). Combining
the social cognitive career theory, contextual factors and entrepreneurship education
programs in intention-based model: a tale of two diverse regions. Journal of
Entrepreneurship in Emerging Economies, 14(3), 392-421. doi:10.1108/JEEE-10-2020-
0367
Polland, M. (2022). Analysis: record numbers of Chinese graduates enter worst job market
in decades. Article retrieved from URL: https://www.reuters.com/world/china/record-
numbers-chinese-graduates-enter-worst-job-market-decades-2022-06-23/.
Shan, J., Zhang, W., & Hu, Y. (2022). More young people in China are seeking a career in
rural areas and local communities, bringing with them knowledge, vitality. Retrieved
from https://www.globaltimes.cn/page/202206/1268325.shtml
Yim, S. H., & Kim, T. H. (2010). The influence of customers' on/off-line word of mouth
activities on their evaluation of brand equity and revisit intention: A focus on family
restaurants. Korea Journal of Tourism Research, 34(7), 61-83.
1259
Holey moley guacamole! Understanding foreign currency

exposure
Valeria Martinez
Dolan School of Business, Fairfield University, Fairfield, CT, USA.
Abstract
International financial management refers to the investment and financing
decisions that managers of a multinational company face in the international
context that they engage in in business. Breuer and Ruiz de Vargas (2021)
highlight the high demand for practical applications in this area. The purpose
of the current work is to introduce undergraduate students to the concept of
currency risk and hedging through a case study on currency risk of the firm
Holey Moley Guacamole. Students analyze the benefits and drawbacks of
different hedging alternatives for the firm’s operating currency exposure. The
exercise here developed is geared towards the introduction of the topic of
currency risk to undergraduate students in a finance cases class, upper level
financial management class, or international financial management class. In
addition to developing this exercise and implementing it in class, student
learning is assessed.
Keywords: International financial management; operating currency exposure;
hedging strategies.

Holey moley guacamole! Understanding foreign currency exposure
1. Introduction
International financial management refers to the investment and financing decisions that
managers of a multinational company face in the international context that they engage in in
business. Breuer and Ruiz de Vargas (2021) talk about the importance of International
Financial Management as a special discipline in the broader finance field and they bring to
light the fact that this is an area of ongoing research with a high demand for practical
applications.
One of the most challenging topics for undergraduate students to grasp in international
financial management is currency risk. This topic is very important in the context of a
company’s international financial management because it has been proven that currency
exposure hedging adds value to a firm (Geyer-Klingeberg et al.; 2020). In addition, existing
finance applications or cases on this topic are dated and geared towards graduate students
(see for example: Porsche Exposed, 2004; BMW Currency Hedging, 2007; and Foreign
Exchange Hedging Strategies at General Motors: Competitive Exposures, 2005). The most
recent example is McCarthy (2016), also geared towards graduate students.
The purpose of the current work is to introduce undergraduate students to the concept of
currency exposure and hedging through a case study on currency exposure. Specifically
identifying and proposing different hedging alternatives for a firm’s operating currency
exposure and analyzing the benefits and drawbacks of each alternative. The exercise here
developed is geared towards the introduction of the topic of currency risk to undergraduate
students in a finance cases class, upper level financial management or international financial
management class. In addition to developing this exercise and implementing it in class,
student learning is assessed.
2. Currency Exposure
Most International Financial Management textbooks (see for example Madura, 2016 and Eun
and Resnick, 2021) and studies on firms’ foreign currency exposure identify three types of
currency exposure: transaction exposure, economic exposure and translation exposure.
Transaction exposure is defined as the sensitivity of a firm’s contractual cash flows on a
foreign currency to unexpected changes in those foreign currencies. This type of exposure
occurs because contracts are set at a fixed price and exchange rates change in time, causing
the value of these set contracts to increase or drop based on changes in the exchange rates.
Economic exposure refers to changes in the value of a firm based on unanticipated exchanges
in exchange rates. Changes in exchange rates can affect a firm’s operating cash flows as well
as the value of the firm’s assets and liabilities such a firm’s receivables, payables and foreign
1262
Valeria Martinez
currency loans. This is why transaction exposure is many times considered a specific form
of economic exposure.
Lastly, translation exposure refers to the potential effect that unanticipated changes in
exchange rates can have on a firm’s consolidated financial statements. Where consolidated
financial statements refer to the translation of all foreign subsidiaries financial statements
from local currencies to the parent’s home currency. This is, when a foreign subsidiary’s
assets and liabilities denominated in foreign currencies change from the perspective of a
parent company, due to unanticipated changed on exchange rates.
2.1. Operating Currency Exposure

The focus of this current exercise is on identifying and hedging operating exposure, an
extremely important part of economic exposure, and the most challenging type of currency
exposure to identify and to hedge in a firm. Operating exposure or exposure from its
operations is defined as the impact of unanticipated changes in the value of foreign currencies
on a firm’s operating cash flows. Operating exposure is much more challenging to identify
than exposure on a firm’s assets and liabilities. In addition, it cannot be readily obtained as a
line in a financial statement but must first be estimated. This is, operating exposure depends
on a firm’s competitive position in the market it which it sells it product and sources its inputs
as well as the firm’s ability to reduce the currency change impact by adjusting markets in
which it sells its product, its product mix, and its sources of inputs. In sum, the more sensitive
a firm’s costs and/or sales are to exchange rate changes the higher the degree of operating
exposure that a firm faces.
It is important to note that changes in exchange rates may not always cause operating
exposure. When changes in exchange rates are offset by the inflation differential between
two currencies, this change in the exchange rate will not cause operating exposure. In this
case, we say purchasing power parity holds or the change in the exchange rate is exactly
offset by the inflation differential.
2.2. Managing Operating Currency Exposure

A firm can use operational or financial hedging strategies to reduce or eliminate their
operating exposure. Financial hedging strategies would include using contracts such as
forwards, futures, options or swaps on foreign currencies. For example, if a firm incurs costs
in a foreign currency, as the foreign currency appreciates those costs will increase. A firm
could set up a forward contract hedge in which they sell the foreign currency forward, such
that as the foreign currency appreciates, they will lose net cash flow (NCF) but gain in their
forward contract position offsetting the NCF loss. The challenge of setting up a financial
hedge is determining the date for which to hedge as well as the amount of foreign currency
that needs to be hedged. In the case of operating exposure the challenge of identifying these
1263
two variables many times lead companies to use operational hedging strategies. In addition,
since currency exposure in a firm’s operations is ongoing, in these circumstances financial
hedges are not a one time deal such as in the case of transaction exposure, but need to be
constantly rolled over or reset to continue to hedge the company’s operating currency risk.
Operational strategies refer to strategies that help firms match assets and liabilities in the
same or correlated currencies to eliminate or reduce the firm’s operating exposure. This is,
by matching exposure of inflows and outflows in the same or correlated currencies, or
reducing that inflow and/or outflow in the foreign currency, we are effectively canceling out
or reducing the effect of the foreign currency change on our cash flows. For example, if a
firm’s source of operating exposure are sales in a foreign currency, the firm could reduce this
exposure by increasing or starting production of goods in that foreign currency such that the
expenses in the foreign currency offset the revenues in the currency thereby reduce or
eliminate the currency exposure. Our focus in this example will be on operational strategies
that help reduce or eliminate the firm’s operating currency exposure. This will become much
clearer through our example of Holey Moley Guacamole.
3. The Exercise: Holey Moley Guacamole!

The objective of this exercise is to determine in which areas of the firm’s cash flow Holey
Moley Guacamole has exposure to foreign currency risk and discuss how they can reduce
this exposure.
Assume you are the owner of Holey Moley Guacamole, a firm that produces and sells
guacamole in the U.S... Your main input are avocados, which you import from Mexico. You
have been operating for 12 years.
Your guacamole sells in major U.S. supermarkets for a price of 6.5 USD for each 8 oz.
guacamole tub. Each avocado crate containing 25 lbs. of avocado costs 800 Mexican Pesos
(MXN). Thus, your guacamole production cost is 64 MXN per guacamole tub, or
approximately the cost of 2 lbs. of avocados. You sell Holey Moley Guacamole only in the
U.S... Your current sales of guacamole are 1,000,000 tubs annually.
Besides the avocados you import from Mexico, you incur an additional 300,000 USD cost of
materials to produce the 1,000,000 tubs guacamole which include other vegetables and spices
needed to make the guacamole and its packaging, among other items. In addition, other
operating expenses include a fixed cost of 100,000 USD and a variable cost of 10% of sales.
Your depreciation expenses are 175,000 USD a year. You currently have 5,000,000 USD
worth of debt that charges a 3.25% interest annually, and your corporate tax rate is 21%.
Lastly, the elasticity of demand for Holey Moley Guacamole is minus 1.5. This is, a one
1264
Valeria Martinez
percent increase in price of Holey Moley Guacamole, decrease the quantity demanded by
1.5%.
To study the sensitivity of Holey Moley Guacamole’s net cash flow (NCF) to changes in the
value of the MXN, we look at the net cash flow assuming the MXN appreciates against the
USD from 0.5 USD/MXN to 0.7 USD/MXN, a 40% appreciation. With this information, we
construct the firm’s net cash flow under the base case scenario and develop alternative
strategies to reduce the firm’s operating currency exposure.
3.1 Identifying Operating Currency Exposure

As we can see in the base case, the net cash flow changes from 1.685 million USD to 674,675
USD when the MXN appreciates from 0.5 USD/MXN to 0.7 USD/MXN. As the peso
appreciates against the dollar, we observe net cash flow decreases. No other items except the
exchange rate between the MXN and the USD change in the base case. We can observe that
the increase in our cost of materials incurred in MXN is the source of the lower NCF.
Although we continue to produce and sell the same amount of guacamole, the appreciation
of the MXN causes an increase in the cost of production thereby reducing the NCF by 60%!
This is a clear sign of operating currency exposure. How can we reduce or eliminate this type
of currency exposure?
In the case of Holey Moley Guacamole, the firm’s operating exposure is in its production
expenses, which is a negative cash flow since it represents a cost for the firm. We will analyze
different ways in which Holey Moley can reduce or eliminate their operating exposure.
1265
Table 1. Operating Currency Exposure Reduction Scenarios
3.2.4.Pass on 50%
of Cost Increase to 3.2.5.Substitute
Consumers Via 50% of Mexican
3.2.2.Pass on 50% Higher Prices and Avocados with
3.2.1.Buy of Cost Increase to Substitute 50% of California
Avocados from Consumers Via 3.2.3.Sell Avocados with Avocados and Sell
3.2. Base Case - MXN Appreciates from California Instead Higher Guacamole Guacamole in California Guacamole in
0.5 USD/MXN TO 0.7 USD/MXN of Mexico Prices Mexico Avocados Mexico
USD/MXN $ 0.05 $ 0.07 $ 0.07 $ 0.07 $ 0.07 $ 0.07 $ 0.07
Sales
U.S. $ 6,500,000.00 $ 6,500,000.00 $ 6,500,000.00 $ 5,460,000.00 $ 6,500,000.00 $ 5,460,000.00 $ 6,500,000.00
Mexico $ 650,000 $ 650,000
Total $6,500,000 $6,500,000 $6,500,000 $5,460,000 $7,150,000 $5,460,000 $7,150,000
Cost of materials
U.S. other materials $300,000 $300,000 $300,000 $300,000 $300,000 $300,000 $300,000
U.S. avocados $1,600,000 $1,120,000 $1,760,000
Mexican avocados $ 3,200,000 $ 4,480,000 $ 2,240,000 $ 3,136,000 $ 4,928,000 $ 1,568,000 $ 2,464,000
Total $3,500,000 $4,780,000 $4,140,000 $3,436,000 $5,228,000 $2,988,000 $4,524,000
Operating expenses
U.S.: Fixed $ 100,000 $ 100,000 $ 100,000 $ 100,000 $ 100,000 $ 100,000 $ 100,000
U.S.: Variable (10% of sales)$ 650,000 $ 650,000 $ 650,000 $ 546,000 $ 715,000 $ 546,000 $ 715,000
Total $750,000 $750,000 $750,000 $646,000 $815,000 $646,000 $815,000
Depreciation $ 175,000 $ 175,000 $ 175,000 $ 175,000 $ 175,000 $ 175,000 $ 175,000
Earnings Before Interest $2,075,000.00 $795,000.00 $1,435,000.00 $1,203,000.00 $932,000.00 $1,651,000.00 $1,636,000.00
and Taxes (EBIT)
Interest expense
U.S. $ 162,500 $ 162,500 $ 162,500 $ 162,500 $ 162,500 $ 162,500 $ 162,500
Mexico $ - $ - $ - $ - $ - $ - $ -
Total $ 162,500 $ 162,500 $ 162,500 $ 162,500 $ 162,500 $ 162,500 $ 162,500
Earnings before Taxes (EBT) $ 1,912,500 $ 632,500 $ 1,272,500 $ 1,040,500 $ 769,500 $ 1,488,500 $ 1,473,500
Income Tax (21%) $ 401,625.0 $ 132,825.0 $ 267,225.0 $ 218,505.0 $ 161,595.0 $ 312,585.0 $ 309,435.0
Net Income $ 1,510,875.0 $ 499,675.0 $ 1,005,275.0 $ 821,995.0 $ 607,905.0 $ 1,175,915.0 $ 1,164,065.0
Add Back Depreciation $175,000.00 $175,000.00 $175,000.00 $175,000.00 $175,000.00 $175,000.00 $175,000.00
Net Cash Flow $ 1,685,875.00 $ 674,675.00 $ 1,180,275.00 $ 996,995.00 $ 782,905.00 $ 1,350,915.00 $ 1,339,065.00
Percentage change in NCF -59.98% -29.99% -40.86% -53.56% -19.87% -20.57%
Net exposure to MXN $ (3,200,000) $ (4,480,000) $ (2,240,000) $ (3,136,000) $ (4,278,000) $ (1,568,000) $ (1,814,000)
3.2 Operating Currency Exposure Reduction Scenarios
3.2.1 Buy Avocados from California Instead of Mexico

Assume we substitute half of our avocado imports from Mexico with U.S. avocados. Also
assume the cost of avocados in the U.S. is 1.6 USD per lb. of avocado. This is equivalent to
the cost of Mexican avocados at the original exchange rate of 0.05 USD/MXN. Since we
only purchase half of our avocados from Mexico, our exposure to the Mexican Peso is now
an outflow of 2.24m USD compared to an outflow of 4.48m USD in the base case scenario
with a 0.07 USD/MXN exchange rate. We can clearly observe that as the value of the peso
1266
Valeria Martinez
appreciates, our cash flows drops by only 30% compared to the 60% drop in NCF in the
original scenario. Thus, substituting 50% of our avocados from Mexico with U.S. avocados
reduces our exposure to the MXN by 30% and decreases the fluctuation in our net cash flow.
3.2.2. Pass on 50% of the Cost Increase to Consumers via a Higher Guacamole Price
The company’s ability to pass on the higher cost of producing guacamole to consumers via a
higher guacamole price will depend on the price elasticity of demand for Holey Moley. We
assume that demand elasticity of Holey Moley is -1.5. This is, a 1% increase in the price of
guacamole reduces the quantity demanded by 1.5%. Given an MXN appreciation of 40%
(from 0.5 USD/MXN to 0.7 USD/MXN), if we pass 50% of this increase in cost to
consumers, and given a price elasticity of -1.5, the price of guacamole would increase by
20% from 6.5 USD to 7.8 USD per guacamole tub, and the quantity demanded would drop
by 30% ( -1.5= Qd/20%). Thus demand for guacamole will be 700,000 tubs instead of the
original 1 million tubs. However, the price per tub in USD will increase to 7.8 USD.
Therefore, total sales in USD are expected to be 5,460,000 USD. This is 1,040,000 USD less
than in our base case scenario. To produce 700,000 guacamole tubs at 2 lb. of avocado per
tub priced at 64 MXN per every 2 lb., the cost of Mexican avocados is 44,800,000 MXN, or
in USD at an exchange rate 0.07 USD/MXN, 3,136,000 USD (44,800,000 MXN*0.07
USD/MXN). Assuming the rest of the costs remain constant, with the exception of the
variable cost which is 10% of sales, our NCF drops by 41% in this alternate scenario. This
compares to an NCF drop of 60% in the base case scenario at an exchange rate 0.07
USD/MXN.
When passing on higher costs of production to consumers, the price elasticity of demand
plays a very important role in the outcome. If the price elasticity of demand were lower, the
reduction in quantity demanded due to the higher prices would also be lower .
3.2.3. Sell Guacamole in Mexico

If it makes sense for the firm’s overall business, they could generate a positive cash flow in
MXN by selling guacamole in Mexico and offset the negative cash flow in MXN from
importing avocados. Assume Holey Moley can sell guacamole in Mexico at the same price
than in the U.S. Also, assume they can generate 10% of U.S. sales in Mexico. Suppose they
will need to increase production by 10% to meet additional demand for guacamole. In this
scenario, we observe the NCF drops by 54% as compared to a drop of 60% in the base case.
In the same vein, exposure to the MXN decreases only slightly from 4.480m USD in the base
case with 0.07 USD/MXN to 4.278m USD in this scenario, given that we would need to
import more avocados from Mexico. Thus, the effort to offset inputs from Mexico with sales
in Mexico is thwarted by the need to import more avocados from Mexico to produce the
added guacamole.
1267
3.2.4. Pass on 50% Cost Increase to Consumers via Higher Guacamole Price and Substitute
50% of Mexican Avocado Imports with California Avocados
Just like the second scenario, passing on 50% of this increase in price to consumers, and
given a price elasticity of -1.5, increases the price of guacamole by 20% (from 6.5 to 7.8 USD
per guacamole tub). Quantity demanded drops by 30% and the demand for guacamole will
be 700,000 tubs instead of the original 1 million tubs. Total sales in USD are expected to be
5,460,000 USD. Moreover, to produce 700,000 guacamole tubs Holey Moley will import
half of the avocados needed from Mexico and source the other half from the U.S. They need
700,000 lb. of avocado from the U.S. at 1.6 USD per lb. for a cost of 1.12m USD; and 700,000
lb. of avocado from Mexico at a cost of 32 MXN per lb. or 1.568m USD at the 0.07
USD/MXN exchange rate. In this scenario NCF drops 20% and the exposure to the MXN is
-1.568m USD.
3.2.5. Substitute 50% of Mexican Avocado Imports with California Avocados and Sell
Guacamole in Mexico
Assume you have 10% additional guacamole sales coming from Mexico and 50% of your
avocado comes from the U.S. The avocado price in the U.S. is 1.6 USD per lb. of avocado
and Mexican avocados are 32 MXN per pound. We increase avocado inputs to account for
the additional production needed to sell guacamole in Mexico. Thus, Holey Moley generates
7.15m USD in sales: 6.5m USD in U.S. sales and 0.65m USD in Mexico sales. Avocado
inputs are 2.2m lb. of avocados needed to produce 1.1m guacamole tubs: 1m tubs in the U.S.
and 0.1m tubs in Mexico. Half of the avocados come from Mexico and half from the U.S...
In this scenario, NCF drops by 21%. The second smallest drop in all scenarios analyzed here,
and second lowest MXN exposure (1.8m USD) out of all scenarios. Overall, the fourth and
fifth scenarios seem to be the best alternatives. They show the smallest drop in NCF and the
lowest MXN exposure out of the five scenarios analyzed.
4. Student Learning
Students worked on a set of two pre and post exercises on economic exposure. The first
exercise in each set was moderately difficult (#1) and the second was difficult (#2). Students
answered the pre-exercises after discussing theory and examples on the topic in class. Then
students worked on the Holey Moley Guacamole Case. Lastly, they worked on the post
exercises. In the pretest, 92% and 28% of students got the correct answer on problems #1 and
#2, respectively. In the post-test, 48% of students got the correct answer on #1 and 76%
obtained the correct answer on #2. This is, students’ scores for the difficult question improved
considerably from the pre to the post-test; however, scores for the moderately difficult
question fell. Thus, overall results from the pre and post-tests are mixed. Further feedback
from students may lead to more clarity in this outcome. Still, cases have shown they add
1268
Valeria Martinez
value beyond traditional lectures. Kester et al. (2005) points out traditional teaching methods
like lectures, may be more efficient in transmitting knowledge. However lectures are not
effective in helping student learn to deal with the unstructured problems they will face in
their professional careers, as cases do. Moreover, Trahan (1993) suggest integrating lecture
and cases into the same courses combines the strengths of both methods and may be superior
to each method individually. Perhaps here lies the answer.
5. Conclusion
Students learn about operating currency exposure through the analysis of scenarios in the
Holey Moley Guacamole case. Results on the effectiveness of using cases vs. lecture are
mixed. Gathering further student information may clarify this issue.
References
Arnold, T., and Buchanan, B. (2004). Interest rate parity in Excel. Journal of Financial
Education, 28-41.
Breuer, W., & Ruiz de Vargas, S. (2021). Some key developments in international financial
management. Journal of Business Economics, 91, 595–615
Eun, C. and Resnick, B. (2020). International Financial Management, 8th Edition, McGraw-
Hill Publishers. New York, NY.
Geyer-Klingeberg, J., Hang, M., & Rathgeber, A. (2021). Corporate financial hedging and
firm value: A meta-analysis. The European Journal of Finance, 27(6), 461-485.
Holden, C. W. (2012). EXCELTM MODELING IN INVESTMENTS, 5th Edition, Pearson.
Holowczak, R. D. (2007). Real time foreign exchange data modeling in international
economics and finance. Journal of Financial Education, 22-35.
Kester, G. W., Dean, R. A., Ding, D. K., Hoover, S. A., & Skully, M. (2005). The perceptions
of students outside the United States on cases versus lectures. Journal of the Academy of
Business Education, 6, 1.
Lei, A. Y., & Li, H. (2012).Using Bloomberg Terminals in a security analysis and portfolio
management course. Journal of Economics and Finance Education, 11(1), 72-92.
Madura, J. (2013). International Financial Management, 13th Edition. Cengage Learning.
Boston, MA.
McCarthy, S. (2016). Foreign Exchange Operating Exposure: A Practical Teaching
Approach. Journal of Financial Education, 42(1-2), 116-136.
Moffett, Michael H., and Petitt, Barbara S. (2004) Porsche Exposed, Thunderbird School of
Global Management, Arizona State University.
Oleaga, Maria (2007). BMW Currency Hedging, IESE Business School University of
Navarra.
Trahan, Emery A. (1993). Bridging the Theory-Practice Gap: An Integrated Case-Study
Approach, Financial Practice and Education, 3 19-22.
1269
Creating a bridge to post-traditional male student success at a

community college
Suffolk County Community College, Riverhead, New York, United States of America.
Abstract
Post-traditional students appear to be the highest growing population of
students on American college campuses. Post-traditional students are 24
years of age and older. Frequently referred to as adult students or
nontraditional students. About 1/3 of the people, who are working and
learning in the U.S., are over the age of 30, which is a very substantial increase
over what we saw in the 1980s and 1990s. Carnevale (2015). Since the
enrollment in colleges has been favoring females since the 1970’s (Borzelleca,
2012), this paper is solely focused on the post-traditional male students as they
attempt college level learning. This research paper investigates what supports
and services are necessary for male post-traditional students to attain success.
During the fall of 2022 a comprehensive qualitative research study was
conducted involving 8 post-traditional male students at a community college
in a suburban New York location.
Keywords: Post-traditional students; male students; adult students;
transformation.

Creating a bridge to post-traditional male student success at a community college
1. Introduction
This paper is the result of a qualitative research project which took place fall 2022 through
spring 2023. A total of 8 Post-traditional male students over the age of 24 were interviewed
to learn about their community college experience. For this research we classified post-
traditional students as students over the age of 24 and any age beyond 24. According to the
National Center for Education Statistics, non-traditional students are usually 24 and older
(2002). A post-traditional student who returns to college is accepting a very real challenge.
These students understand their life will be changed as they enter a new phase of their
educational journey. This research study examined the supports and barriers post-traditional
males, in particular, experience after they decide to attend higher education. The post-
traditional students involved in this study are working to transform their careers through their
educational journeys. Though all were very busy they generously participated in this research
project.
A review of the current landscape of higher education across the United Stated reveals a
changed demographic that has been increasing in number. (Stone, et al., 2016). Ellis (2019)
states this demographic of adult learners displays characteristics unlike traditional students,
and correspondingly this group is called nontraditional (post-traditional) students. The
conceptual framework of this research was guided by Transformative Theory. Creswell and
Poth, (2018) describe the basic principle of Transformative Theory as knowledge not being
neutral and mirrors the power and injustice in society. This research study examined the
transformation process for post-traditional male students. The researcher looked to discover
what this population of male post-traditional students is experiencing at a community college.
In addition, this research study has the potential to develop supportive measures from an
institutional lens.
Considering all the life responsibilities of post-traditional students it becomes important
examine what are post-traditional male students are facing. Goncalves and Trunk (2014)
begin their article Obstacles to Success for the Nontraditional Student in Higher Education
addressing the importance of engagement activities to support college retention. They
specifically focus on the ability or lack of ability of nontraditional students to become an
active part of the college campus. Yet with their schedules how is college engagement even
a consideration? In general, according to Goncalves and Trunk (2014) post traditional
students are highly motivated by their desire to acquire an education. They suggest their
motivation is intrinsic in nature (Goncalves & Trunk, 2014). Overall, post-traditional
students are motived by their desire for self-improvement. They compare traditional age
students who feel forced to attend college by their parents and suggest post traditional
students are more intrinsically driven for many reasons (Goncalves & Trunk, 2014). Post-
traditional students, in particular, are motivated by the ideas of enhancing their self-image
and heightening their self-esteem. This motivation to improve aligns closely with the
1272
Transformative Learning Theory described by Mezirow (1991) as a critical assessment of

oneself and recognizing a change that needs to occur. This feeling of wanting to achieve
more, steered Goncalves and Trunk (2014) to ascertain post-traditional students appreciate
communication with their faculty members and would highly benefit from this type of
interaction. However, based on scheduling and hectic lives post-traditional students do not
engage frequently outside of the classroom, (Goncalves & Trunk, 2014). Post-traditional
students are described as exhausted learners who are stretching work -life boundaries and
enduring economic struggles (Arnold, et al 2021).
Transformative theory speaks to the change that is possible while examining what needs to
occur to make the change possible Creswell and Poth (2018). During these 8 interviews the
researcher looked to discover what supports need to be in place for a post-traditional male to
be successful attending college. Creswell and Poth (2018) continue stating, qualitative
research should have an action-based agenda that has the propensity to change the lives of
the participants and others like them. The males involved in this research were all hoping
their educational experience would transform their lives in one way or another.
The Transformative Theoretical framework, outlined by Creswell and Poth (2018), and
Transformative Learning described by Mezirow (1991) align with the transformative process
and action needed by post-traditional students to achieve success. This transformative
process worked well with this research project to examine the self-reported experiences of
post-traditional male respondents. The goal of this research was to discover ways in which
institutions can have a positive or negative effect on post-traditional male students. Fleener
and Barcinas (2021) suggest a transformation occurs when adult educators explore how
individuals and groups of individuals come together to engage and enact reasonable, possible,
probable and emerging futures. They continue this thought; where a person’s passions are
unveiled, especially as it relates to personal transformation and lifelong learning that includes
openness to new and diverse ideas. (Fleener & Barcinas, 2021). This transformation allows
the individual post traditional student to envision their own future and to create the life they
want. Creswell and Poth (2018) describe the transformative framework to research as a
method for educators to allow the participant to embrace the process while taking an active
role.
3. Methodology
In his section the researcher describes the methodology and research process. The research
study was IRB approved and conducted over the fall and winter semesters. Eight Post-
traditional male students volunteered to participate.
1273
3.1 The Process

The researcher spent the semester investigating the barriers and bridges that are in place for
male students involved in this research. We look to understand what the experience was like
as they returned to college as adult students with life experiences and obligations. The
experience for a male as he attempts to balance work, school and other responsibilities can
be complicated. There are significant financial implications to attending classes. Zaft, (2008)
described the financial constraints adult students feel as they return to college, as difficult.
Congruent with Zaft, Hittepole (2016) notes males often cite finances as impacting their
decision to withdraw. In addition, she notes the lack of support within higher education is
having a significant impact on the success of these male nontraditional students possibly
hindering the transformation to a successful more lucrative career (Hittepole, 2016). Also,
Goncalves and Trunk (2014) described the many responsibilities of older students, that may
hinder success. This research examined the perspective of the nontraditional males as the
researcher intently listened to their stories.
This study’s theoretical framework looks through the lens of a higher education leader to
assess the policies and resources that support or interfere with the successful completion the
higher educational pursuits of these nontraditional male students. The goal of this research
project is to understand the extent to which institutional supports may bring about
transformative change and a heightened awareness of a population that may be overlooked
(Creswell and Poth, 2018).
3.2 Research Questions

Three research questions were used for the research study. These questions framed the study
and act as a guide as we investigated the personal histories and experiences of the study’s
eight participants. Examining these questions helped the researcher to understand the reasons
post-traditional males return to college. In addition, we looked to see what supports they
consider important and what they feel is needed to be successful.
RQ1 What was your motivation for post-traditional male students (at the case-study
community college) to return to college level studies? What are the factors that encouraged
their return?
RQ2 How do participants perceive their decision to return college as being supported by the
university (case study community college)?
RQ3 What barriers do post-traditional male students at the community college in this case
study identify as hindering their achievement of academic? Describe the journey and your
ability to persist regardless of the barriers.
1274
These open-ended questions offered insight into the experience in higher education of the
post-traditional male participants by creating a narrative dialog between the researcher and
each participant.
3.3 Participants
The students who participated in the study met the criteria of the post-traditional male
student, which is beneficial for quality assurance (Creswell & Poth, 2018). In addition to
address the diversity of student populations on the campus the researcher applied maximum
variation sampling method (Creswell & Poth, 2018). Maximum variation sampling occurs
when the sample participants are taken from a large diverse body of possible participants.
An intentional effort was made to select the participants from a diverse group of students.
We used male gender, age, minimum of 12 credits completed, race and ethnicity as
parameters when selecting participants. The researcher made an intentional effort to ensure
the sample accurately reflected the diversity of the campus involved in the study. Maxwell
(2013) suggests defining the dimension of the variation of the population which is most
relevant to the study and selecting individuals that represent the most important and relevant
variations. This method allows for the differentiation of the participants, as each person is an
individual and may come from a completely different geographic area and socioeconomic
background (Creswell & Poth, 2018). The cohort of students interviewed was a remarkably
diverse group, which accurately reflects the surrounding community. In total there were 8
participants- 3 Caucasian, 2 Black/African American, 2 Hispanic/Latinx, 1 Asian/Pacific
Islander.
Table 1. Participant Demographics
Demographic
Information
Name Age Race Gender Ethnicity

Andy 49 White/Hispanic Male Hispanic
Chuck 56 White Male Caucasian

Jose 42 Hispanic Male Hispanic
Nicki 31 Black Male African
American
Santa 30 Black Male African
American
Sky 48 White Male Caucasian
Steve 38 White Male Caucasian
1275
Table 2. Demographics at Case Study Community College.
Student Body
All College Approximately 21,000 students
Gender Female: 56%; male: 44%
Age 18-24: 80%; 25 and over: 20%
Ethnicity White 51.3%
Black/African American: 8.1%
Hispanic: 29.3%
Asian/Pacific Islander: 4.6%

International: 0.4%
American Indican/Alaskan Native: 0.3%

Two or more: 2.0%
Native Hawaiian: 0.1%
Other/Unknown: 3.9%
The case study community college is located in a suburban north eastern area of the United
States within a richly diverse community. Data from the college’s website indicate twenty
percent of the entire student body are age 25 and above.
Through interactive interviews we examined the self-reported experiences of 8 post
traditional male students enrolled full time at the case-study community college. The
individual interviews allowed for open communication in which the participants felt
comfortable sharing their personal story. The community college involved is a large tri-
campus institution. The participants involved in the research were all registered as fulltime
students on one single campus. The campus used in the study is in a densely populated
suburban/urban setting. The surrounding area is ethnically diverse and primarily working
class. The study included interviewing 8 individual students who met the required
demographics. All students interviewed were self-identified males over the age of 24 and
had completed a minimum of 12 credits prior to the fall 2022 semester. In addition, most of
the students were working fulltime and had family obligations.
The interviews all took place in zoom, and most were in the early evenings to accommodate
the participants' scheduling needs. Each interview took between 40 and 60 minutes. All
interviewees filled out an informed consent prior to the meeting and agreed to the conditions.
1276
The interviewees were primarily completed during and directly after the fall 2022 semester.
Students interviewed were all in good standing academically.
4. Findings
The major areas of study of the participants varied from pre-Nursing, History, Business,
Liberal Studies and HVAC. All students’ interviews had very precise goals. They shared
similar aspirations of self-improvement and enhanced career opportunities. Only one student
interviewed was not interested in a career change or professional advancement. Most were
motivated by financial empowerment and greater opportunities that a college degree would
provide. A consensus among the group was the feeling of limited opportunities without a
degree.
Another commonality was the “missed opportunity” in their younger years. The interviewees
spoke of their recognition of the need for good time management skills. Many spoke of their
lack of this time management skill in their youth but have learned to adapt now as they attend
college as adults. “I didn’t have the time management skills I needed in my younger days”.
At this point in their lives when they have so many things to manage, collectively they appear
to have realized the importance or prioritizing and managing their time wisely.
“Really my biggest issue is time,”
In addition, financial resources and the need to increase their earning power was a common
thread. It is important to note the need to increase earnings to match their personal needs as
well as their family’s needs was also a significant factor triggering the student’s return to
college. Three students mentioned the feeling of plateauing in their industry and not being
able to advance in way that would adequately support the needs of the family. “I was applying
for promotion and my boss told not to think of it because of my lack of a degree.” One student
was going through a divorce and wanted to ensure a better life for his child.
Time management and the lack of availability of college resources was a major finding
regardless of the degree path or goal of the students interviewed. In addition, all students
were looking to enhance their sense of self-fulfillment. Six of the eight were genuinely
concerned with their financial security. Most students interviewed felt supported by the
college to some extent, but noted the lack of evening, weekend or online support.
5. Conclusion
This narrative case study research project offered insight into the experiences of the eight
students who participated. The knowledge gained is not generalizable due to the small sample
size and the singular campus, however it is meaningful to gain a greater understanding of the
1277
needs of the post-traditional male student. This type of research can assist college
administrators as they look to recruit more post-traditional males and support their retention.
References
Arnold, C. H., Badenhorst, C. & Hoden, J. (2021) Reviving and Reinstating Adult-Educators’
Agency within Adult and Post-Secondary Programmes. The Irish Journal of Adult and
Community Education P63-73
Borzeleca, D., (2012) The Male-Female Ratio in College Feb 16, 2012,03:26pm EST
https://www.forbes.com/sites/ccap/2012/02/16/the-male-female-ratio-in-
college/#4e25f285fa52
Carnevale, A.P., Jayasundara, T., & Cheah, B. (2013) The college advantage: Weathering
the economic storm. Georgetown University.
Creswell, J.W. and Poth, C. N. (2018) Qualitative inquiry and research design; choosing
among five approaches. Sage Publishing, Thousand Oaks California
Ellis, H. (2019). Pursuing the Conundrum of Nontraditional Student Attrition and
Persistence: A Follow-Up Study. College Student Journal, 53(4) 439-449
Fleener, J. & Barcincus, S. (2021) Democracy’s relationships with the future: transforming
Society’s promise. Journal of Transformative Education 2022 20:3, 176-188
Goncalves, S. & Trunk, D. (2014). Obstacles to Success for the nontraditional student in
higher education. The International Honor Society in Psychology, Vol.19.No 4/ISSN
2164-8204
Hittepole, C. (2016) Nontraditional Students: Supporting Changing Student Populations; A
guide for Chief Academic Officers and Chief Student Affairs Officers, NASPA Journal
Stone, C., O'Shea, S., May, J., Delahunty, J. & Partington, Z. (2016). Opportunity through
online learning: Experiences of first-in-family students in online open-entry higher
education. Australian Journal of Adult Learning, 56 (2), 146-169
https://www.naspa.org/images/uploads/main/Hittepole_NASPA_Memo.pdf
Suffolk County Community College https://www.sunysuffolk.edu/
U.S. Department of Education, National Center for Education Statistics. (2019). Digest of
Education Statistics, 2017 (NCES 2018-070), Chapter 3.
Zafft, C. K. (2008). Bridging the Great Divide: Approaches that Help Adults Navigate from
Adult Education to College. Adult Learning, 19(1–2), 6–11.
1278
EPIC in action, measuring entrepreneurial competencies in higher

education
Connie O’Regan1, Neil Ferguson2, Michelle Millar1, Jenny Mullery2, Natalie Walsh2,
Tony Hall3
1
UNESCO Child and Family Centre, School of Political Science and Sociology, University
of Galway, Ireland, 2Designing Futures, Dean of Students, University of Galway, Ireland,
3
School of Education, University of Galway, Ireland.
Abstract
Increasingly university programmes are introducing a range of experiential
learning based programmes to support students to develop their
entrepreneurial competencies during their time at university.
This paper describes how the University of Galway is utilising the
Entrepreneurial Potential and Innovation Competences Tool (EPIC) to track
the changes in student self-reported competencies having participated in one
of its flagship student entrepreneurial programmes. Based in Ideaslab, the
university’s Human Centre Design Studio, the approach used is experiential,
design centric and informed by D.School, Stanford.
Initial findings from the EPIC surveys completed by 23 students are reported.
These data are part of a university wide initiative and further data will be
collected over the next three semesters. In so doing, we hope to add to the
body of knowledge concerning the utility of this approach to measuring the
changes in entrepreneurial competencies following participating in a
university based entrepreneurial learning activity.
Keywords: Entrepreneurial competency; undergraduate student;
measurement; experiential learning, design centric approaches.

EPIC in action, measuring entrepreneurial competencies in higher education
1. Introduction
Jamieson’s (1984) framework for entrepreneurship education distinguished between
education about enterprise, education for enterprise and education in enterprise. The core
educational approach taken by the Human Centred Design studio in this case study is located
within both the “education for” and “education in ” approaches. In particular, the team utilise
“Design Thinking” as a methodology across their programmes. They focus on fostering
design thinking as a basis for practical engagement in developing innovative solutions to real
world problems (Brown, 2008). Using the Stanford d.school1 approach, students in Ideaslab
learn to apply active problem solving as they work on real life challenges. Working within
this methodology, IdeasLab are part of an international movement to use design thinking
approaches to enhance the provision of entrepreneurial education (Linton & Klinton, 2019;
Mueller & Thoring, 2012; Sarooghi et al., 2019).
Within the University of Galway, the programmes offered by Ideaslab are currently being
enhanced and developed as part of the Designing Futures Initiative, a suite of complementary
and connected initiatives and modules, focused on developing the attributes, dispositions and
skills required of graduates, to best support them in the challenges they face in the world of
today. This overarching project is being tracked using a formative evaluation approach
(Patton, 2010). Specifically, the evaluation team decided to deploy the EPIC tool
(https://www.heinnovate.com) to explore whether participation in IdeasLab fosters the
development of the entrepreneurial competencies of the students who take part.
The Entrecomp framework was developed by the Joint Research Centre (JRC) of the
European Commission with the aim of building “a bridge between the worlds of education
and work, by contributing to a better understanding and promotion of entrepreneurship
competence in Europe” (Bacigalupo et. al. 2016, p.7). The framework consists of 15
competencies and detailed learning outcomes across three thematic areas, Ideas and
Opportunities, Resources and Into Action. In 2018, the European Commission established
the EEEPHEIC project (Evaluation of Entreprenuership Education Program in Higher
Education Institutions and Centres) to facilitate the development of common evaluation and
measurement frameworks for entrepreneurial education (Baggen & Kaffta, 2022). One
specific outcome was the publication of the EPIC tool (Entrepreneurial Potential and
Innovative Competences), based on the Entrecomp framework. It is a course assessment tool
that can be used by education providers to assess the effectiveness of the training provided.
A series of thematic statements are provided across five domains of entrepreneurial
competencies, entrepreneurial intentions and attitudes, enterprising behaviours,
entrepreneurial strategies and education effects. The EPIC tool is freely available on the
1
For further information see https://dschool.stanford.edu/
1280
Connie O’Regan, Neil Ferguson, Michelle Millar, Jenny Mullery, Natalie Walsh, Tony Hall
HEInnovate website. It is a design that it can be used flexibly and modified according to the
cohort of students or learning situation. In this instance, the evaluation developed an online
survey to be circulated to students participating in two Ideaslab programmes across one
semester. This paper describes the programmes provided, the survey tool deployed and some
initial findings from the data collection.
2. Methods
2.1. Programme Description

Two types of programmes were included in this case study. Firstly, 38 students participated
in an enterprise challenge that was delivered through a combination of class room activity,
team based work in students own time and mentoring sessions with external enterprise
partners. Table 1 below provides an overview of the classroom content delivered to challenge
participants.
Table 1: Enterprise Challenge Classroom Content.
Workshop Content
Team building Team warm up activity

Introduction and Completion of Team Canvas
Introduction to Design Thinking Designing Thinking Method

Stakeholder Mapping
Interviewing
Problem Definition Focus on “How Might We”

Practice review and developing HMW statements
Brainstorming & Ideation Affinity Diagrams

How Now Wow approach for Idea selection
Prototyping & Lean Canvas How to create a value proposition

Essentials of a Lean Canvas
Story Telling Essentials of Storytelling

How to develop a pitch
Opportunity to draft out approach
1281
At the outset of the programme, each student was assigned to one of eight separate teams.
These teams are multidisciplinary and span students from first year undergraduate to Phd
level. The challenges consisted of four personas detailing a real life situation for the
enterprise partners. Each partners provided 4 sessions of mentoring to individual teams over
the course of the challenge. These were to guide and support the students in their problem
solving and idea development. Ideaslab staff facilitated the teams to make contact with their
mentors at the arranged times and issued reminders about mentoring sessions at the weekly
content workshops. In addition, the enterprise partners gave their time and expertise to attend
the final team pitches and the selection of the winning ideas. These pitch sessions lasted
approximately 2 hours, with four teams presenting to each of the two partners at two separate
sessions. The challenge ended in week 8 of the semester with the selection of a winning pitch
based on the effectiveness of the proposed solution in meeting the challenge set out by each
of the enterprise partners.
The second activity in this case study is the Ideaslab internship programme is a paid
placement for University of Galway students to work on projects with the Ideaslab team over
10 weeks in semester. Students are encouraged to work up to 4 hours per week on assigned
tasks and are supported by a developmental scholarship for their participation. Students were
recruited through an open call through Ideaslab and the college social media channels early
in the semester. They applied to Ideaslab using a CV and a personal statement. A number
were then selected for interview and 11 students were successful in becoming an Ideaslab
intern. The internship programme offered a range of opportunities for students to get
involved in Ideaslab projects over the course of the semester. At the outset, students were
asked to write a short biography and publish it to the Intern team SharePoint site as an aid to
building a community across the team. They were asked to update their LinkedIn profile and
engage with the Ideaslab social media channels. Students were also asked to begin and
maintain a reflective practice log. In this log, they were asked to identify their goals for the
placement, to make weekly updates on their progress and complete a short reflection of their
experience at the end of the internship. Interns were also able to attend sessions in Ideaslab
on Design Thinking during the placement as well as engage with the Student Success
Coaching2 service.
2.2. Survey Tool

The evaluation team reviewed the online EPIC tool and identified a sub set of 20 statements
across three of the five domains that best matched the course content. A number of
demographic items and open ended items were included. The survey was circulated to all
2
Student Success Coaching at the University of Galway is also part of the Designing Futures Initiative, further information is
available at https://www.universityofgalway.ie/designingfutures/
1282
participants at the end of each of the programmes using Qualtrics. Ethical approval for the
research was secured from the university’s research ethics committee.
2.3. Programme Participation

Across both programmes, a total of 49 students participated. The profile of this group of
participants indicate a high level of representation of both international students (55.1%) and
postgraduate students (53.1%). The Ideaslab team reported that this semester’s participants
was an outliner in terms of international and post graduate students and plan to track
participation levels to monitor this issue next term to establish if this is a recurring trend.
Profile of Participants and Survey Sample

Internship
Enterprise Challenge
Postgraduate
Undergraduate
Science & Engineering
Arts
Business
Medical, Nursing, Health Sciences
Prefer not to say
Non EU
EU
Prefer not to say
Male
Female
0% 20% 40% 60% 80% 100%
Programme Survey Return
Figure 1. Profile of Participants & Survey Respondents.
3. Results
The survey was completed by a total of 23 students across both programmes, a return rate of
46.9%. Figure 1 compares the profile of survey returns to the overall profile of programme
participants, illustrating a representative sample. Table 2 below provides a breakdown of the
EPIC scores, illustrating the initial mean score across the group, the final mean score and
provides the calculated difference. Notwithstanding the small sample size, a number of initial
points can be made. Firstly, the area with the most overall increase in scores is in the “Ideas
and Opportunities” area. This finding makes intuitive sense given the design centric focus
of the programme. It is likely that this trend is reflected in the gains in the sub items on
1283
“looking for new opportunities” and “want to solve problems in new ways”. Secondly, there
is also a positive trend in a number of the items dealing with team engagement and working
with others, “convince others to engage” “actively participate in teamwork”. Although as
will be noted below the feedback on team engagement was developed further by considering
comments made in the open-ended items included in the survey. Finally, the least
development in scores occurred in the item dealing with “estimated budgets”. This finding
does reflect less engagement on this topic during this iteration of the programme.
Table 2: EPIC Scores.
1284
4. Discussion
The EPIC tool did indicate positive impacts of participation on self-reported scores.
However, as noted earlier, this particular sample was quite small (n=23). It will be possible
to collate EPIC scores into SPSS for each of the next three semesters to track score changes
across a larger sample, checking to see if there are different scoring patterns according to
number of events attended or demographics etc.
The tool currently uses a device to allow participants to complete the survey at the end of the
programme with a sliding scale to score each competency at their level both when they started
and also at the end of the programme. A number of students in this cohort reported that it
was difficult to be consistent using this approach and to estimate their level at the start of the
event. As a result, the research team have modified the approach so that next semester
students will rate their competencies on two different surveys, one at the start and one at the
end of the programme.
While the items on the EPIC tool did indicate positive impacts on team engagement, a number
of students indicated tensions with managing the time commitment, maintaining contact
across the team and promoting team engagement in the open-ended survey items. Based on
this feedback, the Ideaslab team have decided to provide an increase focus on team building
on the start of the programmes to promote the positive development of team cohesion. In
addition, as the semester progresses, they will regularly remind participants to link back for
support and advice if challenges with team engagement persist.
There was also interest in including an item in the survey to get feedback from participants
on how they would share a “bonus” with team members as a proxy measure of team
effectiveness. The evaluator will source some options on this issue for the team to consider
and include in the next semester’s data collection.
5. Conclusion
Baggan and Kaffka (2022) recommended that “the capability of European citizens to engage
with the unknown and deal with uncertainty should be developed from childhood.
Experiential or challenge-based learning programmes address this need and its offer should
be ensured via progression lines from primary education to higher education and beyond, in
the form of lifelong learning (both formal and informal) of adults” (p.10). This paper has
described the deployment of the EPIC tool to measure the effectiveness of participation in
two experiential learning opportunities aimed at increasing entrepreneurial competencies in
a Higher Education setting. The initial data from this first cohort indicates that this measure
has potential to track these changes in students’ self-reports. In particular, the tool seems to
be sensitive to changes in those specific areas of design and ideation focus central to these
1285
particular initiatives. Some amendments have been suggested to the deployment of the tool
in the next round of data collection in this project. Further analysis will be carried out to
consider the suitability of the tool with the total cohort of students over the course of the
project.
Funding Acknowledgement
Designing Futures (DF) is a flagship programme which received €7.57m funding through
Human Capital Initiative, Pillar 3 by Ireland’s Higher Education Authority (HEA, 2020).
References
Bacigalupo, M., Kampylis, P., Punie, Y., & Van den Brande, G. (2016). EntreComp: The
entrepreneurship competence framework. Luxembourg: Publication Office of the
European Union, 10, 593884.
Baggen, Y., Kaffka, G. (2022). Entrepreneurial literacy and skills, Publication for the
committee on Employment and Social Affairs,Policy Department for Economic,
Scientific and Quality of Life Policies, European Parliament,Luxembourg
Brown, T. (2008). Design thinking. Harvard business review, 86(6), 84.
Higher Education Authority (2020) Human Capital Initiative Pillar 3 – Innovation and
Agility. https://hea.ie/skills-engagement/human-capital-initiative-pillars-1-2-and-
3/#pillar-3-%e2%80%93-innovation-and-agility
Jamieson, I. (1984). Education for Enterprise. In A. G. Watts & P. Moran (Eds.), CRAC: 19-
27. Cambridge: Ballinger.
Linton, G., & Klinton, M. (2019). University entrepreneurship education: a design thinking
approach to learning. Journal of innovation and Entrepreneurship, 8(1), 1-11.
Müller, R. M., & Thoring, K. (2012). Design thinking vs. lean startup: A comparison of two
user-driven innovation strategies. Leading through design, 151(2).
Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance
innovation and use: Guilford press.
Sarooghi, H., Sunny, S., Hornsby, J., & Fernhaber, S. (2019). Design thinking and
entrepreneurship education: Where are we, and what are the possibilities? Journal of
Small Business Management, 57, 78-93.
1286
The MEM project: 5 years of experiences, challenges, and

outcomes of an international double master-level degree
Roberto Montanari1, Layek Abdel-Malek 4, Gino Ferretti1, Alessandro Bernazzoli2,
Eleonora Bottani1, Andrea Volpi1, Federico Solari1, Letizia Tebaldi1, Giorgia Casella1,
Natalya Lysova1, Michele Bocelli1, Claudio Suppini1, Lucia Orlandini1, Monia Bertoli1,
Andrea Pintus3.
1
Department of Engineering and Architecture, 2International Division, 3Department of
Humanities Social Sciences and Cultural Industries - University of Parma, Italy, 4Department
of Industrial and Manufacturing Engineering - New Jersey Institute of Technology, USA.
Abstract
In the 2016-2017 academic year, a novel Double Degree educational project
named MEM (Master in Engineering Management) between the University of
Parma and New Jersey Institute of Technology (NJIT) had begun. Since then,
three (3) double degree master's programmes have been developed, starting
with the Engineering Management course. To date, 113 students have attended
the MEM programme.
This paper aims to assess the impact of the MEM double degree programme.
To this end, a questionnaire was developed and submitted to the students who
completed the project and entered the workforce (66 students). An analysis of
the survey results was then conducted to determine the impact, if any, of the
MEM programme on the students' academic careers and/or their entry into the
workforce. The results were compared across two (2) sets of students: those
who participated in the programme and the students that graduated from the
standard degree course.
Keywords: Double degree, internationalization, engineering management
education.

The MEM project
1. Introduction
The world today is experiencing major challenges that require multicultural professionals to
help in addressing them. It is no secret that the nature of the present supply chains is very
complex, as they transcend political, cultural, technological, and geographical barriers. In
essence, one of the main fulcrums that the resilience of these supply chains depends on is
humans that have the capabilities to provide the needed nuances for their maintenance.
Higher education is a major supplier of graduates that possess these traits.
As reported in the literature, the past decade has witnessed the development of many joint
international programmes where students receive dual, double, and joint degrees, as well as
several exchange programmes. Among these programmes is MEM. In HEAd 2018
(Montanari et al., 2018), we presented our experience at that time. In this paper, we share the
recent developments, data, and extensions since then.
2. Literature analysis
In recent years global initiatives in higher education have received growing attention. This
growth has resulted in a plethora of publications and conference presentations. This section
reviews some of the pertinent ones.
Tran et al. (2023) have presented an in-depth analysis covering a study of 227 English-written
articles that examine the transition of higher education, with a focus on internationalization.
They report that, while the transition from domestic to international frameworks can be
rewarding, it requires appropriate planning and support. Hynek (2021) has instead focused
on how to achieve effective internationalization. He states that it is essential to build strong
international partnerships by establishing joint degree or double degree programmes that
integrate international students.
Maldonado (2022) describes in his paper the requirements, procedures, benefits, and results
in the development of a dual programme in Mechanical Engineering between two universities
in Mexico and Japan. For the past 15 years, the programme has graduated students who
master three languages (English, Spanish, and Japanese), in addition to international
experiences that have transformed their lives. Similarly, Stegall et al. (2021) have explored
the benefits of integrating two international programmes for undergraduate business students.
They demonstrated how these programmes enrich the students’ experiences and provide
employers with candidates that enjoy global exposure.
Knight (2012) has introduced a framework to improve the mobility of both students and
scholars. The author has pointed out the risks and benefits of internationalization. He also
provides a conceptual structure to analyze the meaning, trends, and opportunities of
international dual, joint, and double degrees. Asgary and Robbert (2010) have presentd a
1288
Roberto Montanari, et al.
cost-benefit analysis of dual international degrees. They also point out the risks and rewards
of such programmes.
In his paper, Oliveira (2011) has illustrated how today’s global markets require engineers
with value-added experiences, particularly in humanities. In this work, the author reports on
the opportunities resulting from the development of a consortium of three (3) universities in
the US and Brazil. An interesting approach to internationalization was proposed in Gulick et
al. (2006), where the authors introduce international co-op/internship programmes that allow
expanded technical knowledge.
For additional insights, the interested reader is referred to (Aasmundtveit et al., 2018),
(Cheng, 2021), (Corno et al., 2016), (Ivanova et al., 2020), (Ober et al., 2021), (Curtis and
Ledgerwood, 2018), (Reardon et al., 2022), and (Wei et al., 2020).
As can be seen from the foregoing, the interest of the community towards the innovation,
experiences, and opportunities that international dual, double, and joint degree programmes
offer is very high. In the following introduction section, we present our own experience and
results of double degrees developed between the University of Parma and the New Jersey
Institute of Technology (NJIT) showing the progress achieved since our earlier paper
presented in HEAd 2018 (Montanari et al., 2018).
3. The MEM Project

The MEM project (Montanari et al., 2018) provides students with an opportunity to receive
two master’s degrees from the University of Parma (UNIPR) and the New Jersey Institute of
Technology (NJIT). The participating institutions collaborate on a non-exclusive basis on the
development of master-level degrees in Engineering, where the Master of Engineering
Management (MEM) dual degree programme (MoU, 2015) being the first one. Since then,
two (2) further Master’s degree programmes were added, one in Mechanical Engineering
(MoU, 2016) and the second one in Engineering for the Food Industry (MoU, 2019.)
As stated before, MEM aims to advance academic exchange and cooperation in the fields of
engineering in order to increase the transnational expertise of European and American
students, enhance job opportunities, competitiveness, cultural exposure, and language skills
(Malek et al., 2014). Students participating in the programme must pass core and elective
courses in both institutions. The host University identifies the set of courses that should be
taken by the participating students, according to each student's specific programme
requirements. The approach allows more flexibility to meet the student’s needs while
fulfilling the graduation requirements of both universities. In designing the study plan for
each student, it is essential, however, to develop the academic paths by appropriately
balancing the length and the workload of the courses offered by the two universities. The
1289
The MEM project
equivalence and the validity of the study plans and academic paths are verified by
administrative advisors in each institution.
4. Methodology
4.1. Aims
The study aims to assess the impact of MEM from different perspectives, related to (i) the
students' academic careers; (ii) the universities involved; (iii) the MEM project; and (iv) entry
into the workforce. To perform this analysis, the phases before, during, and after the mobility
were analyzed. First, the study seeks to identify any correlations between the academic
careers of “standard students” (i.e., attending the standard master’s degree programme at the
University of Parma) and “MEM students” (i.e., attending the double master’s degree). For
instance, it aims to evaluate the “good grade point average” (GPA) of MEM students before
their mobility period and at the end of their academic career, comparing it to the GPA of
standard students. With reference to the University of Parma, the analysis aims to assess
whether MEM contributes to enhancing the attractiveness of academic degrees, thus
increasing the number of students coming from other universities and other regions of Italy.
As far as the MEM project itself is concerned, the study seeks to quantify the satisfaction of
the MEM students in terms of the academic curricula and the usefulness of the skills obtained
for their job. Finally, regarding job placement, the study aims to define the impact, if any, of
the MEM programme on the salary and the time elapsing between graduation and the first
day of work, as well as the positive impacts of attending the project on the placement phase.
4.2. Source of data

Since the academic year 2016-2017, 113 students have been involved in the MEM
programme. 81 of these students have completed their international mobility period, and 66
of them are already working. Based on the data gathered during these years of activity, a
database was built to derive important insights into the project. To conduct the analysis,
different statistical samples were defined based on the project phase investigated (before,
during and after the mobility) (Figure 1). It is important to point out that all data were
processed anonymously, to respect the privacy of the students. Additionally, several data
sources were designed, starting with the UNIPR and NJIT databases. Another data source
considered was the AlmaLaurea survey (AlmaLaurea Interuniversity Consortium and
AlmaLaurea srl). The consortium involves 80 Italian universities and, since 1999, it has been
carrying out annual surveys on the graduates’ profiles, providing a wide-ranging portrait of
their characteristics, university achievements, and employment status. Specifically, the
ranking focuses on the positioning of the graduates on the job market, the characteristics of
the job, the salary, and the usefulness of the skills acquired within the degree courses. In this
1290
study, an analysis of the past four years (2019-2022) was conducted, evaluating the responses
for the Engineering Management degree class (LM-31, 34/S) both on the national level (about
2300 students per year) and with respect to the University of Parma (about 90 students
considered per year). Finally, to determine the impact of MEM, a survey was conducted
through a questionnaire submitted to 66 students who have completed their academic careers
and are currently working. A response rate of 83% was achieved.
Figure 1. Number of MEM students over the years
5. Results
5.1. Academic Career

Based on the UNIPR data (Table 1), except for the COVID period, the number of MEM
students has increased over the years, with 32 students participating in the project during the
academic year 2022-2023. During the COVID period, there were no students enrolled in the
double degree program as none were interested in acquiring the degree through remote mode,
despite being given the opportunity to do so.
1291
The MEM project
Table 1. MEM’s impact on Academic career
Standard Students MEM Students
Bachelor’s degree grade 101.1/110 97.8/110

Master’s degree grade 103.2/110 106.8/110
Master’s degree age 26.50 years old 25.11 years old
% of students graduated on time 81.25% 80.50%
5.2. Parma University

Thanks to MEM, the University of Parma has increased its visibility and attractiveness. In
the first years of MEM only 12.12% of the master’s students came from a different university,
whereas in the last two academic years, the MEM students who did not get their bachelor’s
degree from the University of Parma were 27.7%. This value is higher than the percentage
of standard students coming from other universities (22.1%, 15.4% on a national basis). Not
only the double degree programme has increased its visibility and attractiveness over time,
but now it appears to be more attractive than the standard master’s degree course.
5.3. MEM Project

Based on the survey conducted among the MEM students, the following feedback (Table 2)
was collected, with the ratings on a scale from 1 (completely disagree) to 5 (completely agree)
Table 2. Feedback provided by MEM students
MEM Students
I would do the double degree experience again 4.64/5

I would recommend the MEM project to other students enrolled in the master's
degree course 4.62/5
The double degree enhances the development of soft skills such as adaptation,
flexibility, problem-solving, teamwork, curiosity, empathy, and organization 4.30/5
The skills acquired thanks to the double degree project are adequate for my job
4.29/5
Acquiring double degree is worthwhile, even if not specifically required for
4.54/5
employment
5.4. Job placement

The impacts of MEM on job placement are the following:
1292
Table 3. MEM’s impact on Job placement
Parma National
MEM
Standard Average
Students
Students
Working students after one year 83.0% 85.0% 90.7%

Months to entrance into the job market 3.1 2.8 2.4
% stable contracts after one year 33.5% 37.3% 80.0%
% graduates working abroad 2.2% 2.8% 9.3%
Salary after one year 1541.00€/m 1505.75€/m 1567.71€/m
Salary after two years 1605.00€/m 1590.00€/m 1636.25€/m
6. Conclusion
This article represents an important milestone in the field of double-degree programmes as
there are very few cases in the literature where quantitative data are available to measure the
impact of these projects. It is, indeed, extremely difficult to numerically quantify their impact.
Moreover, an apparent peculiarity has been found regarding the salary. Clearly, it is the job
market that determines its value, and not much can be said about it. However, it is evident
that if, on one hand, students view the project positively despite receiving the same salary as
those who did not undertake the double degree program, it means that, fortunately, "money"
is not their priority. In other words, it is very reductionist to associate one's work with the
salary earned: other values count in a career, such as the role, training, job quality, and the
ability to work abroad, all of which can be achieved through a project of this kind. In addition,
the University of Parma has increased its attractiveness and visibility. Most importantly,
MEM students believe that the MEM project is an extremely positive value in job placement.
References
Aasmundtveit, K. E., Wang, C., Rencz, M., Desmulliez, M. P. Y., Ender, F., & Imenes, K.
(2018). Joint international master in smart systems integration: University collaboration
for improved education. Paper presented at the IEEE Global Engineering Education
Conference, EDUCON, , 2018-April 624-629. doi:10.1109/EDUCON.2018.8363288
AlmaLaurea srl. (2022-2021-2020-2019). Graduates' Employment status. 24th -23th-22th-
21th- Survey - (2022-2021-2020-2019) https://www.almalaurea.it/
AlmaLaurea srl. (2022-2021-2020-2019). Graduates' Profile 2021-2020-2019-2018. 24th -
23th-22th-21th Survey - (2022-2021-2020-2019) https://www.almalaurea.it/
Asgary, N., & Robbert, M. A. (2010). A cost-benefit analysis of an international dual degree
programme. Journal of Higher Education Policy and Management, 32(3), 317-325.
doi:10.1080/13600801003743513
1293
The MEM project
Cheng, L. (2021). Employment trends and forecast (2017-2021) for engineering students in
a double degree joint programme. Paper presented at the ACM International Conference
Proceeding Series, 382-387. doi:10.1145/3498765.3498825
Corno, F., Lal, R., & Hassouna, S. (2016). Building bridges: Enabling intercultural
competences within double degree programs. Turkish Online Journal of Educational
Technology, 2016(NovemberSpecialIssue), 398-405.
Curtis, T., & Ledgerwood, J. R. (2018). Students’ motivations, perceived benefits and
constraints towards study abroad and other international education opportunities. Journal
of International Education in Business, 11(1), 63-78. doi:10.1108/JIEB-01-2017-0002
Gulick, D., Pearson, D., Elliott, G., Oliver, J., & Zimmer, K. (2006). Developing globally-
minded engineers through education and experience: A panel discussion on international
co-op/internship program models. Paper presented at the ASEE Annual Conference and
Exposition, Conference Proceedings,
Hynek, J. (2021). Strategic management of university internationalization. Paper presented
at the Proceedings of the 2021 30th Annual Conference of the European Association for
Education in Electrical and Information Engineering, EAEEIE 2021,
doi:10.1109/EAEEIE50507.2021.9530902
Ivanova, O., Ivanov, M., Kuznetsov, M., Aleksandrov, A., & Revetria, R. (2020). Double
degree M.sc program on industrial safety engineering with aerospace application: A case
study. Paper presented at the International Symposium on Project Approaches in
Engineering Education, 10, 221-226.
Knight, J. (2012). Student mobility and internationalization: Trends and tribulations.
Research in Comparative and International Education, 7(1), 20-33.
doi:10.2304/rcie.2012.7.1.20
Maldonado, D. (2022). Engineering twinning program: Universidad de Monterrey (UDEM),
MEXICO and Nagaoka university of technology (NUT), JAPAN. Paper presented at the
ASEE Annual Conference and Exposition, Conference Proceedings
Malek, A. L., Montanari, R., Bernazzoli, A., Bottani E., & Rinaldi, M. (2014). Experience
and Challenges of International Dual and Joint Degrees. Eighteenth International
Working Seminar on Production Economics. 24-28 February 2014. Innsbruck, Austria.
Malek, A.L., Ballester E., Bernazzoli A., Montanari R., & Sànchez Ruiz, L.M., (2014).
Towards the Global Engineer: a triple degree approach. 4th Valencia Global 2014, 19-20
June 2014, Valencia, Spain.
Montanari, R., Malek, L. A., Andrei, P., Ferretti, G., Valenti, S., Mancinelli, L. M.,
Bernazzoli, A., Bottani, E., Bloom, J. S., & Deek, F. P. (2018). The MEM project:
experiences, challenges and outcomes of an international double master-level degree.
Proceedings of the 4th International Conference on Higher Education Advances
(HEAd’18). https://doi.org/10.4995/head18.2018.8159
Mora, L. M. & Montaner, J.J. (2014). International Education, UMANE Atlantis Program,
EU/US International Bachelor’s Degree. ASEE 2014 Zone I Conference, April 3-5, 2014,
University of Bridgeport, Bridgeport, CT, USA.
Ober, I., Nosek, J., & Joly, P. (2021). About the double degree master's programs between
University Toulouse III-Paul Sabatier and Technical University of Liberec. Paper
presented at the 2021 IEEE International Workshop of Electronics, Control,
1294
Measurement, Signals and their Application to Mechatronics, ECMSM 2021,

doi:10.1109/ECMSM51310.2021.9468839
Oliveira, A. (2011). Key issues of establishing an international engineering exchange
program. Paper presented at the SEFI Annual Conference 2011, 923-932.
Reardon, J., Miller, C., & McCorkle, D. (2022). The effect of student perceived benefits and
obstacles to determine if and where to study abroad. Journal of International Education
in Business, 15(2), 351-372. doi:10.1108/JIEB-05-2021-0060
Steagall, J. W., Falk, P., Gallo, A., & Porter, T. W. (2021). International double degree
programs as accelerators for internationalization: Lessons from the trans-Atlantic
business school alliance. Journal of Teaching in International Business, 32(1), 7-35.
doi:10.1080/08975930.2021.1906378
Tran, N. H. N., Amado, C. A. D. E. F., & Santos, S. P. D. (2023). Challenges and success
factors of transnational higher education: A systematic review. Studies in Higher
Education, 48(1), 113-136. doi:10.1080/03075079.2022.2121813
Wei, S., Tan, C., Koomsap, P., & Huang, C. -. (2020). International dual-degree programs:
Learning experience in student's perspective. Paper presented at the International
Symposium on Project Approaches in Engineering Education, 10 193-198.
1295
Multiple-value governance in (German) higher education: a new

paradigm?
Bernd Kleimann
Research Area “Governance in Higher Education and Science”, German Centre for Higher
Education Research and Science Studies, Hannover, Germany.
Abstract
Shifts in governance frameworks of higher education systems as well as
scholarly debate indicate that New Public Management (NPM) – this long-
standing hegemonial reform framework for the public sector in general and
for higher education (HE) in particular – has passed its peak. This paper will
explain and justify this claim for the case of German HE. It argues in favour
of the assumption that a new governance paradigm – called multiple-value
governance - is on the horizon. This new governance regime partly overthrows,
partly modifies and partly succeeds NPM. In order to corroborate this claim,
a reflection about the organizational consequences of introducing
sustainability as one of many values into the German higher education sector
is outlined at the end of the paper. The latter’s intention is to lay a conceptual
foundation for a research agenda that addresses the hypothesized changes in
HE governance empirically and offers perspectives for further research.
Keywords: Higher education governance; multiple-value governance; new
public management; expansion of higher education; Germany.

Multiple-value governance in (German) higher education: a new paradigm?
1. Introduction
Shifts in governance frameworks of higher education systems as well as scholarly debate
indicate that New Public Management (NPM) – this long-standing hegemonial reform
framework for the public sector in general and for higher education (HE) in particular – has
passed its peak. This paper will explain and justify this claim for the case of German HE. It
argues in favour of the assumption that a new governance paradigm – called multiple-value
governance - is on the horizon. This new governance regime partly overthrows, partly
modifies and partly succeeds NPM. In order to support this claim, the idea of a future
investigation of the organizational consequences of introducing sustainability as one of many
values into the German higher education sector is outlined. The overall ambition of the paper
is to lay a conceptual foundation for a research agenda that addresses the hypothesized
changes in HE governance empirically and offers perspectives for further research.
2. The Trajectory of NPM

New Public Management (NPM) has been the cause of an avalance of reforms affecting the
higher education sector in Germany form the late 1990s onwards. NPM’s powerful
promotion of values like efficiency, effectiveness, and subsidiarity has guided change in
teaching, research and university administration, with varying consequences (cf.
Wissenschaftsrat, 2018). Moreover, in the wake of NPM’s focus on strengthening
governance mechanisms like competition, external guidance and managerial hierarchy (de
Boer et al., 2007), universities have advanced to become strategic actors (Krücken & Meier,
2006). The gain in autonomy and agency has affected both the university’s goals as well as
its internal decision-making and support processes. Additionally, the NPM-driven reshaping
of HE organisations has spurred an increasing vertical and horizontal differentiation of higher
education organisations (Banscherus et al., 2015; Scott, 2015). However, the state in
Germany still occupies a dominant, although more distant position in governing the HE sector
(de Boer & Huisman, 2020).
Today NPM seemingly has lost momentum. This is partly due to the failure of overly lofty
expectations, partly to the successful implementation of NPM steering instruments (which
are now taken-for-granted) and partly to the massive expansion of HE in Germany (Scott,
2015; Wolter, 2014) which is manifested in a growing number of HE organisations in general
(with 426 extant universities today), of private HE institutions, students, graduates, staff,
annual state funding, and publications (Dohmen and Krempkow, 2014; Dusdal et al., 2020;
Wohlrabe, Gralka, and Bornmann, 2019). At the same time, the expansion is closely linked
to a broad dynamics of differentiation which resonates with the multiplying of societal
expectations toward HE. Universities face a vast range of already established or new societal
demands (Schimank, 2001; Henke et al., 2017), including gender equality, health promotion,
1298
Bernd Kleimann
open access, internationalization, accountability and others. These expectations circulate in

different value-centred discourses within and outside HE and science. While NPM has
opened the doors for them, they now tend to form a new governance regime in German HE
that is about to replace NPM.
3. Multiple-Value Governance: the Emergence of a New Governance Regime

While NPM seems to have passed its zenith as a reform framework, the contours of a novel
governance regime – even if still somewhat blurred (Klenk and Reiter, 2019) - have begun
to take shape (Jungbauer-Gans, Gottburgsen & Kleimann, 2023). My claim is that the
emergence of this new governance regime is fuelled by the value-related expansion of HE in
the last decade and the ensuing dynamics of differentiation in HE. The new regime’s core
feature of a value-driven governance of HE institutions is fed by several discourses – like
those on the pluralisation of higher education tasks, on accountability and auditing, on a third
academic mission (Henke et al., 2017; Rubens et al., 2017), or on the concept of "public
value" as a substitute for the efficiency-oriented NPM ideal (Broucker,de Wit, and
Verhoeven, 2017). Empirically, it seems undeniable that HE organisations are confronted
with a fast growing number of heterogeneous value expectations (with values as highly
abstract, unquestioned aspects of preferability; Luhmann, 1985). Partly, these value-oriented
expectations are rooted in long-standing societal debates (e.g. on gender equality or scientific
excellence), partly they are embedded in rather recent discussions (about, for instance, open
access). However, each of the discourses can be described as being related to a specific value
proposition whose origins lie in different societal sectors and milieus. Some examples of
these values are: gender equality, equal distribution of educational chances, diversity, health
promotion, citizen science, knowledge transfer, research and teaching excellence, academic
freedom, open science and open access, social responsibility of research and education,
solutions for climate change, the fostering of democratic attitudes and participation, the
inclusion of refugees or non-traditional students, lifelong learning, digitalization,
internationalisation, the inclusion of disabled students, or the implementation of
sustainability in academia.
As many of these aspects are not new, one may ask what is new about the presumably novel
HE governance regime? The answer is: In contrast to NPM, the new regime is characterized
by two specific features:
1) The regime is based on the widespread assumption that the HE system is able to and should
provide the knowledge that is indispensable for society’s coping with the great political,
social, economic, ecological or health-related and even scientific challenges. While the
political sector is expected to deliver binding decisions on all these matters, the HE system
is expected to deliver (incorporated) knowledge that informs these decisions. As the
1299
discourses on the challenges to which HE shall make its contribution are centered around
abstract values, the new governance regime may be called “multiple-value governance”.
2) The new regime is composed of established as well as new actor constellations and
characterized by a new balance of governance mechanisms. While some of these mechanisms
– like status competition in HE (Krücken, 2019; Brunsson and Wedlin 2021) – have been
strengthened before by NPM (de Boer et al., 2017; Hüther and Krücken, 2018), others are
reinforced by the new regime – like novel professional networks, value-oriented cooperation
with new partners outside academia, competition between universities in favor of societal
values, or the establishment of new value-oriented self-governance committees or
hierarchical bodies with representatives of different societal fields.
The resulting mixture and weighting of governance mechanisms enables actors from various
societal areas to play a role in HE governance and to bring forward their interests concerning
academic knowledge production and distribution. Accordingly, a key element of the research
agenda this paper proposes should be the development of an adapted “governance equalizer”
(deBoer et al., 2007) that accounts for this new configuration of HE governance mechanisms.
In this regard it is important to mention that the multiple-value regime is certainly not a
completely new framework. It partly abolishes, partly continues, and partly reinterprets the
former „pure“ NPM regime, thus constituting a bricolage of, on the one hand, NPM features
with its typical values (effiency, effectiveness, subsidiarity) and governance mechanisms
(like competition for scientific excellence) and, on the other, of a multitude of both new and
established values and recalibrated governance mechanisms. While the ideal type of NPM
has underscored the instrumental dimension of HE governance, the multiple-value regime
accentuates the value-relatedness of HE by linking the latter closely to the aforementioned
substantial, often overlapping or even contradicting value propositions.
To implement these values, the new regime adopts the NPM governance toolbox, for instance
using the competition mechanism to promote gender equality or sustainability development.
At them same time, multiple-value governance enriches this „toolbox“ by employing
alternative forms of governance like, as stated above, novel networks, collaboration patterns
(Olechnicka et al., 2019), additional negotiation systems, or new forms of professional or
collegial self-regulation (for instance through new career paths in academia).
4. Towards a Research Agenda

Against this background, the paper intends to corroborate its claim by focusing on some
structural consequences in German HE organisations that are brought about by the new HE
governance regime. While the effects of NPM on the organizational structures of German
universities have been analysed quite thoroughly (Sağırlı, 2015; Bogumil, 2013), the impact
1300
Bernd Kleimann
of the novel regime on university structures is much less investigated. In order to shed some
light on the possible structural consequences, the paper concentrates on a prominent and
promising example in this regard: the role of sustainability as a complex value proposition.
Sustainability is particularly suitable as a case for value-related changes in HE governance
because, on the one hand, it clearly constitutes a value sphere: sustainability includes a broad
spectrum of ecological as well as economic and social values which have been specified by
the United Nations’ 17 sustainable development goals. On the other hand, it is well known
that the sustainability discourse has had an enormous impact on HE internationally as well
as in Germany (Leal Filho et al., 2021, Leal Filho, 2016 and 2018). Hence, an analysis of
how the sustainability discourse has been adopted through changes in organizational
structures of German universities and through shifts of the latters’ relationships to their
environment seems to be a promising possibility to investigate structural effects of the
assumed multiple-value governance.
To this end, the paper draws on the organization approach of sociological systems theory as
an analytical lens (Luhmann, 2018; Kühl, 2021; Kleimann, 2018). According to this theory,
the main organizational structures of HE institutions are goal programmes (i.e. teaching,
research, third mission), conditional programmes (like administrative rules or formal career
paths), communication channels (primarily represented by the hierarchical structure of
universities), personnel, and organisational culture. It is in recourse to these conceptual
dinstictions that the paper asks whether and – if yes – in which way the demand for
sustainable development has been adopted by German HE organisations through changes in
their organizational structures. These structures include
- novel forms of organizational self-display,
- novel study programmes as products in teaching,
- novel goal programmes in research and third mission,
- novel conditional programmes (i.e. administrative rules),
- novel communication channels, and
- novel demands on academic and administrative personnel.
As several empirical observations of German universities show, one can conclude that
sustainability has affected all structural dimensions of HE organisations in Germany (Leal
Filho et al., 2021; Levesque and Wake, 2021). A preliminary analysis of these structural
changes offers several interesting insights. For instance, it is the characteristic of universities
as multidisciplinary and still rather loosely-coupled systems (Weick, 1976) that allows them
to take up value expectations that actually conflict or contradict each other in social practice.
Furthermore, the velocity and extent with which HE organisations were able to address
sustainability issues indicates that they, due to their particular organisational structures,
represent a type of organization which is highly responsive to heterogeneous societal value
1301
expectations. This is why universities have been called “institutions of society” (Stichweh,
2009). Moreover, universities are not only adressees of the sustainability discourse, but
actively contribute to it by shaping the ideas around sustainability through the elucidation of
its meanings, the empirical analysis of its implications in different scientific fields and
through the communication and distribution of sustainability-related findings via teaching,
research, and third mission. In doing so, universities act as institutional entrepreneurs
(DiMaggio, 1988) which actively shape their institutional environment.
One dissemination mechanism for insights on sustainability are university networks. They
are used by universities, for instance, to provide learning opportunities for sustainable
management in areas where such solutions do not exist so far. Thus, the informative ‘strength
of weak ties’ in these networks allows to get to know successful role models and to tap into
bodies of knowledge for sustainable solutions.
Summing up, one can conclude that focusing on sustainability in German universities
provides exemplary evidence of how the assumed multiple-value governance is
implemented. Universities adopt to a varying degree value expectations in their self-
descriptions, in research, teaching, organizational design, decision-making structures,
administrative rules, and personnel. This observation should encourage further research on
the structural implementation of sustainability and of other value propositions in HE
organisations and is supposed to inspire further studies on the features and the international
distribution of the emerging multiple-value governance regime.
References
Banscherus, U., Engel, O., Mindt, A., Spexard, A., & Wolter, A. (eds.). (2015).
Differenzierung im Hochschulsystem: Nationale und internationale Entwicklungen und
Herausforderungen. Bielefeld: Waxmann.
Bogumil, J. (2013). Modernisierung der Universitäten: Umsetzungsstand und Wirkungen
neuer Steuerungsinstrumente. Berlin: Edition Sigma.
Broucker, B./de Wit, K./ Verhoeven, J.C. (2017). Higher Education for Public Value: taking
the debate beyond New Pubic Management. Higher Education Research & Development,
37(2), 227-240.
Brunsson, N. & Wedlin, L. (2021). Construction competition for status: sports and higher
education. In Arora-Jonsson, S., Brunsson, N., Hasse, R., & Lagerström, K. (eds.):
Competition. What It Is and Why It Happens. (93-111). Oxford, Oxford University Press.
de Boer, H. F., & Huisman, J. (2020). Governance Trends in European Higher Education. In
G. Capano & D. S. L. Jarvis (eds.). Convergence and Diversity in the Governance of
Higher Education (1. ed., 333–354). Cambridge: Cambridge University Press.
de Boer, H. F., Enders, J., & Schimank, U. (2007). Comparing Higher Education Governance
Systems in Four European Countries. In N. C. Soguel & P. Jaccard (Hrsg.). Governance
and Performance of Education Systems (35–54). Dordrecht: Springer Netherlands.
1302
Bernd Kleimann
DiMaggio, Paul J. (1988): Interest and Agency in Institutional Theory. In: Lynne G. Zucker
(Hrsg.): Institutional Patterns and Organizations: Culture and Environment. (3-21).
Cambridge MA: Ballinger.
Dohmen, D., & Krempkow, R. (2014). Die Entwicklung der Hochschulfinanzierung—Von
2000 bis 2025. Konrad-Adenauer-Stiftung.
Dusdal, J., Powell, J. J. W., Baker, D. P., Fu, Y. C., Shamekhi, Y., & Stock, M. (2020).
University vs. Research Institute? The Dual Pillars of German Science Production, 1950–
2010. Minerva, 58(3), 319-342. doi:10.1007/s11024-019-09393-2
Henke, J., Pasternack, P., & Schmid, S. (2017). Mission, die dritte: Die Vielfalt jenseits
hochschulischer Forschung und Lehre: Konzept und Kommunikation der Third Mission.
Berlin: Berliner Wissenschafts-Verlag.
Hüther, O., & Krücken, G. (2018). Higher Education in Germany-Recent Developments in
an International Perspective (1st ed. 2018). Springer International Publishing.
Jungbauer-Gans, M., Gottburgsen, A., Kleimann, B. (2023). Wertebasierte Governance in
Hochschule und Wissenschaft. Beiträge zur Hochschulforschung (1), 42-63
Kleimann, B. (2018). (German) Universities as multiple hybrid organizations. Higher
Education 77(6), 1085-1102. https://www.doi.org/10.1007/s10734-018-0321-7
Klenk, T., & Reiter, R. (2019). Post-New Public Management: Reform ideas and their
application in the field of social services. International Review of Administrative
Sciences, 85(1), 3–10.
Krücken, G. (2019). Multiple competitions in higher education: a conceptual approach.
Innovation, 1-19. doi:10.1080/14479338.2019.1684652.
Krücken, G. (2019). Multiple competitions in higher education: a conceptual approach.
Innovation, 1-19. doi:10.1080/14479338.2019.1684652.
Krücken, G., & Meier, F. (2006). Turning the University into an Organizational Actor. In
G.S. Drori, J.W. Meyer, H. Hwang (Eds.). Globalization and Organization: World
Society and Organizational Change (241-257). Oxford: Oxford University Press.
Kühl, S. (2021). Organizations. A Short Introduction. Princeton et al.: Organizational
Dialogue Press.
Leal Filho, W. (ed.) (2016). Forschung für Nachhaltigkeit an deutschen Hochschulen.
Wiesbaden: Springer Spektrum.
Leal Filho, W. (ed.). (2018). Nachhaltigkeit in der Lehre: Eine Herausforderung für
Hochschulen. Wiesbaden: Springer Spektrum.
Leal Filho, W., Will, M., Shiel, C., Paço, A., Farinha, C. S., Orlovic Lovren, V., Avila, L.
V., Platje, J. (Joost), Sharifi, A., Vasconcelos, C. R. P., Fritzen Gomes, B. M., Lange
Salvia, A., Anholon, R., Rampasso, I., Quelhas, O. L. G., & Skouloudis, A. (2021).
Towards a common future: Revising the evolution of university-based sustainability
research literature. International Journal of Sustainable Development & World Ecology,
1–15.
Levesque, V. R., & Wake, C. P. (2021). Organizational change for sustainability education:
A case study of one university’s efforts to create and implement institution-wide
sustainability competencies. International Journal of Sustainability in Higher Education,
22(3), 497–515. https://doi.org/10.1108/IJSHE-09-2019-0285
1303
Luhmann, N. (1985). A Sociological Theory of Law. London ; Boston : Routledge & Kegan
Paul.
Luhmann, N. (2018). Organization and decision. Cambridge University Press.
Olechnicka, A., Ploszaj, A., & Celinska Janowicz, D. (2019). The geography of scientific
collaboration. Abingdon, Oxon: Routledge.
Rubens, A., Spigarelli, F., Cavicchi, A. and Rinaldi, C. (2017). Universities’ third mission
and the entrepreneurial university and the challenges they bring to higher education
institutions. Journal of Enterprising Communities: People and Places in the Global
Economy. 11(3), 354-372. https://doi.org/10.1108/JEC-01-2017-0006.
Sağırlı, Gülay (2015). Die Einführung des Neuen Steuerungsmodells im deutschen
Hochschulsystem. Erklärungsansätze für den Wandel im Management und der
Verwaltung von Hochschulen. Dissertation, Universität zu Köln.
Schimank, U. (2001). Festgefahrene Gemischtwarenläden - Die deutschen Hochschulen als
erfolgreich scheiternde Organisationen. In E. Stölting & U. Schimank (eds.). Die Krise
der Universitäten (S. 223–242). Wiesbaden: VS Verlag für Sozialwissenschaften.
Scott, P. (2015). Expansion, differentiation and modernisation in contemporary higher
education systems. In U. Banscherus, O. Engel, A. Mindt, A. Spexard, & A. Wolter (eds.).
Differenzierung im Hochschulsystem (43-58). Münster: Waxmann.
Stichweh, Rudolf (2009a): Autonomie der Universitäten in Europa und Nordamerika:
Historische und systematische Überlegungen. In: J. Kaube (ed.): Die Illusion der
Exzellenz. Lebenslügen der Wissenschaftspolitik. 38-49. Berlin: Wagenbach.
Weick, K. (1976). Educational organizations as loosely coupled systems. Administrative
Science Quarterly. 21(1), 1-19.
Wissenschaftsrat (2018): Empfehlungen zur Hochschulgovernance. Köln.
https://www.wissenschaftsrat.de/download/archiv/7328-18.html
Wohlrabe, K., Gralka, S., & Bornmann, L. (2019). Zur Effizienz deutscher Universitäten und
deren Entwicklung zwischen 2004 und 2015. ifo Schnelldienst, 72(21), 15-21.
Wolter, A. (2014). Eigendynamik und Irreversiblität der Hochschulexpansion. In U.
Banscherus, M. Bülow-Schramm, K. Himpele, S. Staack, & S. Winter (eds.).
Eigendynamik und Irreversiblität der Hochschulexpansion (19–38). Bielefeld: wbv
Media. DOI: 10.25656/01:10088.
1304
Development of the “Complex Project” course in the transport

engineering education
Gábor Horváth
Department of Transport, University of Győr, Hungary.
Abstract
In the paper we discuss the course history, the basic goals and principles
defined at the beginning, and we talk about its methodology. For the “Complex
project” course the main parts of the lecturer’s job are the detailed
preparation of the project tender, the operative support of the students’ work,
and they also have to perform the mid-semester and end-of-semester
evaluations. Continous progress is controlled by milestones laid out
throughout the program. The effectiveness of the “Complex project” can be
determined based on the previous years, and the future possibilities will also
be examined. Besides elevating the level of education from the students’
perspective, at the same time it also improves the lecturers’ competences.
Keywords: Practical course; intergrated specific professional knowledge;
project tender; teamwork; presentation and Final report; complex evaluation
system.

Development of the “Complex Project” course in the transport engineering education
1. Introduction
The “Complex Project” course is part of the syllabus for the bachelor program for transport
engineering. This course can be taken after completing the general studies, the basic and
specialized professional courses. This methodological course intergrates the theoretical
knowledge gathered in the previous courses and focuses on solving complex transport
engineering problems.
This course has been around for 15 years. Since its introduction it has evolved and has gone
through experiments and changes. However, its upward trajectory is constant and its
usefulness is apparent on higher levels as well.
This paper examines the fine tuning of these educational methodologies from the beginning
stages to the present. It mentions the definitive changes of educational structure, and it
introduces future developmental plans.
2. The general introduction of the “Complex Project” course

In this section we discuss the course history, the basic goals and principles defined at the
beginning, and we talk about its methodology in general.
2.1. Historical overview

Until the 2000s there was no practical course that intergrated the specific professional
knowledge in transport engineering training. Design and operational control tasks only
occurred as side projects in profession-specific courses, without any correlation. These
include track design, schedule design, vehicle scheduling, software, etc.
During the curriculum developments of the new millennium, an idea came about to create a
course that integrates the profession-specific knowledge. For its design we took inspiration
from national and international examples. One of these was the project-based program of the
university in Duisburg where the courses were built on solving a tasks set out at the beginning
of the program. Due to the wider spectrum of transport engineering, the time constraints made
this deep level of integration impossible, but we collected ideas from these principles. On a
national level we found examples in the transport design programs where students worked
on real-life design projects. We were not expecting these types of projects at the outset, but
the ideas gave us a good starting point.
Similarly to the educational system in its enterity, the online learning that came about during
the pandemic required significant methodological developments. Despite the technical
challenges, positive changes happened, e.g in the methodology of task allocation that will be
explained later.
1306
Gábor Horváth
2.2. The goal and general methodology of the course

As we outlined above, the goal of this course is:
• To develop the ability to use the theoretical knowledge in practice

• To recognize the interference of design processes and methods in complex tasks
• To be able to follow and carry out entire design processes
• To highlight the importance of teamwork and performace-based benefits
• To prepare for the autonomous planning of the thesis by going through the research
methodology together, and to also prepare for the thesis debate by practicing
presentation techniques.
Generally design tasks are created to be worked on by student groups for an entire semester.
Groups are made up of 3-4 students. In our experience teamwork is not realized in groups
with less than 3 people, and groups with over 4 people leave little room for each individual
student which gives way for potential passivity.
The topics of the tasks are based on the two main areas of transport, road (local public
transport) and rail, and students can choose based on their interest. Each topic has a
responsible lecturer as a team lead who coordinates the groups’ work. The in-person “contact
lessons” are designed to be 4 hours/week, so 56 hours over the 14 weeks of the semester. For
the successful completion of the project, students also need to work on the project
individually which makes up about 50% of the overall effort.
The syllabus for the semester consists of consultations, project work outside of the classroom,
and presentations.
3. Details of the teaching method

For the “Complex project” course the main parts of the lecturer’s job are the detailed
preparation of the project tender, the operative support of the students’ work, and they also
have to perform the mid-semester and end-of-semester evaluations.
3.1. The structure of project tenders

Over the years we worked on various project topics, but the structure of the project tenders
was standardized in the early years. It’s important that the design process should focus on the
complex effects of problem-solving, meaning that beside traffic design, the projects involve
technical, legal, economic, environmental and sociological aspects.
At the beginning we were selecting from projects completed by the department and we
updated or recreated them. Later, we assigned tasks recommended by fellow lecturers that
focused on solving traffic-related problems. Sometimes even students recommended project
1307
topics. These resulted in a wide variety of papers but it also meant that different groups had
different levels of difficulty, so the comparisons and evaluations were not fair. To solve this
issue we decided to assign the same tasks to multiple groups which resulted in a healthy
competitive environment but also had the drawback of groups being able to copy sections
from each other. The pandemic brought about a solution for this matter. Due to the
restrictions, in-person planning meetings had to be minimized. We designed a framework
task for both road and rail and put students in groups based on their hometowns, so they could
work on them using their regional data. This meant the solutions were comparable due to
sub-tasks being the same, but the locations examined were different. This proved to be a great
solution so we kept it after COVID.
Upon the allocation of project tasks we provided detailed descriptions to students that gave
clear boundaries for the research. The main elements of the tenders:
• Type of transport (road/rail)

• Project title (recently it’s been a general title, later specified by the location)
• Precedents (motivation)
• Project goal
• Main points of development
• Mandatory pieces of content
• Recommended analytical methodology
• Main evaluation criteria.
These provide a good support and basis for the project development process for the semester.
3.2. Operative project lead

Each group has a lecturer as the project lead. They do not take part in the project development,
but they keep a close eye on their groups and provide occasional guidance.
Project leads take part in the introduction lesson where the tasks are explained, and they help
with orientation and group dynamics by providing detailed information when needed.
Project leads help their group understand the task and draw up a plan for the entire semester.
Based on the members’ competencies they discuss the strong points of each person and select
which part of the task they would be responsible for. They divide tasks both horizontally (can
be completed in parallel) and vertically (interlinked and cannot be completed in parallel).
They list the required resources such as data compilations, maps, national and international
literature, published plans and papers, etc as well as the contacts of institutions, associations
and scholars they need to involve. They also take stock of the required technical assets such
as surveys, traffic countings, recordings, interviews, IT requirements, modelling and
1308
Gábor Horváth
simulation software. Considering the succession of sub-tasks, they create schedule with a
Gannt diagram as the basis of project management.
The students start their work based on the project plan which usually begins with location
visits and bibliography research.
The usual structure of the design process consists of:
• Understanding the background

• Uncovering the problem
• Examining the reasons
• Exploring the possible solutions (analyzing existing designs, looking for example, etc.)
• Examining their applicability
• Developing individual solutions
• Rating versions
• Performing ROI and impact assessment.
During consultations students report on the completed tasks, rate their usefulness with the
lecturer’s help, and determine the next steps.
Project leads invite competent external consultants beforehand to gain access to corporate
information that could help students with the project.
The project lead also provides assistance with the oral presentations. With those, it is
important that:
• All students participate in the preparations

• They keep to the time limitations (but also use up the available time)
• They use the correct terminology
• They have an awareness about their gestures
• They pay attention to grammer and clarity.
Main points of the supporting Power Point presentations:
• Their layout design should be suitable for the topic

• The number of slides should be appropriate for the time limitations
• Visible fonts and figures
• Contrasting elements
• Animations (but not unnecessary ‘fireworks’)
• Practicing pointing techniques (with pointer, cursor).
The spoken and written content has to harmonize with one another (the presenter has to talk
about what’s on the figures and the written text has to support the spoken content). The
project lead assists with the preparation of the written project paper. The paper has a set
1309
length limitation and mandatory content elements. Its format is identical to the regular thesis
format so that students can learn the formatting expectations early. The general matrix of the
semester schedule is shown on Figure 1.
Figure 1. Semester matrix.
3.3. The evaluation process

Continous progress is controlled by milestones laid out throughout the program. The so-
called progress reports (separate for road and rail) presented by the groups at ¼ and ¾ of the
semester provide an update on the progress of the project. These are evaluated by all project
leads. At the halfway point, all groups present a so-called Intermediate report in front of
everybody. Groups present their current results with a Power Point presentation that serves
as a rehearsal for the semester-closing presentation. Along with the project leads, students
from other groups can also give constructive feedback. Besides points set forth in 3.2,
professional opinions and individual requests can be added.
The course requirement is divided into 2 parts:
• Preparation and hand-in of the project paper by the required due date
• Presentation of the results at the Final report, and persuading the client.
The final evaluation will be determined by a ratio of 60% paper and 40% presentation. Both
have a complex evaluation system with points given for certain sections. The format of the
papers is checked by the course lead, 10 points can be given. The project lead evaluates the
content and can give 50 points. The Final report is evaluated by all present project leads.
They can give up to 40 points, and the average of all points given will be taken into account.
The sum of the above give the base of the point for each group, so maximum 10+50+40=100
points. This is multiplied by the number of students in the group (3 or 4), and that is number
that will by divided among the student. The points give to each person will be determined by
the group. When they hand in the paper, each member also hands in a evaluation sheet where
1310
Gábor Horváth
they rate their team member’s work throughout the semester by dividing 100% among them.
For each individual the average of the received % will be considered (to avoid cliques the
project lead can overrule ratios). The points given to each student will be determined by
distributing the total score of the group by the % received. These pre-determined criteria will
give the final mark to the students.
If a group hands in their papers after the deadline, a “penalty” is taken by deducting x% of
the points.
4. Evaluating the course

The effectiveness of the “Complex project” can be determined based on the previous years,
and the future possibilities can also be examined.
4.1. Statistics
During the 15 years between 2008 and 2022, 562 students worked on 148 projects in 148
groups. The evolution of participants and topics by year can be seen on Figure 2.
Figure 2. Number of students and projects 2008-2022.
Figure 3. Grades and course avarages 2008-2022.
1311
Due to the practical nature of the course, the final grades and group averages are higher than
for theoretical courses. In fact, the only ones who failed the class were the no-shows. The
distribution timeline is shown on Figure 3. Over the course of 15 years the course average
has been 4,20 (on a scale of 1 to 5) which show that the student were well prepared and
enthusiastic.
4.2. Developments moving forward

It is apparement that the methodology is continuously improving. As a next step, along with
tasks offered by our department, transport corporations will send us project plans. Students
can apply for these and they will be available for 2-3 groups. We will provide a high level of
guidance from the project leads and the groups will compete with each other to come up with
alternative solutions. Lecturers will give presentations on the procedures that are deemed
highly useful for the potential solutions. Students will separately present their progress every
3 weeks. The client company will take part in the semester-ending evaluation, and the group
they find the best will receive a prize.
We have developed a continuation for this bachelor’s course for the master’s program.
Students will already be familiar with the process of project development, so they will learn
how to lead and coordinate projects.
5. Conclusion
This course is significantly facilitates the transition between theoretical education and
practical application.
It provides a tool for acquiring the methodology that is required for understanding, creating
and debating the thesis for each individual person.
Compared the results of the students at the final exam and thesis, we found that the marks
are significantly better than before the introduction of the subject “Complex project”.
It lays a foundation for the master’s program and improves the professional recognition on
the program.
Besides elevating the level of education from the students’ perspective, at the same time it
also improves the lecturers’ competences.
References
Project tenders 2008-2022 (Department of Transport, University of Győr).
Sylabus of courses 2008-2022 (Department of Transport, University of Győr).
Statistics of courses 2008-2022 (Department of Transport, University of Győr).
1312
Digital identity and body identity: the mutation of the university

environment in pandemic times
Rosanna Perrone, Lucia Pallonetto, Carmen Palumbo
Department of Humanities, Philosophy and Education, University of Salerno, Salerno,
Italy.
Abstract
The pandemic period led universities to face new needs and to reinvent and
readapt their educational activities in online mode to make them available and
continuous. The consideration is about the effects produced in university
students by the pandemic conditions and the exclusion of bodily experience in
the real world. A study was led on a group of 405 subjects in order to evaluate
their emotional, conflictual and social state and the possible emergence of
obstructive behaviour in relation to themselves and others. Thanks to this
study, it’s clear that the bodily experience remains irreplaceable, and in a
moment in which we have been deprived of the body and its resources, we need
to reconsider the relationship between person and his body, re-evaluating it as
a real value and relational foundation, of its innate potential as a means of
learning.
Keywords: Covid-19; corporeality; higher education; idea inventory; body
identity; digital identity.

Digital identity and body identity: the mutation of the university environment in pandemic times
1. Introduction
In the last three years, the global pandemic situation required the academic sphere to radically
and immediately reconsider innovative teaching approaches and tools, in order to ensure the
normal continuation of educational activities.
In the academic field, as in schools, there were radical changes and there was a shift from
face-to-face teaching to distance teaching. In this period out of the ordinary, there has been a
lack of social experiential teaching opportunities, which are fundamental for professional
training, as well as experiences of a motor, playful, sporting and emotional nature, which are
also to be considered as activities at the basis of personal growth.
In the academic environment, the appreciation of diversity and an inclusive spirit create
opportunities for community and cooperation. In pandemic times the bodily identity was put
aside (Francesconi, 2015), to leave room for digital identity, which was necessary to take part
in a didactic relationship, but the concept of corporeity is not secondary, nor is the promotion
of health in universities, which in addition to promoting higher education, should guarantee
quality and well-being in relations with teachers and among students. Therefore, a detailed
examination of the state of predisposition of students to progressively re-appropriate the
potential of the body (Palumbo, 2013), which has been marginalized, is needed. If the
educational relationship can be safeguarded through mediatization, it is worth trying to
explore the abilities of the body and especially the corporeity, understood as a hub of multiple
knowledge (Gamelli, 2011). Consequently, it is the case to introduce a pedagogy of
corporeity, which can transfer in the various educational fields those basic principles an
education to the motor movement in its different forms, which lead to a body culture
conceived as bodily awareness.
It is therefore necessary to reflect on how the university environment has been invested in a
pandemic period of change, in which the student has been forced in the confined space of his
home and his room to face a new concept of self-perception (Gibson, 1979) and self-efficacy,
on how each individual student had to try on his own to take the bodily self, gradually
replaced by a digital self (Lo Piccolo, 2019). As a result of this pandemic period, the
inevitable repercussions on the increased burden of both emotional and physical stress due
to the continuous restrictions caused by the lockdown periods, cannot be ignored. It is
therefore increasingly common to experience stress disorders, and university institutions
cannot fail to take charge of this, because, from an inclusive perspective it is the task of
university institutions to ensure a learning environment that meets both the needs and
expectations of each and every individual.
1314
2. Research
2.1. Objective
The study aims to promote mental health and psychophysical well-being in the academic
environment by assessing perceived self-efficacy in managing internal and external conflicts,
as well as an interpersonal and intrapersonal analysis on a group of subjects attending the
University of Salerno.
2.2. Sample
The subjects observed within the framework of the research are students at the University of
Salerno, (Southern Italy) who were identified in a convenience sample of 405 students
(M=124; F=281) aged between 20 and 25.
2.3. Instrument
The Idea inventory questionnaire by Kassinove et al. (1977) was used as a tool for the
evaluation of cognitive and motor dysfunctional aspects. The test consists of 19 items
(reduced version), on a Likert scale, with scores from 1 to 4. The student is asked to answer
and provide his or her degree of agreement or disagreement with the items indicated.
2.4. Data analysis and results

The Idea Inventory, in this survey, is intended to detect the level of changes implemented at
the mental level in response to an adaptation or non-adaptation in one's context, in this
specific case the academic context. The data were pre-screened to determine the accuracy of
data entry. Valid data were classified into a descriptive statistic (SPSS software) and a
frequency calculation was performed. The significant data are shown in the table (fig 1.)
Thus, the analysis showed that: in Item 22, 45.7% of the students answered that they are
totally agree. This result manifests a lack of resilience to the ability to try again after a failed
attempt (Vaccarelli, 2016), and to not finding comfort in the other as well as having to adapt
quickly to new situations and put oneself in a position to actively react to the changes; in Item
4, 33.3% of the students answered that they totally agree. For almost two years we have been
experiencing a different reality, the result of significant changes that cannot be ignored, of
new pedagogical realities that can be tangible and valuable, and think about themes such as
intercorporeality (Rinaldi, 2003), taking into account the teacher/learner relationship; this
relationship is based on wide aspects concerning eye contact, but also gestures, mindfulness,
synchrony, what goes beyond eye contact, voice, gestures, rhythm, aspects that are
imperceptible in distance mode; in item 8, 45.9% of the students answered that they totally
agree. In this case, the students accumulated more anxiety and less problem solving skills
with regard to their lives, all caused by an inability to relate and open up to the world; in item
9, we note the 29.1% of studens are totally agree. This result can be traced back to the lack
1315
of comparison with the other, and the uninterrupted and limiting interface with the only
display of the mobile phone and computer; in Item 10, 48.6% of the students answered that
they are totally agree; the ability to manage difficulties represents an inherent human skill,
but if not properly exercised, it can lose its effectiveness and cause frustration and discomfort.
In Item 12, 38.5% of the students answered they totally agree, this result shows the
educational emergency that we are going through. Teachers, researchers and the students
themselves have a duty to commit themselves to a renewal of teaching actions and to provide
tools that guarantee the possibility of a constant relationship between bodies to arrive at a
new physical presence as a "still" point in teaching. That is, the presence of a pedagogy that
gives space to new, rather than new, rediscovered epistemological tools: corporeity and
intercorporeity.
1316
ITEM 1 – It is essential to obtain the esteem ITEM 2 –

and affection of the people who are It's terrible when things don't go as they should. F %
important to me F % Totally agree 185 45,7
Totally agree 215 53,1 Quite agree 106 26,2
Quite agree 121 29,9 Quite disagree 73 18,0
Quite disagree 26 6,4 completely 41 10,1
completely 43 10,6 disagree
disagree Total 405 100,
Total 405 100, 0
0
ITEM 3 - It is easier to avoid responsibilities ITEM 4 - When I'm in a difficult situation, I
and commitments than to face them F % easily give up if there is no one to help me out F %
Totally agree 85 21,0 Totally agree 135 33,3
Quite agree 46 11,4 Quite agree 51 12,6
Quite disagree 148 36,5 Quite disagree 148 36,5
completely 126 31,1 completely 71 17,5
disagree disagree
Total 405 100, Total 405 100,
0 0
ITEM 5 - I'm afraid I don't know how to deal ITEM 6 - I take it out if others don't like my
with people in authority in the best way F % appearance or the way I dress F %
disagree disagree
Total 405 100, Total 405 100,
0 0
ITEM 7 - A person of value must always be

competent and comfortable, whatever he does F % ITEM 8 - I get angry if I don't achieve my goals F %
disagree disagree
Total 405 100, Total 405 100,
0 0
ITEM 9 - My unhappiness is mainly caused ITEM 10 - We always need to lean on someone

by the events or people I deal with and have stronger than us F %
no chance to control it F % Totally agree 35 8,6
Total 405 100, 0
0
1317
ITEM 11 - It is a terrible thing not to be able ITEM 12 - When I arrive at a party I feel
to find the perfect solution to our problems F % very bad if no one comes to greet me F %
disagree disagree
Total 405 100, Total 405 100,
0 0
ITEM 13 - I feel like a worthless person when ITEM 14 - I get angry if my opinions are not
I do not succeed well in my work or study F % accepted F %
disagree disagree
Total 405 100, Total 405 100,
0 0
ITEM 15 - I feel sad and rejected when others ITEM 16 - I avoid situations where I have to
neglect me F % make choices because it makes me nervous F %
disagree disagree
Total 405 100, Total 405 100,
0 0
ITEM 17 - I get nervous and need to be ITEM 18 - It's the past that influences my life
helped when I face difficult commitments and and I can't do anything to change it F %
responsibilities F % Totally agree 38 9,4
Total 405 100, 0
0
ITEM 19 - I worry a lot if I can't find the
right solution for my commitments F %
Totally agree 82 20,2
Quite agree 207 51,1
Quite disagree 85 21,0
completely 31 7,7
disagree
Total 405 100,
0
Figure 1. Results of Idea Inventory Test.
1318
3. Discussion and conclusions

This study aims to understand how the possible onset of dysfunctional attitudes can influence
the effective and complete absence of bodily communication. Therefore, it is necessary to
invest in the teacher, so that he or she can grow with and through the body, with the
implementation of transformative and innovative teaching practices as an opportunity for
improvement. Within the university contexts, there is a plurality of epistemologies, and to be
able to imagine a confluent path between these systematically means rethinking not didactics
itself, but the ways in which the university system considers didactics to be central, and how
didactics, as a science of interaction, can be a 'significant opportunity for students. In the light
of the latest studies and directions, it is essential to rethink corporeity not as a dissociating,
secondary element, separated from the rest of education. On the contrary, the Pedagogy of
the Body arises, revisiting the current educational contexts, in which corporeity is scarcely
involved, and proposes instead a greater interconnection and scientific reflection on the
formative and educational value of the body intended as a mediator capable of giving
meaning to the educational experience (Sibilio, 2011). The pedagogy of the body is an
educational and training approach that aims to enhance and strengthen the role of corporeity
in learning contexts. In this perspective, it is necessary to consider a more corporeal scenario,
in which posture, eye contact and the correct use of verbal and non-verbal language are used.
The body can and must be recognised as the primary articulation of all learning, and to do so
it is appropriate to look at corporeity with construction, in order to create a new operational
dimension, aimed at reaching significant dimensions involving the emotional and affective
life, and to be able to explore the body's potential in training processes. Indeed, it can be
stated that «the body plays a crucial role in the development of cognitive abilities» (Kyselo
& Di Paolo, 2015 p.520). The profound crisis experienced during the pandemic period seems
to have affected the self-perception and selfefficacy sphere (Bandura,1997), leading to
continual reflections on the future and eliciting continual questions on how to make up for
lacks and absences caused by virtual teaching. The solution might be to be able to conceive
of the body in a vision of centrality and not as an orthosis of the mind, so as to be able to
understand the real peculiarities of corporeity as a unity and crossroads of multiple learning.
Reflecting on the statement I am my body (Marcel, 1951), then in a future perspective,
teaching strategies should focus on corporeity by moving towards the valorization of
educational practices that offer the possibility of discovering the body's potential by means
of its expressive resources (Berthoz, 2011). The body has for too long been conceptually kept
at the margins of learning processes, and thus needs continuous redemption as part of an
entity. Our body, therefore, is much more than a window on the world, it is a reality, a field
for receiving data of all kinds that noisily pours in, but it is certain that every single piece of
information, therefore, afference will not be lost but somehow somatically incorporated and
reused to face new educational and... life challenges.
1319
References
Bandura, A. (1997). Autoefficacia. Trento: Erickson, 2001.
Berthoz, A. (2011). La semplessità. Torino: Codice.
Francesconi, D. (2015). Pedagogy and cognitive neurosciences in dialog. The example of
bodily experience. Formazione & Insegnamento, 9(1), 179–184. Retrieved from
https://ojs.pensamultimedia.it/index.php/siref/article/view/1150
Gamelli, I. (2011). Pedagogia del corpo. Milano: Raffaello Cortina.
Gibson, E. J. (1979). The ecological approach to visual perception. Boston: Houghton
Mifflin.
Lo Piccolo, A. (2019). L’Università tra apprendimento e occupabilità. Education Sciences &
Society. Open Access, 10 (2). Retrieved from
https://journals.francoangeli.it/index.php/ess/article/view/8669
Marcel, G. (1951), Le mystère de l’être, II. Paris: Aubier
Kyselo, M. & Di Paolo, E. (2015). Locked-in syndrome: a challenge for embodied cognitive
science. Phenom Cogn Sci 14, 517–542 . https://doi.org/10.1007/s11097-013-9344-9
Kassinove, H., Crisci, R., & Tiegerman, S. (1997). Developmental trends in rational thinking:
implications for rational emotive school mental health programs. Jour- nal of Community
Psychology, 5.
Palumbo, C. (2013). La danza-educativa. Dimensioni formative e prospettive educative.
Roma: Anicia.
Rinaldi, C. (2003). Narrazione, intercorporeità e riconoscimento in Danilo Dolci. In s.
costantino (a cura di), Raccontare Danilo Dolci. L'immaginazione sociologica, il
sottosviluppo, la costruzione della società civile (pp. 117-131). ROMA : EDITORI
RIUNITI.
Sibilio, M. (2013). La didattica semplessa. Napoli: Liguori Editore. Snyder, C. R. (2001).
Coping. What works for what. New York: Norton.
Vaccarelli, A. (2016). Le prove della vita. Promuovere la resilienza nella relazione educativa:
FrancoAngeli.
1320
Reflective practice to bridge the theory-gap practice in Human-

Computer interaction classes
Moretlo Tlale-Mkhize1, Janet Liebenberg2
1
Department of Information Technology, Central University of Technology, Free State,
South Africa, 2School of Computer Science and Information Systems, North-West
University, South Africa.
Abstract
There is a need to incorporate reflective practice in students’ studies at higher
education institutions to bridge the perceived theory-practice gap in Human-
Computer Interaction classes. Reflective practice as an approach is employed
for students to make connections between theory and practice. Reflective
practice allows students to evaluate what they have learned through practice.
The action research methodology was used to facilitate the research. All
students who completed the assignment were invited to participate in the study,
and since it was voluntary, 24 students took part. Using reflective sheets, the
students were able to evaluate their current actions to improve in the future.
Data were analyzed using conventional content analysis to interpret the data.
The results indicated that the students were able to reflect on their work and
new ideas they can incorporate into their future work. The relevance of
reflective practice to bridge the theory-practice gap is highlighted.
Keywords: Reflective practice; theory-practice gap; higher education;
Human-Computer interaction.

Reflective practice to bridge the theory-gap practice in Human-Computer interaction classes
1. Introduction
There is a common belief that there is a gap between the Human-Computer Interaction (HCI)
theory and its application. The perceived theory-practice gap is also visible in HCI classes.
Students are faced with the problem of a lack of time allocated for practical work to bridge
the perceived theory-practice gap (Pittarello & Pellegrini, 2017). In HCI education, the
usability and user experience of interactive systems and products are explored, understood,
and improved (Abdelnour-Nocera, Michaelides, Austin, & Modi, 2012). Case studies are
commonly used to teach students how to apply the theory they've learned (Collazos &
Merchan, 2015). RP was identified as the best theory to guide the intervention needed to
assist students to bridge the perceived theory-gap practice. In order to effectively assess what
students have learned from practice, reflective practice is an essential component of reflective
learning (Chaffey, de Leeuw, & Finnigan, 2012). The reflective practice process allows
students to apply theory to practice (Colomer, Serra, Cañabate, & Bubnys, 2020).
Action research has been chosen as the suitable methodology to facilitate the process of RP.
The purpose of AR is to improve the performance of lecturers and the learning experience of
their students (Efron & Ravid, 2019). In order to improve current practices, AR process
participants need to engage in self-reflection and question some interventions.
The aim of the present study was carried out to bridge the theory-practice gap by using RP
in HCI classes. Data collected revealed that the use of RP is relevant for bridging the theory-
gap practice.
2. Methodology
Students enrolled in the second year of an IT program who have registered for the graphical
user interface design module that focuses on Human-Computer Interaction participated in
this study. They were given an assignment as part of the formative assessment. All students
who completed the assignment were invited to participate in the study, but because
participation was voluntary, only 24 students out of 183 participated. Participants completed
a reflective sheet to reflect on their process during the completion of the assignment.
Action research (AR) is an orientation to knowledge creation that arises in the context of
practice and requires researchers to work with practitioners (Huang, 2010). In its initial
formulation, ‘action research’ was defined as a method that enabled theories produced by the
social sciences to be applied in practice and tested based on their practical effectiveness (Carr,
2006). In this research context, students will be involved in projects that will enable them to
learn by practising, in other words, they will be active participants who are going to apply
theory in their practice. The five iterative phases of action research as illustrated in Figure 1
1322
(Baskerville, 1999) are diagnosing, action planning, action taking, evaluating, and specifying
learning.
Figure 1: Action research process adapted from Baskerville (1999).
2.1. Diagnosis phase

During the diagnosis phase, the lecturer/ researcher discovered that the students were making
little use of theory when completing the practical work. The problem identified created some
barriers for students to learn the design principles in order to apply them during their practical
assignments. This also revealed that the students are unable to link the theory learned and the
practical. The performance of the students on theory assessments was better as compared to
the practical assessments. The students studied theory to pass the main assessments and
practical assessments were only formative and no summative assessments were done. The
problem identified from this stage will be used to plan the action that will be undertaken for
the cycle.
2.2. Action planning phase

In order to conduct an effective intervention, it was necessary to first identify and evaluate
the problem. The intervention consists of a plan of action aimed to improve a participant's
attitude toward improving their skills. The possible actions to assist with the problem
identified in the diagnosis phase from the literature are presented in Table 1.
1323
Table 1: Possible interventions.
Recommendations from literature Possible intervention
Students complete a new task through hands-on Students will have to complete
experience (Konak, Clark, & Nasereddin, 2014). hands-on work as part of their
assignment. Students will be
required to do practical work as part
of the assignment
Describe what went well and the problems they The reflective sheet will incorporate
encountered, as well as where they will need to improve the questions:
(Schultz, McEwen, & Griffiths, 2016).
During the course of completing the
assignment, ask the students what
challenges they encountered,
Ask students how they solved the

problems encountered,
Obtain their feedback on how they

would improve their work if they
had more time.
Assignment grading standards are developed properly Students work will be evaluated
(Roldan et al., 2020). based on the use of a rubric
throughout grading.
The work should be communicated

before it is submitted.
The reflection process follows after students completed Students are invited to reflect on the
their work (Konak et al., 2014). completed work
Participants will be asked to reflect

after completing the work.
Each participant will be given a

reflective sheet to complete.
The students attend the theory classes and practical classes thereafter. Using the diagnosis
results, an action plan will be developed to alleviate the identified problem in order to develop
students' skills.
1324
2.3. Action taking phase

During the action taking phase, the next step was to give students a practical assignment to
complete as part of reflective practice. The summary of the actions taken based on the
planning phase is presented in Table 2.
Table 2: Interventions implemented.
Recommendations from Literature Interventions
Students complete a new task through hands-on • Students completed hands-on work
experience (Konak et al., 2014). as part of their assignment.
• Students were required to do
practical work as part of the
assignment
Describe what went well and the problems they The reflective sheet incorporated the
encountered, as well as where they will need to following questions:
improve (Schultz et al., 2016).
• During the course of completing
the assignment, the students were
asked what challenges they
encountered,
• students were asked how they
solved the problems encountered,
• Feedback was obtained on how
they would improve their work if
they had more time.
Assignment grading standards are developed properly • Students' work was evaluated
(Roldan et al., 2020). based on the use of a rubric
throughout grading.
• The work was communicated in
class before it was submitted to
clarify misunderstandings.
The reflection process follows after students completed • Students were invited to reflect on
their work (Konak et al., 2014). the completed work
• Participants were asked to reflect
after completing the work.
• Each participant was given a
reflective sheet to complete.
1325
2.4. Evaluation phase

The data were analyzed using conventional content analysis. Table 3 gives the summary of
the process undertaken to analyze the data.
Table 3: Content analysis process adapted from Zhang and Wildemuth (2009)
Process Application in this study

1. Preparing the data The collected data was available in written form
for use in Atlas.ti.
2. Defining the unit of analysis Conventional content analysis was used.
3. Developing categories Codes that contribute to the same meaning or

context are categorized.
4. Testing the coding scheme on a sample Data from four participants were coded.
5. Coding all the text Coding each text entailed assigning a code to it.
6. Assessing consistency Errors were thoroughly checked, and updates

were made as needed by going over the coding
twice.
7. Drawing conclusions from the data The key categories have been identified and
discussed.
2.5. Specify learning phase

Two categories derived from the data collected for this study will be discussed in this paper.
The challenging parts category and problem-solving category are discussed:
Challenging parts category
In accordance with the challenges encountered, participants highlighted the difficult aspects
of the assignment that they encountered while completing it. The question was designed to
determine whether or not the participants could analyze their progress and identify where
they went wrong. The difficulties that the participants encountered were nearly identical
because they involved understanding the assignment's requirements. Among the challenges
listed were the following:
• problem with understanding the requirements;

• challenge with knowing what the banner is;
• difficulty with knowing where to start; and
• experiencing the work for the first time.
One of the most difficult aspects of reflective practice was revealing how participants acted
when they first encountered the assignment. In this case, reflective practice aided in
1326
determining those other students may be unprepared to complete an assignment. The results
revealed that, in order to engage in reflective practice effectively, you must first thoroughly
understand the assignment's requirements.
Problem-solving category
The problem-solving question sought to ascertain what the participants did to overcome the
difficulties they encountered throughout the difficult sections of the assignment. The
participants reflected on their actions as well as their coping strategies. These included asking
questions, seeking additional assistance, and conducting Internet research. The following is
a summary of actions taken:
• Using the internet and YouTube;

• Contacting the lecturer for clarification;
• Studying theory to understand;
• Finding answers in their study materials; and
• Carefully reading instructions.
Current and prospective students will be encouraged to learn the various strategies that will
help them complete an assignment successfully. Students nowadays face information
overload due to the abundance of information available on the internet, making it difficult for
them to choose appropriate content for their assignments.
3. Discussion
Participants were asked to identify the challenging parts they encountered while they were
completing the assignment. The purpose of this question was to determine whether the work
presented a challenge to the students. The findings indicate that when participants were
completing the assignment, they encountered problems related to the processes and methods
involved. Understanding what is required is crucial since it allows students to focus on the
correct strategy when answering the questions. This statement is supported by Cerdán, Pérez,
Vidal-Abarca, and Rouet (2019) who assert that students need to understand what they have
been asked.
The goal of the problem-solving question was to find out how the participants overcame the
challenges they mentioned. Participants were asked what steps they took to solve the
problem. This allowed participants to reflect on the steps they took to overcome the problems
they encountered. The participants stated that they solved the problems using the theory
work. If students are unable to contact the lecturer, they should read the instructions several
times to ensure that they understand the question.
1327
4. Conclusion
Students will be asked to watch the videos provided again before submitting their work to
ensure that they understand the examples of work relevant to what they are required to
complete in the future. In order to effectively complete their assignments, students will be
encouraged to master a variety of strategies. This paper has revealed that it is possible to
bridge the theory-practice gap in HCI classes by increasing student engagement and
providing activities that encourage hands-on exploration and experimentation. It is essential
to incorporate more interactive activities and discussions into classes so that students are able
to gain a deeper understanding of the material as well as apply it to real-life situations.
Furthermore, instructors should provide resources and activities that cater to the different
learning styles of students. It is through these strategies that the students will be able to gain
a deeper understanding of HCI theory and practice, resulting in a more successful academic
experience. This paper has detailed how to bridge the theory-practice gap in HCI classes.
References
Abdelnour-Nocera, J., Michaelides, M., Austin, A., & Modi, S. (2012). An intercultural study
of HCI education experience and representation. Paper presented at the Proceedings of
the 4th international conference on Intercultural Collaboration.
Baskerville, R. L. (1999). Investigating information systems with action research.
Communications of the association for information systems, 2, 1-32.
Carr, W. (2006). Philosophy, methodology and action research. Journal of philosophy of
education, 40(4), 421-435.
Cerdán, R., Pérez, A., Vidal-Abarca, E., & Rouet, J.-F. (2019). To answer questions from
text, one has to understand what the question is asking: differential effects of question
aids as a function of comprehension skill. Reading and Writing, 32(8), 2111-2124.
Chaffey, L. J., de Leeuw, E. J. J., & Finnigan, G. A. (2012). Facilitating students' reflective
practice in a medical course: literature review. Education for Health, 25(3), 198.
Collazos, C. A., & Merchan, L. (2015). Human-computer interaction in Colombia: bridging
the gap between education and industry. IT Professional, 17(1), 5-9.
Colomer, J., Serra, T., Cañabate, D., & Bubnys, R. (2020). Reflective learning in higher
education: Active methodologies for transformative practices. In (Vol. 12, pp. 3827):
Multidisciplinary Digital Publishing Institute.
Efron, S. E., & Ravid, R. (2019). Action research in education: A practical guide: Guilford
Press.
Huang, H. B. (2010). What is good action research. Action Research, 8(1), 93-109.
Konak, A., Clark, T. K., & Nasereddin, M. (2014). Using Kolb's experiential learning cycle
to improve student learning in virtual computer laboratories. Computers & Education, 72,
11-22.
1328
Pittarello, F., & Pellegrini, T. (2017). HCI and education: a blended design experience.
Multimedia tools and applications, 76(4), 4895-4923.
Roldan, W., Gao, X., Hishikawa, A. M., Ku, T., Li, Z., Zhang, E., . . . Yip, J. (2020).
Opportunities and challenges in involving users in project-based HCI education. Paper
presented at the Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems.
Schultz, K., McEwen, L., & Griffiths, J. (2016). Applying Kolb’s learning cycle to
competency-based residency education. Academic Medicine, 91(2), 284.
Zhang, Y., & Wildemuth, B. M. (2009). Qualitative analysis of content. Applications of
social research methods to questions in information and library science, 308, 319.
1329
The use of TAM in evaluating the effectiveness of network

simulation tools in internet technologies subject
Dina Moloja
Department of Information Technology, Central University of Technology, South Africa.
Abstract
Technology has immensely influenced the educational space around the globe
by using educational tools to assist students with practical work. This became
evident when higher learning institutions reached a standstill due to the
COVID-19 pandemic. However, the effectiveness of these educational tools in
teaching and learning remains unknown. This research aims to use the
technology acceptance model to guide the analysis of the effectiveness of a
network simulator called a packet tracer. This study was conducted in
November 2022 with 82 Internet technology students. A quantitative method
was used in this study. Data were collected through observations and survey
questionnaires. Cross-tabulation was used to analyze the data. The results
revealed that Packet Tracer, as a virtual learning tool, has proven to be an
effective software for supporting the teaching and learning of Internet
technology students. This contributes to the body of knowledge and technology
in the education discipline.
Keywords: Education; evaluation; Packet Tracer; TAM; virtual learning;
technology.

The use of TAM in evaluating the effectiveness of network simulation tools
1. Introduction
The shift in technology has impacted the socioeconomic world; this simply means that Higher
Learning Institutions (HLIs) have also been impacted (Štrbo,2021). Higher learning
Institutions face various challenges during the teaching and learning of computer-related
subjects (Ambiyar, Yondri, Irfan, Putri, Zaus, & Islami, 2019; Sudarsana, Armaeni, Sudrajat,
Abdullah, Satria, Saddhono & Ekalestari, 2019), causing both students and teachers to
participate and interact less in the learning process. This was specifically observed in Internet
Technologies (a computer networking subject). Some of the impediments encountered in the
subject include (1) the limitations of network devices, so students have to take turns in a
group practice, causing some students not to participate; (2) the high price of network devices
needed for practice; and (3) the practice process should take considerable time, because
students must physically configure the network, in addition to the minimal amount of practice
equipment.
Teachers can now use virtual learning technology (V-learning) to overcome problems
experienced by students. As a result, V-learning can strengthen HLIs to become more
competent and prepare students for the global market (Janal, Jalil, & Ahmad, 2020).
Visual learning technology uses visual aids such as videos to deliver educational content
more effectively (Du, Dai, Tang, Hung, Li, & Zheng, 2022). In the same line of argument,
Abdrabou and Shakhatreh (2021) emphasized that V-learning greatly benefits and enhances
the learning process, as interactive effects are used to reinforce the material being studied. It
is therefore considered a great way of learning as it aids in increasing a learner’s interest in a
certain subject, which makes the learning process more enjoyable and keeps the student’s
interest for longer periods.
Packet Tracer (PT) software is a visual tool utilized around the globe. Packet Tracer is a
comprehensive learning software application for teaching skills and concepts related to
computer networks (Muniasamy, Ejlani, & Anadhavalli, 2019). The software can perform
computer network simulations or is often called a computer network. As mentioned by
Abdrabou and Shakhatreh (2021), the software makes learning more comfortable by
providing a realistic network simulation and visualization of the environment. In addition,
this packet tracer software can simulate ongoing real-time updates that highlight the logic
and network activities.
In this study, the researcher used Technology Acceptance Model (TAM) to guide the analysis
of the effectiveness of PT. The technology Acceptance Model is a theoretical framework used
to explain how people accept and use technology. It was developed by researchers at the
University of Massachusetts in the late 1980s and is based on prior work by researchers in
the fields of organizational behavior, cognitive psychology, and information systems (Isaac
et al, 2018)
1332
Dina Moloja
The rest of the paper is organized as follows: Section 2 provides work related to this research.
Section 3 discusses the methodology of this study. Section 4 highlights the theory
underpinning this study. The results are presented in Section 5. The last section provides the
conclusions of the study and recommendations for future research.
2. Related work
Packet Tracer software has been used around the globe to enhance students learning.
Especially during the COVID-19 pandemic (Allison, 2022). A recent review of the literature
found that, since the outbreak of the pandemic, more and more institutions have adopted PT
as a teaching tool to foster remote learning. The review also highlights the importance of
providing adequate training for teachers on how to effectively use the tool to facilitate
learning. Other studies have examined how the use of PT can help to improve student
engagement and performance, as well as its potential to provide hands-on experience to
students who are unable to access physical networking equipment. Finally, there is also
research into the challenges faced by educational institutions when using PT, including the
need for better student support services and more resources to enable effective teaching.
For instance, Ojugo and Eboka (2021) used PT to improve service delivery and performance
dependability by conducting standardized tests, such as throughput tests, application
response-time tests, and availability tests on a campus network in Nigeria.
Another scholar used PT as an effective pedagogy in computer networking courses in
Malaysia. Their study revealed that PT successfully helped their students to understand
several key concepts of computer networking and, at the same time, quash some abstractions
they faced in the course (Noor, Yayao, & Sulaiman, 2018).
A study conducted in Malaysia by Allison (2022) developed a Survey on the Challenges
Faced by the Lecturers in Using PT in Computer Networking Courses. Their idea was to
identify the problems faced by teachers throughout the teaching and learning process, even
with the help of instructional simulations. In their conclusion, the authors proposed that
laboratory work activities need to be improved by employing an engagement taxonomy as
its preparatory guidelines.
Thus, an environment in which any technology or educational tool is adopted or used
provides different results depending on how it is used, who uses it, and their location.
Therefore, it is important to evaluate the usefulness and ease of use of PT software to evaluate
its effectiveness in the context of Internet technology students.
1333
3. Underpinning theory
This study applied TAM as the underpinning theory. TAM is a useful theoretical model for
predicting how technology or a system is used and its effectiveness (Isaac et al, 2018).
According to Allison (2022), TAM is a widely accepted concept that explains the
relationships between users and technology. It is based on the notion that users' attitudes
towards technology are based on their beliefs regarding the utility and convenience of the
technology. According to the TAM, user opinions of usefulness and ease of use are the major
factors in determining if a user will adopt and use a system. The TAM has been used to
understand and anticipate the behavior of users in a range of scenarios, such as technology
acceptance, software engineering, and e-commerce (Isaac et al, 2018).
The technology Acceptance Model assisted the author by guiding the evaluation of the
significant effects of student perceptions, student motivation, perceived ease of use, and
perceived usefulness of the adopted virtual technology in enhancing the accuracy of students’
performance and learning.
It is worth noting that the author only used two constructs from the TAM model. These two
constructs–perceived ease of use and perceived usefulness–allowed the researcher to evaluate
the effectiveness of PT.
4. Methodology
This quantitative research was conducted at a public university in South Africa (SA), where
82 respondents were asked to complete a questionnaire survey. Questionnaires were prepared
using Likert scales with five options from Strongly Agree to Disagree Strongly. Aspects
evaluated include aspects of Internet Technologies, learning, and communication. The
questionnaire before being used for data collection is validated by an expert (expert
judgment). Four experts validate the questionnaire. After the expert validated the
questionnaire, a tryout was conducted.
The researcher also used observations to collect data. The author observed students during
lab sessions and noted their level of engagement with the network simulation tools. 2.
students were asked to complete a questionnaire to assess their level of satisfaction with the
PT. 3. The author monitored the students’ interaction with PT and took notes of their
performance. 4. students were asked to provide feedback on their experience with PT. 5. The
amount of time students spent on the PT was measured. 6. The quantity and quality of the
student’s work completed with PT was assessed. 7. The author monitored the student’s
progress in using the network simulation tools by tracking the grades they receive.
1334
Dina Moloja
The collected data were from October 2022, as Internet Technologies (INT) is a second-
semester subject. The response rate of the study was 100%, which is good compared to other
studies found in the literature.
A Technology Acceptance Model (TAM) was used to guide the data analysis. The following
two TAM constructs were validated: 1. perceived ease of use of PT and 2. Perceived
usefulness of PT.
4.1. Perceived ease of use of Packet Tracer

Perceived ease of use refers to how easy it is for users to learn and use Packet Tracer (Vinay
& Rassak, 2015). It is an important factor in determining user adoption, as users are more
likely to use a technology if they find it easy to use. Packet Tracer is a powerful network
simulation tool that can be used to design, configure, and troubleshoot networks (Janal et al,
2020). Users have reported that the software is intuitive and easy to use, making it a good
choice for those new to network simulation (Isaac et al, 2018).
4.2. Perceived usefulness of Packet Tracer

Perceived usefulness is the extent to which a user believes that using Packet Tracer will
enhance their job performance (Sudarsana et al, 2019). It is based on the belief that Packet
Tracer will help the user become more efficient, accurate, and productive in their work.
Additionally, perceived usefulness may also be based on the user's belief that Packet Tracer
will improve their understanding of networking concepts, as well as increase their overall
knowledge of networking (Isaac et al, 2018).
5. Results
To efficiently evaluate the effectiveness of PT for INT students, it was necessary to determine
whether the V-learning tool was useful. The analysis of PT's perceived ease of use is shown
in Table 1, using the following determinants: installation, connection, addressing,
configuration, and simulation. Based on each determinant, participants were asked questions,
and a yes or no answer was expected.
Numerous studies have demonstrated that perceived ease of use is crucial in the context of
information systems (IS). According to Isaac et al (2018), the definition of perceived ease of
use refers to how much a person believes using a given system or technological equipment
will need little to no effort. Students’ experience of installing PT was 89%, which means that
most of the students were able to install the tool, and 84% of the students were able to connect
the devices. What was interesting about perceived ease of use was the addressing
determinant. The questionnaire revealed that only 67% were able to address the topology and
33% could not. This means that addressing is an element that requires students’ attention.
1335
The configurations and simulations were 74% and 77%, respectively. This implies that for
PT to be more effective, these elements need to be considered so that they can be improved.
Table 1: Perceived Ease of Use of Packet Tracer.
Determinants Question Yes % No %
Installation Was installation 73 89% 9 11%

easy?
Connection Did you 69 84% 13 16%

manage to
connect your
devices?
Addressing Were you able 55 67% 27 33%

to address your
topology?
Configuration Were you able 61 74% 21 26%

to do the
configurations?
Simulation Was simulation 63 77% 5 23%

understandable?
Furthermore, Table 2 shows the perceived usefulness of the packet tracers. There is a claim
made in the literature that perceived usefulness improves with perceived ease of use. The
allegation appeared to be true based on the results of this study. Abdrabou and Shakhatreh
(2021) stated that perceived usefulness is the extent to which a person thinks that employing
a specific system or piece of technology will improve their performance.
According to the results presented in Table 2, all students obtained a new skill, and their
knowledge was improved by using PT for teaching and learning. Although 5% of the students
felt that PT did not save time, they all agreed that it was cost-effective, with a percentage of
100. Additionally, 91% felt that the information provided about PT was complete and
adequate, while 9% believed that the information was incomplete. Thus, efforts should be
made to enhance the information provided.
1336
Dina Moloja
Table 2: Perceived usefulness of Packet Tracer.
Determinants Questions Yes % No %
Skill acquired Did you obtain 82 100% 0 %

any new
skills?
Knowledge Did your 82 100% 0 0%
improvement practical
knowledge
improve?
Save time Does using a 78 95% 4 5%

packet tracer
save time
Cost-effective Is it cost- 82 100% 0 0%

effective to use
a packet
tracer?
Information Do you think 75 91% 7 9%
completeness you have all
the
information
you require
about the
packet tracer?
6. Conclusion and recommendations

Using V-learning tools like Packet Tracer as an instructional aid in computer network courses
is beneficial. Packet Tracer is an advanced teaching platform that provides an interactive and
fun environment for both students and teachers. The findings of the study suggest that most
students believe that using this type of tool can facilitate a better understanding of computer
networking concepts, make studying technical concepts simpler, and enhance their
networking abilities. Furthermore, the ability of Packet Tracer to demonstrate network
algorithms proves its worth as a resource to learn networking principles, concepts, and
routing protocols, which are known to be the most difficult and important topic in this subject.
TAM has been proven to be an effective method of assessing the efficacy of network
simulation tools in Internet Technologies courses. The research indicates that these tools have
been successful in helping students comprehend and apply network theories, as well as giving
them practical experience in designing and testing networks. The results also point to the
1337
benefits of the tools in terms of developing problem-solving skills and improving general
analytical thinking. TAM has also been beneficial in providing feedback on the satisfaction
and engagement levels of students in terms of the tools' effectiveness. Ultimately, TAM has
provided an overall measure of the impact of the tools on student learning and performance.
In conclusion, PT is an effective tool for studying computer network topics, such as technical
and routing protocol principles, and students' exam results can be used to measure the
effectiveness of the Packet Tracer for future use.
References
Abdrabou, A., & Shakhatreh, W. (2021). Teaching Computer Networks to Electrical
Engineering Students by a Lecture-based Course and a Lab Course: A Quantitative
Analysis of Students’ Perceptions. In Proceedings of the 6th International Conference on
Information and Education Innovations, ACM, New York, NY, USA, 24-29.
https://doi.org/10.1145/3470716.3470721
Allison, J. (2022). Simulation-Based Learning via Cisco Packet Tracer to Enhance the
Teaching of Computer Networks. In Proceedings of the 27th ACM Conference on
Innovation and Technology in Computer Science Education, 1, 68-74.
Ambiyar, A., Yondri, S., Irfan, D., Putri, M. U., Zaus, M. A., & Islami, S. (2019). Evaluation
of Packet Tracer Application Effectiveness in Computer Design Networking Subject.
International Journal on Advanced Science Engineering Information Technology, 9(1),
54-59.
Demeter, R., Kovari, A., Katona, J., Heldal, I., Costescu, C., Rosan, A., ... & Thill, S. (2019).
A quantitative study of using Cisco Packet Tracer simulation software to improve IT
students’ creativity and outcomes. In 2019 10th IEEE International Conference on
Cognitive Infocommunications (CogInfoCom), Naples, Italy, 23-25 October 2019, IEEE,
353-358. 10.1109/CogInfoCom47531.2019.9089920
Du, X., Dai, M., Tang, H., Hung, J. L., Li, H., & Zheng, J. (2022). A multimodal analysis of
college students’ collaborative problem-solving in virtual experimentation activities: A
perspective of cognitive load. Journal of Computing in Higher Education, 1-24.
https://doi.org/10.1007/s12528- 022-09311-8
Isaac, Osama & Abdullah, Zaini & Ramayah, T. & Mutahar, Ahmed & Alrajawy, Ibrahim.
(2018). Integrating User Satisfaction and Performance Impact with Technology
Acceptance Model (TAM) to Examine the Internet Usage Within Organizations in
Yemen. Asian Journal of Information Technology, 17(01), 60-78. doi:
10.3923/ajit.2018.60.78.
Janal, M. A. B., Jalil, Z. B. A., & Ahmad, Z. B. (2020). The effectiveness in using packet
tracer simulation software in improving the skills among computer.
System and networks programme students.. International Journal Of Technical Vocational
And Engineering Technology (iJTveT), 2(1), 38-49.
Muniasamy, V., Ejlani, I. M., & Anadhavalli, M. (2019). Student’s performance assessment
and learning skills towards wireless network simulation tool–Cisco Packet Tracer.
International Journal of Emerging Technologies in Learning (Online), 14(7), 196. Doi:
1338
Dina Moloja
10.3991/ijet.v14i07.10351.
Noor, N.M., Yayao, N., & Sulaiman, S. (2018). Effectiveness of Using Cisco Packet Tracer
as a Learning Tool: A Case Study of Routing Protocol. International Journal of
Information and Education Technology, 8, 11-16. DOI:10.18178/IJIET.2018.8.1.1004
Ojugo, A. A., & Eboka, A. O. (2021). Empirical Bayesian network to improve service
delivery and performance dependability on a campus network. IAES International
Journal of Artificial Intelligence (IJ-AI), 10(3), 623-635. 10. 623.
10.11591/ijai.v10.i3.pp623-635.
Štrbo, M. (2021). Modern E-Learning based Information and Communication Technologies
(ICT) as a Part of Teaching Process. In 2021 6th International Conference on Inventive
Computation Technologies (ICICT) (pp. 1212-1218). IEEE. Coimbatore, India.
10.1109/ICICT50816.2021.9358547
Sudarsana, I. K., Armaeni, K. W. A., Sudrajat, D., Abdullah, D., Satria, E., Saddhono, K., &
Ekalestari,
Sudarsana, I. K., Armaeni, K. W. A., Sudrajat, D., Abdullah, D., Satria, E., Saddhono, K., ...
& Ekalestari, S. (2019, November). The implementation of the e-learning concept in
education. In Journal of Physics: Conference Series (Vol. 1363, No. 1, p. 012063). IOP
Publishing.
Vinay, M., & Rassak, S. (2015). A technological framework for the teaching-learning process
of computer networks to increase the learning habit. International Journal of Computer
Applications, 117(4), 1-4. Doi: 10.5120/20539-2904.
1339
Symbiosis between Learning Analytics and digital transformation

Hans Tubbax1,3, Iris Peeters2,3
1
Faculty of Bio-engineering Science, KU Leuven, Belgium, 2Group of Science and
Technology, KU Leuven, Belgium, 3Affiliated with Leuven Engineering and Science
Education Center (LESEC), KU Leuven, Belgium.
Abstract
Higher Education Institutes (HEIs) are undergoing a profound and
widespread digital transformation with the introduction of digital technologies
in education. The introduction of digitised educational environments produced
huge data repositories that could serve learning analytics, for the purposes of
understanding and optimising learning and the environments in which it
occurs. Where the digital transformation of education made learning analytics
better and more qualitative, learning analytics leverages the digital
transformation of HEIs by taking better informed actions.
This paper elaborates on the process of developing a conceptual learning
analytics platform at KU Leuven (Belgium). Where the project started off as a
pure data project, gradually several preconditions emerged, which caused a
lot of side-tracking. The introduction of learning analytics needs a digital
transformation of education. At the same time the digital transformation of
education needs qualitative and quantitative learning analytics to take it to the
next level. As if Learning Analytics and digital transformation live in symbiosis
with each other.
Keywords: Digital transformation; higher education; Learning Analytics;
Data platform.

1. Introduction
Due to the impact of digital technologies, society is undergoing a profound and widespread
digital transformation. This causes disruptive innovation in many different sectors, including
higher education. No Higher Education Institute (HEI) will be able to afford ignoring the
evidence from the data, without losing competitiveness in the future (Hidalgo, 2018). Hereby,
the digital transformation of HEIs is not a goal as such but must serve a purpose. However,
most HEIs are focused more on digitisation and digitalisation than actual digital
transformation, triggered by the introduction of Learning Management Systems (LMS). To
mature towards a complete digital transformation a coordinated pedagogical and
technological shift is needed (Brooks & McCormack, 2020). Yet, the biggest challenge in
this change process is not the technology as this is merely related to budgets and resources,
allocating these in the most optimal way. The pedagogical shift is much harder since it
involves people and the culture of the HEI. Even the Covid-19 pandemic, regularly referred
to as the so-called burning platform that compelled HEIs to use technology and quickly ramp
up their capacity for online teaching and learning, was not enough to induce a full digital
transformation (Pelletier, 2021). Many lecturers that had a bad experience with the forced
crisis-change, revert to the known recipe of traditional classroom education and assessment,
complicating digital transformation. Nevertheless, the changing role of universities
stimulates the urge to change anyway (Barnett, 2010).
Learning analytics plays an important role in this necessary transformation of HEIs and can
be defined as “the measurement, collection, analysis and reporting of data about learners and
their contexts, for purposes of understanding and optimising learning and the environments
in which it occurs” (SoLAR, 2022). The creation of huge data repositories by the introduction
of digitised educational environments such as LMS, MOOCs or other virtual environments
indeed leads to new areas of research and techniques.
The goal is to optimise the learning process (Khalil & Martin, 2016) but at the same time
these new kinds of analytics ask us to reflect deeply on what kinds of learning we value.
Learning analytics do not passively describe sociotechnical reality, they begin to shape it.
(Buckingham Shum, 2014). Using ‘The learning analytics cycle’ (Clow, 2012) Buckingham
Shum (2014) describes how the implementation of Learning Analytics triggers very
fundamental questions. By answering these questions, the university takes decisions on what
kind of education they want to offer, how they want to measure this and what actions they
want to take to improve learning.
1342
Figure 1 "How Learning Analytics shapes education" (Buckingham Shum, 2014)

based on ‘The learning analytics cycle’ (Clow, 2012).
Already in 2012, Greller & Drachsler proposed a generic design framework that can act as a
useful guide for setting up Learning Analytics services in support of educational practice and
learner guidance, in quality assurance, curriculum development, and in improving teacher
effectiveness and efficiency (Greller & Drachsler, 2012).
While interest in learning analytics has grown rapidly among HEIs, the maturity levels of
HEIs in terms of being ‘student data informed’ are only at early stages. The SHEILA
(Supporting Higher Education to Integrate Learning Analytics) project aimed to build a
policy development framework that supports systematic, sustainable and responsible
adoption of LA at an institutional level (Tsai et al., 2018). Both frameworks underline the
importance of the involvement of all stakeholders in a LA project, with a special emphasis
on their privacy.
This inspired the Science, Engineering and Technology Group (SET Group) of KU Leuven
to initiate the project ‘Learning Analytics & Dashboards: in function of the efficiency and
effectiveness of blended learning’ to investigate how existing data on students' digital study
behaviour can be used to give motivating and action-oriented feedback to didactic teams.
During this project it became clear that Learning Analytics was more than just unlocking
some data and producing fancy dashboards. Although many specific issues that arose during
the project were already described in detail in other documents, the project did not move
forward by just pasting together the proposed partial solutions.
This paper elaborates on typical pitfalls and matching insights on the symbiosis between
Learning Analytics and the Digital Transformation. The authors hope to inspire Higher
Education Institutes (HEIs) to implement Learning Analytics and use it as a tool to leverage
the digital transformation in their HEI.
1343
2. Methodology
The project ‘Learning Analytics & Dashboards’ was set up in an agile way, with ‘planning
periods’ of about 10 weeks and associated sub-goals. At the end of each planning period, a
steering committee discusses the results and defines sub-goals for the next iteration. Based
on the SHEILA framework we worked with colleagues collectively to determine the priorities
for the project. With standard data modelling techniques, we've created a dimensional data
model that would cover the needs for those learning processes needing learning analytics the
most.
2.1. From conceptual learning analytics platform to Proof of Concept by involving

stakeholders.
During the project specific feedback sessions were organised with stakeholders whose
opinions were important for the further progress. Depending on the subgoal, participants
were selected from a very broad range of roles and functions within the university. These
sessions allowed qualitative analysis of problem statements, alignment with the needs of the
stakeholders and thus their support of the proposed solutions. They also generated input from
involved lecturers, so that several demo-dashboards could be built, and a conceptual learning
analytics platform was developed.
By involving the different institutional stakeholders in the development, testing, deployment,
and assessment phase of learning analytics tools, most of the resistance to change might
already be mitigated in an early stage of the project. It will be of critical importance for its
acceptance that the development of learning analytics takes a bottom-up approach focused
on the interests of the learners as the main driving force. (Greller & Drachsler, 2012)
However, to scale-up these solutions, and generate proof for the concept, a more structured
and targeted approach is required to get a clear view of the present operational needs that
exist in terms of study data and learning analytics. Through semi-structured 1-on-1 interviews
with innovators and early adopters in Learning Analytics the conceptual learning analytics
platform is surveyed.
2.2. Finding unity and scalability towards a Minimum Viable Product

Where a bottom-up approach helps to increase the support of the stakeholders, there is a risk
that proposed solutions diverge from each other giving a scattered landscape of tools and
dashboards for learning analytics. To create a uniform learning analytics platform, there is a
need for organizational information governance encompassing the processes, standards,
rules, and practices an organization follows. The findings of the project contributed
considerably to the recently initiated KU Leuven working group 'learning data policy
1344
framework' to set out the boundaries and conditions for a centralised learning analytics data
platform.
Simultaneously, the side-project E-COOL (Effective and Consistently Organized Online
Learning environment) sought guidelines to bring systematics in terms of structure, layout,
and communication on the LMS, considering the differences in didactic work forms and
learning objectives. A systematisation in the structure of a course is crucial to create
unambiguous and useful learning analytics. Interpreting digital traces requires knowledge of
course structure and how learning materials are integrated into the learning approach. Even
more importantly when dashboard present cross-curricular data where the information user
can not have a detailed insight in how each course is structured.

Where the learning analytics project started off as a pure data project, the present scope is
much wider. Gradually, several preconditions emerged, which caused a lot of side-tracking:
privacy, data quality, course & curriculum design, structuring the LMS, external tools & LTI
couplings, ... In addition, the context in which the project was set up turned out to be very
complex: the momentum of the pandemic, discrepancy between learning objectives and
learning content assessment, the rollout of formative testing & blended learning, migration
to a new LMS, ...
Nevertheless, a conceptual learning analytics platform and proof of concept could be
developed based on the feedback sessions and interviews. This allowed the generation of
concept dashboards, which fed the feedback sessions and interviews (Figure 2).
Figure 2 Concept learning analytics dashboard.
At the same time, many didactic teams appeared to question the usefulness of Learning
Analytics. A huge gap in engagement for learning analytics between the Innovators & Early
1345
Adopters on the one side and the Early Majority on the other side became very clear. To
tackle this growing resistance to change the focus on the support of the stakeholders became
even more important, together with scalability and sustainability of the proposed solutions.
Where the Early Adopter can still live with teething problems, bugs and beta models, the
Early majority are pragmatists who only use things that are 'Complete' and somehow solve
their problems (Peeters, Grommen & Tubbax 2023). To gain wider support for learning
analytics to justify the deployment of a full learning analytics platform at the university, the
gap between the Early Adopters and Early Majority of learning analytics needs to be bridged.
Additionally, when inventorying other ongoing Learning Analytics investigations at KU
Leuven, typically executed by innovators at a small scale, we noticed that most projects had
a double complexity. Not only were they investigating new technologies or applications to
capture, process and/or visualise study data. They were also looking at innovating the
educational processes that were more suited for blended education. Because innovators do
not always have scalability in mind, but rather seek an ad hoc solution to their own specific
problem, most of these projects were not viable. Even more reason to find a solution and
build a centralised learning analytics data platform that still satisfies them, but which is
scalable.
Building a centralised learning analytics data platform is an essential step towards scalable
analytical capabilities. This requires good information management capabilities to integrate,
extract, transform, and access transactional data (Davenport & Harris, 2007). Whether we
are speaking about the Knowledge Discovery in Databases (KDD) process in datamining
(Fayyad et al., 1996) (Figure 3) or of the extract, transform and load (ETL) process in
Business Intelligence & Analytics (Vassiliadis, 2009), the base of qualitative analytics is
materialised by the analytical database on which visualisations, reports, Machine Learning
Training, etc. for the data consumers can be built.
Figure 3 Overview of the steps constituting the KDD process (Fayyad, Piatetsky-Shapiro et al. 1996).
The building of such an analytical database can only be achieved when the processes that
would fill the source databases are digitalized sufficiently, so that the database is rich enough
to draw well-founded conclusions and to take data-driven action. This is why the working
group 'learning data policy framework' has put effort in defining a typology and inventory of
which study data should be integrated in learning analytics. Furthermore, through the
SHEILA framework priorities on learning analytics policy were agreed on.
1346
The results of this seed project will be used as a blueprint for a scale-up project in which the
learning analytics strategy and system are implemented and deployed. The partnership in the
current project with people of the Educational Policy Units and ICTS is crucial for the
continuation toward the scale-up project.
4. Conclusions
Building a learning analytics strategy for HEIs is so much more than merely aggregating
some data and building generic dashboards. The maturing of the learning analytics and the
evolution from digitisation towards digital transformation clearly go hand in hand. As HEIs
are maturing in their digital transformation, more qualitative learning information is
generated, and more accurate learning analytics become feasible. By providing more and
better information about the study progress of a student, the HEI can improve their digital
transformation in education, on every level.
This sounds like a positive spiral every HEI would want to enter. However, there are some
conditions and pitfalls. For instance, the streamlining of these processes and standardising
the information in a central learning data store, is essential to transform the institution's
operations, strategic directions, and value proposition. To develop this centralised learning
analytics platform, involvement of all stakeholders seemed necessary, in an iterative process.
Moreover, the implementation of learning analytics and the digital transformation of the HEI
must be managed in sync with each other so that both exist in symbiosis. Otherwise, the
positive spiral turns in to a negative vicious circle, with one change running ahead from the
other, bringing unused solutions. A challenging mission in a challenging world that keeps
HEIs competitive for the future.
Acknowledgements
The authors would like to thank the Group of Science, Engineering and Technology and
LESEC for the networking opportunities and their financial support. All members of the
steering committee of the mentioned Learning Analytics project are thanked for their
substantial contribution.
References
Barnett, R. (2010). Being a University (Foundations and Futures of Education). Routledge.
Brooks, C. D., & McCormack, M. (2020). Driving Digital Transformation in Higher
Education.
Buckingham Shum, S. (2014, 25th June 2014). Learning Analytics: Welcome to the future
of assessment? (EdMedia2014 Keynote). EdMedia, Tampere, Finland.
1347
Clow, D. (2012). The learning analytics cycle: closing the loop effectively. International
Conference on Learning Analytics and Knowledge - LAK ’12,
Davenport, T., & Harris, J. (2007). Competing on Analytics: The New Science of Winning.
Harvard Business School Publishing Corporation.
Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). The KDD process for extracting
useful knowledge from volumes of data. Communications of the ACM, 39(11), 27–34.
https://doi.org/10.1145/240455.240464
Greller, W., & Drachsler, H. (2012). Translating Learning into Numbers: A Generic
Framework for Learning Analytics. Educational technology & society, 15(3), 42-57.
Hidalgo, R. (2018). Analytics for Action: using data analytics to support students in
improving their learning outcomes. In G. Ubachs & L. Konings (Eds.), The Envisioning
Report for Empowering Universities (pp. 6-8). European Association of Distance
Teaching Universities (EADTU).
Khalil, M., & Martin, E. (2016). What is Learning Analytics about? A Survey of Different
Methods Used in 2013-2015 Smart Learning Conference, Dubai, UAE.
Pelletier, S. (2021). Strategic Tech Decisions During the Pandemic.
https://connect.chronicle.com/rs/931-EKA-218/images/StrategicTechResearchBrief.pdf
SoLAR. (2022). About SoLAR : What is Learning Analytics? Retrieved first of April from
https://www.solaresearch.org/about/what-is-learning-analytics/
Tsai, Y.-S., Whitelock-Wainwright, A., Muñoz-Merino, P., Moreno-Marcos, P., Rubio-
Fernández, A., Delgado-Kloos, C., . . . Kollom, K. (2018). SHEILA - Supporting Higher
Education To Integrate Learning Analytics. In Research Report November 2018 (pp. 44).
Vassiliadis, P. (2009). A Survey of Extract-Transform-Load Technology. International
Journal of Data Warehousing and Mining, 5, 1-27.
1348
Six colours for inclusion: results of an explorative activity in a

simplexity approach
Department of Human, Philosophical and Educational Sciences, University of Salerno,
Salerno, Italy.
Abstract
Schools and teachers are called upon, in the age of complexity, to control a
plurality of problems related to the numerous stimuli arising from the personal
and environmental influences of students. This implies a new way of working
in the direction of creativity, which therefore becomes a teaching tool as well
as a human process. This paper presents the results of an exploratory activity,
conducted on 657 teachers, as part of the training activities for teaching
support to pupils with special educational needs at the University of Salerno.
The aim was to investigate whether teachers in training can develop lateral
thinking, based on suggestions derived from de Bono's (2015) studies and,
specifically, his "Six Hats for Thinking" theory, highlighting points of
agreement with the simplexity theory proposed by Berthoz (2011). This
represents a different way of using colour to think, act and include.
Keywords: Inclusion; lateral thinking; divergent thinking; simplexity;
creativity.

Six colours for inclusion: results of an explorative activity in a simplexity approach
1. Introduction
The school, in the age of complexity, due to the presence of a plurality of stimuli, of
innovations. In this context the teacher must support the students to decipher of all the stimuli
and problems, as well as search for strategies to deal with them (Sibilio, 2020), trying to find
innovative and non-linear strategies (Sibilio, 2012; Sibilio & Zollo, 2016), which are on the
great line of creativity (de Bono, 2015a). In this sense the logic of creative thinking, if
declined in educational experiences included in the construction of the teacher's
competences, could prove to be an interesting strategy for the resolution of problematic
situations.
In this regard, it is possible to find elements of commonality with the scientific reflections of
Alain Berthoz, a physiologist of perception and action who, in his theory of simplexity
(Berthoz, 2011), hypothesises that the solutions devised by living organisms to decipher and
deal with complexity can be considered valid and applicable to the entire class of adaptive
complex systems. One of the principles of simplexity described by Berthoz is deviation, i.e.
a rule that attempts to solve the problem, but in an original way, and by using, and
experimenting with, a plurality of trajectories, perhaps even unknown ones, to arrive at the
resolution of the problem itself. The prerequisite for the use of deviation in education can be
identified in the subject's ability to deal with the complexity of problematic situations through
the use of creativity (Goleman et al., 2017), divergent thinking (Guilford, 1950) and lateral
thinking (de Bono 2015; 2015a), which have shown interesting educational implications and
are corroborated by a rich scientific literature. In the present study, we will show precisely
the results of an exploratory survey conducted on a sample of 657 students, attending the
training courses for the support qualification for Secondary School at the University of
Salerno, in order to identify whether in the course of their action they are able to exercise
lateral thinking, thus maturing transversally innovative competences for problem solving and
for the development of inclusive operational trajectories, using de Bono's (2015) "Six Hats
for Thinking" as a tool for action, according with Berthoz's (2011) simplexity detour.
2. Colours, divergence and inclusion

The first theory on the concept of divergent thinking dates back to the American psychologist
Guilford, who open to new solutions and capable of giving rise to a new associative mode of
elements to achieve objectives, placing it alongside the convergent thinking that had
characterised scientific research up to that time. According to the scholar's perspective,
therefore, convergent thinking operates within established schemes, tackles the problem with
a given method and, through the latter, finds the only possible solution; divergent thinking,
on the other hand, acting outside the established schemes, allows one to approach the problem
with a new approach, arriving at original solutions and identifying the creative process with
1350
the typical dynamics of problem solving (Zollo, Kourkoutas & Sibilio, 2015). Creativity is
an exclusive characteristic of a few exceptional minds and becomes the hallmark of human
thought, the natural expression of the individual's inner self. A further theorist, who made
creativity the cornerstone of human action, was de Bono (2015). The Maltese psychologist
and physician, in fact, applying creativity to the world of business and linking it to corporate
competitiveness, defines it as "the ability to think and act differently that can be developed
in a systematic and deliberate way by anyone willing to put the principles of lateral thinking
into practice" (de Bono, 2015a, p. 28). An application of creativity for the development of
problem solving skills. A scholar, through the metaphor of hats, learns how to approach
problems by assuming different points of view and proposes six different perspectives from
which an idea can be generated. This premise is useful to understand how through the creative
and divergent process one can also produce results in the recognition of the other from
oneself, in the awareness that the hat worn by the other from oneself represents his personal
mode of expression, his value to be recognised, accepted and supported, in a suitably
inclusive vision.
Edward de Bono's reflection starts from the way we approach problematic situations,
considering in many cases only one point of view and thus reducing the possible solutions.
According to the scholar, each way of solving a problematic situation can be compared to a
hat that not only defines a certain type of thinking, but also has its own colour (de Bono,
2015); therefore, instead of trying to cover all aspects with thinking, it is possible to separate
the various types of thinking and carry them out distinctly. The colour that distinguishes hats
(white, black, green, red, blue, yellow) represents a much broader mode of expression; it
takes on a symbolic significance. It becomes, in fact, a means of guiding the individual to
activate mechanisms of perception of diversity, of different ways of facing reality, using
colour as an interpretative key, as a tool for accepting the different way of looking at the
world, as a different method of valuing diversity, adapting to a thought that is not lateral, but
inclusive, respectful of all and each one. Von Goethe (1970) himself considered colour as
the actions of light capable of expressing passions and actions. Colour, therefore, is a
representation of oneself that only finds meaning in the relationship with the other, in the
awareness that with the other the total process of recognition of belonging to a nature is
realised. Colour, therefore, expresses the sensitivity of the individual, strong feelings, both
positive and negative, capable of providing insight into us and others (Carluccio, 2008).
Kandinsky himself in "The Spiritual of Art" (1989) explains the semanticity of colour,
according to which black is synonymous with despair, meaning and the end, from which
other colours unravel, a symbol of re-birth and continuation, the beginning of new worlds to
explore. Colour, consequently, is an expression of conditions, of experiences, of personal and
shared experiences, a means of re-encountering and integrating different colours which, in
union, in each case give rise to chromatic balance. Declined in the educational field, the
exercise of lateral thinking theorised by de Bono would require experiential itineraries for
1351
the construction of the teacher's professional skills, to be carried out through activities that
introduce the subject's ability to exercise flexibility and detour in didactic transposition. This
finds points of contact with Berthoz's theory of simplexity (2011).
3. Creativity in a simplexity perspective

From the descriptions conducted so far on the concepts of creativity, divergent thinking and
lateral thinking, interesting points of reflection emerge on possible didactic declinations of
these theories in the perspective of a simplexity didactic, i.e., a teaching method capable of
deciphering and coping with the complexity of training processes and educational contexts.
Simplexity starts from the assumption that human organisms can flexibly adapt to
problematic situations, using operational patterns and rules of use. If we want to connect
simplexity with lateral thinking and, consequently, creativity, it is possible to identify
significant analogies (Guilford, 1967; Goleman et al., 2017). Among the distinctive traits of
creative, divergent, lateral thinking as analysed are flexibility, reliability, and detour,
identified by Berthoz as properties and rules of simplexity, i.e., simplexity tools and rules
that enable deciphering and coping with complexity, including educational complexity.
Flexibility and reliability to change are, in fact, configured as simplexity properties of the
didactic system and of didactic and inclusive action, which must "be capable of perceiving,
capturing, deciding or acting in many ways (vicariousness) depending on the context,
compensating for deficits, coping with new situations" (Berthoz, 2011, p. 9). These are,
therefore, tools that allow one to disentangle oneself from the protean difficulties of the
learning experience, seizing precisely from the problematic situation the opportunity to
broaden knowledge through action. In a similar vein, de Bono (2016) proposes using thought
not to solve individual problems, but to grasp new interpretations of reality.
The search for different adaptive solutions and alternatives to usual situations constitutes, in
this sense, the expression of a freedom of choice in the broad repertoire of possible solutions
to avoid getting lost in complexity. Berthoz precisely sees this as the rule of simplexity,
grafted onto the search for exactly this plurality of original solutions to problems (Zollo,
Kourkoutas & Sibilio, 2015). In this sense, further simplexity rules such as inhibition, the
rule of refusal and the principle of detour appear useful to bring out the potential of creative
thinking, which precisely requires the ability to inhibit and reject automatic and immediate
solutions, identifying flexible operational strategies that, through ancillary complexity, bring
out new modes of didactic action capable of favouring the learning process. Such didactic
actions are also suitable for favouring inclusive pathways (Sibilio, 2013; 2015). In the light
of these reflections and considering the systemic perspective outlined, the acquisition of
creative thinking skills, consistently with a simple vision of didactics, could be a valid
strategy to promote training interventions aimed at deciphering complexity, favouring
effective and inclusive courses of action.
1352
4. Description of an exploratory survey: methodology
4.1. Research question and objective

On the basis of the theoretical premises presented, a research investigation was carried out,
initiated on the basis of the question: Is the teacher, both in training and working at school,
able to exercise lateral thinking, which is also important for the maturation of creative
thinking and useful for applying it during the teaching phases in order to guide the pupil to
the development of problem solving? The question was therefore posed as an objective to
investigate whether teachers in training, in the process of obtaining the qualification to teach
support aimed at the Secondary School, are able to recognise creative thinking in themselves,
understood as "the set of attitudes, expressions and techniques that allows one to
transversally cut through the schemes of an asymmetrical self-organised system to generate
new conceptions and perceptions" (de Bono, 2015a, p. 201) and are predisposed to use it.
4.2. Participants and instrument

The research, eminently exploratory in nature, was conducted with a sample of 657 students,
without a purposive selection, who from September 2022 to July 2023 are following as part
of the Enabling Pathways on support for Secondary School at the University of Salerno,
characterised by both a general and special pedagogical-methodological-didactic framework
(General Didactics, Special Didactics, Special Pedagogy, Methodology and Educational
Design). For data collection, a Google form has been prepared, structured in two parts. The
first part surveyed the age of the participants and their gender, in order to identify the gender
variable with respect to the answers provided. A second section of the questionnaire asked,
instead, to self-attribute one of the six hats described by De Bono, on the basis of the
characteristics of the hat that seemed more representative.
4.3. Research phases

The activity proposed to the teachers in the four training groups was divided into three phases.
In the initial phase of the course, were explained the main studies on creativity and divergent
thinking, focusing in detail on Edward de Bono's theories of lateral thinking and presenting
the 'Six Hats for Thinking' method as a tool for exercising creative thinking. Subsequently,
was illustrated Alain Berthoz's theory of simplexity.The first phase, characterised above all
by theoretical reflections on the subject, was followed by a practical-operational moment: the
teachers, in fact, already previously involved in group activities of various types, were
proposed an exercise aimed at exercising lateral thinking and, therefore, creative thinking,
centred on the declination of the properties and rules of simplexity in general didactics, in
disciplinary didactics and in special didactics. In the third phase, after providing the
theoretical foundations with regard to creativity, were carried out a series of group training
experiences that, through discussion, highlighted the importance of awareness of diversity
1353
and perceptual plurality in teaching. Subsequently, teachers were asked, based on Edward de
Bono's (2015) "Six Hats for Thinking" method, to indicate on a pre-filled table using the
integrated Google Forms function, the colour of the hat that best described their personal
characteristics in the problem-solving phase, specifying, through the indication of the colour,
the approach considered prevalent with respect to the others.
5. Data analysis
The analysis of the data, conduced usung the method of descriptive statistics, made it
possible, first, to detect how the teachers in training within the framework of the Enabling
Pathways on support for Secondary School have different perceptions of their own approach
to solving problematic situations. Being an experience aimed at understanding the teachers'
tendency to use creativity for a lateral approach to metacognitive and inclusive didactics, to
overcome the rigidity of logical models linked to vertical thinking very often used, specific
attention was paid to the colour of the hats that fully represent the two types of thinking:
white (vertical thinking) and green (lateral thinking).

Regarding the data about the age of participants, it should be stress that the 76% of total
participant of four groups has an age between 31 and 50 years, versus 24% that has an age
between 25 and 30 years. In addiction, the data about the gender of participants shows that
82% of participants was female, versus 18% of male gender.
About the data from the first group, it should be noted that the highest prevalence was
attributed to the red hat (29%), and the yellow hat (21%). The blue hat also found a good
percentage of attribution (18%). The last two hats identified as their own were green (13%)
and white (13%). As far as the second group is concerned, a prevalence of the red hat (29%),
followed by the blue hat (23%) should also be noted. An almost equal percentage emerges
between the yellow (15%) and black (14%) hats. Here too, the white hat (10%) and the green
hat (9%) are in the minority in terms of self-attribution compared to the others. As far as the
results of the third group are concerned, there was again a prevalence of the red hat (25%),
followed by an equal percentage of 17% of yellow and white hats. This was followed by a
17% attribution of blue hats. The lower attribution of the black (14%) and green (12%) hats,
representative of lateral thinking, should be highlighted. As far as the last group is concerned,
it should be noted that the red hat (31%) is the predominant one, followed by the blue hat
(25%). Then comes the white hat (16%), followed in the last positions by the green hat (13%),
yellow (10%) and black (5%).
1354
From the analysis of the data, it emerges that the way of acting, reflecting, of the teachers is
aimed at a prevalence of emotions, the red hat being the prevalent one in all four groups. It
also emerges that the vertical thinking hat, i.e. white, does not turn out to be one of those
most attributed, a sign that the teachers are attempting a change of course in the way of acting
and operating, attempting to overcome a vertical tendency in the way of approaching the
problem, highlighting an initial form of non-linearity (Sibilio, 2012; Sibilio & Zollo, 2016)
through recourse to emotions, to positivity, also experimenting with a possible
interconnection of the six hats through organisation, highlighted with the choice of the blue
hat. However, the road to be taken so that the green hat, i.e., that of lateral thinking, is
preponderant, is still uphill, as the teachers do not show, for all four groups, a general
tendency to wear such a hat, which is useful for exercising creativity, while also respecting
the detour from operational standardisation.
It is also to be hoped that black can increase its percentage. The hypothesis of the rejection
of the black hat is its association with negativity. In truth, the black hat requires that a person,
in each problematic situation, is also able to foresee weaknesses in order to construct suitable
paths to intervene and avoid them, even then by resorting to creativity. This kind of aspect is
necessary for those who, working in the world of Special Educational Needs, can identify the
elements of weakness in the pupil, highlighting their strengths as a tool for building a good
inclusive curriculum. In relation to sociodemographic analisysm emerged that in the first
group it was also highlighted that the female participants had a convergence towards the
choice of hats associated with creativity (60%), with a positive outlook on life, while the
male participants attributed to themselves hats relating to the sphere of rationality and order
(40%). A similar process also occurred in the other groups, where it emerged that 70% of the
participants converged towards the red hat, an indication of how the participants see
themselves as inclined towards emotion compared to the male participants. The results that
emerged for the other two groups were not very dissimilar. From this we denote that there is
still a perspective according to which girls are generally associated with emotion, with
creative action, while men instead with a more rational and linear action.
7. Conclusions
In the situation of complexity, dynamism and plurality that characterises today's educational
contexts, the exercise of creative thinking could represent a valid problem-solving strategy.
The production of new and alternative ideas, the ability to approach problems in an
innovative way, flexibility, adaptation to change and detour are not only the peculiar
characteristics of creativity, but also fit coherently into the perspective of inclusion and in the
perspective of a simplified didactics. The results of the training activity carried out with the
teachers within the framework of the Enabling Pathways have, however, highlighted, through
the research carried out, which was exploratory in nature, a low predisposition on the part of
1355
the teachers to exercise creative thinking and to use flexibility, adaptation to change and
detour in the teaching field. In the light of this scenario, starting from the awareness of the
importance, in teacher training, of creative thinking, which, as emerges also from the Profile
of Inclusive Teachers (2012), allows for the development of effective and flexible strategies,
and due to the various cognitive styles of learners, it would be appropriate to work more on
creativity (Zollo, Kourkoutas & Sibilio, 2015).
In conclusion, the results presented if, on the one hand, they solicit further and broader
studies, on the other hand, they encourage, on the basis of the critical issues that emerged,
scientific reflection on the necessary and possible planning of specific training itineraries
aimed at the acquisition, by teachers, of didactic detour skills and the identification of flexible
and alternative strategies that can be capitalised on in the activity of coping with the
educational complexity in the educational environment.
References
Berthoz, A. (2011). La semplessità. Torino: Codice.
Carluccio, C. (2008). Disegno arte e immagine. Napoli: CUES.
De Bono, E. (2015). Sei cappelli per pensare. Milano: Bur.
De Bono, E. (2015a). Creatività e pensiero laterale. Milano: Bur.
De Bono, E. (2016). Il pensiero laterale: come produrre idee sempre nuove. Milano: Rizzoli.
European Agency for Development in Special Needs Education (2012). Profile of Inclusive
Teachers, Odense, Denmark: European Agency for Development in Special Needs
Education. Disponibile su: http://www.european-agency.org/sites/default/files/Profile-
of-Inclusive-Teachers.pdf.
Goleman, D., Kaufman, P., & Ray, M. (2017). Lo spirito creativo. Milano: Rizzoli.
Guilford, J.P., (1950). Creativity. American Psychologist, 5 (9): 444-454.
Guilford, J.P., (1967). The Nature of Human Intelligence. New York: McGraw-Hill.
Kandinskij, V. (1989). Lo spirituale nell'arte, a cura di Pontiggia, E. Edizioni Se.
Sibilio, M. (2012). La dimensione semplessa dell’agire didattico. Traiettorie non lineari
nella ricerca. Nuovi scenari interdisciplinari, atti di convegno Sirem, 10-14.
Sibilio, M. (2013). La didattica semplessa. Napoli: Liguori.
Sibilio, M. (2015). Simplex didactics: a non-linear trajectory for research in education. Revue
de synthèse, 136(3), 477-493. DOI: 10.1007/s11873-015-0284-4.
Sibilio, M., & Zollo, I. (2016). The non-linear potential of didactic action. Education
Sciences & Society-Open Access, 7(2). Retrieved from
https://journals.francoangeli.it/index.php/ess/article/view/3947.
Sibilio, M. (2020). L'interazione didattica. Brescia: Scholé.
Von Goethe, J. W. (1970). Theory of colours. Mit Press.
Zollo, I., Kourkoutas, E. E., & Sibilio, M. (2015). Creatività, pensiero divergente e pensiero
laterale per una didattica semplessa. Educational reflective practices, 1, 5-17
1356
Cybersecurity education in European higher education

institutions
Mirva Salminen, Niko Candelin, Kaisa Cullen, Sari Latvanen, Marianne Lindroth,
Teemu Matilainen
Department of Information and Communications Engineering (DICE), Aalto University,
Finland.
Abstract
This research paper presents some of the findings of an ongoing Cyber Citizen
Initiative project (2022–24), which benchmarked civic cybersecurity education
and training in all European Union (EU) member states. The research paper
focuses on cybersecurity education in higher education. The study concluded
that cybersecurity education varies across the EU. Whereas some countries
have dozens of higher education institutes providing education in
cybersecurity, some others have only a few institutions and educational
programmes for the topic. In general, the educational programmes tend to be
specific and focus on technical skills. A wider understanding of cybersecurity
as a societal concern, civic cybersecurity skills, and discipline geared
cybersecurity competence building may be lacking from the curricula.
Keywords: Cybersecurity competence; cybersecurity skills; cybersecurity
education; higher education; European Union.

Cybersecurity education in European higher education institutions
1. Introduction
The overarching digitalization of societies has made the strengthening of cybersecurity
competence at all levels of society and across its sectors a priority in many countries. For a
longer period, this has meant the build-up of cybersecurity focused educational programmes
and curricula in universities and universities of applied sciences, as well as the integration of
cybersecurity awareness and skills into basic and secondary education. More recently, the
need of teaching civic cybersecurity skills to everyone in society has become emphasized.
This research paper presents some of the findings of an ongoing Cyber Security Initiative
project lead by Aalto University, Finland. In its first phase (in 2022), the project benchmarked
the 27 European Union (EU) member states regarding the teaching of civic cybersecurity
skills. Civic cybersecurity competence is understood as “[t]he knowledge, skills and abilities
required to operate in a cyber environment” (Limnéll et al., 2023, 2, 142). Everyone living in
a digitalised society needs such competence. Data collection was implemented by extensive
online research supplemented with interviews, conference and seminar attendance, and a
scoping literature review.
This research paper focuses on findings regarding the teaching of cybersecurity skills in
formal, higher education in selected member states. The countries in the sample provide a
view to cybersecurity education across the union: Finland, Estonia, Germany, France,
Romania, Spain, and Greece. Alongside providing geographical coverage, the selected
countries rank relatively high in cybersecurity indices and/or host pivotal EU cybersecurity
institutions. The research paper asks: How is cybersecurity competence currently built at
universities and universities of applied sciences in the selected countries? The study includes
both educational programmes in cybersecurity and civic cybersecurity training provided to
all students in higher education.
2. Cybersecurity training in higher education in selected EU member states

There is no universally agreed practice of benchmarking (Chen et al., 2020). It can be
understood as a process of improvement through adaptation and substitution of a process by
another recognized as better (Maire et al., 2008, 765). In the Cyber Citizen Initiative project,
benchmarking meant establishing the state of civic cybersecurity education across the EU to
find out adoptable practices. This research paper focuses on higher education institutions
providing cybersecurity education, the content of educational programmes, and the ways in
which national cybersecurity strategies strive to improve cybersecurity education.
2.1. Finland
Finland's goal is to be a country of excellence in cybersecurity. One of its main strategic
policies is the “[d]evelopment of cyber security competence – everyday skills and top skills
1358
Mirva Salminen, Niko Candelin, Kaisa Cullen, Sari Latvanen, Marianne Lindroth, Teemu Matilainen
as cyber security safeguards" (The Security Committee, 2019, 8). National measures to
enhance cybersecurity competence include, inter alia, the strengthening of cybersecurity and
information security related education at all educational levels (The Security Committee,
2019, 4, 9). Higher education degrees focusing on cybersecurity are offered by both
universities and universities of applied sciences. However, topics related to cybersecurity are
part of several other higher education degree programmes as well (Limnéll et al., 2023).
Universities of applied sciences provide technical cybersecurity education. In terms of
content, this education (eight bachelor’s and four master's programmes, as well as
specialisation, supplementary, and conversion training) is comprehensive and serves the
needs of the industry. (Lehto, 2022, 9, 61-62.) There are three university degree programmes
focused on cybersecurity, all master’s programmes (provided by the universities of Jyväskylä
and Turku and Aalto University) (Limnéll et al., 2023). In addition, cybersecurity is included
in the structure of many degree programmes as optional or compulsory studies and many IT
programs offer a possibility to study broader cybersecurity studies. Several free cybersecurity
courses are also offered to everyone by the Finnish Institute of Technology (FITech), which
is a network university. Its founding members include seven universities and technology
industry associations. (Lehto, 2022, 10, 72, 76, 85, 92.)
2.2. Estonia
The promotion of a cyber-literate society is one of the goals in Estonia’s cybersecurity
strategy. To achieve this goal, the academia must be involved in cooperation with the public
sector and companies. Cybersecurity is taught at all educational levels as part of the
development of digital skills. Lack of cybersecurity specialists and a small number of new
talents in the field are recognised as challenges which must be solved, for example, by
including cybersecurity in intensified IT studies. (Republic of Estonia, 2019, 7-8, 15, 17.)
Studying cybersecurity in higher education is possible in two master's programmes. Tallinn
University of Technology (Tallinna Tehnikaülikool, TalTech) runs a master's programme
"Cybersecurity" in cooperation with the University of Tartu (Tartu Ülikool). The students
study under high-level cybersecurity practitioners from universities, industry, law
enforcement, CERT, and the NATO Cooperative Cyber Defence Centre of Excellence
(Taltech, 2023). The international master’s programme “Cyberus Erasmus Mundus Master
in Cybersecurity” is administered by the University of South Brittany in France (Université
Bretagne Sud) and TalTech is responsible for a part of the training. Students can specialise
either in software cybersecurity or IoT cybersecurity. (Université Bretagne Sud, 2023.)
2.3. Germany
According to the German cybersecurity strategy, responsible behaviour in cyberspace and
the opportunities and risks brought by internet are an essential part of digital citizenship
1359
skills. Therefore, digital education must be integrated into the country's entire education
system. (BMI, 2016, 10.) The organization of and decision-making in the school system are
delegated to the education ministries of the 16 states. Thus, each state has different
regulations regarding the curriculum, the provision of study subjects and degrees, and the
transition between school forms. The federal government's goal is that by graduation young
people have sufficient knowledge and skills related to IT security. The aim is to increase and
to expand course offerings in IT by establishing more study places at universities and
supporting leading institutions, especially in computer science (BMI, 2016, 10).
IT studies are offered in almost all universities, which also include comprehensive
information and cybersecurity study units. For example, University of Saarland provides a
master’s degree in cybersecurity, which offers students an opportunity to deepen their
cybersecurity knowledge comprehensively and choose a specialization in the areas of
cryptography, privacy, software security, systems and networks, formal methods, or legal
aspects of cybersecurity (Universität Des Saarlandes, n/d). The Cybersecurity Higher
Education Database (CyberHEAD) by the European Union Agency for Cybersecurity
(ENISA) lists altogether five bachelor’s and master’s degrees related to cybersecurity.
2.4. France
The French white paper on defence and national security (2013) already highlighted the
influence of increasing the number of trained information and cybersecurity experts on
national security. In addition, it was to be ensured that information and cybersecurity were
integrated into higher education in computer science, with the aim of preventing
vulnerabilities in information systems and promoting vigilance and response against cyber
threats (Poupard, 2016). To achieve these goals, The Agence nationale de la sécurité des
systèmes d'information (ANSSI) created the CyberEdu programme, which purpose is to
provide resources for cybersecurity training. This activity led to the creation of the CyberEdu
association to develop and maintain the original programme and to grant approval to courses.
In autumn 2019, CyberEdu published 78 certified training courses (Poupard, 2016).
The national cybersecurity strategy (2015) also highlights the importance of digital security
as part of higher education. Higher education institutions should have a training unit
dedicated to digital security. Contemporarily, French universities offer several programmes
focused on digital and cybersecurity, such as MSc in Cybersecurity (CySec), MSc in
Computer Science in Computer Security, and Master in Data & Security Science. According
the CyberHEAD database (2023), France is offering 11 different master’s degrees and
postgraduate courses related to cybersecurity.
1360
2.5. Romania
The Romanian cybersecurity strategy (2021) defines goals and identifies components that are
relevant for the functionality of digital services and their safe utilization. The aim is to create
a necessary framework for the future development of state administration, the business
environment, the national economy, and the education and research sector. The government
is investing in the development of education technology as part of the implementation of the
Digital Education Action Plan of the European Commission for 2021–27. It has allocated
resources for the improvement of digital pedagogical skills, educational resources, and
physical and other resources. (Directoratul National de Securitate Cibernetica, n/d.)
In an EU funded Cyber_Education project, Romania's nine largest universities were invited
to create common and modern curricula for cybersecurity and educational laboratories. The
project was launched by the Bucharest University of Economics, which specialises in training
cybersecurity experts. There are 54 public and 35 private universities. Universities
specialising in cybersecurity education are Technical University of Cluj-Napoca (n/d)
teaching Information and Computing System Security and University Politehnica of
Bukharest (n/d) teaching Coding and Storage Theory of Information Master.
2.6. Spain
Spain emphasizes the importance of national cybersecurity culture, which is one of the five
main goals and materialized in two of the seven lines of action in Spain’s National
Cybersecurity Strategy (2019). Cooperation between public and private organizations and
the media should be encouraged and strengthen human and technological skills promoting
appropriate professional skills. Line of action 5 boosts research, development, and innovation
support in digital and cybersecurity programmes, including universities and research centres.
It also identifies needs for professional skills in cybersecurity and fostering collaboration
between educational and training institutions by enhancing such elements as continuous
training and university education. It also pays attention to identifying, encouraging, and
retaining cybersecurity talents. Line of action 7 develops cybersecurity culture, such as digital
literacy and quality and truthfulness of information. In schools, raising awareness and
cybersecurity training are adapted to all levels, fields, and subjects. (Gobierno de Espãna,
2019, 38, 52-53, 56-57).
According to ENISA’s CyberHEAD database (2023), cybersecurity is taught in 23 different
higher education programmes (one bachelor’s programme, while the rest are master’s
programmes). Spanish National Cybersecurity Institute INCIBE (2021) has listed 168
institutions that provide training in cybersecurity of which 49 are universities. As examples,
Barcelona’s Ramon Llull University’s (2023) Master in Cybersecurity on-site study syllabus
includes subjects such as ethical hacking, security analysis, and cyber intelligence.
Mondragon University provides a master’s degree in data analysis, cybersecurity, and cloud
1361
computing. The studies include ethical hacking, internships in companies, and cybersecurity
management. (Mondragon Unibertsitatea, 2018.)
2.7. Greece
Of the five strategic goals in Greece’s cybersecurity strategy two (goals 4 and 5) address the
ambitions for higher education. For instance, the state will offer support to research and
development at the academic level. Such supportive actions include, for example, the reform
of curricula to cover more cybersecurity issues and enhanced collaboration between
academic and research institutions. One of the main flagship activities is the development of
an Education and Awareness Action Plan, which will include an analysis of the current
situation and an outline of targeted actions and activities for academic institutions. In relation
to academic education, emphasis is given on the preparation of future cybersecurity
executives. Suitable undergraduate and postgraduate study programmes are developed, and
student attendance is incentivised. (National Cybersecurity Authority, 2020, 63-69.)
Several state universities and private colleges offer studies in cybersecurity. There are 24
universities which are all accredited by the state. Only education given at universities is
considered higher education. (Study in Greece, 2023.) The CyberHEAD database lists four
universities and five master’s programmes (ENISA, 2023). For example, the International
Hellenic University offers a master’s degree in Cybersecurity, which main topics are digital
forensics, intrusion detection and network security, and security audit and compliance
(International Hellenic University, n/d). The University of the Aegean has a master’s
programme in Information and Communication Systems Security. Courses include network
security, database security, cryptography, privacy, cybersecurity, and forensics. (University
of the Aegean, 2022.)
3. Concluding remarks
Cybersecurity education varies significantly between the EU member states (for the full
qualitative and quantitative benchmarking, see Limnéll et al., 2023). Whereas countries like
Spain have dozens of higher education institutes teaching cybersecurity, others like Estonia
have only a few institutions and educational programmes for the topic. The numbers ought
to be contextualised, for example, with the size of the population and hence the estimated
national need for expertise. In general, the educational programmes tend to be specific and
focus on technical skills. A wider understanding of cybersecurity as a societal concern, civic
cybersecurity skills, and discipline geared cybersecurity competence building may be
lacking, even if some countries have also integrated these perspectives into curricula. Thus,
the educational breadth and depth are to be considered as well. For strengthening civic
cybersecurity competence, however, higher education is only a part of the puzzle. Alongside
experts, a skilled general population is needed to support secured digitalisation of society.
1362
References
Bundesministerium des Innern und für Heimat (BMI). (2016). Cyber security strategy for
Germany. Retrieved February 10, 2023, from
https://www.enisa.europa.eu/topics/national-cyber-security-strategies/ncss-
map/national-cyber-security-strategies-interactive-map?selected=Germany.
Chen, F., Lyu, J., & Wang, T. (2020). Benchmarking road safety development across OECD
countries: An empirical analysis for a decade. Accident Analysis and Prevention, 147,
205752. Doi: 10.1016/j.aap.2020.105752.
Directoratul National De Securitate Ciberneticá. (n/d). Directoratul National de Securitate
Cibernetica. Retrieved November 26, 2022, from dnsc.ro.
European Union Agency for Cybersecurity (ENISA). (2023). Cybersecurity Higher
Education Database CYBERHEAD. Retrieved January 31, 2023, from
https://www.enisa.europa.eu/topics/education/cyberhead#/.
Gobierno de Espãna, Presidencia del Gobierno. (2019). National Cybersecurity Strategy
2019, 38, 52-53, 56-57. Retrieved January 31, 2023,
fromhttps://www.dsn.gob.es/eu/file/2989/download?token=EuVy2lNr.
International Hellenic University. (n/d). MSc in Cybersecurity. Retrieved January 31, 2023,
from https://www.ihu.gr/ucips/postgraduate-programmes/cybersecurity.
Lehto, M. (ed.). (2022). Development Needs in Cybersecurity Education: Final report of the
project. University of Jyväskylä. Retrieved January 31, 2023, from
http://urn.fi/URN:ISBN:978-951-39-9469-3.
Limnéll, J., Alasuutari, M., Candelin, N., Cullen, K., Halonen, O., Helenius, M., Hermunen,
T., Lappalainen, J., Latvanen, S., Lindroth, M., Matilainen, T., Palonen, O.-P.,
Riiheläinen, J., Salminen, M., & Virkkunen, P. (2023). Cyber citizen skills and their
development in the European Union. Aalto University. Retrieved April 24, 2023, from
https://cyber-citizen.eu/en/materials/.
Maire, J.-L., Bronet, V., & Pillet, M. (2008). Benchmarking: methods and tools for SME.
Benchmarking: An international journal, 15(6), 765-781. Doi:
10.1108/14635770810915931.
Mondragon Unibertsitatea. (2018). Data Analysis, Cybersecurity and Cloud Compting.
Retrieved February 2, 2023, from https://www.mondragon.edu/en/master-degree-data-
analysis-cybersecurity-cloud-computing.
National Cybersecurity Authority. (2020). National Cybersecurity Strategy 2020-2025.
Ministry of Digital Governance, Hellenic Republic, 63-69.
Poupard, G. (2016). Processus pour l’obtention du label secnumedu. Peremier Ministre.
Retrieved February 10, 2023, from https://www.ssi.gouv.fr/uploads/2016/05/anssi-
secnumedu-p-01_v2_processus.pdf.
Premier Ministre. (2015). French National Digital Security Strategy. Retrieved February 10,
2023, from https://www.ssi.gouv.fr/actualite/the-french-national-digital-security-
strategy-meeting-the-security-challenges-of-the-digital-world/.
Ramon Llull University. (n/d). Master in Cybersecurity. Retrieved February 1, 2023, from
https://www.salleurl.edu/en/education/master-cybersecurity.
1363
Republic of Estonia. Ministry of Economic Affairs and Communications. (2019).

Cybersecurity strategy 2019-2022. Retrieved February 10, 2023, from
https://ccdcoe.org/library/strategy-and-governance/?category=cyber-security-strategies.
The Security Committee. (2019). Finland’s Cyber Security Strategy 2019. Government
Resolution 3.10.2019. Retrieved February 6, 2023, from
https://turvallisuuskomitea.fi/en/finlands-cyber-security-strategy-2019/.
Spanish National Cybersecurity Institute (INCIBE). (2021). Instituciones que imparten
formación en ciberseguridad en España. Retrieved January 31, 2023, from
https://www.incibe.es/sites/default/files/paginas/talento/catalogos-formacion/catalogo-
instituciones.pdf.
Study in Greece. (2023). Bachelor’s/Master’s/PhDs. Retrieved February 1, 2023, from
https://studyingreece.edu.gr/studying/studies-taught-gr/bachelors-masters-phds/.
Taltech. (n/d). Cybersecurity (MSc). Retrieved February 2, 2023, from
https://taltech.ee/en/cyber-msc.
Technical University of Cluj-Napoca. (n/d). Information and Computing System Security
(SISC) – Cybersecurity Master Program at Technical University of Cluj-Napoca.
Retrieved November 26, 2022, from utcluj.ro.
Université Bretagne Sud. (n/d). Cyberus Erasmus Mundus Joint Master in Cybersecurity.
Retrieved February 2, 2023, from https://master-cyberus.eu/.
University of the Aegean. (2022). Ασφάλεια Πληροφοριακών και Επικοινωνιακών
Συστημάτων. Retrieved January 31, 2023, from
https://msc.icsd.aegean.gr/masters/security/.
University Politehnica of Bukharest, Faculty of Applied Sciences. (n/d). Coding and Storage
Theory of Information Master, Prezentare. Retrived November 26, 2022, from tcsi.ro.
Universität Des Saarlandes. (2023). Cybersecurity Master of Science, Solving the challenges
of today and providing security for tomorrow. Retrieved February 8, 2023, from
https://cysec.uni-saarland.de/master/cybersecurity/.
1364
The impact of digital platforms enhancement on global virtual

teams’ engagement across selected business schools
Emil Velinov1, Juergen Bleicher2, Vincent Montenero3
1
Department of Marketing and Management, Skoda Auto University, Czech Republic,
2
Department of Economics, Duale Hochschule Baden-Wuerttemberg, Germany, 3MAIS
Institute at Technical University of Prague and CEFRES Institute, Czech Republic.
Abstract
This research explores the effect of the utilization and improvement of digital
platforms in teaching and learning, through a global virtual project that
involves business schools from five nations. The study concentrates on how
technology used in the education process at various business schools has
impacted the engagement of undergraduate and graduate students in global
virtual team work during the post-COVID period. Additionally, the paper
examines how the formations of global virtual teams of students has increased
significantly their motivation in the subjects such as international business,
strategic management and cross-cultural management . The study is based on
a blended learning international collaboration project involving 114 students
participating in global virtual collaboration through various digital platforms
in International Management and International Business classes. The results
demonstrate that through computer-supported collaborative learning, students
from all participating business schools were highly committed and motivated
in carrying out their assignments. The paper demonstrates that the integration
and use of digital platforms in both face-to-face and online classes in
International Business and International Management within this virtual
collaboration has offered students new knowledge and sustained their
motivation throughout the project.
Keywords: Global virtual teams; business schools; engagement.

1. Introduction
The ongoing global pandemic has brought about numerous changes in education, particularly
in higher education, where the integration of latest technologies has accelerated. At the same
time, students have altered their expectations of colleges and universities as new global and
regional trends have become the norm (Arora et al., 2020). Experiential learning has become
prevalent in business schools, with classes in international business and international
management being enhanced through global virtual teams, allowing students and instructors
to collaborate with counterparts globally, regardless of geographical barriers (Fayed et al.,
2021). The prevalence of digital platforms has also emerged thanks to exponential
advancements and the COVID era, significantly altering classrooms globally by
incorporating the latest technologies to manage this digital transformation (Velez-Calle et al.,
2020).
The main objective of this paper is to highlight the rapidly growing impact of digital
platforms on students' commitment in international business and international management
courses. The research covers five countries and five different business schools, including
both undergraduate and graduate students who collaborated virtually. The paper focuses on
determining the factors that motivate these five international business school students to
engage in virtual cooperation in the fields of international management, strategic
management, and cross-cultural management.
The experiential learning referred to in the paper involves two main projects: X-Culture and
BLIC (Blended Learning International Cooperation). The BLIC project was established in
2014 by the Baden-Wuerttemberg Cooperative State University (Dualle Hochschule) in
Germany. The aim of the BLIC project is to engage undergraduate and graduate students in
developing a business strategy to internationalize German companies by entering emerging
and advanced foreign markets. The BLIC concept aligns with a program established by the
German Ministry of Foreign Affairs to promote competitiveness, digitalization, and
internationalization of small and medium-sized German companies. Thus, Dualle
Hochschule Baden-Wuerttemberg (DHBW) established BLIC as an international project,
intensifying its international cooperation with international partner business schools through
a program promoting global virtual collaboration, taught at both the undergraduate and
graduate levels.
The BLIC program, has expanded to six countries across three continents, attracting students
from the US, South Africa, Iceland, the UK, the Czech Republic, and Estonia. The program
focuses on global virtual collaboration, where students act as managers and management
consultants and conduct cross-cultural studies and business strategy analyses for German
small- and medium-sized enterprises (SMEs). The program is an innovative teaching method
in international business classes, as it provides students with opportunities to improve their
1366
intercultural, language, technical, management and working skills. Business schools globally
are incorporating new approaches in their curriculum, such as virtual projects with partner
universities, to provide better preparation for their students and to foster international
experience. The paper's research questions focus on the impact of experiential learning and
the introduction of digital technologies on the motivation of bachelor and master students to
participate in virtual global teams from different business schools. The use of digital learning
platforms allows scholars to receive constant feedback from students and gather detailed data
on their engagement in global virtual collaboration.
2. Literature Review
2.1. Digital Platforms impact on Global Virtual Teams

In recent years, the use of digital platforms in teaching international business courses has
become increasingly popular. Adopting supportive technologies to facilitate learning has
become a widespread practice in higher education. The integration of online and blended
learning platforms into international business courses has greatly impacted and improved
education systems (Erpenbeck et al., 2015; Northey et al., 2015; Nortvig et al., 2018; K.
Smith & Hill, 2018). The use of innovative technology in higher education creates a future-
focused teaching and learning environment across business schools (Becker et al., 2018).
According to a meta-analysis by Chen et al. (2018), computer-supported collaborative
learning (CSCL) has resulted in a positive impact on knowledge acquisition, skill
development, and student perception (as shown in Table 1). During the pandemic crisis, the
rapid development of technology-assisted learning and teaching accelerated as education
providers sought to maintain their programs (MMB Institut, 2021; Wipper & Schulz, 2021).
The pandemic served as a catalyst in transforming traditional, factory-model education into
a student-centered approach that prioritizes individualized lifelong learning. In this context,
professionalizing learning involves leveraging technology to enhance it (Hattula et al., 2021;
Horn & Staker, 2017).
Nortvig et al. (2018) conducted a review of literature on e-learning and blended learning and
found that a common approach is to define online learning as the opposite of face-to-face
learning (Ryan et al. 2016). Online learning is described as being flexible in terms of time,
place, and pace, and is facilitated by web-based technologies that replace traditional brick-
and-mortar classrooms (Bernard et al., 2014; Chigeza & Halbert, 2014; Israel, 2015; Northey
et al., 2015; Potter, 2015). Pellas and Kazanidis (2015) highlight the use of learning
management systems (LMS) and virtual learning environments in online learning
environments. Virtual learning environment is characterized by involving the students in
face-to-face, blended learning modules or courses by using various digital platforms such as
MS Teams, Moodle, Canvas and others. The online learning environment consists of entirely
1367
online mode teaching and learning, which prevailed during global pandemic across the
business schools.
A technology-rich learning environment supports the underlying pedagogical approach. In
online learning settings, instructors have the opportunity to provide feedback and guidance
to facilitate learning outside the classroom. By combining traditional teaching methods with
technology-assisted methods, the learning environment can be enriched and both approaches
can benefit from each other. Under certain conditions, blended learning can be more effective
than online or face-to-face learning alone (Pellas and Kazanidis, 2015; González-Gómez et
al., 2016). In the BLIC program, students collaborate in virtual global teams and experience
realistic scenarios such as client-consultant relationships. This student-centered approach
prepares them for the global workplace and a blended learning approach was deemed the
most appropriate teaching method (Velinov, Bleicher, & Forrester, 2021). In the student-
centered approach, the focus is on the students, as they are working as managers or
consultants and they have to perform their tasks either in teams or individually. Thus, it
requires from them to learn constantly and to communicate explicitly and clearly with their
counterparts globally.
Despite the growing interest in blended learning, research in this area is still in its early stages
(Smith, 2018). Smith (2018) describes blended learning research as practical, specific, and
focused on outcomes. Despite the widespread dissemination of blended learning concepts in
recent decades, it is still considered a specialized rather than a general phenomenon.
According to Smith and Hill (2018), there is no standard understanding of blended learning,
and a joint conceptual framework is needed. However, they highlight meaningful
commonalities in definitions that emphasize the combination of online and face-to-face
learning (Graham, 2006; Garrison and Kanuka, 2004; Allen and Seaman, 2014). Different
definitions highlight different distributions of online and face-to-face teaching, but all
acknowledge the integration of virtual, technology-enabled communication with the
interventions of humans (Eaton, 2020; Graham, 2009; Le Rossignol, 2009; Nortvig et al.,
2018).
3. Methodology
This study collected data from 114 students from five business schools in Germany, UK, US,
Iceland, the Czech Republic, and Estonia who were enrolled in business and management
programs. The data was collected through an online questionnaire by applying Likert scale
and they were distributed via Google Forms and analyzed to determine the impact of digital
platforms and teaching innovations on student engagement in GVTs. The study also
compares the students' perceptions of the effectiveness of digital platforms and teaching
approaches before and during the pandemic period (2017 and 2022).
1368
The paper evaluates the effect of utilizing modern digital platforms in teaching and learning
in a select group of business schools on student engagement in Global Virtual Teams (GVTs)
as part of the BLIC and X-Culture programs. These programs take place each semester in the
same five business schools and offer students an opportunity to participate in experiential
learning cycles. These experiential learning cycles represent learning cohorts, which is either
module or entire course on international business, strategic management or cross-cultural
management stretched out from one month to four months. The BLIC program is smaller in
scale, with a focus on global virtual teams made up of bachelor and master male and female
students in the age of 18-25 years old who serve as managers and consultants, conducting
cross-cultural and market analyses. On the other hand, the X-Culture program is larger and
includes a wider range of project activities.
In the field of business education, courses in International Management and International
Business are widely taught at universities and business schools worldwide. These courses are
taught not just in English but also in local languages. Students in these courses are given a
variety of tasks, such as individual assignments, group presentations, written assignments,
peer reviews, critical thinking essays, and other deliverables, which help them develop their
analytical and team-working skills, as well as their entrepreneurial skills. The universities are
putting significant effort into internationalizing their curricula, which is a trend that is seen
across business schools and universities worldwide. Students connect with their foreign
counterparts mainly through virtual means, such as Skype, Facetime, WhatsApp, Viber, and
WeChat, in their Global Virtual Teams and group assignments. The majority of students
involved in the BLIC project rely on digital platforms for communication and also use
WhatsApp group chats and search engines such as Google for information (see Table 1 and
2).
The paper uses previous research by Chen et al. (2018) on the effects of computer-supported
collaborative learning (CSCL) on students' perceptions as its methodology. However, this
study stands out as it focuses on the impact of the global pandemic on the implementation of
global virtual teams (GVTs) where students from universities worldwide collaborate
remotely. Three key aspects are emphasized in the definition of blended learning: the students
have a degree of flexibility in terms of time, place, path and pace; this flexibility is combined
with the need for in-person learning under supervision; and finally, the authors highlight the
importance of a cohesive learning experience. The design of a blended learning course in
International Business must be such that content is taught both online and in person and that
the teaching methods complement each other seamlessly.
1369
4. Results
Tables 1 and 2 below explain what is the students’ perception on BLIC and X-Culture
projects and different factors have been evaluated by using the Likert scale. These factors
include utilization and intensity of using digital platforms by students for conducting their
tasks in GVTs. Also, it is evaluated students’ perception on their communication with other
students and teachers by the usage of wide varieties of digital platforms and communication
channels. Both tables represent consolidated data from all business schools within the field
research. The research findings indicate that students have a positive view of their
experiences in the BLIC and X-Culture projects. During the nationwide shutdowns in 2022,
the students were required to stay at home, but the adoption and use of digital platforms and
online learning were critical for them to stay connected with their team members and
complete their assignments. The students reported satisfaction with their team's performance
and the support they received from lecturers and team coaches through the search engines
such as Google, Bing and others reaching at the value of 6,289 out of 7 (see Table 1). The
lecturers were also evaluated highly by the participants, and the communication via
community or conference technologies was ranked high, indicating a strong level of
interaction between students and lecturers reaching at the value of 4,596 out of 7 (see Table
2).
Student interest in the X-Culture and BLIC projects was fluctuating throughout the academic
year, due to their busy schedules with jobs and multiple group assignments in each subject.
The students also expressed fear about working virtually with peers from other countries, as
it required them to do extra work, such as reading online information and maintaining virtual
contacts. However, students' motivation to work in GVTs remained high ,as they value the
communication with the teachers and students counterparts abroad by using available icloud
digital platforms, which enable the students to perform their tasks and assignments efficiently
(see the value of 4,026 on the cloud-based groupware for collaborative work.
5. Discussion
The paper describes how GVTs are a recent trend in International Management and
International Management courses at business schools and these courses are appealing to
students. The universities have been participating in the BLIC program since 2017 and all
students in the study sample worked virtually with their peers from bachelor and master
programs. The majority of Czech students had good English language skills, allowing them
to communicate and collaborate effectively with their international counterparts. Around 15-
20% of the 40 students per semester were international, with the majority being Russian-
speaking students who had proficiency in both English and Czech.
1370
When it comes to students' reluctance to work in GVTs, the main causes are fear of
communicating outside of their comfort zone, low knowledge of foreign cultures and work
habits, limited time to dedicate to GVT work, and most commonly, discomfort speaking in
English. Many of the students who participated in GVTs also had student jobs alongside their
studies and found it challenging to balance both. The students felt that the GVTs took up too
much of their time, and often asked their international management instructors if the X-
Culture or BLIC projects were mandatory or optional. The Czech students in particular had
low self-esteem when it came to GVT tasks and assignments and were not eager to
collaborate with non-Czech students.
Table 1. Usage of digital media and technologies by the students within BLIC and X-Culture
Programs.
N Min Max Means Std.Dev.
Utilization intensity of technologies and 114 1.000 7.000 3.149 1.915

tools: 1. Online libraries from the university

tools: 2. Internet search engines
(e.g.Google, Bing)

tools: 3.Social media portals (e.g.Facebook)

tools: 4.Video channels (e.g.YouTube
videos from experts)

tools: 5 Community/ Conference
technologies (e.g.Zoom,MS
Teams,Skype,WebEx,etc.)

tools: 6.Cloud-based services
(e.g.GoogleDrive,DropBox,OneDrive)

tools: 7.WeChat,Whats App, Viber App,etc.

tools: 8.Learning portals from the
university (e.g.Moodle,Canvas,Blackboard)
Source: Own elaboration *1=Not at all or very low utilization intensity, 7=Very high utilization intensity.
1371
Table 2. Digital tools for communication and interaction between students and teachers within
BLIC and X-Culture projects.
N Min Max Means Std.Dev.
Utilization intensity of technologies 114 1.000 7.000 4.105 2.212

and tools: 1. Online libraries from the
university

and tools: 2. Internet search engines
(e.g.Google, Bing)

and tools: 3.Social media portals
(e.g.Facebook)

and tools: 4.Video channels
(e.g.YouTube videos from experts)

and tools: 5 Community/ Conference
technologies (e.g.Zoom,MS
Teams,Skype,WebEx,etc.)

and tools: 6.Cloud-based services
(e.g.GoogleDrive,DropBox,OneDrive)

and tools: 7.WeChat,Whats App,
Viber App,etc.

and tools: 8.Learning portals from the
university
(e.g.Moodle,Canvas,Blackboard)
Source: Own elaboration *1=Not at all or very low utilization intensity, 7=Very high utilization intensity.
International Management and International Business courses, particularly when it comes to

Global Virtual Teams, can be demanding for students. They need to adjust to a large amount
of instructions, establish contact with their team members from around the world, and
maintain virtual cooperation for at least one semester. The challenges faced by global virtual
teams extend beyond pedagogical practices and require lecturers and tutors to understand the
broader context of students' readiness to cooperate and how it is influenced by online
communication (Johnson, 2013). Furthermore, the implications of cultural values and
1372
practices on student aversion to cross-cultural teamwork, as well as the role of trust-building

in virtual teams, need to be analyzed in order to help students achieve a committed
performance in virtual teams and networks (Hakanen, M., Kossou, L., & Takala, T., 2016).
6. Conclusion
The focus of this article is the impact of experiential learning in global virtual teams (GVTs)
on student motivation, as seen through the lens of the BLIC project. The authors believe that
incorporating international business education into hands-on, adjustable, and experimental
learning will bring more value to universities. Despite its small size and focus on partnership
and agile testing, the BLIC project has been successful in motivating students by providing
opportunities to develop business skills, strategic thinking, and communication abilities in
hybrid learning environments. Both the BLIC and X-Culture projects have positively
impacted student motivation by providing hands-on experience and international business
knowledge.
This paper has both academic and theoretical implications. From an academic perspective,
the authors found that experiential learning through GVTs has increased motivation among
students at the selected business schools, largely due to the use of various digital platforms.
The theoretical aspect of the paper highlights the role of digital platforms and virtual
interaction in GVTs and experiential learning. The findings suggest that virtual collaboration
enhances learners' mindset, digital inclusion, and motivation. To maximize the benefits of
virtual collaboration, students were encouraged to document their learnings through a
structured reflective report. The future development of GVTs is crucial in shaping high-
quality international business courses and keeping students motivated to participate and
contribute in class.
References
Allen, I. E., & Seaman, J. (2014). Opening the Curriculum: Open Educational Resources in
US Higher Education, 2014. Babson Survey Research Group.
Arora, A. K., & Srinivasan, R. (2020). Impact of pandemic COVID-19 on the teaching -
learning process: A study of higher education teachers. Prabandhan: Indian journal of
management, 13(4), 43-56.
Bernard, R. M., Borokhovski, E., Schmid, R. F., Tamim, R. M., & Abrami, P. C. (2014). A
meta-analysis of blended learning and technology use in higher education: from the
general to the applied. Journal of Computing in Higher Education, 26(1), 87–122.
https://doi.org/10.1007/s12528-013-9077-3
Chen, J., Wang, M., Kirschner, P. A., & Tsai Chin-Chung (88). The Role of Collaboration,
Computer Use, Learning Environments, and Supporting Strategies in CSCL: A Meta-
Analysis. Review of Educational Research, 2018(6), 799–843.
1373
Chigeza, P., & Halbert, K. (2014). Navigating E-Learning and Blended Learning for Pre-
service Teachers: Redesigning for Engagement, Access and Efficiency. Australian
Journal of Teacher Education, 39(11). https://doi.org/10.14221/ajte.2014v39n11.8
Eaton, M. (2020). The Perfect Blend: A Practical Guide to Designing Student-Centered
Learning Experiences. International Society for Technology in
Education.https://ebookcentral.proquest.com/lib/kxp/detail.action?docID=6191041
Erpenbeck, J., Sauter, S., & Sauter, W. (2015). E-Learning und Blended Learning:
Selbstgesteuerte Lernprozesse zum Wissensaufbau und zur Qualifizierung. Essentials.
Springer Gabler.
Fayed, I., & Cummings, J. (2021). Teaching in the post COVID-19 era. Springer
International Publishing. https://doi. org/10.1007/978-3-030-74088-7.
Graham, C. R. (2006). Blended learning systems. The Handbook of Blended Learning:
Global Perspectives, Local Designs, 1, 3–21.
Graham, C. R. (2009). Blended learning models. In Encyclopedia of Information Science and
Technology, Second Edition (pp. 375–382). IGI Global.
Horn, M. B., & Staker, H. (2017). The blended workbook: Learning to design the schools of
our future. Jossey-Bass.
Nortvig, A.‐M., Petersen, A. K., & Balle, S. H. (2018). A Literature Review of the Factors
Influencing E-Learning and Blended Learning in Relation to Learning Outcome, Student
Satisfaction and Engagement. The Electronic Journal of E-Learning, 16(1), 44–55.
Pellas, N., & Kazanidis, I. (2015). On the value of Second Life for students’ engagement in
blended and online courses: A comparative study from the Higher Education in Greece.
Education and Information Technologies, 20(3), 445–466.
https://doi.org/10.1007/s10639-013-9294-4
Potter, J. A. (2015). Applying a Hybrid Model: Can It Enhance Student Learning Outcomes?
Journal of Instructional Pedagogies, 17.
Ryan, S., Kaufman, J., Greenhouse, J., She, R., & Shi, J. (2016). The effectiveness of blended
online learning courses at the community college level. Community College Journal of
Research and Practice, 40(4), 285–298.
Smith, A. C. (2018). Developing Online Learning in the Helping Professions: Online
Blended and Hybrid Models (1st ed.). Springer Publishing Company.
https://ebookcentral.proquest.com/lib/kxp/detail.action?docID=5247361
Smith, K., & Hill, J. (2018). Defining the nature of blended learning through its depiction in
current research. Higher Education Research & Development, 38(2), 383–397.
https://doi.org/10.1080/07294360.2018.1517732
Velinov, E., Bleicher, J., & Forrester, P. L. Creating and Managing International Virtual
Teams of Students in Management Education. In Developments in Virtual Learning
Environments and the Global Workplace (pp. 124–140). https://doi.org/10.4018/978- 1-
7998-7331-0.ch007 (Original work published 2021)
1374
Evaluating the impact of strategies on students’ perceptions of

digital transformation – a case study of a Swedish higher
education institution
Ellen Mårtensson1, Carina Ström Hylén1, Henning Brink2, Sven Packmohr1
1
Department of Computer Science and Media Technology, Malmö University, Sweden,
2
Department of Organization and Information Systems, Osnabrück University, Germany.
Abstract
Digital transformation (DT) in Higher Education Institutions (HEIs) affects
the learning environment with digitally-enhanced teaching methods, student
assistance, and administration processes. HEIs develop strategies to exploit
the chances offered by DT. Our study investigates the connection between the
strategic work of HEIs on DT and how students perceive the results of this
work in their daily studies. We applied a case study design on a Swedish HEI
to gather our results. Results show that students are somewhat positive about
the strategic work but still perceive digitalization as a barrier to collaborating
with peers and lecturers. Our research contributes to knowing if the time and
effort spent on an HEI’s DT impact the student stakeholder group. By bringing
forth ways of improvement, we generate new knowledge about DT processes
in HEIs. Thus, we inspire educators and administrators in this industry by
putting forward lessons learned and improvements.
Keywords: Higher education; digital transformation; strategy; students’
perception; case study.

Evaluating the impact of strategies on students’ perceptions of digital transformation
1. Introduction
In Higher Education Institutions (HEIs), digital transformation (DT) affects the learning
environment with digitally-enhanced teaching methods, student assistance, and
administration processes (Reid, 2014). DT relates to integrating technology in all areas within
an organization, thereby transforming its operations to deliver better service (Weerawardane,
2021). To succeed on a journey to a digitally mature organization, a roadmap pushed by a
digital strategy is needed (Gobble, 2018). HEIs have specific strategies, resources, and
support mechanisms related to their DT. Students are an essential stakeholder group directly
affected by strategies and resources. It is of utmost importance for HEIs to know how their
students perceive the progress of their DT journey. Thus, our research questions are: To what
extent are strategic documents and students' perceptions regarding DT related to each other?
What should HEIs focus on to improve students’ perception of DT?
We choose a case study approach to investigate how Malmö University (MaU) in Sweden
works with its DT and how its students perceive it. In terms of the number of students, MaU
is among the ten largest universities in Sweden. MaU is actively working towards developing
digital competencies in staff, as well as promoting flexible learning environments for students
through blended learning (Malmö universitet, 2022). Especially the faculties of Teaching and
Technology drive research on digital learning. These factors contributed to the choice of
Malmö University as an adequate case.
The following presents a background section on DT in higher education (HE). After, we
inform about the case study design. The result section presents aspects of policy documents
combined with findings from survey data. Within the discussion, we will enhance results
with the current literature and suggest areas of improvement before concluding our paper.
Besides learning and teaching, several other dimensions need to be considered in DT, such
as administrative tasks, human resourcing, marketing, and DT governance (Benavides,
Tamayo Arias, Arango Serna, Branch Bedoya, & Burgos, 2020). DT is a highly complex
process. Hence, it is essential to look at the process holistically and place the different
stakeholders at the center of it (Benavides et al., 2020). Many different elements depend on
each other, e.g., managers, government, faculty, digital platforms, information systems,
teacher training units, teachers, and students (Benavides et al., 2020). DT might shift the roles
of teachers and students, as pedagogical concepts such as flipped classrooms are easier to
pursue with digital technologies. DT enables autonomous, flexible learning concepts and
enhances collaboration (Benavides et al., 2020). The DT affects all the different procedures
and services. Hence, HEIs require a complete institutional strategy (Gobble, 2018).
1376
Ellen Mårtensson, Carina Ström Hylén, Henning Brink, Sven Packmohr
Leaders of HEIs have identified four superior goals connected to DT: (1) improving students’
learning environment, (2) increasing operational efficiency, (3) stimulating innovation in
education, and (4) increasing computing power for cutting-edge research (Alenezi, 2021).
These goals imply the importance of integrating digital tools into existing systems to rethink
the concept of HEIs. However, DT has mostly been seen as a support for the existing learning
and teaching environment instead of contributing to a greater disruptive change (Alenezi,
2021). Similarly, some scholars suggest that students’ digital competence should be
promoted by using digital tools early in all courses (Gulliksen, 2017). In doing so, HEIs help
meet future societal demands for advanced digital skills in the population (Gulliksen, 2017).
The Swedish Higher Education Authority suggested the creation of a national strategy for
digitalization in HE (Universitetskanslersämbetet, 2022). While today there is no nationally
DT strategy for HEIs, local alternatives can be found in universities’ guiding documents.
Regularly, these documents focus on digital and/or blended learning. Other aspects of DT
were seemingly not included (Ljungqvist, 2018). Goals in the documents do not always
coincide with teachers' experiences. A comparison of Swedish university teachers’
experiences with universities’ documents regarding digital teaching identified discrepancies.
Often, the documents anticipate digital teaching to be efficient and time-saving. Instead, the
teachers’ experiences indicate an increased workload (Ljungqvist, 2018). Teachers’ and
students’ experiences of digitalization at Gothenburg University show that the Covid-19
pandemic boosted the use of digital teaching tools. Even if digital teaching existed, it was
seldom used before (Wackenhut & Gillette, 2022). The pandemic has removed many
constraints. The future will show how future guiding documents will be influenced by the
learnings of enforced digitalization during the pandemic. A DT strategy cannot encompass
wordings in a document. Organizations must perform strategically by strategizing, having a
strategy, and being strategic (Gulbrandsen & Just, 2020). Words, resources, and stakeholders
are essential to consider when an organization's strategic efforts are to be reviewed.
3. Method
We investigated MaU´s DT process in 2021 by using a case study design. According to Yin
(2013), case studies are performed in five distinguished steps: (1) Plan by identifying why a
case study is preferable; (2) Design by defining the units to study; (3) Prepare by screening
potential cases, (4) Collect evidence through data sources; and (5) Analyze by identifying
patterns and developing an analytical framework. As our topic is complex and needs a holistic
observation by understanding a real-world case in an authentic context, we deem a case study
to be an appropriate approach (step 1). The units being studied (step 2) are MaU’s policies -
as an expression of its strategic DT work - and the perceptions of MaU’s students regarding
its DT. This study can be seen as a single case study preparing for further research (step 3)
and informed suggestions for MaU. For a case study, it is expected that different materials,
1377
such as documents and interviews, must be collected (Creswell & Poth, 2017). We collected
several documents (step 4) containing Malmö University´s DT strategies and resources, such
as the official document “Strategy 2022” (Malmö University Board of Governors, 2017),
which contains the university’s goals and visions between 2018 and 2022. Additionally, we
examined strategy documents before 2018 to follow up on its transformation (Lindquist,
2017). We inspected its organizational structures. Regarding the student's perceptions of the
university´s DT, we have used data collected from a survey conducted in 2021 with
participants of semesters two and five from Malmö University within the study program
Bachelor in Media Technology (Packmohr & Brink, 2022). In total, 156 students participated.
Participants had to rate 30 DT-related items on a five-point Likert scale (Vogelsang, Brink,
& Packmohr, 2020). From the survey items, we have selected 13 as they reflect on the
strategy documents. We analyzed different materials (step 5) to learn how the documents
influence the university’s operations and affect the students’ perceptions. We use analytical
generalization to reject, modify, corroborate, or advance existing theories (Yin, 2013).
4. Results
MaU’s strategy document from 2018 to 2022 (Malmö University Board of Governors, 2017)
stated that the primary stakeholders were staff and students. Relations to digitalization are
found in the focus areas of identity and role in society and the university environment. Within
these areas, the strategy states specific goals such as digital competence and offers. MaU
wants to generate quality in education and research by making the best of digital
improvements. Further goals regarding developing the digital work and learning environment
to meet present and future requirements for digital skills can be found. The document states
that the university will use the potential of digital technologies. MaU possesses several
Advisory Boards besides its faculties. One of them is the Advisory Board for Digitalisation.
Its mission is to manage the work of digitalization and point out directions to achieve the
vision of being at the forefront of digital technology in education, research, and operational
support (Malmö Universitet, 2023). The Board for Digitalisation advises the vice-chancellor
and the University Board. The chairman of the advisory board is the dean of the Faculty of
Technology. Representatives from different departments are part of the board. The Advisory
Board of Digitalisation interprets digitalization as more than mere integration of different
tools. Instead, digitalization is a cultural change. The application of digital technologies
facilitates work and enables new initiatives in the academic environment (Malmö Universitet,
2023). Other contributions have been made, such as appointing a university-wide
digitalization officer or other activities organized by the Center for Teaching and Learning,
such as workshops, implementation of digital exams, or courses such as Collaborative
learning in digital learning environments. In September 2020, the university launched the
so-called Digihub, where students and staff members can borrow digital equipment. Students
1378
can connect to other students as digital mentors to get help with online studies, mainly
regarding tools such as dlp and collaborative tools, and content production. Digihub is
gathering different resources and offers in the same place, such as media production, IT
support, mentors, and pedagogical support. Since 2018, MaU has used Canvas as its digital
learning platform, which provides a rather open structure. There are many possibilities for
designing a course in terms of using dlp tools and layouts. MaU offers courses on using
Canvas for staff and students via the platform itself (Malmö Universitet, 2023).
Table 1. Survey results.
(#) Item in keywords Agree Rather Neither Rather Disagree

agree nor disagree
(1) necessary technical means to
64,7% 21,2% 9,0% 4,5% 0,6%
use digital course offerings
(2) university moving forward in
24,4% 42,3% 23,7% 9,0% 0,6%
terms of DT
(3) sufficiently trained in the use of
9,6% 20,5% 24,4% 26,9% 18,6%
the dlp
(4) digital teaching has negative
34,6% 30,8% 11,5% 16,7% 6,4%
effects on collaboration with peers
(5) difficulties in motivating to
20,5% 26,9% 10,3% 22,4% 19,9%
participate in digital courses
(6) not enough resources for digital
9,6% 19,9% 32,1% 23,7% 14,7%
teaching
(7) lecturers have IT skills to handle
14,7% 24,4% 24,4% 26,9% 9,6%
dlp
(8) university offers digital services,
25,6% 41,0% 21,2% 9,6% 2,6%
which support my studies
(9) digital teaching worsens
35,3% 32,1% 15,4% 12,8% 3,8%
exchange with lectures
(10) University has clear vision/DT
5,8% 24,4% 45,5% 17,9% 5,8%
strategy
(11) University management
19,2% 35,9% 39,7% 3,2% 0,6%
supports DT
(12) learning culture at university
7,7% 21,2% 32,1% 26,3% 12,2%
has not changed due to DT
(13) university strives to constantly
10,9% 40,4% 34,6% 12,2% 1,3%
improve to transform digitally.
From a larger survey (Packmohr & Brink, 2022), we selected 13 items related to MaU’s DT
work. For an exploration of the data, table 1 represents the frequencies of the possible degrees
of agreement/disagreement. Students are confident that they possess the necessary technical
means (1). On the contrary, students perceive rather insufficient training on the dlp (3).
1379
Collaboration through digital channels with fellow students (4) and lecturers (9) is perceived
as negative. In comparison, perceptions of problems in motivation (5) are fairly evenly
distributed. Regarding resources (6), students tend to have a slightly positive perception. A
balanced perception is visible regarding lecturers’ skills (7). With MaU’s strategy (10), there
is a tendency toward the middle. Additionally, the engagement of the top management (11)
is perceived as positive. Regarding the learning culture (12), there is a tendency toward the
middle. On a positive note, students perceive MaU as open to improvements within DT (13).
5. Discussion
Considering the wording regarding DT within MaU’s strategic documents, there is a high
overlap with definitions from the field (Reid, 2014). MaU wants to use DT to affect the
university's culture, which students do not entirely perceive as changed yet. One part of a
changed learning culture is a digital-nomad pattern (Nurhas, Aditya, Jacob, & Pawlowski,
2022), which offers students the opportunity to study and gain other experiences
simultaneously. MaU might more explicitly communicate such opportunities to students.
MaU’s Advisory Board of Digitalisation comprises representatives from several
departments, which ensures a broad administrative perspective. However, neither students
nor teachers are included. Scholars highlight the importance of having all stakeholders
involved (Benavides et al., 2020). Monitoring of the two stakeholder groups within the core
operations seems to be imperative. Students find digital courses challenging regarding
collaborations with peers and motivating themselves, despite MAU offering the course
Collaborative learning in digital learning environments for teachers. A solution is to enable
more interpersonal exchange and counseling (Draxler-Weber, Packmohr, & Brink, 2022).
Instead of setting up a top-down solution, co-design methods with the intended users as an
active part of the design process (Sanders & Stappers, 2008) could be a way to address these
challenges. The students perceive MaU as at the forefront of its digitalization. It seems MaU’s
efforts on DT are paying off. Most students judge MaU´s digital services to support students
during their studies as positive. This might be connected to the visibility of DigiHub, which
is available for guidance. Looking at DigiHub, a recommendation could be an even stronger
connection to the universities core operations. Practically, DigiHub’s personnel could
become part of courses. The students give scattered answers regarding teachers' knowledge
of working with the digital platform. One reason might be that Canvas’ open structure enables
a great degree of freedom in how courses can be designed and executed. This might lead to
the widespread perception of teachers' competencies but might also relate to the dilemma
between standardization and adaptation (Aili & Bjarnason, 2020). Another problem is the
time lecturers need to adapt to digital teaching (Universitetskanslersämbetet, 2022). Even if
courses and guides are available, developing competencies takes time, especially in faculties
that are less prone to technical and/or pedagogical education. The dilemma of standardization
1380
and adaptation might also reflect on the answers regarding sufficient training. As there is no
objective standard on the dlp, students need to be trained for adaptability. It seems the
university is giving the students the impression that it continuously improves on its
digitalization. Still, the Covid-19 pandemic might have triggered many changes, which
students perceive as an effort of the university (Boström, Collén, Damber, & Gidlund, 2021).
6. Limitations and Conclusions

To analyze efforts on a strategic level, our study connected a university's strategy documents
to its students' perceptions. In general, the perception of students is positive. Regarding
collaborations and motivation, we found more negative perceptions. Thus, recommendations
are to include stakeholders more in the DT process. Even if a single case is examined, a case
study should draw general conclusions (Yin, 2013). Most universities need help with their
DT. MaU is a common example of an HEI that is pursuing its DT, leading to the fact that our
recommendations could be applicable in diverse settings. A specific advantage could be the
Swedish government’s digitalization vision for the whole state. As this is a single-case study,
further research could include a multiple case study design (Yin, 2013) or collect additional
data, e.g., instructors’ perceptions. In addition, more in-depth statistical analyses could
provide further insights. Our quantitative data was collected during the Covid-19 pandemic.
Students might be positively biased toward MaU’s digitalization work.
References
Aili, C., & Bjarnason, E. (2020). Lärares digitala arbetsmiljö vid Lunds universitet:
Slutrapport för LärDA projektet 2019/2020.
Alenezi, M. (2021). Deep Dive into Digital Transformation in Higher Education Institutions.
Education Sciences, 11(12), 770. https://doi.org/10.3390/educsci11120770
Benavides, L., Tamayo Arias, J., Arango Serna, M., Branch Bedoya, J., & Burgos, D. (2020).
Digital Transformation in Higher Education Institutions: A Systematic Literature Review.
Sensors, 20(11), 3291. https://doi.org/10.3390/s20113291
Boström, L., Collén, C., Damber, U., & Gidlund, U. (2021). A Rapid Transition from Campus
to Emergent Distant Education; Effects on Students’ Study Strategies in Higher
Education. Education Sciences, 11(11), 721. https://doi.org/10.3390/educsci11110721
Creswell, J. W., & Poth, C. N. (2017). Qualitative inquiry and research design: Choosing
among five approaches. (Fourth edition.). Sage Publications.
Draxler-Weber, N., Packmohr, S., & Brink, H. (2022). Barriers to Digital Higher Education
Teaching and How to Overcome Them—Lessons Learned during the COVID-19
Pandemic. Education Sciences, 12(12), 870. https://doi.org/10.3390/educsci12120870
Gobble, M. M. (2018). Digital Strategy and Digital Transformation. Research-Technology
Management, 61(5), 66–71. https://doi.org/10.1080/08956308.2018.1495969
Gulbrandsen, I. T., & Just, S. N. (2020). Strategizing communication: Theory and practice.
Printing Solutions.
1381
Gulliksen, J. (2017). Institutionalizing human-computer interaction for global health. Global

Health Action, 10(sup3), 1344003. https://doi.org/10.1080/16549716.2017.1344003
Lindquist, H. (2017). Malmö universitet 2018: Mångfald, kreativitet och
samhällsengagemang. Malmö: Malmö högskola. SwePub.
Ljungqvist, M. (2018). Diskurser om undervisning med digitala verktyg: En kritisk
granskning av relationen mellan universitetslärares beskrivningar och lärosätens
styrdokument. Högre utbildning, 8(2), 90. https://doi.org/10.23865/hu.v8.1343
Malmö universitet. (2022). Vision, mål och strategi 2025. Retrieved January 5, 2023, from
https://mau.se/om-oss/vision-mal-och-strategi/
Malmö Universitet. (2023). Digitalisering | Medarbetarwebben. Retrieved January 5, 2023,
from https://medarbetare.mau.se/ledning-och-organisation/digitalisering/#accordion-
28713
Malmö University Board of Governors. (2017). Malmö universitet Strategy 2022. Retrieved
from https://mau.se/contentassets/c8c53c0e8d464830ba798ed2485a7c17/malmo-
university-strategy-2022_eng.pdf
Nurhas, I., Aditya, B. R., Jacob, D. W., & Pawlowski, J. M. (2022). Understanding the
challenges of rapid digital transformation: The case of COVID-19 pandemic in higher
education. Behaviour & Information Technology, 41(13), 2924–2940.
https://doi.org/10.1080/0144929X.2021.1962977
Packmohr, S., & Brink, H. (2022). Comparing online and on-campus students’ perceptions
of the digitalization of higher education institutions. 8th International Conference on
Higher Education Advances (HEAd’22). Presented at the Eighth International
Conference on Higher Education Advances. Editorial Universitat Politécnica de
Valéncia. https://doi.org/10.4995/HEAd22.2022.14670
Reid, P. (2014). Categories for barriers to adoption of instructional technologies. Education
and Information Technologies, 19(2), 383–407. https://doi.org/10.1007/s10639-012-
9222-z
Sanders, E. B.-N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design.
CoDesign, 4(1), 5–18. https://doi.org/10.1080/15710880701875068
Universitetskanslersämbetet. (2022). Satsa på en nationell strategi för digitalisering av högre
utbildning och forskning. Retrieved December 30, 2022, from
Universitetskanslersämbetet website: https://www.uka.se/om-oss/
Vogelsang, K., Brink, H., & Packmohr, S. (2020). Measuring the Barriers to the Digital
Transformation in Management Courses – A Mixed Methods Study. In R. A. Buchmann,
A. Polini, B. Johansson, & D. Karagiannis (Eds.), Perspectives in Business Informatics
Research (pp. 19–34). Cham: Springer International Publishing.
https://doi.org/10.1007/978-3-030-61140-8_2
Wackenhut, A. F., & Gillette, M. B. (2022). Covid-19 and (re)learning teaching: Never let a
crisis go to waste. Högre Utbildning, 12(1), 52. https://doi.org/10.23865/hu.v12.3562
Weerawardane, D. (2021). Digital transformation of Higher Education: What’s next? New
Vistas, 7(2). https://doi.org/10.36828/newvistas.147
Yin, R. K. (2013). Case study research: Design and methods. (5. ed.). SAGE.
1382
My first six months (MF6M): industrial design students

perceptions of their first semester experience of learner-centred
design studios
Helen McLean1, William Dim2, Soumitri Varadarajan2, Caroline Francis2
1
Learning & Teaching, RMIT University, Australia, 2Industrial Design, RMIT University,
Australia.
Abstract
This paper reports on a learning and teaching project investigating the
experience of Industrial Design students in their first semester of study in three
courses that were reimagined to increase students’ agency in shaping their
learning. In this five year project, My First Six Months (MF6M), teachers
aimed to create a learner-centred environment by using facilitative, rather
than directive, teaching strategies in a studio-based context. Focus groups
with students explored their experience of the courses and unearthed valuable
insight into students’ expectations about learning at university and the design
discipline. Students were overwhelmingly positive about collaborating in their
projects, while diverse expectations about learning and teaching roles and
bemusement about the different emphasis on assessment featured prominently
in students’ reflections. Recommendations arising from their perceptions
include providing an ongoing narrative about independent learning
throughout each course and ensuring support and expectations for
independent learning are more explicitly designed into the course delivery.
Keywords: Industrial design pedagogy; progressive education; learner-
centred.

MF6M: ID students perceptions of their first semester experience of learner-centred design studios
1. Introduction
My First Six Months (MF6M) is a five-year learning and teaching initiative that commenced
in 2018. The initiative involved reimagining three first-year studio courses with learner-
centered pedagogies in an undergraduate Industrial Design degree at an Australian
metropolitan university.
The objective of the foundation year courses was to introduce students to learner-centered
learning and teaching methods during their first semester, with the aim of establishing their
independence as learners in these courses and their future studies. The initiative sought to
depart from the traditional tertiary education model, which relied on an assumed or
unconscious enculturation of students in design practice (Webster, 2007).
A purposeful shift in classroom power dynamics from the teacher to the learner (Goodyear
& Dimitriadis, 2013) promoted personal and social construction of knowledge (Bandura,
1997). The ‘students as partners’ (Matthews, 2017) approach was employed from their first
steps, to engage their independence as lifelong learners into these courses (McLean &
Varadarajan, 2020) and its objective was to cultivate students’ independence and self-
regulation as learners (Bandura, 1997), fostering their individual development and
confidence (Boud, 2000).
The MF6M courses were deliberately designed to challenge the roles of both learners and
teachers. The students received less direction from tutors in leading and evaluating learning
than it was assumed they would be familiar with, as first-year university students. Tutors
were instructed not to "talk at students" by way of lectures or seminars but to embody
partnering dispositions and work alongside learners to support and prompt but not lead
learning. Despite diverse modes and contexts of practice over the life of the initiative, the
approach to learning and teaching remained consistent.
This paper addresses a period in the course where classes in each course were combined as a
weekly three-hour studio workshop for approximately 60 students enrolled. The teaching
team (three tutors) physically located themselves to be among students in the workshop space
to facilitate and guide the weekly activities where students needed to be highly active and
participatory in exploring, experimenting, researching and constructing design solutions
collaboratively.
Each course provided an extensive set of digital resources for students to explore according
to their own interests and needs for researching, prototyping and making design solutions.
Site visits to professional studios and industry locations along with evening talks with
professionals were also offered as additional learning opportunities.
1384
Helen McLean, William Dim, Soumitri Varadarajan, Caroline Francis
2. Realistic
A key feature across the MF6M approach involved students working in small groups to
develop responses to design briefs, setting their own solutions and standards according to
their combined skills and knowledge. Classes were free from the direction and external
influence of an expert or ‘master’ and students were provided with a range of digital
resources, site visits and expert talks that were not prescribed throughout each course but
offered for students to curate and engage with for their own needs and interests. Using
inherent design processes of pinup, peer review and self-assessment, there was an underlying
objective for students to also begin developing their own understanding and language of
design concepts and standards nurtured through emerging and ongoing dialogues about
quality and standards of their work (Boud, 2000; Sambell, McDowell, & Montgomery,
2012). At heart, these methods sought to seed students’ independence and to avert a ‘master’
and ‘apprentice’ model of learning that may become construed through power relations often
established in assessment practices like the studio review or presentation (Webster, 2007).
At the outset of MF6M it was assumed that the introduction of learner-centred methods would
stimulate shifts in students’ expectations about roles and responsibilities in learning and
teaching.
3. Methodology
A research proposal was designed to investigate and evaluate the student experience of
MF6M. It acquired University Ethics approval and was led by an academic developer who
was external to the School that the project was based in.
Focus groups were used to acquire students’ perceptions of their experience in these courses
and students were invited, rather than selected, to participate. The focus groups were
promoted by posters and in-class visits by the academic developer, as well as prompts from
teaching staff to students. Of the approximately 60 students enrolled across the three courses,
five students from those courses responded and opted in. The diverse student sample of five
was randomly assembled into two groups, of whom four were school leavers - three females
(one who had previously engaged in Industrial Design study in Semester 2, 2017) and two
males (one who was mature-aged in his mid-twenties), all were domestic students.
The participants signed consent forms and were advised they could cease involvement in the
study at any stage and that their participation was anonymous, their contribution would be
de-identified and what they disclosed would have no impact on their assessment results for
this and other courses in their studies.
Each of the interviewed students had experienced at least one of the courses in the MF6M
project, though did not need to have had previous experience of the program to share
1385
perceptions about the MF6M course they were enrolled in. Questions based on the following
topics: Structure of the Course, Learning Activities and Learners and Teachers Roles, were
issued before the focus group, for participants to consider.
Transcripts of the focus groups were commercially transcribed, analyzed using NVivo for
themes, and shared with participants for member-checking. None of the participants had any
objection or comments about the veracity of the transcribing or that the content of the
interviews should be changed or removed. To further correlate the focus group responses,
Course Evaluation Survey (CES) qualitative comments were also thematically analyzed. An
inductive approach was used to allow unanticipated responses and insights to emerge, in
addition to the deductive process for identifying initial perceptions.
4. Findings
Following is an analysis of the students’ perceptions that emerged from the above-mentioned
topics of the focus group questions.
4.1. Structure of the Course

Students indicated that they liked the course structure, which consisted of an initial three-
week ‘intensive’, a seven-week project development phase, and a two-week portfolio
preparation period. Weekly classes had an open approach with collaborative group activities
instead of lectures and seminars, which students enjoyed for the sociability, interaction, and
diversity of feedback from the tutors, particularly around assessment tasks and outcomes.
Although some students were still uncertain about expectations, they generally enjoyed the
class structure as it encouraged active discussion and sharing of work and feedback. Engaging
with their peers' work inspired and motivated those students, allowing for the construction of
shared standards and expectations, rather than relying on external frameworks. This often
served to inspire or motivate students through a sense of competition or awareness of new
possibilities for their own work.
However, the ambiguous nature of the project thematic still left some wanting clearer briefs
and more formative feedback from the teaching team to establish confidence in requirements
and assessment tasks. Overall, students voiced that the design studio activities such as
presentations and pin-ups were useful for facilitating learning and keeping on projects track.
4.2. Learning Activities

The students referred to learning activities that are inherent to design practice (pinups,
presentations, site visits, peer feedback and review), and other activities that were designed
to develop their independence as learners and emerging professionals (group work, self-
assessment, industry and site visits).
1386
Design Practice: Students in the course found the pinup and work presentation practices to
be positive and motivating experiences that allowed them to interact and engage with their
peers and the teaching team. Despite some initial uncertainty about the requirements, students
soon recognized the value of these activities in providing formative feedback for improving
their work and progressing ideas.
While students appreciated the peer feedback and review process, they also felt that it could
be constraining. Some students found it challenging to provide constructive feedback, as P4
noted that some students tended to focus too narrowly on one aspect of the work. Similarly,
P3 found that the feedback they received could be obvious at times. All students
acknowledged that giving and receiving feedback was a valuable skill that they needed to
develop further.
Activities for Learning for Independence: Students were overwhelmingly positive about
working in groups and found it to be a valuable learning experience. They appreciated the
opportunity to collaborate and recognized that working with others had multiple benefits for
developing their design skills and socialization. Though, student P3 was also aware of the
challenges of shared work, with some students feeling uncertain about what they could claim
for their individual assessment.
While students were positive about peer feedback and review activities, they also wanted
more ‘expert’ critical feedback from tutors to scaffold their understanding of standards in the
discipline. Students P2 and P4 also expressed bemusement at the social dynamics and
competitiveness that sometimes emerged in the peer review process, such as when peers
would not accept constructive feedback.
Self-assessment was seen as a valuable tool for developing self-regulation skills and
reflecting on learning, but student P1 and P3 were not as enthusiastic about it or fully
confident in the veracity of the process. There was a sense of uncertainty about the standards
they were assessing to and how to judge their own work. Despite this, students generally
found the MF6M course structure to be engaging and collaborative, although students P4
suggested that clearer briefs and more formative feedback from the teaching team would have
been helpful to contextualize the outcomes of self-assessment.
4.3. Learner and Teacher Roles

Facilitating rather than Directing: First-year students had mixed responses to tutors not
taking the lead in directing their learning. Some enjoyed the freedom to follow their own
interests and adapt briefs to their skills, while others expected more specific guidance,
particularly for managing their time and maximizing their efficiency in the learning process.
In addition to guidance, students also expected tutors to help them understand difficult
concepts and theoretical constructs. They liked hearing tutors give their opinions to challenge
1387
them. For example, student P4 found a class particularly memorable when the course
coordinator gave a provocation about design and waste. Student P4 also mentioned, although
it annoyed some, they thought it was a useful wake-up call.
Guidance and Clarity: While students were trusting to engage in introductory activities that
inducted them to the design discipline and the program, they wanted more clarity about the
specific requirements and standards for learning activities and tasks that were oriented to
assessment. They also expected specific guidance for managing their first encounters with
design education protocols and practices like the pinup and presentation.
Evaluating Learning: During the focus group, students expressed confusion about the self-
assessment process and its relationship to their final grade. Student P1 was unclear about
their role in the process and how it related to the criteria for assessment. There was also
uncertainty about the moderating process and its purpose. Student P3 recounted how a tutor
explained that if a self-assessment was an outlier to the trend, that it would be brought back
in line. Regardless, some students remained unsure about the inconsistencies they observed
in how peers were assessing their work. For those who wanted to participate thoughtfully,
they were uncertain about the standards expected and how to assess their work. Students P1
and P4 concurred that although they successfully constructed an understanding of a
benchmark, that there was still a lack of definitive measure.
5. Discussion
Students highly valued the social opportunities in their courses and were overwhelmingly
positive about group work. As first year students, they demonstrated a maturity in managing
social and group dynamics under pressure while working collaboratively in groups. It was
perceived as an inherent aspect of design practice. Leveraging each other's skills, learning’s
and talents through the constant public presentation and discussion of projects, it seemed they
were able to garner more appropriate, implicit and direct feedback, which produced a higher
quality of work outcomes. These identified benefits clearly resonate with the qualities that
Ashford-Rowe, Herrington, and Brown (2014) associate with authentic assessment.
The most significant and underlying issue to emerge from the interviews was that a number
of students were still unsure about the intended purpose of the self-assessment process. Their
perceptions revolved around inconsistencies and uncertainties running through the process,
such as: lack of awareness of moderation, students ignoring peer feedback and grading
themselves high, or others taking the process seriously and genuinely reflecting on their
learning but getting little confirmation of standards. As first year students who mostly had
not experienced self-assessment previously, these doubts about the process are
understandable and the issues they describe have been acknowledged by others, including
Sambell et al. (2012).
1388
The dominance of grades in shaping students' attitudes towards learning is notable, as

reflected in their inclination to equate the process of self-assessment with the final grade
(Biggs & Tang, 2011; Sambell et al., 2012). This focus on grades detracted from the cognitive
process of reflecting on their own work. Additionally, students were influenced by social
dynamics and competition, through seeming arrogance or ignorance, ‘appeared’ to result in
some awarding inflated grades, thereby raising concerns about the fairness of the moderation
process, even though they knew that extreme grading would be moderated by the teaching
team. It may be that at the time of the focus groups, students hadn’t received their final grades
for the course and thus had not been able to ‘close the loop’ or examine the gap between their
own estimations and the moderation process, and thus be confident in the integrity of it.
The language used by students throughout the interviews suggests that most were not
immediately ready to be independent ‘partners of learning’, a key intention of the MF6M
initiative. Although comfortable to be creative in exploring ideas, problem solving and
prototyping for learning activities, they tended to convey notions of dependency on tutors,
often expressing need for ‘clarity’, ‘guidelines’, ‘consistency’, ‘standards’, ‘benchmarks’,
particularly around assessment tasks, or ‘explain a little bit more’, ‘more feedback’, ‘I didn’t
really know where I was at’ for learning activities. However, the language they use to
describe the deficiencies that they perceived also points clearly to needs that are relational
and dialogic, resonating with strategies that Matthews (2017) proposes as essential for a
successful ‘students as partners’ model. Additionally, as first year students in semester one,
they did display strong abilities to set goals, problem solve and complete work, suggesting
that with some clearer guidance in the set up and defining of expectations, independence and
partnering as learners is a clearly achievable goal for the next stages of the program.
6. Recommendations
The following key points emerged from the student experience and suggestions for
addressing them:
1. Establish clear context and ongoing narrative to reorient students towards increased
independence and self-responsibility as learners and partners in learning.
2. Provide a clearer definition of responsibilities for teachers and students to align
expectations.
3. Offer guidance and assistance for students to navigate and engage with the extensive
range of resources available.
4. Provide explicit guidance, feedback, and benchmarks early on to manage anxiety,
support awareness of standards and expectations, and gauge student development.
5. Socialize assessment as a learning process to embed a changed mindset and
understanding of lifelong assessment literacy.
1389
6. Ensure integrity of self-assessment results through moderation and reflection on the

process of self-assessment and responding to feedback, potentially ‘unhooking’ grades
from the assessed product.
7. Reflect and share experiences of ‘partnering’ at the end of the course to confirm
milestones and expectations around independence and self-regulation development.
8. Establish a contract of expectations and responsibilities to support busy students in
planning their engagement with the course's rich experiences and resources.
7. Conclusion
In summary, the students who participated in the focus groups had positive perceptions of
the three courses within MF6M, recognising the courses' purpose of preparing them for the
complexities of the world. They valued the emphasis on group work and its relevance in both
their design education and personal development. Some students even acknowledged the
courses' aim to promote innovative thinking and social responsibility within the design field.
However, the students also shared aspects of their experience that did not meet their
expectations, including uncertainty, feeling overwhelmed, and discrepancies between their
assumptions and the course expectations. As this project aimed to introduce new social norms
and responsibilities for learning, it is crucial for students and teaching staff to establish and
maintain a culture of effective communication and mutual support throughout the program.
Doing so will ensure that students remain confident and committed to their learning not only
during their time at university but throughout their future lives.
References
Ashford-Rowe, K., Herrington, J., & Brown, C. (2014). Establishing the critical elements
that determine authentic assessment. Assessment & Evaluation in Higher Education,
39(2), 205-222.
Bandura, A. (1997). Self-efficacy : the exercise of control. New York: W.H. Freeman.
Biggs, J., & Tang, C. (2011). Teaching for quality learning at university: What the student
does (4th ed.). Maidenhead: McGraw-Hill Education.
Boud, D. (2000). Sustainable Assessment: Rethinking assessment for the learning society.
Studies in continuing education, 22(2), 151-167.
Matthews, K. (2017). Five Propositions for Genuine Students as Partners Practice.
International Journal for Students As Partners, 1(2).
McLean, H., & Varadarajan, S. (2020). My First Six Months: Students’ Perceptions of
Learner-Centred Design Studios. In (pp. 143-153). Singapore: Springer Singapore.
Sambell, K., McDowell, L., & Montgomery, C. (2012). Assessment for Learning in Higher
Education. London, UNITED KINGDOM: Taylor & Francis Group.
Webster, H. (2007). The Analytics of Power: Re-presenting the Design Jury. Journal of
Architectural Education, 60(3), 21-27.
1390
Student perceptions of a remotely operated motor-driven

generator in engineering education
Nicolaas Luwes1, Leanri van Heerden2, Walter Commerell3
1
Centre for Sustainable SMART Cities, Central University of Technology, Bloemfontein,
South Africa, 2E-Learning and Educational Technology Central University of Technology,
Central University of Technology, Bloemfontein, South Africa, 3Institute for Energy
Technology an Energy Economics University of Applied Sciences Ulm – Technische
Hochschule Ulm, Ulm, Germany.
Abstract
Smart technologies in engineering education provide students with relevant
real-world experience and safeguard the quality of instruction during
emergency remote teaching. This paper aims to have students complete a
motor-driven generator practical via a RemoteLab and measure their
perceptions of the experience. A survey was conducted with five-point Likert
scale questions asking students to indicate their satisfaction with the useability
and effectiveness of the RemoteLab, then given a list of action verbs from
Gloria Willcox’s feeling wheel to select what best described their emotions
during the exercise, and finally open-ended questions to expand on their
experience. Results showed students found the system easy to use and effective
and most felt joyful and powerful emotions during the exercise. There was
some indication that they still preferred contact sessions on campus which
shows that a RemoteLab may be best used as an intervention when the contact
mode of delivery is suddenly disrupted.
Keywords: Engineering education; RemoteLab; online learning; Gloria
Willcox feeling wheel; (4IR) Fourth Industrial Revolution.

Student perceptions of a remotely operated motor-driven generator in engineering education
1. Introduction
Engaging students hungry for new experiences have become a challenge in an online teaching
and learning environment, especially in subject fields where practical experience is
dependent on working with expensive, or hard-to-come-by, tools and machinery. Practical
experience is crucial for deeper learning and building student confidence (Bradberry & De
Maio, 2019). Engineering students studying renewable energies need to experience the
relationship between their theoretical lectures and practical laboratory exercises to fully
assimilate the new skills. Renewable energies have large variants in test conditions, thus field
tests on systems are often analysed under real-world conditions in research and training.
Distance laboratories as teaching tools were becoming more widely used for imparting
practical skills to students in online teaching and learning delivery but saw an enormous boost
in the field of e-learning in education during the COVID-19 pandemic when universities and
other institutions of learning, were forced to shift their delivery to online environments
(Gamage, et al., 2020). Creating a laboratory that allows students to practice their newly
acquired theoretical knowledge in real-world conditions requires smart technology that
enables a user to remotely control the machinery in the laboratory from any location over a
network. The fourth industrial revolution uses modern smart technology in the automation of
traditional and modern manufacturing including the Internet of Things (IoT). IoT is a new
technology paradigm thought of as a global network of machines and devices that can interact
with each other, bridging the gap between the world of the Internet and our physical plane
(Kumar, Tiwari, & Zymbler, 2019). The development of IoT protocols can be used in the
construction and implementation of a RemoteLab.
For this paper, the two universities – the Technical University of Ulm (THU), Germany and
the Central University of Technology (CUT), South Africa – collaborated on developing a
didactic concept RemoteLab that allows students access to the prototype motor-driven
generator from any location via the Internet. Students from both universities used the
RemoteLab and were subsequently asked to complete a questionnaire gathering their
perceptions of the experience. The students’ perceptions of the useability of the system will
assist the lecturers in shaping and improving the RemoteLab. Gathering perceptions as
qualitative data will help the researchers answer their research question on how students feel
about using a motor-driven generator in the RemoteLab and more richly describe the human
experience (Paradis, O'Brien, Nimmon, Bandiera, & Martimianakis, 2016).
This paper will first review relevant literature on practical experience in an online
environment, the RemoteLab, and describe participant attitudes. Then the method
underpinning the research is discussed after which the paper is closed with the results of the
survey and the conclusions drawn from the data.
1392
Nicolaas Luwes, Leanri van Heerden, Walter Commerell
2. Literature Review
Online practicals can be performed either by virtual or remote laboratories. In the case of
virtual the exercise is simulated via a computer whereas, in a remote laboratory, the physical
operation runs in real-time and only the control thereof is facilitated by a computer which
can be in a different geographic location connected to a network accessed by the user
(Mohammed, El Zoghby, & Elmesalawy, 2020). For the engineering lecturer, a remote
laboratory is preferable as the student experiences the operation actively and is not only an
observer as the simulation runs through the operation. Practicing with the physical
machine/tool, albeit from different a location, allows the student to problem-solve and
troubleshoot on the spot which enriches the learning experience. It allows students the
opportunity to work with machines/tools that they may otherwise not have access to, like the
motor-driven generator used in this paper.
It can be difficult for students to describe how they are feeling about classwork as emotions
can be deemed as separate from a “work” state of mind. As the engineering field often attracts
people with a more analytical mindset this is especially true for engineering students. The
Gloria Willcox feeling wheel, was inspired by Joseph Zinker's and Robert Plutchik's
comparison of emotions to colors but adjusted to action verbs, can be used to frame questions
with the correct vocabulary so that participants can identify their feelings regarding the
specific experience (Willcox, 1982). Gloria Wilcox developed action words linked to feelings
in a wheel diagram using the four basic emotions: scared, sad, mad and glad but with glad
expanded into joyful, powerful, and peaceful (Zinker, 1978) (Plutchik, 1982). These action
words can be used to frame survey questions and so assist the participant with putting a name
to the emotion they experienced.
3. Methodology
The development and implementation of RemoteLab motor-driven generator using Google
cloud can be seen below:
1393
Figure 1. The final prototype RemoteLab
The prototype RemoteLab is built on a typical motor-driven generator lab experiment. The
user would adjust the variable power supply and variable load as indicated in the
experimentation documentation. In the prototype RemoteLab this was replaced by
controlling the inputs via a LabVIEW CRIO and the electronic control inputs of the supply
and load via new custom software. The new Human-Machine Interface (HMI) can be
controlled onsite and a second digital twin HMI with the same look and feel was programmed
that can be controlled online. The Google cloud space is the backbone or in-between to the
remote twins and the main controller. Of these servers, Google sheets and Google scripts
were used.
An exploratory study is employed along with descriptive statistics involving quantitative
analysis of the collected data. Data was collected with surveys using five-point Likert scale
questions measuring students’ perceptions of the RemoteLab experience, from being “Not
satisfied at all” to being “Extremely satisfied”. The Likert scale, developed in 1932, is still
widely used today to measure participant attitudes. From the five- or seven-point scale the
participant can select their strength of feeling, for example whether they agree or disagree
with the statement (Joshi, Kale, Chandel, & Pal, 2015).
Student perceptions of the learning environment can be predictive of learning and indicative
of motivation to learn (Wallace, Kelcey, & Ruzek, 2016). Participants were then given the
opportunity to select multiple feeling wheel action verbs in response to the question: “Which
of the following emotions would you say you felt while working on your generator
remotely?” to gauge their emotional mindset during the exercise.
1394
As stated, the RemoteLab prototype was built by converting a typical motor-driven, generator
lab experiment to an online RemoteLab prototype with local and online HMI’s. The prototype
was evaluated by a sample group of 10 students, selected by asking for volunteers from both
the German and South African cohorts. They were asked to do the typical experiment using
the online prototype RemoteLab. The hardware is stationed in the German ULM campus and
German and South African students were accessing it from their homes and/or a computer
facility on the respective campuses. After the experiment was completed, the sample was
asked to do a perception survey via Google Forms. Ethical clearance was obtained for the
survey.
4. Results
Out of the 10 participants, 8 were from the CUT, and 2 were from ULM. Of these, 40% were
between the ages of 26 and 30, and 60% were between 18 and 25. Being all under the age of
30 one would expect a certain amount of tech-savvy, but 80% indicated that they have never
worked in a RemoteLab before.
Results indicated that the RemoteLab useability (labeled as very easy, easy, normal, hard,
very hard) bell-curved with 10% saying it was very easy and 40% indicating it was easy to
use, with the other 50% stating normal, no hard or very hard.
Perceptions delivery mode 60% of students preferred to have hands-on classes with both the
lecturer and peers in the same location. So even though they found the RemoteLab easy to
use, they still preferred the traditional contact sessions.
The satisfaction pole on the ability to still test their theoretical knowledge in a practical
exercise even when remote, (labeled as extremely satisfied, satisfied, neutral, dissatisfied, not
satisfied at all) the majority responded to being satisfied with the bell curved from normal to
extremely satisfied.
Perceptions, from the sample, on working with the physical equipment remotely indicated a
bell curve toward more satisfied. Working with the actual hardware is a positive experience
as renewable energy experimentation, in its nature, has large variants in test conditions that
are not always possible with simulation or purely theoretical calculations. This tends to aid
in learning if students see the external and environmental factors producing a deviation in
theory calculations.
As stated in the normal operation of the experiment students would have had to manually
change the variable power supply and variable load. For it to be a RemoteLab, this system
needed to be automated first, with a new window HMI. Perceptions from the sample on the
new graphic interface saw the curve leaning to satisfied and extremely satisfied but with some
neutral and even one dissatisfied indicated, which needs to be investigated.
1395
The following questions and results were designed using the feeling wheel.
Figure 2. Recorded emotions from participants and interpret to the feelings wheel
Most of the area occupies the ‘joyful’ to ‘powerful’ section, with a smaller area covered in
the ‘scared’ and ‘mad’ sections. The negative emotions could be ascribed to this, being most
of the participants’ first attempt at a practical exercise in a RemoteLab, which may have
caused frustration while learning to navigate the new environment. Note that mostly positive
feelings were recorded, thus the minority indicated frustration should cause too much of a
concern, but a solution could be that more technical training on the use of the remote system
is perhaps needed before the exercise is undertaken in the future.
The survey was concluded with some open-ended questions to allow participants space to
expand on their experience. To see if learning took place the question: “Do you think it’s
important to test your theory on real world physical equipment? Motivate.” was asked. The
sample mostly replied yes with some indicating the following: “Yes, for us to have more
accuracy and have proof of what is calculated theoretically can also be achieved remotely”,
“Yes, it helps to put theory into practice while in turn gauge one's understanding of the
work.”, “Definitely, because everything you've learned you need to practice it in the real
world”
The question: “How do you see the difference between a software simulator and a remote
physical system for experimentation?” was asked to see if the sample could distinguish
between a simulator and a remote laboratory, and most were able to as seen in the following
excerpts: “Remote physical systems give you the real-life experience or expectation”, “With
the remote physical system we can see the actual machinery and hear a sound in the
background unlike on the simulator”, “Physical systems will be more accurate, since it
allows for the practical variable”. Some apprehension was also recorded with one participant
1396
stating that: “The only difference is that with a remote physical system one is more cautious
as one wrong move could blow the machine”. This answer shows a need for instructing the
students on possible issues and troubleshooting to instill confidence. One participant
preferred the simulation stating: “It is interesting that there is a real object running at the
other end but if the simulation would run with fewer lags, I would prefer the simulation”
which suggests further investigation into the use of simulators may yield valuable insight.
Participants also showed a sense of the exercise’s value, when asked for “Any comments?”,
one stated gratefully: “Thank you for organizing the remote Lab experience it was a great
exposure to see how remote-control machinery can be done as we are in the 4IR era.” And
another wrote: “This opens a door for a lot of possibilities, to a place that has limited
resources for students to learn and expand their understanding of theory content.”
5. Conclusion
The purpose of this paper was to gauge engineering students’ perceptions of using a
RemoteLab to operate a motor-driven generator. Perceptions were gathered with a survey
conducted after the experience. The survey used five-point Likert scale questions to
determine the students’ satisfaction with the useability and effectiveness of the RemoteLab.
Then the students’ emotional mindset during the exercise was determined by asking them to
select feeling wheel action verbs from a list that most described their emotions. Finally, they
were given the opportunity to expand on their experience with open-ended questions.
The Likert scale questions indicated a high level of satisfaction with the system and that
students found it easy to use and effective. However, when asked about their preference
between the RemoteLab and the traditional contact sessions, the majority were inclined
toward a physical class in the same location as the lecturer and their peers. Most students
selected joyful and powerful from the feelings list, indicating a positive mindset overall, with
some mentioning feeling scared and mad, most likely due to it being their first RemoteLab
experience. During the open-ended questions, most responses were positive and showed that
students were ready to use the RemoteLab when necessary. The following quote from one
participant represents the general feeling best: “The experience was amazing, and I hope to
do it again”.
This paper showed that RemoteLabs, like other 4IR technologies, can be useful in
engineering education when circumstances force remote learning, but that students would
still prefer the on-campus contact sessions with the lecturer and peers. The use of the feeling
wheel to gauge emotional state during the exercise gives valuable insight into the mindset of
the students during this new experience and along with the data from the Likert and open-
ended questions will help the lecturer improve their teaching and learning practice and better
prepare the curriculum for possible future disruptions, like the past COVID-19 pandemic.
1397
References
Bradberry, L. A., & De Maio, J. (2019). Learning by doing: The long-term impact of
experiential learning programs on student success. Journal of Political Science
Education, 15(1), 94-111. doi:https://doi.org/10.1080/15512169.2018.1485571
Gamage, K. A., Wijesuriya, D. I., Ekanayake, S. Y., Rennie, A. E., Lambert, C. G., &
Gunawardhana, N. (2020). Online delivery of teaching and laboratory practices:
Continuity of university programmes during COVID-19 pandemic. Education Sciences,
10(10), 291-299. doi:10.3390/educsci10100291
Joshi, A., Kale, S., Chandel, S., & Pal, D. K. (2015). Likert Scale: Explored and explained.
Current Journal of Applied Science and Technology, 7(4), 396-403.
doi:10.9734/BJAST/2015/14975
Kumar, S., Tiwari, P., & Zymbler, M. (2019). Internet of Things is a revolutionary approach
for future technology enhancement: A review. Journal of Big data, 6(1), 1-21.
doi:https://doi.org/10.1186/s40537-019-0268-2
Mohammed, A. K., El Zoghby, H. M., & Elmesalawy, M. M. (2020). Remote controlled
laboratory experiments for engineering education in the post-COVID-19 era: Concept
and example. 2nd Novel Intelligent and Leading Emerging Sciences Conference (pp. 629-
634). Giza: IEEE Xplore. doi:10.1109/NILES50944.2020
Paradis, E., O'Brien, B., Nimmon, L., Bandiera, G., & Martimianakis, M. A. (2016). Design:
Selection of data collection methods. Journal of Graduate Medical Education, 8(2), 263-
264. doi:10.4300/JGME-D-16-00098.1
Plutchik, R. (1982). A psychoevolutionary theory of emotions. Social Science Information,
21(4-5), 529-553. doi:https://doi.org/10.1177/053901882021004003
Wallace, T. L., Kelcey, B., & Ruzek, E. (2016). What can student perception surveys tell us
about teaching? Empirically testing the underlying structure of the tripod student
perception survey. American Educational Research Journal, 53(6), 1834-1868.
doi:https://doi.org/10.3102/0002831216671864
Willcox, G. (1982). The feeling wheel: A tool for expanding awareness of emotions and
increasing spontaneity and intimacy. Transactional Analysis Journal, 12(4), 274-276.
doi:https://doi.org/10.1177/036215378201200411
Zinker, J. (1978). Creative process in gestalt therapy. New York: Vintage Books.
1398
Ideas on digitally supported individualization of teaching and

learning for evolving competency requirements
Faculty of Electrical Engineering and Information Technology, OTH Regensburg,
Regensburg, Germany.
Abstract
The world is changing rapidly, mainly due to the digitalization of all areas of
living. A huge amount of information is accessible via the Internet, and since
it is no longer possible for individual humans to keep track of it, artificial
intelligence (AI) is analyzing this data. In this rapidly changing world, students
have to be educated for a successful career during their whole working life.
These boundary conditions lead to completely new challenges for the
education of students that are unprecedented in this form. Digitalization in
education can help to cope with these challenges but can only be a means, not
a goal. Personal interaction with students remains the most important task in
education to address individual weaknesses and further develop strengths and
talents. With the increasing amount of openly available information and the
consequently increasing diversity of experiences within the group of students,
differentiation is advancing to become the key to successful education.
Digitization can help with this challenging task and support communication
between students and their experienced instructors. But computers cannot
replace human interaction and attempts to improve teaching efficiency by
replacing this communication with electronic means endangers the learning
success for complex concepts. This article analyzes education demands and
possibilities for digitally supported teaching and learning.
Keywords: Digitalization; higher education; student competencies;
differentiation.

Digitally supported individualization of teaching and learning
1. Introduction and Motivation

More and more information is being generated and is accumulating as common accessible
and searchable knowledge. In all fields of professional working life, topics are becoming
more and more complex. Experts are concentrated on smaller and smaller fields of
knowledge. This exponential growth of information and the apparent availability of nearly
all information on the Internet leads to a completely new scope of teaching and learning.
The idea and motivation for this article arose during personal discussions about the demands
for teaching and learning in this fast-changing world with an exponentially increasing amount
of information. This article postulates the need for a fundamental change in higher education
as the consequence of the described change in information availability. The authors postulate:
Theorem 1 There is enough information on any subject available: A plethora of books,
internet pages, online videos, scientific papers, blogs and podcasts are out there for any
teaching topic – easily accessible and not too difficult to find.
The conclusion is that the traditional role of a lecturer has no benefit for future learning:
Corollary 1 The benefit of lecturers in the modern learning process is not to provide
additional material or repeat facts.
The goal of this article is to think about new teaching scenarios with benefits for learners and
new roles for lecturers. The next Section 2 explains how the modern information society
changed the diversity of students. Section 3 describes expected shifts in graduates’
competencies that are necessary for their later working life and Section 4 explains the
resulting changes in study goals for students. Section 5 describes the impacts of digitalization
on students’ education and Section 6 shows a way toward a digitally supported individual
teaching and learning. The last Section 7 gives a summary and a conclusion.
2. Change in Diversity of Students

The diversity discussion is focusing on differences in talent, social background, and personal
living situation. Due to the information society, the diversity discussion needs to enter a new
scale: In former times a few books and school education were the main sources of information
for students. There was a mainstream of competencies induced by the standardized teaching
at school. Pupils received the same information, had the same learning goals and passed the
same tests. Some, from today’s point of view, small differences between students originating
from diverse social backgrounds have been discussed (Ahmadaliev et al., 2022; Ugras, 2018).
However, in the modern information society, things are changing fundamentally. The school
has lost its effect of standardizing the competencies. The contribution of schools to the
information supply has become less and less important, as even very young children can learn
1400
from the internet. Every individual has consumed a different set of information, developed
different competencies and built a different world model. In consequence, students are not
only having social diversity, but they have developed an individually unique set of
knowledge, competencies and an individual way of thinking. This is the main source of
diversity and makes conventional teaching approaches less effective.
Theorem 2 The main source of future student diversity will result from information diversity.
3. Shift of Graduates’ Competencies inWorking Life

The working environment of professional life is changing rapidly. Berger and Frey (2015)
write that “Jobs that are not at risk of computerization have something in common: they
require an understanding of human heuristics or involve the creation of novel ideas - that is,
they require social or creative skills.” and “Over the next decade, the computerization of
middle–income jobs will shift to mainly substituting for low-income workers, while the
demand for problem–solving, creative and social skills is likely to increase further.”
Dondi et al. (2021) write that citizens benefit from a set of fundamental skills:
• add value beyond what can be done by automated systems and intelligent machines
• operate in a digital environment
• continually adapt to new ways of working and new occupations
They state that the two major factors of proficiency with a higher likelihood of employment
are “Adaptability” and “ Coping with uncertainty”.
ChatGPT was asked “Which competencies are needed for future work?” with the answer:
“There are several competencies that are likely to be important for future work, including:
1. Technical skills, such as programming and data analysis.
2. Interpersonal and communication skills, such as the ability to work effectively in
teams and to communicate complex information clearly.
3. Problem–solving and critical thinking skills.
4. Creativity and the ability to adapt to change.
5. Emotional intelligence, including the ability to manage stress and navigate
emotional dynamics in the workplace.
6. Digital literacy, including proficiency with technology and understanding of how it
is used in various industries.
7. Cultural competency, including the ability to work effectively with people from
diverse backgrounds and cultures.
8. Continuous learning, the ability to learn new skills and technologies quickly and
adapt to new situations.”
1401
This answer by ChatGPT is not generated by its own research and ChatGPT doesn’t
understand the real meaning of this question; this answer is produced by using a huge data
set implemented inside an artificial neural network. ChatGPT can’t generate new knowledge,
but it helps to extract and summarize existing knowledge.
Theorem 3 As AI programs gain more and more skills for finding, summarizing and
analyzing knowledge and competencies of graduates for a successful career will change
fundamentally.
Where AI can achieve acceptable and reliable results, it will do so faster and cheaper than
humans. Remaining tasks for humans will be those where AIs can’t deliver sufficient quality.
4. Resulting Changes in Study Goals
4.1. Information Selection and Evaluation, not Information Gathering

The amount of information is increasing exponentially and the Internet gives access to a large
part of mankind’s knowledge. Search machines show all available information on the internet
on request and ChatGPT from OpenAI (2023) is the next level. This reverses the teaching
paradigm of the past where a lecturer had to generate new content with information
specifically prepared and arranged for the students. The capacity that was used to achieve
this is now free for other activities in teaching that have gained importance: Focusing on
topics that are essential in a digital world with overwhelming information flows.
Theorem 4 The job of lecturers will be to teach students how to gather valuable information,
how to evaluate and connect these to find solutions. Lecturers need to assist in building up
mental working models by setting up connections between gathered pieces of information for
gaining competencies.
Guiding students will consist of enabling them to find the required facts for a given challenge,
evaluate the correctness of the information, and discard misinformation. From this
information, a creative process of problem-solving should be started.
4.2. Information Selection and Evaluation, not Information Gathering

It is expected that most routine work of academic professionals can be done by AI. Real
social interaction, self-reflected and critical thinking, and creativity can not be performed by
today’s AI. They are trained on a data-set and can manipulate the output according to trained
rules and user interaction. But they can not reflect on their work based on a world model and
so the systems are limited in their ability to create novel ideas (Boden, 1998). Mazzone and
Elgammal (2019) write that “clearly, machine learning and AI cannot replicate the lived
experience of a human being; therefore, AI is not able to create art in the same way that
human artists do.” This says that AI and humans act principially different.
1402
Shneiderman (2020) “endorses a Human-Centered AI approach for designing and developing

systems that support human self-efficacy, encourage creativity, clarify responsibility, and
facilitate social participation.” Real social interactions, critical thinking, and creativity as well
as responsibility are not possible with current AI approaches. But AI programs can help
humans to do a better job when human-AI cooperation is done in an appropriate way.
Theorem 5 Social interaction, critical thinking, and true creativity will be future key
competencies and gain in importance. These will be complemented by the competencies to
work in the digital domain and to interact with AI computer programs.
Humans will be needed in these important areas and consequently there will be an increased
need to develop the competencies required to perform such tasks during their education.
5. Impacts of Digitalization on Education
5.1. Increase of Efficiency

Routine jobs in teaching can be performed well by AI and thus the efficiency of knowledge
transfer can be increased. Barbosa et al. (2022) predict in the future working trends for 2050
that it will be possible to train a large number of students using MOOCs. These teaching
programs that mainly focus on knowledge can be automated. Examples are trainings for jobs
where special knowledge is important and learners have acquired the competencies for self–
organized learning as well as the domain–specific thinking models before. Asimov (1957)
described in his science fiction novel “Profession” that knowledge is easy to transfer to
humans but to learn how to gain competencies is a long and individual process.
Theorem 6 Many approaches introducing digital elements into teaching and learning focus
on increasing ‘efficiency’, i.e. educating more students with the same amount of resources
and in consequence reducing the direct contact between lecturers and students. This can be
only done when teaching knowledge, not competencies.
5.2. Shift of Focus in Education

As the transfer of knowledge will become an increasingly self-driven, and media and
tool-supported process, there will be a shift of focus for the human-driven teaching towards
skills and competencies like critical and creative thinking as well as social interaction. This
part of teaching can not be handed over to AI and builds up indispensable competencies to
further develop the state of the art of any discipline. One important factor driving human
activity is to ‘do things in the right way’. The definition of ‘right’ however, is based on a
complex rule set based on the observation of nature and social interaction.
1403
Theorem 7 The teaching of knowledge can probably be automated by AI, but the teaching of
skills and the development of personality i.e. individual thinking models require interaction
with the complex world models of experienced humans to be efficient.
Claxton (2023): “As for ChatGPT, we quickly learned that its greatest skill is lying.” Maybe
lying is not the correct wording, as there is no real understanding of the content by the AI
model but it selects and synthesizes information from the internet according to algorithms.
The AI has no idea if any information is true or not, not speaking of seriously complex
concepts. There is also no conscience of the AI; ethical limits are implemented by algorithms
and result in mere censorship but there is no understanding of ethics itself. AI systems are
largely unable to foresee the consequences of their actions, which leads to overoptimism and
as many reports show, blunt lying in case of a lack of information. With this, the AIs will
probably increase the amount of misleading information being available on the Internet and
thus increasing the risk of misleading other AIs.
Theorem 8 The teaching of the future will mainly promote social interaction, critical and
creative thinking, reflection about their own work and prediction of consequences.
5.3. Personalization of Learning

With AI software that helps lecturers with routine work for a high number of students, there
will be more time for the personalization of teaching and learning. Barbosa et al. (2022)
predict in the future working trends for 2050 that learning will be more personalized with
tailored learning plans. This new freedom should be used by lecturers to shape a new era of
learning. Lecturers and students can be assisted by AI software, other media, or can do
teaching and learning using pure human interaction. The prerequisite for this new era of
teaching is a new set of competencies and a new mindset for lecturers and students. The
future of competency-oriented learning will require redefining the role of lecturers:
Theorem 9 The work where lecturers can’t be replaced by AI is the ability to identify
individual ways of thinking and to provide specifically adapted guidance towards more
competency. This can be done with the assistance of AI software.
This implies that experienced lecturers must get into an intense academic discussion with the
learning individuals to be able to observe their specific thinking models and draw the right
conclusions, how to support them:
Corollary 2 The job of lecturers will be to guide students to compensate for existing deficits,
develop individual strengths, and their personality.
This makes it clear that while increasing the efficiency of teaching by possibilities of
digitalization is very useful, it is extremely important to take care not to reduce the contact
between students and lecturers. This would endanger the effectiveness of the unique
1404
competency of human instructors to individually guide students and thus reduce the effect of
the teaching work: The construction of a physical and ethical world model to validate the
own activities happens in observation of the physical reality and direct social interactions.
Also, the systems are largely unable to foresee the consequences of their actions, which leads
to over-optimism and as many reports show, blunt lying in case of a lack of information. This
will increase the need for humans to be able to verify the validity of the information.
Corollary 3 To be didactically useful, digital elements need to be integrated in such a way
as to increase and intensify the intellectual contact between lecturers and students.
Tools that lecturers can use to improve their impact on the learning process are very valuable.
Corollary 4 Digitalization in education is an important supporting tool for an effective
educational approach but it is not a goal of teaching to integrate as many digital elements
as possible.
The introduction of digital elements might lead to a distraction from the actual teaching goals,
or to an increased separation of lecturers and students playing with computers instead of
interacting in discussions. Careful didactic design of a learning environment fitting to the
intended learning outcomes will therefore become an even more challenging task.
Theorem 10 Digitalization should never be the main goal of teaching, but a supporting tool
to achieve maximal competencies of students.
6. Digitally Supported Individualization of Teaching and Learning

Digitalization will give a valuable contribution to education. It releases from routine work
and gives more time and freedom for individual coaching of students. Higher education has
already undergone an evolution during the last decades as many teaching techniques have
been developed and applied that allow for more individualized instruction like project-based
learning, or activating concepts as inverted classroom schemes. AI will lead to the next big
step in teaching and learning. With the integration of AI software into the learning process,
both the efficiency and effectiveness of teaching and learning can be drastically improved.
This evolution has to be developed further and their application has to be extended to achieve
the goal to intensify the contact of thinking models of lecturers and students to support young
people in developing their world models. The authors think that:
1. Gathering knowledge using digital tools with the aid of AI computer programs. This
can be highly automated without the individual support of an instructor.
2. Developing students’ competencies is an individual process under the guidance of
an instructor. This process can’t be automated by using digital means.
1405
7. Summary and Conclusion

The authors see fundamental changes in the way how students will learn in the future as
required learning outcomes and available methodologies are changing completely.
Digitalization will force a change in the mindset of lecturers and learners for education to
remain effective. Competencies in working life that have been essential in the past will be
obsolete. Many tasks will be performed much better and cheaper by AI in the future. Other
competencies that correspond to the nature of human beings will become increasingly
important. Consequently, successful education will need to undergo fundamental changes.
Education of human nature will be much more important than it was in the past. Integration
of human interaction and AI computer programs into courses will be a key development for
future learning programs to increase the efficiency and effectiveness of teaching and learning.
References
Ahmadaliev, D., Metwally, A., Yousef, A. M. F., & Shuxratov, D. (2022). The Effects of
Educational Robotics on STEM Students’ Engagement and Reflective Thinking, 1–7.
Asimov, I. (1957). Profession. Astounding Science Fiction, 59(3), 8–55.
Barbosa, C. E., de Lima, Y. O., Costa, L. F. C., dos Santos, H. S., Lyra, A., Argôlo, M., da
Silva, J. A., & de Souza, J. M. (2022). Future of work in 2050: Thinking beyond the
COVID-19 pandemic. European Journal of Futures Research, 10(1), 1–19.
Berger, T., & Frey, B. (2015). Bridging the skills gap. In T. Dolphin (Ed.), Technology,
globalisation and the future of work in Europe: Essays on employment in a digitised
economy (pp. 75–79). Institute for Public Policy Research.
Boden, M. A. (1998). Creativity and artificial intelligence [Artificial Intelligence 40 years
later]. Artificial Intelligence, 103(1), 347–356.
Claxton, M. (2023). PAINFUL TRUTH: AIs and the age of digital lying. Retrieved January
22, 2023, from https://www.langleyadvancetimes.com/columns/painful-truth-aisand-
the-age-of-digital-lying/
Dondi, M., Klier, J., Panier, F., & Schubert, J. (2021). Defining the skills citizens will need
in the future world of work. McKinsey. Available at: defining-theskills- citizens-will-
need-in-the-future-world-of-work.pdf (mckinsey.com)(Accessed: 30 June 2021).
Mazzone, M., & Elgammal, A. (2019). Art, creativity, and the potential of artificial
intelligence. Arts, 8(1), 26.
OpenAI. (2023). ChatGPT. Retrieved January 17, 2023, from https://chat.openai.com/chat.
Shneiderman, B. (2020). Human-centered artificial intelligence: Three fresh ideas. AIS
Transactions on Human-Computer Interaction, 12(3), 109–124.
Ugras, M. (2018). The Effects of STEM Activities on STEM Attitudes, Scientific Creativity
and Motivation Beliefs of the Students and Their Views on STEM Education.
International Online Journal of Educational Sciences, 10, 165–182.
1406
Using technology innovation and blended delivery for student-

centred learning in large undergraduate classes
Faculty of Architecture, Building and Planning, The University of Melbourne, Australia.
Abstract
Delivery methods of Higher Education classes have been scrutinised globally
during the COVID-19 pandemic as academics and students were forced to shift
rapidly to online delivery modes. In a post-COVID-19 scenario innovative
teaching and learning approaches are required to rethink how large student
cohorts can be educated online and on campus in a meaningful way, thereby
reenvisioning the traditional lecture and its wider class context. This paper
reports on the approach taken by the authors who restructured and redesigned
an existing second-year undergraduate subject in an Australian architecture
faculty. They collaborated with specialist learning designers to develop a
blended mode of large-class delivery that simultaneously addresses their
students’ desire to engage with subject content flexibly and asynchronously,
whilst benefitting from in-person interaction in the classroom. The new subject
was subsequently delivered for the first time at the point of writing this paper.
Keywords: Blended teaching; innovation; pedagogy; polling; student-centred;
post COVID-19.

Using technology innovation and blended delivery for student-centred learning in large classes
1. Introduction
The delivery of classes in tertiary education has long been ripe for a major overhaul. Since
the start of the millennium, education designers have been proposing new modes of teaching
that take into consideration the needs of current students while making better use of new
communication methods facilitated via high-end technology. Universities around the world
investigated novel subject-delivery methods and implemented online classes, yet most of
their efforts can best be described as half-hearted and cautious, given the existing
proliferation of face-to-face learning as the predominant model of student engagement.
The cautious attitude towards online teaching changed dramatically with the sudden arrival
of the COVID-19 pandemic in early 2020. The transformative character this presented to the
tertiary education sector globally is well documented elsewhere (Ewing, 2021; Neuwirth et
al., 2021; Singh et al., 2021 & 2022); it has had a major impact on the acceptance and quality
of online, hybrid, and blended teaching approaches. As universities around the world are
entering the post-pandemic era, many lessons learned emerge that point towards the need to
reflect on what worked, what didn’t, and what needs to be approached differently. Tertiary
Education institutions are having a watershed moment when weighting off between the
benefits of an on-campus experience and the flexibility inherent to flipping their classrooms.
As society changed irreversibly due to COVID-19, what is the best model for delivering
content in higher education? Have University campuses lost ground as places for encounter,
knowledge creation and dissemination? How does one counter isolation of students, instead
focusing on the campus experience? Are there still benefits for students to attend classes in
person? The research presented in this paper considers many of the above questions, as part
of the author’s redevelopment a large-cohort second-year undergraduate subject that
interlocks as design-enabling subject with design studios taught in an architecture faculty.
2. Methodology
In the absence of existing frameworks that guide large-class subject development in higher
education within the Post-COVID 19 context, the authors conducted an in-depth literature
review to assess innovative learning approaches pre, during, and post COVID 19. The
literature review focused on structural issues associated with designing a learner-centred
subject design, as well as analysing the role of technology in facilitating blended learning
modes and cognitive flexibility among students.
In order to complement the learnings from others with the lived experience and challenges
faced in their home institution, the authors carried out an in-house review of the subject they
took over, thereby consulting with former tutors and students on what worked well, and what
needed to change. This review focused on leaving the assessment aspect of the subject
1408
unchanged, yet focused on the opportunities of delivering the class as via a flipped-classroom
model The subject counts as a ‘design-enabling’ subject, meaning that its purpose is to teach
students skills that can be applied on other subjects, notably by intersecting with their design
studio activities. For that reason, the authors conducted a series of workshops with relevant
studio coordinators (among others) to fine-tune the curriculum towards their specific needs.
Due to the focused effort required to reinvent the content of the subject in a short matter of
time, the authors sought and gained research funding from their home institution. It allowed
them to run a series of workshops with learning-design experts, who advised on optimal
pathways to introduce blended and other learning techniques, and to employ research
assistants to work on the curriculum over a 3-month period. As proven elsewhere to support
architectural education (Brandao et. al, 2021), the authors used the tool ‘Miro’ for interactive
whiteboarding, brainstorming, and diagramming to advance the development of the subject.
3. Background
Researchers seem to agree that the disruptiveness of COVID-19 on the way universities
deliver their teaching content is simultaneously an opportunity for a major rethink of how
classes get taught in higher education. At the outset of the pandemic, Pellegrini, Uskov, and
Casalino (2020, p.222) called for a radical transformation of learning approaches to make
room for new skills, cultures and reference models. They predicted that young people would
be unwilling to go back to their usual way of working, once they experienced the advantages
of studying at home; they warned that invaluable insights offered by onsite lessons would get
lost without asynchronous and carefully prepared support (p.240). The validity of the
traditional lecture as the predominant instrument of knowledge transfer for large classes had
previously been challenged by Garrison and Kanuka (2004), who questioned the willingness
of students to commute to campus to receive a one-directional presentation that does not
require them to engage with the subject content in a stimulating and challenging way. In the
context of knowledge transfer, a distinction between learning from lecture material as a way
for gaining knowledge, and interaction with subject matter experts who actively challenge
students’ thinking needs to be made. Whereas the former assists learners in building up
explicit knowledge, the latter emphasizes on implicit understanding applied by learners in
changing contexts. Halpern and Hakel (2003) refer to the sharing of implicit knowledge as
an essential tasks of higher education to allow students to implement solutions independently.
The validity of retaining the in-person lecture format in post COVID-19 times remains in
question. Ewing (2021, p.42) discusses concerns voiced by Asia-Pacific education leaders
ho propose moving away from the lecture/tutorial format all together. Yet she sees
prerecording a lecture and putting the video online not seen as a solution either: We will have to
do much better than simply providing an online recording… academics will need to spend
1409
time developing more engaging online programs with the assistance of learning designers.
Singh et all. (2022) add that: The reason for being on-campus and face-to-face should
outweigh the perceived convenience of participating remotely (p.310). Questions
surrounding the future of the lecture and other on-campus learning hence point towards the
level of bi (or multi) directional interaction and stimulation students experience during the
lecture. Singh et al. (2021) add: Students must feel compelled to participate in-person
because the learning is incredibly dynamic, interactive, or uses equipment or immersive
experiences that cannot be accessed remotely (ibid).
Based on existing literature, learning designers will find it hard to identify a clear pathway
forward in their configuration of student-centred learning approaches that best fit their
subject. On one hand, research seems to suggest that knowledge-retention rates are higher
when individually placed learners draw on online resources (Li & Lalani, 2020). Yet other
research points towards the benefits of an on-campus experience that fosters students’ sense
of belonging (Neuwirth et. al, 2021, Singh et. al. 2021), that provides opportunities for peerto-
peer instruction (Carmichael & MacEachen, 2017), better interaction between faculty and
students (Alshahrani & Ally, 2016), and the establishment of ‘cognitive presence’ where
meaning is constructed through sustained discourse (Garrison et al., 1999).
4. Case Study: Large-cohort design-enabling subject (LCDES)

The impetus for the authors’ research emerged from recent staff changes within their home
institution that required a redesign and reconfiguration of a second-year large-cohort subject
that plays a strategic role within their faculty bachelor’s degree. The subject is embedded
within a suite of correlated subjects, and its function is to teach foundational skills that feed
into a range of other subjects and design studios within the degree. Enrolment numbers for
this subject range between 300 and 350 students per semester, and changes to the curriculum
required a complete redesign of the subject contents.
The redesign was required for two main reasons. Firstly, a perceived mismatch between the
learning outcomes of the subject, which focuses on training students to better perform in a
design studio setting, and the needs of correlated subjects. Secondly, based on the desire of
the authors to re-evaluate the lecture and tutorial format (and associated learning material)
that had been developed during the COVID-19 pandemic, where the subjects was delivered
online only. Coordinators of the correlated design studios highlighted the need for change,
as the LCDES did not fulfill its design-enabling function, instead running as a design studio
that competed with their class. This sentiment was shared by students who lamented the
extensive workload associated to overlapping submission deadlines between these subjects.
In recognition of lessons learned during COVID-19 (and beyond) the authors aimed to
develop a new delivery mode for the LSDES to address the above issues. They were awarded
1410
internal funding from within their home institution to allow for a consolidated 3-month period
for the subject redesign. Supported by a group of teaching-design specialists within their
home institution and drawing on available literature, the authors conceived a guidance
strategy to address several aspects of the subject redevelopment:
• A flexible, learner-centred teaching setup using a semi-flipped classroom.
• A combination of weekly asynchronous and synchronous student activities.
• A dynamic delivery mode that combines individual and asynchronous pre-class
learning material (covering knowledge accumulation: what is…?), large-class
interactive lectures (explaining the purpose of this learning activity: why are you
studying…?), and small-group tutorial classes (coaching and directing students at
applying knowledge for the development of assignments: how do I …?).
• A fundamental rethink of the traditional lecture format into a highly interactive and
engaging event that includes in-class polls, group discussions, and co-teaching.
• Targeted use of information and communication technology (Learning Management
System/videos/in-class polling/etc.) to support the above learning activities.
Figure 1: Diagram illustrating the interaction between lecture, tutorial and workshop activities.
1411
The starting point for advancing the strategy in greater detail was developing new learning
outcomes and associated, assessment tasks. Four learning outcomes were defined, with three
key tasks aligned to them (see Figure 1).
Figure 2: Miro board developed to structure lecture, tutorial and workshop activities, in association with
assignment guidelines and deliverables.
The authors subsequently mapped out all inherent learning requirements for the twelve-week
teaching period (see Figure 2). These were divided into what students need to learn, why it is
relevant to learn such material, and how they can achieved certain tasks via dedicated
processes and techniques. These three focui break down as follows: What is captured as a set
of four weekly ten-minute presentations that are pre-recorded. The associated videos are
placed on the LMS and students are asked to engage with the material before each class as
the semester progresses. Each video contains a question/provocation that serves as starting
point for further conversation during the weekly Monday morning lecture. There, the subject
matter experts curate a highly dynamic discussion to contextualise the why behind the
learning material. The lecturer facilitates discourse, invites opinions from students in the
room and responds to ad-hoc feedback from the large cohort. In order to make this work,
real-time polling (via Poll Everywhere) is used to gather input from the students and foster
deep understanding among learners (Stover et al., 2015 ). Benefits of on-campus interaction
are tapped into by grouping students in pairs and asking them to discuss their views before
responding to the real-time poll. The weekly Monday lecture is followed by a two-hour
tutorial class later that day and a second tutorial class on Thursday. Drawing on the pre-
lecture content and the in-lecture experience (and associated polling output), tutors work with
groups of up to sixteen students to explain in greater detail how they can engage with the
processes and techniques in preparation for their assignment. Groups of four students are
formed and the tutor will engage with one group at any point in time whilst other groups
discuss their progress with their peers.
1412
5. Discussion
Preliminary observations from the setup of the LCDES highlight that the bespoke nature of
the subject to be developed required a unique response to design a matching pedagogy that
includes different modes of content delivery. Neither the pre-2020 face-to-face delivery, nor
the online delivery approach taken during COVID-19 offered a feasible pathway in setting
up the learning environment for students. The decisive ‘back to campus’ move by the author’s
home institution required an immediate response to develop the differentiated and student-
centred learning blended teaching approach.
The authors were fortunate to receive support and in-house funding by the University of
Melbourne FlexAP Academic support team to redesign the subject from scratch between the
3-month semester break. Mapping out all individual learning outcomes and associated
subject content and subdividing it into designated sets of information communicated via
different media, represents a highly labour-intensive effort. The authors redeveloped all three
assignments from scratch, which translated in detailed mapping of before class and in-class
activities by educators and students. When configuring those, a previous student (who had
since become a tutor in the subject) proved to be pivotal as a sounding board to weave-in a
student’s perspective, thereby fostering the student-centric teaching approach desired by the
authors. The redesign for the LCDES had additional consequences outside the pedagogy and
modes of delivery: Class tutors needed to be given an opportunity to become familiar with
the (discussion and polling) outcomes emerging from the lecture to prepare for their tutorials
later that week. At the start of semester tutors needed to get introduced to the logic behind
the blended learning strategy and learn how to adjust their pedagogy accordingly.
Due to the complexity of class setup, a focus on timetabling tutorials and associated
classroom spaces proved to be essential. Learning designers needed to consider the often
predefined and restricted staff and room availability and work within those constraints.
6. Conclusions
This paper proposes a way to develop an interface and interlock (pre-)lecture, tutorial, and
workshop activities in a large-cohort undergraduate architecture subject by means of blended
delivery modes and technology tested and used before and during COVID-19. The subject
restructuring specifically addresses the risk of a lack of student engagement with traditional
lectures, in particular if there is a disconnect with assignment preparation and activities
during tutorials and workshops The availability of technology, including applications that
strengthen learning both online, as well as in-class plays an essential part in designing the
best possible environment in higher education classes.
1413
Moving forward, the authors aim to include future feedback from students to support the
developed interactive lecture format (with pre-recorded material and in-class polling) to
gauge the feasibility of transforming traditional lectures within undergraduate subjects.
Learner-centric blended delivery of classes and supporting material seems to be a useful
approach to develop a more dynamic and engaging academic environment for students.
References
Alshahrani, K., & Ally, M. (2016). Transforming education in the gulf region: Emerging
learning technologies and innovative pedagogy for the 21st century. Routledge.
Brandao, E., Adelfio, M., Hagy S. and Thuvander, L. (2021). Collaborative Pedagogy for
Co-creation and Community Outreach: An Experience from Architectural Education in
Social Inclusion Using the Miro Tool. Advances in Human Dynamics for the
Development of Contemporary Societies, AHFE 2021, 118–126. ISBN: 978-3-030-
80414-5.
Carmichael, D., & MacEachen, C. (2017). Heuristic evaluation of the use of Blackboard &
Facebook groups in computing higher education. International Journal of Modern
Education and Computer Science, 6, 1–8.
Ewing, L. (2021). Rethinking Higher Education Post COVID-19. The Future of Service Post-
COVID-19 Pandemic, Volume 1, 37-50. doi: 10.1007/978-981-33-4126-5_3.
Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based
environment: Computer conferencing in higher education. The Internet and Higher
Education, 2(2), 87–105. doi: 10.1016/S1096-7516(00)00016-6.
Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative
potential in higher education. The Internet and Higher Education, 7(2), 95–105. doi:
10.1016/j.iheduc.2004.02.001
Halpern, D. F., & Hakel, M. D. (2003). Applying the science of learning to the university and
beyond. Change, 35(4), 36-41. doi: 10.1080/00091380309604109.
Li, C., & Lalani, F. (2020). The COVID-19 pandemic has changed education forever: This
is how. World Economic Forum. https://www.weforum.org/agenda/2020/04
/coronaviruseducation-global-covid19-online-digital-learning/ last accessed 10.02.2023.
Neuwirth, L.S., Jovic, S. & Mukherji, B.R. (2021). Reimagining higher education during and
post-COVID-19: Challenges and opportunities. Journal of Adult and Continuing
Education, 27(2), 141-156. doi: 10.1177/1477971420947738.
Pellegrini, M., Uskov, V. & Casalino, N. (2020) Reimagining and re-designing the Post-
COVID-19 Higher Education organizations to address new challenges and responses for
safe and effective teaching activities. Law and Economics Yearly Review, Queen Mary
University of London, 9(1), 219-248. ISSN 2050-9014.
Singh, J., Evans, E., Reed, A., Karch, L., Qualey, K., Singh, L., & Wiersma, H. (2022)
Online, Hybrid, and Face to Face Learning Through the Eyes of Faculty, Students,
Administrators, and Instructional Designers: Lessons Learned and Directions for the
Post-Vaccine and Post-Pandemic/COVID-19 World, Journal of Educational Technology
Systems, 50(3), 301–326. doi: 10.1177/00472395211063754.
1414
Singh, J., Steele, K., & Singh, L. (2021). Combining the Best of Online and Face-to-Face
Learning: Hybrid and Blended Learning Approach for COVID-19, Post Vaccine, & Post-
Pandemic World. Journal of Educational Technology Systems, 50(2), 140-171.
doi:10.1177/00472395211047865.
Stover, S., Noel, D., McNutt, M. & Heilmann, S.G. (2015). Revisiting Use of Real-Time
Polling for Learning Transfer. Journal of Teaching and Learning with Technology, 4(1),
40-60. doi: 10.14434/jotlt.v4n1.13002.
1415
Postgraduate degree program in social business as a new

knowledge tool in entrepreneurship and social impact
Universidad Autónoma de Baja California, Mexico.
Abstract
Inside the new theories and learning models, we can identify social business
as a valuable tool to develop entrepreneurial skills focused on solving social
problems effectively and profitably, and also create innovative solutions that
generate a positive impact on community.
A social business arises with the purpose of solving a social problem using the
market system; It is created on a non-profit basis. Even so, the business model
is sustainable, guaranteeing its operability. Social businesses combine social
and economic objectives, seeking to generate a positive impact on society and
the environment. They emerge as an alternative to the traditional way of doing
business.
The following paper aims to explore the concept of social business, its
importance and how a postgraduate degree in social business can be useful
for those entrepreneurs looking to develop skills and knowledge within this
innovative area.
Keywords: Social business; entrepreneurship; teaching.

Postgraduate degree program in social business
1. Introduction
This new innovative approach has come to change the way in which businesses are conceived
and the role they can develop in society, since companies no longer focus only on maximizing
profits and recovering their investment, but from the beginning they are born with a different
conception: a positive social impact. Among the motivations for the creation of this type of
business is to contribute to solving challenges such as poverty, inequality, environmental,
among others.
At a time where traditional economic theories and structures have not been sufficient to
generate satisfactory answers to today's social problems, it is necessary to rethink from the
academic field, the assumptions, objectives, scope, and tools to address these challenges.
Therefore, it is pertinent that the Autonomous University of Baja California (UABC) offers
highly specialized programs that address these issues, such as a Master's and Doctorate
program in Social Business (MDNS). This project contributes to the efforts of the university
to offer programs immersed in the local, national, and international avant-garde.
The academic background of the MDNS program has its origin in the performance of social
linkage and innovation that the UABC has carried out through the Faculty of Economics and
International Relations (FEyRI) since 1999 with the emergence of the Program of Research,
Assistance and Teaching of Micro and Small Enterprises (PIADMyPE), followed by the
Center for Research, Assistance and Teaching of Micro and Small Enterprises (CIADMyPE)
and now with the UABC-Yunus Center for Social Business and Wellbeing project, an
organization dependent on the FEyRI.
These microenterprise training programs are developed as a collaborative public policy
between the local government and the university to promote formal entrepreneurship, help
microentrepreneurs in vulnerable contexts and thus improve the country's welfare prospects.
It is recognized as an innovative model of assistance to provide non-financial services to
microentrepreneurs where university professors and trainees collaborate (Mungaray Lagarda,
Osorio Novela, & Ramirez Angulo, 2021)
This institutional effort becomes increasingly relevant and necessary as we look at the
increase in the population living in poverty, either measured by income or by access to basic
goods such as food, education, health and infrastructure, by which is an issue of concern for
various actors in society, both globally and at national, regional or local level (Chavez
Nungaray & Hernandez Gomez, 2015).
2. Social business as a new model of entrepreneurship with social impact

Neoclassical firm theory focuses on the concept that firms have a primary goal, which is to
maximize their profits which leads to the long-term largest market share. There is an
1418
emphasized responsibility towards shareholders, considering that innovation is a key factor

within the process. And it is under these conditions, through which it is established that the
only rational objective pursued by the firm is to maximize its profits (Morales Sánchez,
2009). Unlike traditional firms, which focus on maximizing their profits and long-term
growth, social businesses consider social objective as a fundamental aspect of their business
model.
This innovative approach has led to a new way of understanding the role of firms in society
and has prompted the creation of a new theoretical model that focuses on sustainability,
innovation, and social responsibility. This theoretical model recognizes that firms can and
should be agents of positive change in society and that economic sustainability and positive
social impact are not mutually exclusive.
International evidence shows that the motivations of firms, individuals and entrepreneurship
are more complex, diverse, and transcendental than what dominant theories propose. firms
can allocate part of their resources for philanthropic actions, community improvements or
finance foundations that do not pursue any profit. On the other hand, there are social activists,
community leaders, politicians or academics who put the common interest before individual
interest (Yunus, 2013).
Muhammad Yunus, an economist and university professor, founded Grameen Bank in
Bangladesh. The Grameen Bank, also known as the "bank of the poor" is a pioneering
microfinance institution in the field of social entrepreneurship that aims to reduce poverty in
Bangladesh (De Diego Rábago, 2018). Since the structure of the financial system was
designed to extend credit only to people with purchasing power, this limited the options of
vulnerable people to start a business that would help their livelihood. In 2006, Muhammad
Yunus was awarded the Nobel Peace Prize for this project.
Within Yunus' philosophy a social business can be Type I or Type II. Type I refers to a
business where investors reinvest their profits to try to solve a social problem; while Type II
are businesses that arise from need, also known as subsistence enterprises (microenterprises),
businesses that have the purpose of providing income to people who are in a situation of
poverty. In this sense, in the first case, it is the nature of the products, services, or operating
systems of the business that creates social benefit. This kind of social business might provide
food, housing, health care, education, or other worthwhile goods to help the poor. With the
second type of social business, goods or service produced might or might not create a social
benefit. The social benefit created by this kind of company comes from its ownership.
Because the ownership of shares of the business belongs to the poor or disadvantaged, any
financial benefit generated by the company's operations will go to help those in need (Yunus,
2013).
1419
Social business are managed under seven principles: try to improve society as the main
objective and not to generate profits, be financially profitable, respect the environment,
investors recover their investment and the surplus is used to finance innovation and business
development, employees receive competitive salaries and share the spirit of goodwill (Yunus,
2010).
In Mexico, social businesses have gained ground in recent years to address social and
environmental challenges in the country; on issues such as access to education, rural
development, employment, and environmental sustainability (Rubalcava de León & Zerón
Félix, 2020).
Both social and traditional businesses have their strengths and weaknesses, however, social
businesses offer unique advantages compared to traditional businesses, such as its focus on
sustainability, social problem solving, its fusion with the community and social and
environmental responsibility. These models can be an effective way to achieve long-term
change.
Importantly, social businesses also face unique challenges, such as balancing their social and
economic goals and the need for financing. That is why it becomes a topic of interest that
social entrepreneurs receive training and formal instruction that allow them to further raise
their chances of success.
3. Master and PhD in Social Business

The Master's and Doctorate program in Social Business is proposed as a space to link the
concepts of financial and economic sustainability used by traditional businesses, but now
applied to self-sustainable businesses and projects, which do not pursue monetary benefits,
but to solve some social problem.
The professionalizing nature of the MDNS program allows to identify, disseminate, and
replicate the best practices, local, national, and international, of the actors that already
implement actions and have experience, from the private or public sector, such as activists,
community or social leaders, non-governmental organizations and public agencies of all
levels focused on social and economic development. At the same time, it provides its students
with the technical and theoretical tools for the identification, diagnosis, design,
implementation, and evaluation of a social business, while incorporating strategic concepts
into decision-making.
The objective of the programs is to train professionals of the highest level who are interested
in detecting, analyzing and developing projects associated with social businesses, which
contribute to mitigate problems of education, food, health and well-being of society, through
1420
entrepreneurial and solidarity activities, contributing to the improvement of the quality of

people and the community.
Therefore, the profile of the student who is sought to enter the program must have as main
characteristics a sensitivity to social problems and an interest in contributing to the formation
of knowledge, which allows him from this motivation to design, manage and develop
innovative strategies applied to the operation of a social business.
The Master's and Doctorate program is designed with a duration of two years and three years
respectively. The evaluation system is based on objectives and contribution to the graduation
profile; therefore, the evidence of a result is required to infer that the student has achieved
the objectives that have been established in the syllabus proposed in the program. Within this
syllabus are the compulsory and optional subjects, presentation of progress of the social
business project, stays and intervention, as well as any other academic activity endorsed by
the holder of the subject or the Social Business Project Committee.
It is expected that the stays and professional intervention will preferably be in local areas of
Baja California and contribute on the student's work in the area they define together with
their terminal project tutor. Those who receive them within the different sectors in these stays
and professional interventions are called operational collaborators, who work through
systemic frameworks the active participation of the learning topics seen in class.
MDNS program will be characterized by strengthening in the students the professionalizing
tasks applied in the framework of the two and four seminars that contemplate the curriculum
of the master's and doctorate, respectively, which give rise to the realization of a semester
colloquium of advances of terminal works, where the students are evaluated by committees
of experts.
The curriculum of the master's degree privileges the practical and operational approach, while
the doctorate emphasizes, in addition to the previous ones, the analytical-comparative sense;
both totally aimed at solving social problems. A substantive difference with respect to other
programs is that it offers a balance in topics and contents both in theory and in tools for
measuring and evaluating impact.
Studying a postgraduate degree in social business can be useful for those looking to develop
specific skills and knowledge in this field. Develop training in issues of social responsibility,
sustainability and social innovation that allow students to have a broader perspective on the
role of companies in society; without neglecting the business tools needed to create and
manage successful social businesses.
The curriculum of the master's degree includes one year of subjects and one year of stays and
professional intervention. The subjects that students take are development problems,
1421
quantitative methods, methods of social intervention, resource management, solidarity

economy, social cost-benefit analysis and leadership and management.
In the case of the doctorate, the first year is of subjects and the next two correspond to stays
and professional intervention. The subjects they take are studies of development problems,
quantitative and qualitative methods, social business project seminars.
4. Social projects as a result of postgraduate degree in Social Business

Within the MDNS program, there has been the opportunity to train professionals who can
create innovative business solutions that address social and environmental problems of
utmost importance, as well as a management and development.
• Hopetruck: cooking up new stories: This social business project aims to reintegrate
young people who have been in conflict with the law or who are at risk, through a
business model that consists of a training process in the gastronomic area, when they
conclude they can practice for a period of three months in a Food Truck to later be
placed in different allied companies after an evaluation, thus achieving their social and
labor reintegration.
• CUALLI: Solutions for the financial sustainability of Healthcare Centers in Baja
California: The aim of this social business is to provide biodegradable cleaning
products to social care centers in marginalized regions to facilitate their access to
essential hygiene products. At the same time, landfill centers are installed, which do not
generate waste where these centers will receive the necessary inputs and tools through
cross-subsidy consignment.
• Moon Jams: A Business Model for Sustainable Community Development: Business
model focused on the integral recovery of artisanal food processing to promote
sustainable community development in the High Mountains Region of Veracruz. This
project seeks to identify artisanal food processing owned by the community to which
value can be added and are potentially marketable. It also seeks to link with community
agents who have resources to develop processes efficiently.
• Route 2: Civil Association that supports, empowers, and encourages young women in
their personal development through workshops focused on mental health along with an
art therapy methodology, in turn, the importance of the subject is promoted in schools.
The objective is to promote in the community the importance of taking care of their
mental health, their personal and professional development in young people, creating a
link with specialized people.
1422
5. Conclusions
The opening of a professionalizing graduate program provides the opportunity to acquire
knowledge within a real context, to study the phenomena within their conjuncture. Likewise,
it is a reality that there are no postgraduate programs in social business in the region, so
presenting an offer in this regard would contribute to the development of local, regional,
national, and international research, which allow the creation of strategies and consequently
improve the quality of life of vulnerable sectors.
The creation of this program provides useful information and research and serve as a means
of linking projects for priority sectors to be supported by the university. In this way, research
would be provided in favor of the neediest social sectors, focused on increasing the quality
of life of the population in poverty.
In this way, the MDNS program represents for the state of Baja California and the region an
important project in terms of linkage, research, application, and professionalization, by
generating highly trained human resources that can respond to the most preponderant social
problems of the region. As an academic project, it has the task of continuing to bear fruit in
applied science of high impact in the field of economic and regional development, specialized
in social business, through the collection and analysis of data and information collected from
primary sources, to produce quality and innovative specialized academic materials.
By continuing to support and foster the growth of social businesses, we can contribute to a
fairer and more sustainable future for all. The opening of a postgraduate course with these
characteristics would bring several benefits to the university, including the competitive
advantage over other universities, since no other offers it, which would strengthen the
leadership of the UABC nationally and internationally, and would pay for the sense of
solidarity and social responsibility that has always characterized our excellent university.
References
Chavez Nungaray, E., & Hernandez Gomez, E. (2015). Indicios de la Situación
Socioeconómica y de Acceso a la Vivienda en la Frontera Norte de México. . En Bienestar
y Pobreza en América Latina: Una Visión desde la Frontera Norte de México. . UABC &
Once Rios .
De Diego Rábago, P. (2018). Grameen bank: El banco de los pobres. El impacto de una
herramienta para erradicar la pobreza y dar poder a la mujer. Revista de ciencias juridicas
y sociales, 15-54.
Morales Sánchez, M. (2009). Teoría económica evolutiva de la empresa: ¿una alternativa a
la teoría neoclásica? Problemas del Desarrollo. Revista Latinoamericana de Economía,
162-183.
Mungaray Lagarda, A., Osorio Novela, G., & Ramirez Angulo, N. (2021). Service-learning
to foster microenterprise development in Mexico. Emerald Insigh, 14.
1423
Rubalcava de León, C., & Zerón Félix, M. (2020). The Social Enterprise: Mexican Context.
Revista Científica Multidisciplinaria de Prospectiva.
Yunus, M. (2010). Building social business. The new kind of capitalism that serves
humanity´s most pressing needs. New York: Public Affairs.
Yunus, M. (2013). Creating a world without poverty. Social business and the future of
capitalism. Dhaka: Subarna.
1424
Embeddedness of students with special educational needs in

higher education
Department of Educational Studies, MTA-DE-PARTNES Research Group, University of
Debrecen, Hungary.
Abstract
Previous research has examined the institutional and extra-institutional
embedding and network of students, however, we still know little about the
network of students with special educational needs. What makes this research
unique is that it compares the characteristics of SEN-A and SEN-B students.
Therefore, the embeddedness of students with special educational needs within
the institution is investigated using quantitative research method. The research
sought an answer to the question of what characteristics can be used to
describe the network of students with special educational needs, and whether
there is a difference between SEN-A and SEN-B students. Based on the results,
there are significant differences in the case of students with special educational
needs, whether they are embedded more strongly in the environment inside or
outside the institution.
Keywords: Special educational needs; higher education; social capital.

Embeddedness of students with special educational needs in higher education
1. Introduction
The correlation of the student networks with academic performance indicators has already
been investigated by many researchers (Bourdieu 1988, Coleman 1988, Pusztai 2011). They
show that students perform significantly better when they are in harmony with their
environment (Astin 1993). Several studies have proven that the social capital created by
student networks promotes a successful school career and admission to higher education. The
academic embeddedness also promotes this during studies, especially the relationship with
the other students (Pusztai 2011). In the case of students with special educational needs, it
has also been proven that the positive attitude of the students - in addition to the attitude of
the teachers - is important and is also a condition for successful integration (Pető and Ceglédi
2012).In terms of relationships outside institutions, the most important are the contemporary
circle of friends, the family, and voluntary and religious communities. (Utasi 2002, Albert
and Dávid 2007). In the case of pupils and students with disabilities, we do not have
information on friendships outside the institution, but there are already researches on
contemporary friendships that also include the institutional environment. Hrabéczy (2020) in
his research examining families with special educational needs in the fourth grade found,
among other things, that children with special educational needs spend less time with their
peers outside of school compared to the average. In our research, in connection with
Coleman's social capital theory, we examine the inter- and intragenerational network of
relationships of students with special educational needs inside and outside the institution, so
in the following, we will review the relationships within the institution between peers,
between students and teachers and with other institutional administrative staff we present the
results concerning relationships outside the institution, and the results concerning peers and
the family.
1.1. Definition of the target group

According to the OECD, there are three main groups of special educational needs, SEN-A
(disabilities), SEN-B (difficulties), and SEN-C (disadvantages). The types of disabilities
classified as students with disabilities in the Hungarian Higher Education Act can be placed
in two groups in the OECD category system, the SEN-A and SEN-B categories. The division
based on this is illustrated in Table 1. In the course of the research, we will later compare
students with special educational needs belonging to the SEN-A and SEN-B categories.
Table 1: Hungarian students with special educational needs in the OECD SEN category system.
SEN-A SEN-B
Motor, sensory (vision, Other psychological developmental disorders ( learning,

hearing), speech disability attention or behavior control disorder), autism spectrum disorder
Source: own editing based on OECD (2004) and Nftv 2011.
1426
2. Methodology
2.1. Hipotheses
In addition to the family, other intergenerational and intragenerational relationships of
individuals also have a major impact on academic progress and higher education integration,
which has been supported by numerous researches. These factors also prove to be decisive
for students with special educational needs (Pusztai 2011, Pusztai and Szabó 2014, Hrabéczy
2019). However, integration into different communities can be of varying degrees and
difficulties due to stereotypes and prejudices, depending on the type of specific educational
need (Séllei 2015, 2018). Knowing these, we assume the following:
a) SEN-A students have a stronger intergenerational relationship system than SEN-B
students, but SEN-B students have a stronger intragenerational relationship system.
b) In the case of students with special educational needs, the relational capital within the
family will be decisive from the point of view of higher education embeddedness.
2.2. Methods
To investigate this, we implemented a large sample questionnaire data collection using the
snowball method (N=331). The database from this data collection is referred to as the
IncludED2020 database, referring to the possibility of participation in inclusive education
and the year of data collection.
There is no database available to researchers that can be used as a sampling frame for such
research and would provide access to students due to the GDPR. Based on the anonymized
statistical data, aaproximately 2,000 people in Hungary belong to this category, of which we
tried to reach 10%. Since probability sampling was not possible in the absence of a sampling
frame, we resorted to snowball sampling. Keeping in mind, first of all, that the different types
of special needs appear among the respondents in accordance with the real proportions,
therefore our sample represents the reality in terms of SEN types. However, we were not able
to achieve representativeness in terms of training areas. In spite of these limitations, the data
collection proves to be an investigation with an exceptionally large number of elements both
at the Hungarian and international level. The indicator of this is that we cannot find any
research in the literature that would have carried out a large number of questionnaire data
collection by interviewing students with disabilities, we can only find analysis of national
statistics and interview research with a smaller sample. The choice of method is also justified
by the fact that there are many students with disabilities in higher education who do not
register their disability either at the time of admission to higher education or during their
studies in the administrative systems of higher education, thus these students would remain
hidden from the researcher's eyes by the approach through higher education institutions.
1427
However, we also consider it important to include these people in the research, since we have
very little information about their presence and the reasons for not registering their disability,
but by asking them, another aspect of the studies of these students can be investigated.
3. Results
To test our hypotheses, we performed a cluster analysis. The cluster centers are illustrated in
Table 2. The analysis was carried out along two dimensions, with the previously discussed
inter- and intragenerational, as well as intra- and extra-institutional relations items. After that,
the variables were standardized and then included in the analysis.
Based on Table 2, it can be seen that, with the help of the included variables, four embedding
types emerged in the case of students with special educational needs. The following
distribution can be seen in relation to embeddedness along the clusters (Table 3).
Based on Table 3, it can be seen that there is a significant difference in embeddedness among
students with special educational needs. SEN-A students are overrepresented among those
with institutional intergenerational embeddedness, and among them the lowest proportion are
those characterized by a lack of social capital. On the other hand, the opposite can be
observed in the case of SEN-B students. They are overrepresented among those with a lack
of social capital, while among them the proportion of students with institutional
intergenerational embeddedness is the lowest. It coincides with our assumption that the
intergenerational embeddedness of SEN-A students and the intragenerational embeddedness
of SEN-B students will be stronger, however, the high proportion of SEN-B students with a
lack of capital is an unexpected result. According to Coleman (1988), the lack of social capital
is an important risk factor of dropping out. Our result in the light of Coleman’s results shows
us, that SEN-B students can be a more disadvantaged sutiation during their studies, caused
by the fewer supporting opportunities and a weaker social network.
A comparison was made along these clusters along the lines of socio-economic background,
achievement before higher education and the characteristics of the course and institution
visited by the student. Due to space limitations, the tables representing these results are not
published in the study, but our results are discussed in the conclusions.
1428
Table 2: Cluster centers (N=343).
Institutional Family Lack of

Intra-
inter- inter- social
generationals
generational generational capital
parent: Financial support 0,011 -0,696 0,082 -1,054
parent: Meet with friends 0,209 -0,113 0,395 -0,816
parent: Knowing how the children
-0,016 -0,455 0,124 -1,061
spending their free time
parent: Encouraging to study 0,082 -0,654 0,357 -1,071
parent: Conversation about culture 0,591 -0,172 0,150 -0,752
parent: Cultural programs together 0,425 -0,293 0,353 -0,685
parent: Conversation about books 0,483 -0,176 0,316 -0,917
parent: Relationship with instructors 0,509 -0,050 -0,149 -0,302
parent: Conversation about future
0,339 -0,311 0,491 -0,834
career
parent: Cooking for the student 0,259 -0,394 0,418 -0,414
parent: Interested in their child’s
0,021 -0,776 0,327 -1,361
studies
parent: Drives their children to classes 0,598 -0,219 -0,037 -0,309
parent: Involve student in
-0,077 0,056 0,387 -0,741
housekeeping
instructor: Helps to ooperate with other
0,904 -0,129 -0,464 -0,078
students
instructor: Helps with studies 0,593 0,087 -0,362 -0,274
instructor: Helps with career choice 1,099 -0,339 -0,417 -0,070
instructor: Conversation about
0,624 -0,082 -0,280 -0,156
student’s personal problems
instructor: Encouraging to study 1,094 -0,204 -0,483 -0,176
instructor: Takes abilities into account 0,720 0,095 -0,392 -0,403
students: Talking about problems
-0,074 0,134 0,007 -1,811
related to studies
students: Spending free time regularly -0,004 0,294 -0,321 -1,017
students: Conversation about student’s
0,093 0,238 -0,193 -0,911
personal problems
students: Lend books and notes -0,175 0,234 -0,312 -0,584
students: Talking about future career 0,329 0,309 0,107 -1,087
students: Inquires in case of absence 0,388 0,444 0,030 -0,689
students: Studying together 0,124 0,323 -0,160 -0,763
1429
friends: Talking about problems

-0,549 0,369 0,179 -0,682
related to studies
friends: Spending free time regularly -0,375 0,253 0,107 -0,167
friends: Conversation about student’s
-0,448 0,284 0,285 -0,389
personal problems
friends: Lend books and notes -0,236 0,287 -0,096 -0,345
friends: Talking about future career -0,297 0,224 0,226 -0,514
friends: Studying together -0,232 0,317 -0,011 -0,264
Forrás: IncludED2020
Table 3: Distribution of embeddidness clusters in the comparison of students with special

educational needs (Chi-square test, p<0.05), (%, N=331).
SEN-A SEN-B
Institutional intergenerational 30,5% 15,8%
Intragenerational 28,4% 32,1%
Family intergenerational 29,8% 32,1%
Lack of social capital 11,3% 20,0%
N 141 190
Source: IncludeED2020. Note: For underlined values, the value of adjusted residuals is greater than 2.
4. Conclusions
In this research, we explored the components and characteristics of the higher education
embeddedness of students with special educational needs, as well as the relationship between
higher education and pre-higher education networks with certain factors of admission to
higher education. In the course of the analysis, we were able to separate four types of
embeddedness based on the strength of relationships within and outside the institution, as
well as inter- and intra-generational characteristics.
The largest proportion of SEN students with institutional intergenerational embeddedness are
men, but among the SEN-B students belonging to this cluster there are still more women. In
terms of socioeconomic background, the members of this cluster typically came from big
cities and were mostly the children of parents with higher education, who have a better than
average subjective financial situation. This cluster is characterized by the fact that among
them we most likely come from single-parent families. They demand extra points mainly in
the form of preferential treatment and additional performance.
Embedded in the environment inside and outside the institution is typical for students with
special educational needs embedded in the intragenerational environment. There is a higher
proportion of women within this cluster, and the proportion of SEN-B students from mosaic
families is also significant. They are overrepresented from villages and towns, and although
1430
they are not in this cluster in the highest proportion, they are mainly children of parents with
secondary education. They are mostly characterized by a good financial situation, however,
students with a poor financial situation are overrepresented among SEN-B students within
this cluster.
Intergenerational embeddedness within the family is more typical of female students, and
students with special educational needs coming from whole families are included in this
cluster in the largest proportion. Among them, we can most likely find students coming from
smaller towns, who are the children of parents with secondary education, and who are
characterized by a good subjective financial situation.
The fourth cluster includes those students with special educational needs who lack social
capital and are characterized by a lack of interactions with their environment in all examined
respects. It is more common for male students with special educational needs, including
mainly SEN-B students. Among them, we find the largest proportion of students with special
educational needs who were not raised by their parents, and mainly the children of parents
living in smaller or larger cities, but with primary education, who are characterized by a poor
subjective financial situation.
By capturing the characteristics of these clusters and mapping the dimensions of their entry
into higher education, we obtained a more accurate picture of the relationship between the
networks of students with special educational needs and their entry into higher education.
In this connection, we assumed that SEN-A students have a more extensive intergenerational
relationship system, and SEN-B students are more characterized by intragenerational
embeddedness. This assumption was partially confirmed, as our results confirm that SEN-A
students have a stable institutional intergenerational relationship system, who, recalling their
secondary school studies, report less negative teacher behavior towards them, and it is more
typical for them that their interactions with instructors are also positive and more frequent.
Also, SEN-A students are the least characterized by the lack of capital. On the other hand, in
the case of SEN-B students, we did not clearly demonstrate a greater degree of
intragenerational embeddedness. Although the level of intergenerational embeddedness will
be higher in their circles, those struggling with a lack of capital are still overrepresented
among them.
We also assumed that the social capital within the family plays a decisive role in the case of
embeddedness in higher education. However, we assumed that the existence of relationships
within the family will be a more significant resource for SEN-A students than for SEN-B
students. However, in the case of the effect on embeddedness in higher education, it became
visible that family relationships in themselves result in few differences between the two
investigated groups, and we did not experience the supporter's effect on embeddedness in the
entire sample either. In the case of socio-economic background factors, however, the impact
1431
on higher education embeddedness emerged. However, among the examined students, the
quality and quantity of the institutional relationships established before higher education, i.e.
the connection with high school teachers and high school classmates, proved to be more
important. However, it has been proven that in the case of SEN-A students, parental
involvement has a stronger effect on admission to higher education than in the case of SEN-
B students. The social capital between SEN-A and SEN-B students within the family may be
the one that can compensate for the disadvantage arising from the disability during the
preliminary selection process.
Acknowledgments
"The research on which this paper is based has been implemented by the MTA-DE-Parent-
Teacher Cooperation Research Group and with the support provided by the Scientific
Foundations of Education Research Progam of the Hungarian Academy of Sciences."
“Supported by the ÚNKP-22-3 New National Excellence Program of the Ministry for Culture
and Innovation from the source of the National Research, Development and Innovation
Fund.”
References
Albert Fruzsina & Dávid Beáta (2007). Embert barátjáról. A barátság szociológiája.
Századvég, Budapest.
Astin, A. W. (1993): What Matters in College: Four Critical Years Revisited. CA: Jossey-
Bass, San Francisco.
Bourdieu, P. (1988): Homo Academicus. Stanford University Press, Stanford.
Coleman, J. S. (1988). Social Capital in the Creation of the Human Capital. American Journal
of Sociology, 94(1), 95–120.
Hrabéczy Anett (2020). Sajátos nevelési igényű hallgatók bekerülési és bennmaradási esélyei
a felsőoktatásban In: Dobi, Edit (szerk.) Juvenilia VIII. Debreceni bölcsész diákkörösök
antológiája Debrecen, Magyarország: Printart-Press
Pusztai Gabriella & Szabó Diána (2014): Felsőoktatási hallgatók és fogyatékossággal élő
társaik. Kapocs, 13. 4. sz. 23-36.
Pusztai Gabriella (2011): A láthatatlan kéztől a baráti kezekig. Tanulói értelmező közösségek
a felsőoktatásban. Új Mandátum, Budapest.
Pető Ildikó & Ceglédi Tímea (2012): A pedagógusok SNI-vel szembeni attitűdje SACIE-vel
mérve. A Senti- ments, Attitudes and Concerns about Inclusive Education Scale (SACIE).
Iskolakultúra, 22(11), 66–82.
Séllei Beatrix (2018). Szemléleti akadálymentesség a felsőoktatásban. Opus et
Educatio, 5(1).
Utasi Ágnes (2002): A társadalmi integráció és szolidaritás alapjai: a bizalmas kapcsolatok.
Századvég, 24. 2., 3–26.
1432
Black digital humanities in interdisciplinary undergraduate

teaching on diversity, gender, and sexuality
Magdalena J. Zaborowska1, Juan J. Rodríguez Barrera2
1
Departments of American Culture and Afroamerican and African Studies, University of
Michigan-Ann Arbor, United States, 2Department of American Culture, University of
Michigan-Ann Arbor, United States.
Abstract
Two undergraduate courses (2020-23) introduce students interested in the
humanities and computing to the life, works, and intellectual and material
legacy of the world-famous African American writer and activist James
Baldwin (1924-1987). Cross-listed with the Afroamerican and African Studies,
American Culture, Digital Studies, and English Departments, these courses
utilize an open-access digital collection documenting Baldwin’s life and his
selected works. Through innovative and experiential application of literary
history in conversation with the emerging fields of Black Digital Studies and
Black Digital Humanities, students develop projects that build (and build on)
a growing, open-access archive. Published on the ArcGIS StoryMaps platform,
these projects achieve two important higher-education goals: (1) They produce
student-driven knowledge on an internationally renowned Black figure
accessible to non-academic users; and (2) they confirm the importance of
humanities and diversity literacy as invaluable skillsets in the modern
workplace.
Keywords: James Baldwin; literature; black digital humanities;
interdisciplinary; undergraduate.

Black digital humanities in interdisciplinary undergraduate teaching
1. Introduction
This paper explores innovative teaching and learning in two undergraduate courses cross-
listed between the Departments of Afroamerican and African Studies, American Culture,
Digital Studies, and English at the University of Michigan-Ann Arbor in the United States.
The courses introduce undergraduate students interested in the humanities and computing to
the life, works, and rich intellectual and material legacy of the African American writer and
activist James Baldwin (1924-1987). Limited access to Baldwin’s papers and no writer’s
museum in the United States make this key twentieth-century American author hard to teach
outside of his published works. The courses intervene in this situation by combining
innovative interdisciplinary learning and production of new knowledge on this writer in the
form of an open-access digital archive.
Teaching methods harness students’ considerable digital skills to make engagement with
themes in the humanities, especially from antiracist and antiheteronormative perspectives, a
creative and collective endeavor. The authors believe that college instructors can significantly
advance inclusivity in the classroom, as well as foster humanities literacy and civic
engagement, by taking full advantage of students’ familiarity with a range of digital media
and tools. In that sense, learning and teaching also become a cross-generational dialogue.
The courses are: “Preserving James Baldwin’s Legacy through Black Digital Studies” (AAS
498-in action seminar) and “What’s Love (Sex+Race) Got to Do with It: Reading James
Baldwin’s Fiction in the Twenty-First Century” (AAS 498-in action seminar). Designed by
Prof. Zaborowska, a literary and cultural studies scholar with over three decades of scholarly
and teaching experience in interdisciplinary humanities and cultural studies, these courses
reflect an innovative and experiential focus on the ways in which African American literary
history can and should be taught, i.e., in a rich conversation with the emerging fields of Black
Digital Studies and Black Digital Humanities.
The courses utilize and expand through students’ projects the rich archive amassed for over
two decades by Prof. Zaborowska. Alongside more traditional materials like published texts,
the courses draw on an open-access digital collection mounted on the website of U-M
Library, as well as the resources of the National Museum of African American History and
Culture/Smithsonian/NMAAHC in Washington D.C. They are supported by IT and
instructional staff, thus making sure that students also acquire new computing skills as they
develop their projects.
1434
Magdalena J. Zaborowska, Juan J. Rodríguez Barrera
Pictured: Homepage of Chez Baldwin Writer’s House Digital Collection on U-M Library’s website.
The authors have collaborated on teaching techniques that combine interactive lectures,
discussion, multimedia material and writing, and public-speaking assignments that will be
described in this presentation. The courses also include guest speakers, and culiminate in
university-wide symposia open to the public in which students present their work. The long-
term goal for the various iterations of these two courses is to create a virtual writer’s house-
museum for Baldwin that, instead of bricks and mortar, will be constructed from student
projects showcased on an open-access website. The work of the course will be available to
all, and especially to local K-12 communities in Ann Arbor, Ypsilanti, and Detroit. Our James
Baldwin’s Virtual Writer’s House-Museum will in time become an anti-racist teaching tool
and legacy all its own.
2. Teaching Archive
Inspired by Prof. Zaborowska’s recent monograph, Me and My House: James Baldwin’s Last
Decade in France (Duke UP, 2018), the archive for both courses combines several types of
resources for students’ projects, with the goal of combining instructional goals with public
humanities ones. The materials included in the digital collection, for example, consist of
thousands of high-definition digital photographs taken between 2000 and 2014-18 on
research trips to St. Paul-de-Vence, France, where Baldwin’s house was located, and where
he spent his last prolific sixteen years. As Prof. Zaborowska argues in the aforementioned
book and in an essay in the American Quarterly, these images of Baldwin’s house, known
locally as “Chez Baldwin,” and of its contents are key to the preservation efforts of this
writer’s material legacy in the United States and beyond. Taken weeks before that building
was lost to developers and demolished in 2014, the images of the structure preserve its quality
1435
and processes of deterioration over time. The objects from the house that were meant to be
discarded, were instead salvaged by a friend of Baldwin’s who allowed them to be
documented via digital photography. PhD candidate Rodríguez Barrera was team leader in
the processing of metadata for the collection and oversaw undergraduate research assistants
as they went about caterorizing and ensuring the high quality of the various digital materials.
The resulting archive includes documentation of the building, an extensive collection of vinyl
records, jewelry, reference books, artwork, even furniture, as well as the contents of a private
library, phone logs, bills, receipts, and other ephemera.
Documentation of research trips to the site of the house, interviews with subjects who knew
the writer, and documents gathered at research trips to the Schomburg Center for Research
in Black Culture in NYC, where the James Baldwin Papers are located, complement the
resources for both courses, alongside the writer’s published works and diverse artwork
created to commemorate Baldwin by male and female artists.
The ever-expanding virtual Baldwin’s “writer’s house-museum” resulting from the student
course work published via StoryMaps will become a hub for teaching this writer, offering a
student-centered alternative to a brick-and-mortar structure. The students’ creative
productions—e.g., collaboratively produced podcasts and social media posts, written
explorations, recorded interviews, video, or virtual-reality implementations of Baldwin’s
household archive, etc.—will make the surviving Baldwin’s life matter a foundation for new
knowledges and interpretations of his ideas on equality, inclusion, tolerance, and
humanitarianism.
3. Course Descriptions
In addition to offering the public a glimpse into Baldwin’s everyday writing life, student
projects from both courses encode and interpret the objects with which the author surrounded
himself, as well as the books that he read and collected in his library. Serving as one of the
founding blocks in the learning process, the collection organizes the students’ final projects
into a website of resources that will grow with each iteration of the courses. This lasting
archive of knowledge on Baldwin is rooted in the present historical, cultural, and political
moment, reflecting both courses’ different approaches to Baldwin.
In both courses, students work with the ArcGIS StoryMaps platform to create original,
individually desgined final research projects. All students, all levels of experience, and all
fields of study are included in the course, including architecture, literature, general
humanities, social sciences, and STEM fields. Student final projects showcased at university-
wide symposia (held on Zoom as of this writing) include guest speakers who comment on
students’ work.
1436
3.1. AAS 498: “Reconstructing James Baldwin’s Legacy in the Digital Now”
“Reconstructing” taps into the recent resurgence of this important national and international
figure as the subject of new scholarship and as an African American popular culture icon
(Black Twitter, Black Lives Matter). Students work with a Canvas course site that includes
media and other learning tools and resources (e.g., the films I Am Not Your Negro, If Beale
Street Could Talk, James Baldwin: The Price of the Ticket, I Heard It through the Grapevine,
James Baldwin: From Another Place, etc.), including online collections and other African
American Studies-related digital projects. We also host 3-4 guest speakers via Zoom, and
students have a chance to introduce the speakers.
The students work individually and in groups to explore some of the following larger themes
in literary and cultural studies: (1) recasting national identity through intersectional
approaches; (2) the centrality of domestic settings to African American authorial creativity;
(3) the importance of Black women writers to Baldwin’s later works; (4) Baldwin’s relevance
to the articulation of androgynous and trans identities today; (5) Baldwin’s theorizations of
popular media such as cinema, photography, and TV; and (6) Baldwin in fashion and style
and as a creative inspiration.
3.2. AAS 498: “What's Love (Sex+Race) Got To Do with It: Reading James Baldwin’s
Fiction in the 21st Century”
“What’s Love” explores Baldwin’s philosophy of identity, love, and creativity that has
inspired trans people and academics, musicians and painters, dancers and theater actors, and
film directors and documentarians. This course approaches Baldwin’s ideas through a Black
queer lens—Black Queer Humanism (BQH)—while beholding the many musical, literary,
and visual inspirations that came to Baldwin from such eminent artists as Bessie Smith,
Aretha Franklin, Nina Simone, Alice Walker, Maya Angelou, Toni Morrison, Audre Lorde,
Josephine Baker, Beauford Delaney, Vincent Van Gogh, Vertamae Smart-Grosvenor, Glenn
Ligon, Janet Mock, Billy Porter, BLM, among others.
Students read Baldwin’s novels written between 1953 and 1979, his selected short stories
(“Sonny’s Blues,” “Going to Meet the Man”), and several key essays. Baldwin’s fiction,
which has been often misunderstood and dismissed by critics, in fact examines and
anticipates the complex ways in which we approach racialized gender, sexuality, and erotic
attraction in our own historical moment. Like “Reconstructing” (see section 3.1), “What’s
Love” invites students to develop final projects that engage broader themes in literary and
cultural studies, such as the centrality of domesticity to African American artistic production
and Baldwin’s theorizations of popular culture.
1437
4. Conclusion
These courses offer invaluable tools to today’s young people to enter a diverse and complex
world with strong humanities and digital-literacy skillsets. Students get to explore a major
international Black queer activist, intellectual, and literary figure, all while honing their
reading, writing, and public-speaking skills. Their final projects get published on StoryMaps,
thereby offering them an accomplishment to include on their resumes. Students explore
intersectional identity in ways that few other courses offer. For example, by exploring
Baldwin’s daring plots involving interracial couples and queer and other non-normative
sexual attachments, students wrestle with his careful dissections of class, religiosity,
colorism, and family violence—all of which a politics of respectability often kept hidden and
unspoken as much within intimate and local communities as on the national scene at his time,
and even in ours. With wars raging in Europe and Africa, and police brutality and gun
violence escalating in the United States, our students need humanities skills and new and
exciting learning tools more than ever.
References
Baldwin, James. (1953). Go Tell It on the Mountain. New York: Knopf.
---. (1956). Giovanni’s Room. New York: Dell Pub. Co.
---. (1962). Another Country. New York: Dial Press.
---. (1963). The Fire Next Time. New York: Dial Press.
---. (1964). Blues for Mister Charlie. New York: Dial Press.
---. (1965). Going to Meet the Man (Short Stories). New York: Dial Press.
---. (1968). Tell Me How Long the Train’s Been Gone. New York: Dial Press.
---. (1974). If Beale Street Could Talk. New York: Dial Press.
---. (1979). Just Above My Head. New York: Dial Press.
---. (1998). Collected Essays. New York: Library of America.
I Am Not Your Negro. (2017). Los Angeles: Magnolia Home Entertainment.
If Beale Street Could Talk. (2019). Beverly Hills: Twentieth Century Fox Home
Entertainment.
James Baldwin: From Another Place. (2007). Hudson: Hudson FilmWorks, Inc.
Thorsen, Karen & Dempsey, Douglas K. (1990). James Baldwin: The Price of the Ticket.
San Francisco: California Newsreel.
Zaborowska, Magdalena J. (2009). James Baldwin’s Turkish Decade: Erotics of Exile.
Durham: Duke University Press.
---. (2018). Me and My House: James Baldwin’s Last Decade in France. Durham: Duke
University Press.
1438
Digital Resources
_ Chez Baldwin: https://quod.lib.umich.edu/b/baldwin1ic?page=index
_ StoryMaps basics: https://doc.arcgis.com/en/arcgis-

storymaps/gallery/?rsource=https%3A%2F%2Fwww.esri.com%2Fen-
us%2Farcgis%2Fproducts%2Farcgis-storymaps%2Fstories
_ Selected student projects:

https://storymaps.arcgis.com/stories/43f933f23153477885ab114bf7ff7b08
https://storymaps.arcgis.com/stories/b47cba93c8444183997a915e3670b652
https://storymaps.arcgis.com/stories/09cf1dfb9e38447a9b3aaec89b3cc1a4
1439
Audiovisual and accessible translation from a transdisciplinary

insight: curriculum design and professional practice for
intercultural communication
Department of Translation and Interpreting, University of Malaga, Spain.
Abstract
Audiovisual translation and Media Accessibility are fields of Translation
Studies where academic research and professional practice directly influence
the translator’s training.
In this paper, we address the curriculum design of two modules on Audiovisual
Translation and Media Accessibility in an undergraduate level in terms of
concepts, modalities, text types, methodology, learning outcomes and
software. The main goal is to analyze to what extent professional practice
conditions training for intercultural communication.
We find out that there are labor multidisciplinary activities that demand new
functions from the audiovisual translator and we conclude with a reflection on
the transdisciplinary insight that training necessarily acquires in these fields.
Keywords: Audiovisual translation; media accessibility; translator’s training;
employability; intercultural communication.

Audiovisual and accessible translation from a transdisciplinary insight
1. Audiovisual Translation and Media Accessibility in Translation Studies

Audiovisual Translation (AVT) is the denomination of a translation modality that can be also
named in many different ways: dubbing and subtitling, film or cinematographic translation,
multimedia translation, subordinate translation, translation for television, translation for the
screen... Principally due to its wide scope, the most internationally agreed term is AVT, used
as an umbrella term (Díaz-Cintas & Remael, 2021).
This terminological acknowledgement determines the definition of the concept, that needs to
be broad enough to cover a wide spectrum of texts and broadcast channels but limited enough
to not overlap other misleading modalities of translation that could be integrated under the
same umbrella, like webpages localization. In this sense, AVT can be defined as a translation
modality whose texts are characterized for offering information through the simultaneous
combination of two different channels, the auditory canal and the visual channel, and for
being multimodal texts in which at least two codes converge, the linguistic and the visual,
sometimes also integrating the musical code (Chaume, 2012).
Media Accessibility (MA), on the other hand, refers to those practices that allow users with
sensory impairments (visual and hearing) to enjoy “full access, without obstacles or barriers,
to information, entertainment and training offered not only by audiovisual media (film, video
and DVD, television, and the internet), but also by other communicative situations such as
theatre, opera, conferences, museum exhibitions, etc.” (Díaz-Cintas, 2010, p. 158; our
translation). From a more universalist account, MA embraces a “broader conception of
accessibility as an instrument for the human rights of all”, and it is not confined to “any
specific group of people or barriers” (Greco & Jankowska, 2020, p. 62).
AVT and MA, therefore, have a plural nature prone to approaches from different perspectives
that converge in the three valuable poles of all translatological and translation activities:
academic research, professional practice and training.
The academic grounds are settled within Translation Studies, a discipline that is characterized
by its interdisciplinarity, understood as the tendency to establish links with other academic
disciplines with which it cooperates in approaches and methods in order to progress in
scientific knowledge.
In the specific fields of AVT and MA, this disciplinary convergence reveals itself (i) in the
current research trends highlighted in international academic conferences, highly permeable
to the inclusion of research perspectives from other academic fields (e.g. Linguistics,
Semiotics, Sociology, Film Studies or Reception Studies); (ii) by integrating research
methodologies derived from disciplines such as Statistics or Sociology; (iii) through the most
cutting-edge research instruments like the ones used for eye-tracking experiments (generally
targeted towards marketing studies, professional training, sociological studies or neurological
1442
studies); (iv) in order to implement their results in everyday realities, such as in the form of
subtitling for the D/deaf and the hard-of-hearing (SDH), audio description for the B/blind
(AD), surtitling for the opera or theatre plays or respeaking for live broadcasting; (v) making
its first attempts in immersive SDH and AD environments, mainly intended for the theatre
and implying virtual reality technology; (vi) or comprising diversified recipients (e.g.
children, languages learners, the elderly, audience with mental impairments, etc.).
2. Translator’s training in Audiovisual Translation and Media Accessibility
2.1. Transdisciplinarity
When it comes to the training in AVT and MA, they become a transdiscipline in their own
right as they traverse disciplinary boundaries to combine those pedagogical approaches that
best suit them according to real professional demands, being the most significant ones:
i. The approach by learning objectives and tasks (Delisle, 1993), which brings
together the intention pursued in the translation process and finally achieved in the
translation product by means of various pedagogical activities. That is the case of
the following learning outcomes intended for undergraduate translation trainees in
the modules subject to revision: “Trainees will be able to translate various types of
audiovisual texts to different extents, identify different types of documents and the
translation challenges that may be posed by a certain project, analyze their ability to
translate each translation assignment and produce a translation of an acceptable
quality in a reasonable period of time”.
ii. The functionalist approach (Nord, 2005), which understands translation as an act of
intercultural communication, and in doing so the translation trainee should develop
a general competence of “Acquisition of knowledge of other cultures and
development of linguistic and cultural interaction and mediation skills”, as it is
stated in the modules.
iii. The constructivist and social approach (Kiraly, 1995 and 2000), whose axis is the
collaboration between trainers and trainees alike, that is, the collaborative learning;
in this case, trainees are fostered to engage themselves in developing a transversal
competence, namely, the “Ability to work in a team and to relate to other people
from the same or from a different professional field”.
Nowadays many public institutions, private agents and professional associations, both at
national or international levels, offer specialized training in AVT and MA that favors the
very much needed emergence of the transdisciplinary audiovisual translator. In an
undergraduate level, Spanish universities currently offer modules in AVT, with or without
MA in their study programs, as part of the curricula of the degrees in Translation and
Interpreting.
1443
2.2. Curriculum Design

Precisely, we contextualize this study within two modules of AVT that are offered in year 3
and year 4 of the Degree in Translation and Interpreting at the University of Malaga (Spain).
Being the first module compulsory and the second module optional, the total trainees’
workload is 150 hours per module, out of which 45 hours are devoted to classroom training.
The main objective of these modules is to awaken and deepen the inter and intralingual
translation competence in AVT and MA from English into Spanish, in the case of AVT and
MA, or from Spanish into Spanish, solely for MA. To do so, three specific objectives are
followed:
i. To apply theoretical, terminological and instrumental bases (documentation and
computer tools) in the translation of audiovisual texts.
ii. To understand the translation of audiovisual texts as a professional activity.
iii. To reflect on the translation process from a linguistic and a pragmatic perspective.
Together with the learning outcomes and competences formulated above, these objectives
lead to the design of a very specific curriculum, where transdisciplinarity inevitably rules.
The coordination between the two modules is mandatory so that there is a direct relationship
between the time and resources that are invested in the training process and the intended
learning outcomes.
For this reason, the following table sums up this effort in terms of concepts, modalities, text
types, methodology, learning outcomes and software that are actually used in both year 3 of
the Degree (first year of AVT) and year 4 of the Degree (second year of AVT and first year
of MA).
1444
Table 1. Audiovisual Translation and Media Accessibility Curriculum Design.
Year 1 Year 2
Concepts
AVT theory AVT professional environment:
AVT modalities o Global panorama and job prospects
Analysis of the audiovisual text • Profession
AVT professional environment: - Project management
o Self-employment • Associations
o Estimates • Training
o Invoices • Challenges
Modalities
Subtitling: Subtitiling for VOD platforms
o Machine translation subtitling and Dubbing for VOD platforms
postediting Multimedia translation:
Dubbing: o Localization and translation of video
o Voice over/narration games
Media accessibility:
o Subtitling for the D/deaf and the hard-
of-hearing
o Audio description for the B/blind
Text types
Subtitling: Subtitiling and dubbing for VOD platforms:
o Movies o Movies
o Trailers o TV series
Dubbing: Multimedia translation:
o TV series o Localization and translation of video
o Advertising spots games
o Corporate videos • Video games
o Voice over/narration Media accessibility:
o Interviews o Subtitling for the D/deaf and the hard-
o Reality shows of-hearing
o Instruction manuals • Documentaries
• Tutorials
o Audio description for the B/blind
• Animation (short or feature films)
• Informercials
Methodology
o Direct translation EN>ES with/without o Direct translation EN>ES with/without
transcripts transcripts
o Intralingual Translation ES>ES (SPS and
o British EN + American EN + other varieties AD) with/without transcripts
o Clips 1:30-3:00 minutes long o British EN + American EN + other
o Difficulties: repetitions, hesitations, varieties
overlapping voices, speed, humour, puns, o Clips 3:00-4:00 minutes long
foul language, terminology, calques, songs o Difficulties: style, register, dialects,
o Assignments: same text types with added sociolects, idiolects, multilingual texts,
difficulties neutral Spanish, cultural references,
o Estimates and invoices sound effects (SPS)
o Assignments: same text types with added
difficulties
o Estimates and invoices
1445
Learning outcomes
Subtitling: Subtitiling for VOD platforms:
o Subtitling according to an academic style o Subtitling according to professional style
guide (e. g. Díaz-Cintas and Remael, 2021): guides (e. g. Netflix or BBC)
• Spotting (with/without transcripts) • Review of previous competences and
• Formal and textual features introduction to new ones
• Translation techniques • Subtitling without spotting lists
• Spatial and temporal features (with/without transcripts)
o Proficiency in subtitling software (with o Proficiency in subtitling sofware (with
subtitles embedding) subtitles embedding)
o Proficiency in project management
Subtitling for the D/deaf and the hard-of-
hearing
o Subtitling according to UNE 153010 and
comparison with professional style
guides
o Subtitling without spotting lists
(with/without transcripts
o Proficiency in subtitling sofware (with
subtitles embedding)
Dubbling (voice over/narration): Dubbing for VOD platforms:
• Dubbing according to an academic style o Dubbing according to professional style
guide (e. g. Chaume, 2012): guides (e. g. Netflix or Deluxe)
• Dubbing (with/without transcripts) • Review of previous competences and
• Translation techniques introduction to new ones
• Synchronization and adaptation o Proficiency in dubbing and video editing
• Text segmenting software (with recording)
• Dubbing symbols Audio description for the B/blind:
• Proficiency in dubbing and video editing o Audio description according to UNE
software (with recording) 153020 and comparison with
professional style guides
o Audio description with/without dubbing
(with/without transcripts)
o Proficiency in dubbing and video editing
software (with recording)
o Proficiency in project management
Localization and video game translation:
o Multimedia translation: subtitling,
dubbing, voice over/narration,
localization…
o Proficiency in Excel and subtitling,
dubbing and video editing software (with
subtitles embedding)
Software
Subtitling: Subtitle Wokshop Subtitling for VOD platforms: Aegisub
Machine translation subtitling and Subtitling for the D/deaf and the hard-of-
postediting: DeepL, Google Traductor, hearing: Aegisub
YouTube
Subtitles embedding: Handbrake Subtitles embedding: Handbrake
Dubbing: Audacity + Avidemux /Openshot Dubbing for VOD platforms: Audacity +
Shotcut
1446
Voice over/narration: Audacity +Audio description for the B/blind: Audacity

Avidemux/Openshot + Shotcut
Project management: Airtable, Google
General softaware: Any Video Converter/Freemake Video Converter/Handbrake, VLC
2.3. Professional Practice

The curriculum design table is reviewed and updated every year based on the needs of AVT
and MA as a professional practice, on the actual university budget and the trainees’
information technology (IT) possibilities. The latter explain the choice of free and open
source software compatible with Mac and Windows from the myriad of tools available. The
former delves into the current demands of the labor market, that require from the audiovisual
translator to have professional skills in order to carry out multidisciplinary activities linked,
above all, to the handling of new technologies. It is the case of computer-assisted translation
tools, machine translation tools, the translation and localization of video games and subtitle
post-editing.
In addition, specifically designed AVT tools are a must in the professional skills that any
audiovisual translator should master regarding IT. That is the case of specific subtitling
software or video editing software useful for dubbing.
Regarding the concepts, modalities, text types and methodology implemented in both years,
apart from channeling them towards the professional environment, all are sequenced from
the general to the particular, they are graded in progressive levels of accomplishment and
they do not overlap, trying to cover the widest spectrum of learning outcomes.
These multidisciplinary activities demand new functions from the audiovisual translator,
whose professional practice may appear camouflaged in different terminology. As a matter
of fact, if we search under “translation” in the Netflix Jobs webpage, we will find different
positions with no trace of “translation” in their denominations, but they would certainly
benefit from the translators’ professional skills; not to mention that the very same positions
are also found under other searching terms, more AVT-related, like “subtitling” or
“dubbing”: Content Localization Project Manager, Localization Director, Localization
Producer – Games, Localization Project Manager, Localization Project Manager – Studio
and Marketing, Manager, Film & Series Marketing, Product Manager, Localization, or
Senior Animator & Rigger – Games Studio. Being most of these positions related to projects,
diverse work teams are highlighted in the lateral menu, some of them belonging to adjacent
fields, not specifically to AVT itself, like marketing and advertising, economic sciences, data
engineering or even research. This is the case of Consumer Insights, Creative Content,
Creative Marketing Production, Creative Production, Data Science and Engineering, Netflix
Games Studio, Partner Marketing, Partnership, Product Management, or Strategy and
Analysis.
1447
3. Final remarks: An Intercultural Communication Crossroads

AVT and MA can be understood as an intercultural communication crossroads where the
academic interdisciplinarity is understood as a flow of parallel knowledge that is integrated
into AVT and MA; the professional multidisciplinarity refers to the versatility of the
audiovisual translator’s professional skills, and the training transdisciplinarity relates to the
figure of the audiovisual translator as a whole different from the sum of the parts that
compose them.
This configuration poses further challenges to the AVT and MA trainer, who must always
make an effort to update their curriculum design in relation to the work demands of the
prospective audiovisual translator while not neglecting the academic reflection on the
translation processes.
As for the latter, envisaging the turn towards Accessibility Studies as a different field from
Translation Studies (Greco & Jankowska, 2020) would allow more academic deepening in
both fields separately and in cooperation; as for the former, encouraging the collaboration of
educational institutions with AVT and MA companies in the form of, for instance,
employability and entrepreneurship actions (Enríquez-Aranda & García Luque, 2023) or
professional training for trainers (Bolaños-García-Escribano, Díaz-Cintas & Massidda, 2021)
would bring training closer to the professional context. The route ahead offers large
possibilities starting as it does from an intercultural communication crossroads.
Acknowledgements
This paper is part of the Project “The training in audiovisual and accessible translation
oriented towards the professional practice: from university to business” (PIE22-014) within
the Action “INNOVA22. Educational Innovation Groups and Projects 2022-2024”
(University of Malaga, Spain).
References
Bolaños-García-Escribano, A., Díaz-Cintas, J. & Massidda, S. (2021). Latest advancements
in audiovisual translation education. The Interpreter and Translator Trainer, 15(1), 1-12.
doi: 10.1080/1750399X.2021.1880308.
Chaume, F. (2012). Audiovisual Translation. Dubbing. Manchester: St. Jerome.
Delisle, J. (1993). La traduction raisonnée. Manuel d’initiation à la traduction
professionnelle de l’anglais vers le français. Ottawa: University of Ottawa.
Díaz-Cintas, J. (2010). La accesibilidad a los medios de comunicación audiovisual a través
del subtitulado y de la audiodescripción. In L. González & P. Hernúñez (Coords.), Actas
del IV Congreso El español, lengua de traducción (pp. 157-180). Madrid: ESLETRA.
Díaz Cintas, J. & Remael, A. (2021). Subtitling: Concepts and Practices. London: Routledge.
1448
Enríquez-Aranda, M. & García Luque, F. (2023). Training Media Accessibility

Professionals: An Employability and Knowledge Transfer Experience. 1st International
Conference on Didactic Audiovisual Translation and Media Accessibility (TRADIT23).
Madrid: UNED. Retrieved March 24, 2023, from
https://canal.uned.es/video/64240f263056d52c060adee2?track_id=64240fe83056d52c0
d0f5ba2.
Greco, G. M. & Jankowska, A. (2020). Media Accessibility Within and Beyond Audiovisual
Translation. In L. Bogucki & M. Deckert (Eds.), The Palgrave Handbook of Audiovisual
Translation and Media Accessibility (pp. 57–81). Basingstoke: Palgrave MacMillan. doi:
10.1007/978-3-030-42105-2_4.
Kiraly, D. C. (1995). Pathways to Translation. Pedagogy and Process. Kent State University:
Kent State University Press.
Kiraly, D. C. (2000). A Social Constructivist Approach to Translator Education. Manchester:
St. Jerome Publishing.
Nord, C. (2005). Training Functional Translators. In M. Tennent (Ed.), Training for the New
Millenium. Pedagogies for Translation and Interpreting (pp. 209–223). Amsterdam: John
Benjamins. doi: 10.1075/btl.60.17nor.
1449
Setting learners up for success: a universal design for learning

approach to industry placement assessment
National Centre of Excellence in Furniture Design and Technology, Atlantic Technological
University, Ireland.
Abstract
The diversity of learners has never been more pronounced with the
accessibility to learning through remote means. Universal Design for Learning
(UDL) endevours to address some of the challenges posed by their diverse
needs. This research aims to explore the perceptions of learners and their
academic supervisor on the use of UDL principles as part of the assessment
strategy of an Industry Placement module undertaken in a higher education
institution. The learner participants were offered assessment feedback in a
number of different formats by the supervisor participant; namely typed and
video. The results of the research indicate a very positive response from both
stakeholders. Multiple means of representation and action/expression were
found to help break down barriers for diverse learners and to set them up for
success. Scaffolding and content creation methods that are comparable in
terms of workload need to be provided to increase adoption by both
stakeholders.
Keywords: Universal design for learning; multiple means of representation;
multiple means of action & expression; assessment strategy; industry
placement.

Setting students up for success: a UDL approach to industry placement assessment
1. Introduction
Offering learners equal opportunities when it comes to their educational journey seems like
something that should be the norm rather than an exception. However, in reality, this is not
always the case. Novak (2022, pg. 2) reports that learners often experience “academic,
behavioural, social-emotional, cultural, and linguistic barriers” to learning. These diverse
needs of learners require a customised approach to learning goals, course content, delivery
methods, and assessment. The Universal Design for Learning (UDL) framework developed
by Meyer and Rose in the 1990s aims to cater to the diverse needs of learners (Meyer, Rose,
and Gordon, 2014). Novak (2022, pg. 19) defines the Center for Applied Special Technology
(CAST) UDL framework as “an expression of a belief that all students are capable of learning
and that instruction, when crafted and implemented with this belief in mind, can help all
students succeed in inclusive and equitable learning environments”. The UDL framework
applies a three-pronged approach of multiple means of engagement, multiple means of
representation, and multiple means of action/representation (CAST, 2023). Multiple means
of engagement enable learners with different motivations to become interested and to
successfully complete their programme of learning. Sensory disabilities, learning disabilities,
language, and culture all influence how learners understand and comprehend content
presented to them. Multiple means of representation set out guidelines to support educators
in offering choices to learners. Lastly, the provision of multiple means of action and
expression empowers learners to choose the means of communicating their learning in a way
that matches their strengths. The importance of choice is reinforced by Merrill and Gonser
(2021) when they state, “By centering choice, educators signal openness to negotiating the
middle ground and offer students scaffolded opportunities to practice decision making,
explore their academic identity, and connect their learning to interests and passions”.
The challenges of remote learning have also become more prominent in recent years and are
of interest to this study. While remote delivery has made learning more accessible it has also
brought many challenges. There is no in-person contact, and therefore fewer opportunities
for informal communications, for clarification and feedback purposes, to take place between
learners and lecturers. Both parties need to proactively arrange formal online meetings and/or
send emails. This lack of engagement choice is a challenge for diverse learners and a barrier
to success. On the other hand in on-campus settings, informal communications happen
organically - learners approach lecturers at the end of lectures, chance meetings in the
corridors, and/or dropping into a lecturer’s office. These and other informal communication
methods provide the on-campus learners with multiple means of communication with their
lecturer. Peer-to-peer exchanges are also more prevalent in on-campus learning. The lack of
effective interaction in an online environment is confirmed by Tarhini et al (2013). For
remote delivery, information representation often defaults to a typed format – such as
email/learning platform notifications, assessment briefs, and feedback. While a typed format
1452
is also used as part of face-to-face delivery it is supplemented by other diverse means of

communication not feasible and/or applied in remote delivery. This combination of reduced
means of communication and reduced means of information representation poses a challenge
for remote learners with the diversity of remote learners and their needs far more pronounced
when compared with face-to-face learning. UDL literature highlights the value of supporting
learners “It’s about eliminating barriers so every student can succeed” (Novak, 2022, pg. 33).
The overall aim of this research was to explore the perceptions of learners and academic
supervisors on the implementation of UDL principles as part of the assessment strategy of
an Industry Placement module undertaken in a higher education institution. This included
exploring learners’ views on the possibility of being offered alternative means of expression
when submitting assessments in the future and the experience of an academic supervisor
using multiple means of representation when providing assessment feedback. The research
was developed to address the needs of diverse learners who are engaged in remote learning,
specifically on professional placement, who have been found to be under-represented in
higher education in Ireland (Waters & Rath, 2022). The stakeholder’s viewpoint will help
to understand how the current assessment strategy is meeting the needs of the learners and
academic supervisors and what impact UDL principles can have. It is hoped learners’
engagement and success can be improved.
2. Methodology
A case study approach was employed for this research. This involved exploring the
participants’ experiences of assessment when UDL principles were applied to a module they
were engaged with. Yin (2018, p.15) defines “a case study is an empirical method that
investigates a contemporary phenomenon (the ‘case’) in depth and within its real-world
context”. A stage 3 Industry Placement module was selected as the case study for this
research. Learners undertake the Industry Placement module – worth 40 out of 60 credits -
from September to May. They are required to attend the industry placement company four
days a week and online lectures one day a week. An academic supervisor is assigned to the
learner to guide and assess their learning and engagement whilst on industry placement. The
learners typically have no face-to-face contact with their academic supervisor except for one
meeting during the academic year that takes place in their placement company.
Communication is via email and MS Teams meetings. Assessment of the module includes
learners submitting weekly placement reflective journals, to a personal learning software
called PebblePad, detailing their learning experiences. The academic supervisor provides
feedback on a weekly basis which the learners must respond to in subsequent journal
submissions. The current assessment strategy stipulates the learners submit a typed reflective
journal with up to 8 supporting images on PebblePad by 9pm at the latest each Sunday and
the academic supervisor provides feedback via a text box and the option to add an attachment
1453
in PebblePad by 6pm at the latest each Friday. Learners are not offered alternative means of
expression when submitting their reflective journals and multiple means of representation are
not being utilised by the academic supervisors when providing feedback to the learners.
In semester 1 of the 2022-2023 academic year an approach was made to one of the researchers
by a learner on the Industry Placement module who reported they were struggling to
comprehend and respond to typed feedback from the researcher who was their academic
supervisor. Specifically, interpreting passages of text was identified by the learner as being
an issue. This was causing the learner a significant amount of anxiety and stress. The learner
also disclosed they had several learning difficulties the academic supervisor was unaware of.
The means of representation were not effective for this learner’s needs. In response, UDL
principles were piloted to alter how feedback was provided to this learner by the academic
supervisor and later rolled out to a number of other learners. This included focusing on
providing clear and structured feedback, adding numbering to separate out points of
information and where a response was required, recording feedback using MS Teams video,
providing the video link, attaching a copy of the video and the transcript of the video to the
feedback form in PebblePad, and encouraging the learners to engage with learning support
where appropriate. The application of UDL principles to this module set out to address the
shortfallings identified and reduce barriers for diverse learners to succeed in the module.
Purposeful sampling was used for this research, with the academic supervisor participant the
researcher, and the three learner participants being supervised by the academic supervisor
participant. This was conducted as a pre-pilot study with a view to conducting a full pilot in
the 2023-2024 academic year with a full cohort of learners (n=36 and academic supervisors
n=7). Hence the small sample size in this case.
Data was gathered from the learners in the form of an interview conducted by the researcher
who was also the academic supervisor. The learner participants were asked about their views
on the current assessment strategy and the initial UDL principles tested including 1) their
experiences of the UDL principle employed in the assessment of the Industry Placement
module, 2) their perceptions of the barriers and support for student success for diverse
students undertaking remote learning using UDL principles. A semi-structured interview
was used with a combination of open and closed questions The duration of the online
interviews was 10 minutes. An online interview using MS Teams facilitated learner
participants placed in companies in different geographical locations to complete the
interview. The academic supervisor participant engaged in personal reflection on the
experience from their perspective documented through a video and engaged in dialogical
reflection with the learner participants.
Ethical issues of the research pertained to the risks of 1) actual and perceived student coercion
due to the power balance of the researcher as the academic supervisor of the learner
1454
participants, and 2) disclosing confidential information provided by the participants

(including personal and/or sensitive data). These risks were mitigated by taking appropriate
measures in the research method including obtaining ethical approval from the institutions’
ethics approval committee, informing the learner participants that participation in the study
would confer no academic advantage or disadvantage for them, providing information sheets
and an opportunity to ask questions, voluntary signed consent, the right to withdraw up until
the interview transcripts were collated and data anonymized, and the anonymising of all
personal/organisation names provided in the interviews.
3. Findings & Discussion

The findings of this research will be discussed from the perspective of the learners first,
followed by the perspective of the academic supervisor.
3.1. Learner Findings

Of the three learner participants, all three submitted their reflective journals in a typed format
as stipulated by the assessment strategy. When asked if they ever asked to submit using a
different means of expression Respondent 1 (R1), Respondent 2 (R2) and Respondent 3 (R3)
said no. R1 commented, “They've asked me to do this in writing, so I'll just do it in writing.
Keep my head down and get it done”. All three respondents expressed an interest in
submitting using different means of expression. R2 is cited as saying “I haven't thought about
it for other[s] kinds of assessment” and “maybe I would consider something more creative,
like a presentation or [or] even a video”. A similar response from R1 states “maybe a video
a week would be useful, … and kind of it would be a bit…less stressful”. R1 ranked their
preference of reflective journal submission formats as follows: “Audio or video, probably
first and then face to face, then writing… probably last”. There was some caution expressed
by R2 “I've been most confident in the writing just because that's what we do all day, every
day and even at work now, writing emails every day”. R3 was “definitely [be] open, open to
doing a different method” but reported being worried about the quality of the submission
using different formats citing “my grades might take a hit for a little bit” but “I would
eventually get better at it and it would probably even exceed my typing potential”. On the
other hand, R1 spoke about hoping it improved the standard of their submissions when they
said “Maybe I'd feel more connected to it, so maybe I'll put… more of an effort …Yeah, it
would make it more colorful and more kind of interesting”. The issue of support to create
reflective journals using formats other than typed was then discussed. While R1 felt confident
they did not need support creating videos as “it's not something I'm entirely unfamiliar with”
and “it's quite an easy kind of an easy medium to…to take on board”, the other two
respondents felt support was needed. R2 cited needing technical support while R1 said “I
would like to experience all methods to see what I am best with”. Specifically getting to
1455
experiment with the methods and become competent without the pressure of being assessed
was suggested.
The participants were asked for their perceptions of the change from typed to video feedback
from the academic supervisor. All were very positive with R3 making comments such as it
“just help[ed] me keep up with it”, “it was definitely a massive improvement”, “that was a
massive weight on my shoulder” and “the spoken word was easier to consume”. It’s
noteworthy that this respondent has learning difficulties and found typed feedback difficult
to comprehend. R2 said they did not need to receive video feedback but “it feels more
personal” and “It’s just friendly through the video”. Likewise, R1 reported that they found
both typed and video feedback were equally effective but feedback via video was “nice
because I can tell tones” and “I can tell if there’s something umm you want to tell me that I
might miss in the writing… which is interesting”. R1 also cited in reference to video feedback
“I really liked it because I found it easier to remember…I’d be working in the
workshop…remember something you said”. Both R1 and R2 confirmed they do not have any
learning difficulties. So, while R1 and R2 said they did not need video feedback if they had
a choice they would choose receiving video feedback.
3.2. Academic Supervisor Reflections

In terms of the means of representation utilised by Industry Placement module learners, the
academic supervisor participant had no preference as to the means of expression they would
receive for review and assessment. As per the assessment rubric for the reflective journals,
the academic supervisor participant encouraged and found it beneficial when R1, R2, and R3
used supporting images, along with the typed feedback to assist the understanding of the
subject matter being discussed in the journal. Their reflection on providing multiple means
of representation for the reflective journal was a positive one with the option of the learning
being able to choose to be promoted. The academic supervisor comments “help the student
to find what they’re best at and how they can maximize their grades”.
The academic supervisor had never provided video feedback to learners of any module
previously. Instead, typed feedback was used as was common practice among the participant
and their academic peers in higher education. While there was no question that the academic
supervisor would not apply their knowledge of UDL principles to change the feedback format
in the hope the difficulties R3 was experiencing could be resolved. However, there was a
preconception on their part that it may take longer than typed feedback and there may be a
drop in the quality of the feedback. The experience of the academic supervisor participant of
creating video feedback was that it did involve a learning curve and initially, it took longer
than typed feedback. However, the academic supervisor reports they got faster at creating
videos with practice. Secondly, with practice, the academic supervisor reflected their diction
improved and delivery become more natural. Transcripts required post-video editing. Over
1456
the course of the semester, as the academic supervisor became more proficient with creating
and editing video feedback they reflected “it is marginally longer than with the written
one…the transcript is probably the hardest bit… I go through it and I delete the time stamps”.
Learner participants reported they watched the feedback video and read the accompanying
transcript. For academic supervisors to engage with video feedback the academic supervisor
participant asserts “a supervisor [needs to] be comfortable in whatever media they use for
feedback… that you know it supports the student, but it's not onerous and it's not a stress for
them either”. The academic supervisor has questioned whether both means of expression are
equal in terms of quality “I could do a lot more editing and very carefully choose my words
and language in a written one. That I couldn't do in the video unless it was going to be
recorded 100 million times”. When questioned all three learner participants reported they did
not notice a drop in content and quality when the means of representation changed from typed
to a video format.
Like the learner participants, the academic supervisor participant did report they preferred
video feedback “I enjoy it as well” and “I felt more connected with the students by me
speaking to them”. Interestingly, when R1 and R2 were asked towards the end of the semester
which means of representation they preferred R2 said “It was a nice change and I'd be happy
to get the feedback in that format in the future - whatever is easiest for you”. Interestingly,
the academic supervisor participant was now in the position where choice has been handed
back to them by a number of the learner participants as to what means of representation they
wished to use to provide feedback. On reflection, the academic supervisor participant decided
“I could have given written feedback to a number of those students, [but] I opted not to
because I actually enjoy doing the video feedback”.
4. Conclusions & Recommendations

Although only a small pre-pilot study the perceptions of the learners and academic supervisor
stakeholders indicate the UDL interventions in the Industry Placement module assessment
strategy were beneficial and supportive of the participating learners. These interventions
included focusing on providing clear and structured feedback, adding numbering to separate
out points of information and where a response was required, recording feedback using MS
Teams video, providing the video link, attaching a copy of the video and the transcript of the
video to the feedback form in PebblePad, and encouraging the learners to engage with
learning support. In particular, it ensured that learners with learning difficulties could fully
engage with the reflective journal assessment element of the module and it set them up for
success. UDL choice empowered the learners to be resourceful, and knowledgeable and to
take ownership of their learning for the Industry Placement module. Any changes to industry
placement journal submissions and feedback should not take learners/academic supervisors
longer than current means of expression and representation. Remote learning was found to
1457
pose a challenge for students with diverse learning needs. It is more difficult to access
multiple means of representation and expression, to link in with peer support, and to engage
with institutional learning supports. The importance of face-to-face visits cannot be
underestimated to ascertain UDL needs.
Five recommendations emerge from the research namely; 1) Ensure the Industry Placement
assessment strategy provides learners with a choice when submitting their reflective journal
submissions and update the personal learning software PebblePad to accept them, 2)
Encourage both learners and academic supervisors to explore multiple means of expression
and representation, 3) Provide scaffolding for learners and academic supervisors to ensure
they are competent in alternative means of expression and representation, 4) Develop
methods for creating content using multiple means of expression comparable in terms of
workload, 5) Conduct a wider study with a full group of learners and academic supervisors
in a future iteration of the Industry Placement module to ascertain the impact of UDL
principles on the assessment strategy on learning success.
References
Center for Applied Special Technology [CAST], (2023). About universal design for learning.
Retrieved February 7, 2023, from https://www.cast.org/impact/universal-design-for-
learning-udl
Merrrill, S., & Gonser, S. (2021). The importance of student choice across all grade levels.
Edutopia. Retrieved from https://www.edutopia.org/article/importance-student-choice-
across-all-grade-levels
Meyer, A., Rose, D.H., & Gordon, D. (2014). Universal design for learning: Theory and
Practice. Wakefield, MA: CAST Professional Publishing.
Novak, K. (2022). UDL now! A teacher’s guide to applying universal design for learning (3rd
ed.). Massachusetts, USA: CAST Professional Publishing.
Tarhini, A., Hone, K., & Xiaohui L. (2013). Extending the TAM model to empirically
investigate the students’ behavioural intention to use e-learning in developing countries.
2013 Science and Information Conference (SAI), 732–737.
Waters, B., & Rath, V. (2022). Students with disabilities on placement – Guidance on the
provision of reasonable accommodations on practice-based placements in professionally
accredited programmes. Dublin, Ireland: AHEAD Educational Press.
Yin, R.K. (2018). Case study research and applications – Design and methods. California:
Sage Publications.
1458
The UNIVAC project: implementing the user journey approach in

accessibility research at university
Blanca Arias-Badia1, Irene Hermosa-Ramírez1, Mar Ogea2, Mireia Oliver1, Daniel
Segura3, Ana Tamayo4, Sergi Torner1
1
Department of Translation and Language Sciences, Pompeu Fabra University, Spain,
2
Faculty of Philosophy and Letters, University of Cordoba, Spain, 3Catalan Association for
the Promotion of Accessibility, Spain, 4Department of English and German Philology and
Translation & Interpretation, University of the Basque Country, Spain.
Abstract
Work for inclusive, quality education and the reduction of inequalities is
among the prioritised lines of action promoted by the Sustainable Development
Goals of the United Nations (2015). In the context of higher education, persons
with sensory disabilities who study and work at universities face daily
accessibility challenges when interacting with their pervasive digital
environments. The ongoing UNIVAC research project aims to qualitatively
analyse the measures taken by Spanish universities to overcome such
challenges, as well as to explore areas of improvement in accessibility at
university. This paper presents the project’s aim and scope, and proposes the
adoption of the cross-disciplinary user journey approach as a form of
methodological innovation in accessibility research in higher education.
Keywords: UNIVAC project; accessibility; user-centered research; user
journey; persons with disabilities.

The UNIVAC project: implementing the user journey approach in accessibility research at university
1. Introducing the UNIVAC project

Today, there is a growing interest in the international arena to work for inclusive, quality
education. Work for inclusive education and the reduction of inequalities is consistent with
the Sustainable Development Goals (SDGs) proposed by the United Nations (2015), which
establish, among others, the following objectives:
By 2030, ensure equal access for all women and men to affordable and quality
technical, vocational and tertiary education, including university.
By 2030, eliminate gender disparities in education and ensure equal access to all levels
of education and vocational training for the vulnerable, including persons with
disabilities, indigenous peoples and children in vulnerable situations.
Build and upgrade education facilities that are child, disability and gender sensitive
and provide safe, nonviolent, inclusive and effective learning environments for all.
In this context, the project Sensory accessibility at the Spanish university: current needs and
prospective solutions (henceforth UNIVAC) aims to analyse the response given by Spanish
universities to the needs of persons with sensory disabilities (PSDs). In a context of
increasing digitisation, which has been exacerbated by the Covid pandemic, the project
intends to offer a reliable diagnosis of the reality of our university system. Such a diagnosis
is a necessary first step towards the implementation of measures to guarantee PSDs’ equal
opportunities and their right to quality education. UNIVAC proposes an empirical study
based on a wide set of qualitative data aimed at revealing both the needs of PSDs and the
actions currently taken to cater for them in the public universities of three autonomous
communities in Spain, namely Andalusia, Catalonia, and the Basque Country.
The project focuses on the needs of PSDs belonging to the three main types of university user
groups, namely, students, teaching and research staff, and administration staff. The object of
study are not PSDs in themselves, but rather the environments in which they carry out their
daily activities and, more specifically, the digital or digitised environments (the physical
classroom in which digital material is used, the software used by university staff, the virtual
classroom, registration management applications, and so on). This approach is aligned with
the social and the human-rights conceptions of disability, in accordance with which disability
is perceived as an experience of avoidable exclusion linked to the environment (Albert &
Hurst, 2012; Huete, 2013). In the project’s framework, the source of any access barriers is
not an individual with disabilities, but rather an inaccessible society, which is not prepared
for a digital transition that takes everyone into account. Thus, any interventions to improve
accessibility need to take place on the environment. In this context, key digital accessibility
services designed for PSDs come into play: the project will report on the degree of
implementation of services such as captioning, subtitling, translation and interpretation in
1460
Blanca Arias-Badia et al.
sign languages, audio description, the audio induction loop, audio guides, alternative text,
adaptation of texts to the Braille system, large print, or speech synthesis, among others.
Apart from aiming to respond to a social need—the current context of digitisation should not
leave PSDs behind; in this case, by keeping them apart from the university system—, the
project seeks to advance in both the theoretical and descriptive knowledge gathered in the
field of inclusive higher education. Through reports published by entities that work for the
inclusion of persons with disabilities in higher education in Spain, such as the Universia
Foundation1, we have access to statistical data that draw attention to the number of students
who denounce situations of inaccessibility faced during their university studies. This kind of
quantitative information does not by itself allow to map the types of accessibility challenges
faced and propose solutions, as the data must be complemented with qualitative information.
The innovative approach of the project is precisely the qualitative insight that is gathered
from PSDs through focus groups, semi-structured interviews and student diaries, all applying
the user journey approach. Ultimately, UNIVAC is expected to empirically contribute to
policy-making. In this regard, the participation of inclusion staff in the project is another
strong suit.
2. Aims and scope of the project

As has been mentioned, the project aims to offer a diagnosis of the current situation in regard
to the challenges and measures taken to provide inclusion for PSDs studying and working at
university. More specifically, the project has three main objectives: the first one is to describe
the different types of needs described by users with different profiles (year of study, area of
knowledge, work field, type of disability). This objective will allow us to interpret the
quantitative data already available today in regard to disability and higher education, such as
the dropout rates of students with disabilities, as well as the underrepresentation of PSDs in
MA and PhD degrees (Fundación ONCE, 2022). In relation to the staff, the project will focus
on identifying accessibility challenges and solutions in current workflows.
The second objective of the project is to explore the accessibility services available at this
stage, and those on demand by users. Likewise, the project will describe the degree of
satisfaction with the present services, both on the users’ part and on the providers’ (e.g.
inclusion service units at the universities under study, teaching staff providing accessible
materials to students with disabilities). Research actions connected to this objective will
reveal whether access services are applied ex ante, i.e., in the design phase and in accordance
with the principles of universal design, or ex post, after the user’s demand, as an adaptation
1
<https://www.fundacionuniversia.net>. [02.02.2023]
1461
of a previously designed, inaccessible service or product (Greco, 2019; Arias-Badia et al.,

2022).
The third objective of the project is connected to its intensive dissemination—as has been
argued in the specialised literature, in the case of projects aimed at improving the accessibility
of environments or services, it is particularly important for the results to reach and be
“understandable” by the main potential beneficiaries of the research conducted (Greco &
Jankowska, 2019, p. 8)—in this case, PSDs, university students, and university staff. The
right of all citizens to participate in scientific progress and to enjoy its benefits and
applications is included in fundamental texts such as the Universal Declaration of Human
Rights (art. 27) or the International Covenant on Economic, Social and Cultural Rights (art.
15b). Likewise, the project wants to go beyond raising awareness about the need to pay
attention to accessibility and raise specific applicable proposals; the latter is in line with the
appeal recently made in the European TRACTION project to European researchers working
on accessibility (Orero, 2020).
3. Methodological approach
The study adopts a qualitative methodology. This section briefly reports on the types of data
collected for the purposes of the project, as well as on the approach adopted in the research
actions involving participants.
3.1. Data under analysis

Two types of data will be collected and analysed: data reported by informants and data
derived from the analysis of digital materials. The data reported by informants will include
the following:
• the recordings and transcripts of focus groups, with active students, with
graduated students, and with staff;
• the recordings of the observation of the interaction with digital media by PSDs,
to complement the self-reported challenges identified in the focus groups;
• questionnaires to representatives of the three stakeholders involved in the project;
• semi-structured interviews with representatives of non-profit organisations
working for the rights of PSDs;
• semi-structured interviews with accessibility professionals who work with users
of sensory accessibility services;
• diaries and longitudinal monitoring of a selection of users. A detailed follow-up
will be carried out on the experiences related to sensory accessibility of two
participants from each of these groups: students with disabilities, university staff
1462
with disabilities and teaching staff (with or without disabilities) with experience
in teaching PSDs.
The data derived from materials analysis will consist of the following elements:
• main website (homepage) of the universities of the participants in the focus

group;
• website presenting the studies of the participants in the focus group;
• website presenting the inclusion service unit of the focus group participants’
universities;
• registration application of the focus group participants’ universities;
• main website of the virtual campus of the focus group participants’ universities;
• accessibility guidelines and training material offered to university staff;
• selection of teaching materials used by the students participating in the focus
group;
• selection of software and digital documentation frequently used by the staff with
disabilities participating in the focus group.
The first data set (derived from social research instruments) will be analysed through
thematic analysis (Braun & Clarke, 2012) using the computer-assisted qualitative analysis
software ATLAS.ti. As explained in the following section, the qualitative data coding will
incorporate the phases of the user journey approach as themes. A content analysis will be
applied to assess the second data set (derived from the above-cited materials and documents:
university websites, accessibility guidelines available to university staff, teaching materials,
etc.). This analysis will also be conducted with the aid of ATLAS.ti.
3.2. The user journey approach

The term user journey is employed in the field of User Experience Design (UX) to describe
the prototypical path followed by users to reach a goal when consulting a website (Endmann
& Keßner, 2016). Similarly, the notion of the customer journey, from the field of Business
research (Tueanrat et al., 2021), is used in state-of-the-art industry research conducted with
accessibility users as informants to establish which processes call for the implementation or
improvement of accessibility services.
UNIVAC will incorporate this cross-disciplinary notion in the design of all the research
activities involving user participants, as well as in the description of the results of the project.
The project is interested in mapping the phases of the student/worker journey through
university in which inaccessibilities arise. Thus, the interviews, focus groups and
questionnaires will contain blocks on the following aspects:
1463
§ discovery: this block will contain questions about situations of (in)accessibility

related to the phase in which users realised their need or desire to study or work at
university.
§ information: this block will contain questions about situations of (in)accessibility
related to the phase in which users tried to find out about possible studies / jobs.
§ selection (“sale”): this block will contain questions about situations of
(in)accessibility related to the phase in which users opted for studies/jobs.
§ follow-up (“acquisition”): this block will contain questions about situations of
(in)accessibility related to studies / job performance.
In sum, the UNIVAC project combines several qualitative methodologies to 1) describe the
state of sensory accessibility measures in Spanish universities applying a user journey
approach, 2) to cross-check university staff (both professors and inclusion staff) and students’
experiences with data derived from the analysis of digital materials, and, ultimately, 3) to
encourage policy change regarding current shortcomings in (digital and in-person)
accessibility services in higher education settings.
Acknowledgements
This paper is part of the UnivAc project (ref. TED2021-130926A-I00), funded by the Spanish
Ministry of Science and Innovation and the Spanish Research
Agency/10.13039/501100011033 and the European Union “NextGenerationEU”/PRTR. The
lead author is a member of TRADILEX, a research group recognised by the Catalan
Government (2021SGR00952).
References
Albert, B., & Hurst, R. (2012). Disability and a human rights approach to development.
Briefing paper. https://hpod.law.harvard.edu/pdf/human-rights-approach.pdf.
Arias-Badia, B., Bestard-Bou, J. J., & Hermosa-Ramírez, I. (2022). Designing, Making, and
Validating Accessible Products and Services: An Updated Account of Users’
Perspectives. Journal of Audiovisual Translation, 5(2), 57–75.
Braun, V., & V. Clarke. (2012). Thematic analysis. In H. Cooper (Ed.) APA Handbook of
Research Methods in Psychology, Vol. 2: Research Designs (pp. 57–71). APA books
Endmann, A., & Keßner, D. (2016). User Journey Mapping – A Method in User Experience
Design. i-com, 15(1), 105–110.
Fundación ONCE. (2022). Informe 7. Principales resultados. Observatorio sobre
discapacidad y mercado de trabajo en España.
https://www.odismet.es/sites/default/files/2022-12/INFORME%207.pdf
1464
Greco, G. M. (2019). Accessibility studies: Abuses, misuses and the method of poietic
design. In C. Stephanidis (Ed.), International conference on human-computer interaction
(pp. 15–27). Springer.
Greco, G. M., & Jankowska, A. (2019). Framing media accessibility quality. Journal of
Audiovisual Translation, 2(2), 1–10.
Huete, A. (2013). La exclusión de la población con discapacidad en España: Estudio
específico a partir de la Encuesta Social Europea. Revista Española de Discapacidad,
1(2), 7–24.
Orero, P. (2020). Born accessible: beyond raising awareness. Policy paper.
https://ddd.uab.cat/pub/estudis/2020/222130/Policy_Paper_final.pdf
Tueanrat, Y., Papagiannidis, S., & Alamanos, E. (2021). Going on a journey: A review of the
customer journey literature. Journal of Business Research, 125, 336–353.
United Nations (2015). Sustainable Development Goals. Quality education.
https://www.undp.org/sustainable-development-goals/quality-education
1465
About change: how institutionally aligning online pedagogy,

design and technology impacts higher education teachers
Universidad Andres Bello, Santiago, Chile.
Abstract
Emerging empirical evidence in e-learning research is revealing the
advantages of aligning learner-centred pedagogical practice with social
learning-based instructional designs and appropriate interactive technologies
to bring about systemic changes to learning online. This inquiry forms part of
a larger qualitative action research study. Our aim was to to determine the
impact of such a cross institutional change initiative on a group of instructors,
teaching in multi-disciplinarian fields in a higher education technical institute
in Chile. Findings from questionnaires, field notes and in-depth faculty
interviews revealed that evolvement towards teachers’ roles as mediators and
identities as empowered faciltators and community builders occurring over the
initial years of the initiative were accompanied by encouraging signs of the
development of 21st century skills and identities on the part of students. In our
post pandemic era, as online learning becomes more the norm, understanding
such effective pathways to excellence in digital spaces is paramount.
Keywords: Higher education; distance learning; online instructional design;
21st century pedagogy; social interactive technologies; identity.

How institutionally aligning pedagogy, design and technology online impacts teachers
1. Introduction
In the ‘new reality’ of the post pandemic era, more institutions than ever are incorporating
online learning as a permanent offering in their programming. How do they avoid the well-
documented pitfalls that befell many online learning contexts in the pivot to distance learning,
and indeed even before (Zhang et al., 2022)? In recent decades, promising theories have
emerged that seek to provide solutions for guaranteeing effective learning in online spaces.
An ongoing debate around ways of connecting these theories with ‘real’ practice has surfaced
as a result (Pange et al., 2011). More recently, scholars have argued that advances in creating
this connection can be achieved by aligning instructional designs (ID) and pedagogical
practices with contemporary learning theory and with the use of social media technologies in
online settings (Picciano, 2017; Margaryan et al., 2015). Teachers are vital in these change
processes. Knowing the voices of teachers from an institution that has managed to implement
such an aligned ID/pedagogical practice /technology model (Charbonneau-Gowdy et al.,
2022) could add valuable insight into the challenges teachers face in such change processes
and the kinds of viable support mechanisms that are required for the theory/practice
connection to be realized and effective online learning to be assured. Indeed, scholars
examining the various levels of teachers’ preparedness and abilities in online spaces and the
impact on learners (Vlachopoulos & Makri, 2021) are pointing to a paucity of research in this
area.
Our study was intended to respond to this call. From 2020 to 2022, the Chilean technical
institution we report on in the study launched a collaborative macro/meso/micro initiative to
address inconsistencies in the frameworks of their distance learning programs. Institutional
policies professed that programs were preparing students for the highly social workworlds in
which they were inhabiting – contexts that require networking, strong communication skills,
critical and innovative thinking in technical areas. Yet, in reality its programming emphasized
students working online in individual silos accumulating information and being evaluated on
their abilities to absorb and retain this knowledge. The change initiative, with the internal
support and drive of one of the researchers, herself a senior administrator, involved taking
concrete steps across all program areas to align online instructional designs, pedagogical
practices and uses of technology with current 21st century social-learning based theory
(Charbonneau-Gowdy, 2022). At the same time, teachers were supported in a process to
transform their practices from an emphasis on content and evaluation of information retained,
to one in which promoting learner agency, social learning through community building and
the engaged use of interactive technologies, were integral. The aim of this study was to
understand the experiences of teachers in this newly installed institutional trajectory. We
focused on academic staff employed in the institution (n=73 of 150) during the period from
2020 to 2023 and how they related to the changes both in teaching and student learning that
1468
they were being called upon to implement and promote. The following questions guided the
inquiry:
1. In what way did this group of teachers relate to the new direction in online learning
practices being established in the institution?
2. In their view, what changes, if any, did they find themselves making in their roles as
teachers over this period?
3. What changes, if any, did they witness in learners’identities and practices?
2. Theoretical Underpinnings
With the experience that many in education have had online during the recent COVID-19
pandemic, few would dispute that the use of technology leads unquestionably to effective
learning. For the last two decades, e-learning scholars have extolled the advantages and
virtues of technology-based learning (Castro et al., 2019). Yet, it became clear during the
pandemic that these claims presupposed certain contextual conditions – pedagogical, design-
based and technological. Gathering from reviews of scholarship documenting educational
experiences lived online during the pandemic, these conditions were in too many cases absent
(Vlachopoulos et al., 2022). As Adinda et al. (2020) have stressed, it is only in the
epistemological conformity of these conditions with contemporary learning theory and 21st
century goals that effective online learning can be achieved.
Essentially, these theories and goals a) place learners and their agency at the centre of
learning processes; b) are predicated on the understanding that learning is a complex social
interactive phenomenon; c) involve learners in community collaborating on co-constructing
knowledge based on their individual social contexts and experiences; and d) have important
implications for learners’ competencies and identities in a highly dynamic, technology-driven
society (Charbonneau-Gowdy et al., 2019). It is evident that the roles teachers play in
ensuring these kinds of scenarios for learners in their classrooms deviate substantially from
traditional ones. The challenge has been how to support teachers in making the transition to
these new roles.
In the last two decades, theorists in e-learning have proposed models to support teachers, and
other stakeholders, in recognizing the importance of such practices and conditions in digital
spaces (see Picciano, 2017). While each distinct, these theories share a common focus
relevant to our study – that is, to advance the concept of online learning environments as
spaces where teachers share a role with students in deliberating ideas, where social presence
is evidenced through teacher/student engagement in discussion boards, blogs, wikis, and
videoconferencing, where a teacher’s aim is to promote knowledge construction and the
development of abilities, including metacognitive ones, rather than the
dissemination/assimilation of information, where teachers foster the collaboration of learners
1469
to solve problems through discourse, and where their role is to be active facilitators and an
essential part of the learning community (Picciano, 2017).
Theories can propose frameworks for effective teaching and learning online, yet recent
studies show it is through solid instructional designs that closely reflect these contemporary
e-learning theories that ‘real’ pedagogical practice can be impacted. A case in point,
Margaryan et al. (2015) assessed the instructional design quality of 76 distance learning
programs, (MOOC’s). They found the majority “faired poorly” in aligning contemporary
theory-based design to online practice. The framework and key questions used in their
analysis (Table 1) allowed us to draw inferences about the roles and pedagogical practices of
teachers in our study.
Table 1: Framework for evaluating instructional designs used online (Margaryan et al., 2015).
Guiding Principle Description
Problem-centred Learners learn skills in the contexxt of real-world problems
Activation Learners activate their existing knowledge and skills for

developing new skills
Demonstrated Learners learn when exposed to “real” examples of new

skills to be learned rather than information
Application Learners have opportunities to apply their new skills to solve

problems
Integration Learners have opportunities to reflect on, discuss and defend

their new skills
Collective knowledge Learners contribute to collective knowledge
Collaboration Learners collaborate with other to build knowledge
Differentiation Learners have options according to their individual needs

Authentic resources Learners are put in real world situations
Feedback Learners are given regular feedback
3. Methodology
A qualitative participatory action research (PAR) methodology was chosen for the study for
several reasons: i) the underlining incentive involved this approach to respond to challenges
and the “potential in the approach to create meaningful and authentic change for those
involved” (Barcelona, 2020, p. 9); ii) the emphasis the approach places on drawing together
the macro, meso and micro levels of the institution, a critical component of sustained change
1470
initiatives in online teaching and learning (Charbonneau-Gowdy et al., 2019); iii) its
suitability for investigating complex human activity and for uncovering participant voices
(Denizen et al. 2005).
3.1. Research Design

Table 2 illustrates the various overlapping phases of the study and the tools used to collect
the data in each phase. These research phases coincided with the 3-pronged program changes
in Instructional Design, technology tools and pedagogical practices to which teachers were
in parallel adapting and in which they were playing a key role. Essentially, the institutional
changes included: a) adding synchronous videoconferencing sessions in all courses for
building learning communities; b) providing increased opportunities and resources for
student collaboration on both learning assignments/projects and assessment processes; c)
using group project-generated media as course content ; d) incorporating forums, padlets and
career-designated community sites into courses for students to exchange ideas and opinions;
e) creating separate institution-wide faculty and student online community sites.
3.2. Context and Participants

The study took place in Chile, considered an economically stable country one of only 2
members of OECD in South America, yet with deep socio-economic divisions and where a
majority of citizens have unequal access to quality education. Teachers in general are poorly
paid and lack employment stability resulting in a recent concerning national drop in interest
in the profession. The institution where the study took place employs on average 150-200
faculty members – 46% permanent 54% non permanenet, the latter contractual many of
whom must supplement their incomes by teaching elsewhere. Students are all working adults,
the majority from vulnerable disadvantaged backgrounds, both educational and economic.
The privately-owned technical institute, founded in 1985, has offered 100% online programs
since 2017, one of few prior to the pandemic to do so. Technical courses are offered in 15
career streams organized in 5 areas: administration, education, industrial, health and social.
Seventy-five teachers from a cross-section of these areas responded to the survey during
Phase 4 of the study, and 9 of this group opted to participate in the in-depth interviews.
Teachers’ voices (n=105) were aso represented in the various field notes collected over the
phases of the study (Table 2). Participants represent 3 groups of experience – those with
experience before and during changes, those employed at the outset of the change and those
who joined in the second year of the process.
1471
Table 2: Four phases of the study.
Phase Type of Data Description
Phase 1: Introducing Field notes: Collaborative discussions with

ideas for change: representative teachers, heads of
Minutes of meeting
September & October department and management on
2020 improvements to virtual classroom
resources (n= 20)
Phase 2: Bringing Field notes: Teacher awareness-raising actions with

awareness to social information delivery (articles, infographics,
Documents
learning in the online capsules)
modality: November
2020 to August 2021
Phase 3: Institutional Field notes (2020): Online collaborative capacity development

trainings in relation Minutes of meetings sessions with teachers (n = 60)
to social learning in
Field notes (2021): • Sessions to promote the
online modality:
development of collaborative
November 2020 to Meeting notes
capacities with teachers (n = 57)
December 2022
Digital correspondence (2021).
• Communication in digital
environments (n=37) (2021).
• Sessions to foster intentional
feedback and interaction with
students (n=59).
Field notes (2022): The Role of the Teacher as Coach: The

Learning Revolution (n=75) – workshop
Workshop observations
sessions on teaching strategies to generate
meaningful learning
Phase 4: Follow-up Field notes: • Minutes of meetings by area

and teacher support: January 2022 (5 areas).
Documents
March 2021 to • Minutes of teacher feedback on
January 2023 individual support sessions
Teacher surveys (January Teacher feedback surveys on ID-pedagogy-

2023) technology institutional changes (n=73)
Teacher interviews 2 recorded ZOOM 60-minute feedback

sessions - teachers reacting to their roles in
(January 2023)
the changed programs and their
observations of students (n=9)
1472
3.3. Data Collection and Analysis

Data was collected using the various qualitative tools listed in Table 2. The data was analyzed
using grounded qualitative coding methods involving a combined inductive-deductive
process (Miles et al., 2014). After establishing a conceptual framework, a series of iterative
steps were taken that included: i) inspecting the data sets for those data that could inform the
research questions; ii) multiple readings and considerations of the data sets; iii) condensing
and coding the data for key concepts and ideas that related to the theoretical framework and
literature review; iv) identifying and refining salient or common themes from coded data; v)
forming a conceptual framework that could be corroborated by findings.
4. Analysis and Discussion of Findings

Based on data from all data sets, the investment that the institution made not only in the
alignment of design/pedagogy/technology but also in a systematic ongoing and regular
support (Table 2) of faculty’s efforts to apply appropriate changes in their teaching, reaped
positive results - encouraging signs of development in teachers, and ultimately in their
students. Two themes emerged from our anaylsis of the various data sets: 1) the evolving
roles and identities of teachers and 2) the evolving roles and identities of students.
4.1. The evolving roles and identities of teachers

Change leaders in the institution were well aware that the sustained success of their 3-pronged
initiative to ensure that their distance programming was aligned with current learning theory
laid essentially in the hands of the faculty and their teaching practices. Evidence of a
trajectory on the part of a majority of teachers (see Table 3) in terms of their roles was
unpacked from the data analysis. Early data suggests that many saw their roles at the time as
content developers and information distributers, responsible for motivating students to absorb
knowledge and provide prescriptive answers to questioning. For the most part, faculty
members in these early stages considered and fostered learning as an autonomous process
and took little initiative to engage with students in virtual classes or online in forums. Over
the course of the PAR project, the ongoing support and guidance teachers received led to
considerable development of their teaching practices that reflected new strategies and roles
as evidenced in: i) increased focussing on learner-generated content, ii) promoting of group
work where students would co-construct new knowledge in their fields and think critically,
innovatively, metacognitively and iii) exploiting interactive tools to engage with their
students and to build enriched professional relationships and communities of practice. In
tandem with these changed roles, teachers assumed more confident and engaged identities as
online learning guides and facilitators. It is important to note that over the course of the 2
years, faculty’s active engagement online went from 69% in 2021 after a year of support
training to 90% in 2023.
1473
Table 3: Evolving Teacher Roles.
Initial Teacher Roles Evolving Teacher Roles Representative Excerpt

from Data
• Viewing one’s role as • Increased focus on “I feel that the

content expert not on promoting learner- accompaniment allows me to
teaching expertise generated content work with active
• Focus on content • Strategies that foster methodologies so that my
development learners’ co-construction of students can create, carry out
• Expository pedagogical knowledge and their and put into practice what
strategies learning they have learned" (Teacher
• Closed questioning in • Intentional interactions to interview January, 2023).
forums encourage critical thinking “There is evidence of fewer
• Emphasis on teaching not and debate expository classes and greater
on learners and learning • Role oriented to mediate strategies for students to
• Fostering of individual learning through active construct learning… For
student motivation and methodologies example, in the forums they
autonomous learning • Increased understanding of went from closed questions to
learning as a collaborative questions that favor
constructive process interaction and debate" (Field
• Recognition of virtual notes, teaching managers,
environments as spaces to 2022).
promote engaged learning
“Most of the teachers
generate active participation,
open questions in the forums,
providing feedback and
generating new challenges"
(Area Head Meeting,
Teaching Direction, October
2021).
4.2. The evolving roles and identities of students

The teachers’ evolving roles and identities seemed to coincide with transformations they
themselves witnessed in their students. An earlier report provides a full account of the change
process and the resulting transformations in students (Charbonneau-Gowdy et al. 2022). One
teacher sums up the early identities of many students in the following: "I have been able to
see that many students who arrive, who are older people and after a long time they decide to
study. So, they arrive first with that insecurity, low self-esteem and with the fear of failure...
they get frustrated and want to drop out" (Teacher interview, January, 2023). Such
observations of insecurity and low self-esteem among many incoming students were short
1474
lived. Teachers’ comments after a few months of observing students exposed to new teaching
practices revealed more empowered, engaged individuals convinced of their role as key
agents in their learning development and the value of social learning to that development - to
being critical, analytical and innovative thinkers, strong communicators and self-directed
learners. As one teacher observed: “At the beginning, very few students were connected, now
there are more and more, and everyone participates. I think this helps them in the professional
interactions they will have in their future, they are able to generate their own ideas and
reflections" (Teacher feedback questionnaire, January, 2023). It is this and many similar
observations from teachers that attest to the fact that the changes they made in their practices
were instrumental in empowering and engaging learners in activities that had 21st century
benefits for their skills in the immediate and long term.
5. Conclusion
It is not unusal for insitutions to revamp their programs in an effort to improve both
educational and financial results. Typically, curriculum reform initiatives begin as a macro-
level priority with minimal input from meso or micro level stakeholders, except in the reform
plan’s operationalization agenda. Ultimately, the result is often what scholars call “the
implantation gap.” (OECD, 2020, p. 4). In the case of the institution in this study, the process
was decidedly different. Besides initial and regular ongoing collaborative input from all
levels - macro level leaders, meso level administrators and micro level teachers and students,
a systematic and organized plan was installed to accompany and support teachers, and indeed
students, over the two-year rollout period. Based on the strong evidence we uncovered, this
long-term guided effort combined with the epistemological objective to align design,
pedagogy and technology resulted in clear signs of a transformation in teachers’ roles and
identities – from detached deliverers of information to confident and engaged collaborators
and guides in learning, and in students – from disadvantaged and impeded by low self-esteem
to thinking and behaving as 21st century learners, engaged and confident in ways that
reflected well for their futures. The significant drop in student attrition compared to pre-
changes, from 40% to 24.2% based on latest institutional figures, is further evidence of the
benefits of the approach. While we recognize the context specific nature of our study, we
believe these results add to the call for similar initiatives in diverse levels and contexts to add
to the empirical evidence for a way forward to educational changes in online learning.
References
Adinda, D. & Mohib, N. (2020). Teaching and instructional design approaches to enhance
students’ self-directed learning in Blended Learning environments. The Electronic
Journal of e-Learning, 18(2), pp. 162-174.
1475
Barcelona, A. (2020). An analytic hierarchy process for quality action researches in

education. International Journal of Evaluation and Research in Education, 9(3), 517-523
Castro, M. & Tunibay, G. (2019). A literature review: efficacy of online learning courses for
higher education institutions using meta-analysis, Education and Information
Technologies, 26, 1367-1385. doi: 10.1007/s10639-019-10027-z
Charbonneau-Gowdy, P., & Chavez, L. (2019). 3-M Model for uncovering the impact of
multi level identity issues on learners’ social interactive engagement online, Electronic
Journal of e-Learning, 17(2), 131-143.
Charbonneau-Gowdy, P., & Galdames, C. (2022). The impact on learners’ identities of
aligning pedagogy, design and technologies with theory in online courses. Proceedings
of the 21st European Conference on e-Learning (ECEL 2022), 51-60.
Denzin, N.K., and Lincoln, Y.S. (2005). Introduction: The discipline and practice of
qualitative research. In N.K. Denzin and Y.S. Lincoln (Eds.), The Sage handbook of
qualitative research (2011 ed.). Thousand Oaks, CA: Sage.
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open
Online Courses (MOOCs), Computers and Education, 80, 77-83.
Miles, M. B., Huberman, A. M., & Saldaña, J. (2014). Qualitative data analysis: A methods
sourcebook (3rd ed.). Sage.
OECD (2020). Curriculum reform: A literature review to support effective implementation.
Organization for Economic Cooperation and Development. Working Paper No. 239.
Pange, A., & Pange, J., (2011). Is e-Learning based on learning theories? A literature review.’
International Scholarly and Scientific Research and Innovation, 5(8), 932- 936.
Picciano, A. G. (2017). Theories and frameworks for Online Education: Seeking an
integrated model. Online Learning, 21(3), 166-190.
Vlachopoulus, D. & Makri, A. (2021). Quality teaching in online Higher Education: The
perspectives of 250 online tutors on technology and pedagogy. International Journal of
Emerging Technologies in Learning, DOI: 10.3991/ijet.v16i06.20173
Zhang, L., Carter Jr, R. A., Qian, X., Yang, S., Rujimora, J., & Wen, S. (2022). Academia's
responses to crisis: A bibliometric analysis of literature on online learning in higher
education during COVID‐19. British Journal of Educational Technology, 53(3), 620-646.
doi: 10.1111/bjet.13191
1476
Psychometric properties of the Spanish-language version of the

Agentic Engagement Scale (AES): a preliminary study
Estefania Guerrero1, Maite Barrios1, Chuen Ann Chai1, Juana Gómez-Benito1, Alba
Aza2, Georgina Guilera1
1
Department of Social Psychology and Quantitative Psychology, Universitat de Barcelona,
Spain, 2Department of Personality, Evaluation and Psychological Treatments, Universidad
de Salamanca, Spain.
Abstract
In 2011, Reeve and Tseng operationalized the measurement of the ‘agentic
engagement’ construct in five statements. Later, the preliminary scale was
empirically validated (Reeve, 2013) to assess the students' active contribution
during the education process. This work presents a preliminary study of the
psychometric properties of the Spanish version of the Agentic Engagement
Scale (AES). Using a sample of 194 participants, we analyzed item responses
distribution, factor structure, internal consistency, and evidence based on the
relationship between the AES and other measures of engagement and
personality traits. The results confirmed its one-factor structure and its
reliability in terms of internal consistency. In conclusion, the Spanish version
of the AES seems to be a promising scale to assess agentic engagement in
Spanish-speaking students.
Keywords: Agentic Engagement Scale; measurement; unidimensional;
Spanish.

Psychometric properties of the Spanish-language version of the Agentic Engagement Scale (AES)
1. Introduction
Academic engagement is a multidimensional construct formed by behavioral, emotional,
cognitive, and agentic aspects (Reeve & Tseng, 2011; Morcillo-Martínez, Infantes-Paniagua,
García-Notario, & Contreras-Jordán, 2021). Its importance in education has grown due to its
potential to promote learning and prevent school failure and dropout (Rodríguez-Fernández,
Ramos-Díaz, Madariaga, Arrivillaga, & Galende, 2016).
Reeve and Tseng (2011) defined the concept of agentic engagement as the edifying students’
input to the process of receiving instructions. In this way, agentic engagement is a unique
and proactive type of engagement that generates continuous dialectical transactions between
teacher and student (Reeve, 2013). In other words, it involves “the most frequent ways
students proactively and constructively engaged themselves in the flow of the day’s
instruction” such as by asking questions, expressing preferences, and providing suggestions
(Reeve & Tseng, 2011). In 2013, Reeve developed the Agentic Engagement Scale (AES) to
measure these interactions through five items that showed a unifactorial structure (α =.84).
Recent studies have already administered the AES in Spanish-speaking samples, but they
were limited to sample populations of secondary (Cuevas, Sánchez-Oliva, & Fernández-
Bustos, 2016) and primary school students (Morcillo-Martínez et al., 2021), which prevents
generalizing the findings to university students. Our investigation fills this gap in the
literature. Reeve and Tseng (2011) have found moderate correlations between the agentic
engagement construct and other types of engagement (e.g., behavioral engagement,
emotional engagement, and cognitive engagement). In addition, some personality traits such
as extraversion, agreeableness, and conscientiousness have shown a positive correlation with
general engagement (Qureshi, Wall, Humphries, & Bahrami, 2016). Finally, several
researchers have focused on the contribution of engagement to student outcomes such as
academic achievement (e.g., Kimbark, Peters, & Richardson, 2017; Lardy, Bressoux, & De
Clercq, 2022). Notably, Reeve (2013) has found the potential of agentic engagement to
explain independent variance in student achievement.
The aim of this study is to conduct a preliminary assessment of the psychometric properties
of the Spanish version of the AES in a sample of university students. Specifically, we analyze
item responses distribution, factor structure, internal consistency, and evidence based on the
relationship between the AES and other measures of engagement and personality traits.
2. Method
2.1. Participants
The sample consisted of 194 students (154 women [79.4%], 38 men [19.6%], and 2 non-
binaries [1.0%]). Their age ranged from 17 to 48 years (M = 20.6; SD = 3.83). At the time of
1478
Estefania Guerrero et al.
the study, 95.4% were enrolled in bachelor’s degrees, 3.6% were postgraduate students
(Master's degrees/Postgraduate courses), and 1.0% were enrolled in doctorate programs.
2.2. Measures
The original English-language version of the AES (Reeve, 2013) includes five items
measuring students’ dialectical and transactional participation in class (e.g., “During class, I
express my preferences and opinions”). Item responses are presented on a 7-point Likert
scale, ranging from 1 (completely disagree) to 7 (completely agree). Items were translated
and adapted into Spanish by means of parallel translation and reconciliation procedures.
The Spanish version of the Utrecht Work Engagement Scale for Students (UWES–9S;
Schaufeli & Bakker, 2004) was administered. This nine-item scale is the standard tool for
assessing work engagement characterized by vigor, dedication, and absorption (e.g., “When
I’m doing my work as a student, I feel bursting with energy”). Items are scored on a 7-point
Likert-type scale, ranging from 0 (never) to 6 (always). The UWES-9S has been shown to be
internally reliable (α = .84) and have a three-factorial structure that seems invariant across
gender and educational levels.
The Spanish version of the Mini International Personality Item Pool–Five-Factor Model–
Positively Worded (Mini-IPIP-PW; Martínez-Molina & Arias, 2018; original version by
Donnellan, Oswald, Baird, & Lucas, 2006) was applied. This instrument includes 20 items
to measure the Big Five domains: extraversion (e.g., “I am the life of the party”),
agreeableness (e.g., “I sympathize with others’ feelings”), conscientiousness (e.g., “I get
chores done right away.”), emotional stability (e.g., “I am relaxed most of the time.”), and
openness to experience (e.g., “I have a vivid imagination”). Items are rated on a 5-point Likert
scale, ranging from how well each statement described them from 1 (not at all) to 5
(completely) to evaluate each personality trait. There is sufficient support for the Spanish
Mini-IPIP-PW’s factorial validity and composite reliability (≥ .90). Its associations with
engagement suggest convincing correlations (Qureshi et al., 2016).
A sociodemographic questionnaire was included to get participants’ information about
gender, age, current studies, and the admission grade point average (GPA). Even though
average grades seem biased (De Clercq, Galand, Dupont, & Frenay, 2013), they are the most
common institutional information about student academic performance (Lardy et al., 2022).
2.3. Procedure
The data were collected between October and November 2022 using a convenience sample.
Undergraduate university students received the invitation to participate in the study on their
virtual campus. They received a link to an online questionnaire (hosted on the Qualtrics
software platform: https://www.qualtrics.com) that included the instruments listed above and
were asked to share the link with others on their social media (e.g., Twitter, Instagram). The
1479
inclusion criteria were (1) being a university student at the time of responding to the
questionnaire, and (2) having a proficient level of Spanish. Before starting the questionnaire,
all participants were informed about voluntary participation, anonymity, and confidentiality
in the study, and gave online informed consent.
The study followed the Declaration of Helsinki (World Medical Association, 2001) and was
approved by the bioethics commission of the University of Barcelona.
2.4. Analysis Strategy

The statistical data analysis programs Jamovi 2.3.21 and the lavaan package (Rosseel, 2012)
were used to analyze the psychometric properties of the AES.
Descriptive statistics of the AES were obtained at the item level, including mean, standard
deviation, and percentage of item endorsement, skewness, and kurtosis coefficients, as well
as corrected item-total correlations. We assessed the dimensionality of the AES through
confirmatory factor analysis (CFA). We used the weighted least squares means and variance
adjusted (WLSMV), which is suitable for ordinal items. The goodness of fit was assessed by
means of the Comparative Fit Index (CFI; ≥ .90/95 reflecting acceptable-to-excellent model
fit), and the Standardized Root Mean Square Residual (SRMR; ≤ .06 suggesting good model
fit) (Hu & Bentler, 1999).
We calculated the scale-level descriptive statistics (i.e., mean, standard deviation, skewness,
and kurtosis values) for the EAS, the UWES-9S, and the Mini-IPIP-PW. Furthermore, the
internal consistency of these scales was examined using Cronbach’s alpha (α) and
McDonald’s omega (𝜔) coefficients.
Validity evidence based on relationships between the AES and other measures was calculated
using Pearson correlations relating the AES total score with the UWES-S9, the admission
GPA and the Mini-IPIP-PW trait scores.
3. Results
3.1. Item-level Descriptive Statistics

Table 1 informs about the distribution of the AES item scores. All the items spanned the
entire range of the scale (i.e., 1-7), but participants less frequently responded to the
“completely agree (7)” option. Skewness and kurtosis coefficients suggested non-significant
departures from normality, even though items have a moderate positive asymmetry. All
corrected item-total correlations were high.
1480
Table 1. Item-Level Descriptive Statistics of the SP-AES.
PIE (%)
Item M SD SK K rjx
1 2 3 4 5 6 7
I1 2.97 1.76 23.7 29.4 11.9 10.3 12.4 10.3 2.1 0.60 -0.90 .70
I2 3.01 1.83 21.1 33.5 12.9 8.2 9.8 9.3 5.2 0.77 -0.62 .70
I3 3.29 1.84 22.2 20.6 13.9 10.8 17.5 11.9 3.1 0.29 -1.21 .57
I4 3.27 1.90 23.2 20.1 14.9 11.9 16.0 6.2 7.7 0.45 -0.95 .61
I5 2.77 1.72 29.4 28.4 10.3 10.3 12.9 6.7 2.1 0.72 -0.67 .67
Note. I = Item; M = Mean; SD = Standard deviation. PIE (%) = Percentage of Item Endorsement; each statement
is rated on a 7-point Likert-type scale. SK = Skewness; K = Kurtosis. rjx = Corrected item-total correlation.
3.2. Factor Structure

The one-factor model fits the data well (CFI = .972, SRMR = .048). Figure 1 shows the path
diagram of the one-factor model of the AES. All factor loadings were high and statistically
significant.
Figure 1. Path diagram of the one-factor model of the AES.
3.3. Scale-Level Descriptive Statistics and Internal Consistency

Table 2 shows the scale-level descriptive statistics (i.e., mean, standard deviation, skewness,
and kurtosis values) and the internal consistency of the AES total score. The internal
consistency was evaluated using Cronbach’s alpha (α) and McDonald’s omega (𝜔)
coefficients, both suggesting that scores provided by the AES are reliable.
1481
Table 2. Descriptive statistics, internal consistency coefficients, and correlations of the AES with
engagement, academic achievement, and personality traits.
M SD SK K α ω r
AES 15.3 7.11 0.51 -0.27 .84 .86 -
UWES-9S 31.7 10.8 -0.11 -0.64 .92 .92 .39**
Admission GPA 8.02 1.00 -0.23 0.16 - - .02
Mini-IPIP-PW
Extraversion 10.3 3.63 0.21 -0.69 .76 .76 .22*
Agreeableness 15.8 3.15 -0.73 0.16 .83 .83 .11
Conscientiousness 12.5 3.62 -0.20 -0.53 .82 .83 .10
Emotional 9.65 3.42 0.26 -0.69 .74 .75 .12

stability
Openness to 13.4 3.80 -0.21 -0.40 .83 .83 .25**

experience
Note. M = Mean; SD = Standard Deviation; SK = Skewness; K = Kurtosis; α = Cronbach’s Alpha; ω =
McDonalds’ Omega; r = Pearson Correlation. * p < .01; ** p < .001 (two-tailed).
3.4. Validity based on relations with other variables

Descriptive statistics and internal consistency coefficients of the other measures of
engagement, academic achievement, and personality traits are shown in Table 2, also their
bivariate correlations with the AES. The total score of the AES was moderately associated
with students' work engagement (r = .39). Contrary to what we expected, the AES was not
related to the admission GPA. According to the relation of AES with personality traits, the
extraversion and openness to experience factors were significantly related to AES.
4. Discussion
This study shows the initial psychometric evidence of the AES in a Spanish university
students sample. The item-level descriptive statistics scores of the Spanish version of the
AES yielded similar distributional properties to those observed in the original scale (Reeve,
2013). Furthermore, the internal consistency in our results aligns with the original scale and
the Spanish versions of the AES for secondary school students (Cuevas et al., 2016) and
primary school students (Morcillo-Martínez et al., 2021).
Moreover, our investigation revealed that the one-factor structure of the AES (Reeve, 2013)
showed a good model fit and high factor loadings of the five items. These results are similar
1482
to other validity studies of the AES (Cuevas et al., 2016; Morcillo-Martínez et al., 2021) that
reinforce our confidence in the future replication of this structure in larger sample sizes.
Our findings support the correlation between the AES and students' work engagement in line
with the study of Reeve and Tseng (2011), which highlighted the correlation between the
AES and other engagement types. On the other hand, we observe a positive relationship
between AES and extraversion similar to Qureshi et al. (2016). However, we found low
correlations between AES and agreeableness or conscientiousness. In the future, more
research may be performed to identify early predisposition to engagement.
Finally, the absence of a correlation between the AES total score and academic achievement,
as measured by admission GPA, may be attributed to the possibility of grade average bias,
as suggested by De Clercq et al., (2013). Additionally, it is worth noting that the admission
GPA represents past information, specifically the mark used for university admission, rather
than current academic performance. As other authors have found (Lardy et al., 2022), it
would be expected that the relationship between engagement and current academic
achievement would be higher.
5. Conclusion
The Spanish version of the AES appears to be a promising instrument for measuring agentic
engagement. It corroborates good psychometric properties: adequate factor structure,
convincing internal consistency, and validity evidence based on relations with students' work
engagement and extraversion personality traits. These findings have theoretical and applied
implications in education to promote programs and interventions in this field.
References
Cuevas, R., Sánchez-Oliva, D., & Fernández-Bustos, J. G. (2016). Adaptación y validación
de la escala de compromiso agéntico al contexto educativo español. Revista Mexicana de
Psicología, 33(2), 135-142.
De Clercq, M., Galand, B., Dupont, S., & Frenay, M. (2013). Achievement among first-year
university students: an integrated and contextualized approach. European Journal of
Psychology of Education, 28(3), 641–662. doi: 10.1007/s10212-012-0133-6
Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The Mini-IPIP scales:
Tiny-yet-effective measures of the Big Five factors of personality. Psychological
Assessment, 18(2), 192–203. doi:10.1037/1040-3590.18.2.192
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure
analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A
Multidisciplinary Journal, 6(1), 1–55. doi:10.1080/10705519909540118
Kimbark, K., Peters, M., & Richardson, T. (2017). Effectiveness of the student success course
on persistence, retention, academic achievement, and student engagement. Community
1483
College Journal of Research and Practice, 41(2), 124–138.

doi:10.1080/10668926.2016.1166352
Lardy, L., Bressoux, P. P., & De Clercq, M. (2022). Achievement of first-year students at the
university: a multilevel analysis of the role of background diversity and student
engagement. European Journal of Psychology of Education, 37(3), 949–969.
doi:10.1007/S10212-021-00570-0
Martínez-Molina, A., & Arias, V. B. (2018). Balanced and positively worded personality
short-forms: Mini-IPIP validity and cross-cultural invariance. PeerJ, 6(9).
doi:10.7717/PEERJ.5542
Morcillo-Martínez, A., Infantes-Paniagua, Á., García-Notario, A., & Contreras-Jordán, O. R.
(2021). Validación de la versión española de la escala de compromiso académico para
educación primaria. RELIEVE - Revista Electrónica de Investigación y Evaluación
Educativa, 27(2), 1–20. doi:10.30827/RELIEVE.V27I2.22769
Qureshi, A., Wall, H., Humphries, J., & Bahrami Balani, A. (2016). Can personality traits
modulate student engagement with learning and their attitude to employability? Learning
and Individual Differences, 51, 349–358. doi:10.1016/J.LINDIF.2016.08.026
Reeve, J. (2013). How students create motivationally supportive learning environments for
themselves: The concept of agentic engagement. Journal of Educational Psychology,
105(3), 579–595. doi:10.1037/A0032690
Reeve, J., & Tseng, C. M. (2011). Agency as a fourth aspect of students’ engagement during
learning activities. Contemporary Educational Psychology, 36(4), 257–267.
doi:10.1016/J.CEDPSYCH.2011.05.002
Rodríguez-Fernández, A., Ramos-Díaz E., Madariaga J. M., Arrivillaga A., & Galende N.
(2016). Steps in the construction and verification of an explanatory model of psychosocial
adjustment. European Journal of Education and Psychology, 9(1), 20–28.
doi:10.1016/j.ejeps.2015.11.002
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of
Statistical Software, 48(2), 1–36. doi:10.18637/jss.v048.i02.
Schaufeli, W., & Bakker, A. (2004). UWES Utrecht Work Engagement Scale - Preliminary
manual. Occupational Health Psychology Unit, Utrecht University. Utrecht: Utrecht
University.
The jamovi project (2020). Jamovi (Version 2.3.21) [Computer Software]. Retrieved from
https://www.jamovi.org.
World Medical Association. (2001). Declaration of Helsinki ethical principles for medical
research involving human subjects. Bulletin of the World Health Organization, 79(4),
373–374. https://doi.org/https://apps.who.int/iris/handle/10665/268312
1484
Education 4.0: MiReBooks - Development and Usage of Mixed

Reality for Teaching Mining in Higher Education
Resources Innovation Center, Montanuniversität Leoben, Peter-Tunner-Straße 15, 8700
Leoben, Austria.
Abstract
In order to meet workplace demands in relation to Industry 4.0, the MiReBooks
project was established in 2019. The main focus of this paper is to present the
project’s outcomes in the field of Education 4.0 whereby, Mixed Reality (MR)
Mining Engineering educational material was developed for teachers, this in
order to integrate MR in classroom instruction with the aim to enhance student
understanding and application of knowledge to increase their competence in
the specific field of the subject. An account of a MR application in a mining
lecture is presented, together with student feedback, providing grounds to
ascertain that the integration of such technology can improve student learning
in the achievement of lesson/course Learning Outcomes.
Keywords: Mixed Reality; Mining; Education 4.0; Teaching & Learning;
Engineering.

Education 4.0: MiReBooks - Development and Usage of Mixed Reality for Teaching Mining
1. Introduction
The economic age of rapidly advancing technology, digital transformation and workforce
demands, set exceptional challenges on educational systems to prepare graduates for their
future careers (Teo et al. 2021). The advancement of Industry 4.0 requires educational
institutes to transform teaching and learning approaches and to conceptualise new learning
content to meet the demands of industry now and in the future. This approach further requires
the training of transversal skills which permit learners to not only acquire technical
knowledge but also with opportunities to train soft skills and promote an adaptive mindset
and readiness to flexibly engage with new methods to deal with the intricacies of
technological advancements and uncertainty (Ralph et al. 2022).
Ergo, Higher Education Institutes (HEIs) play a pivotal role in contributing to industry
required competences by ensuring the workforce rediness of their graduates. Within the
discipline of engineering education, with a particular focus in the field of mining, traditional
based teaching and learning techniques prevail. Frontal based theory lectures are
supplemented, when possible, with onsite mine visits, which limits the student’s possibility
to interact with the mining operations at a meaningful level.
In 2019, the ‘Mixed Reality Handbooks for Mining Education’ project was launched
(MiReBooks). This being one year after the release of the Digital Education Action Plan
2018-2020, from the European Commission. This action plan laid the foundation for
innovative education, setting out the skills, knowledge and attitudes citizens need for life,
including digital competence (Eu Comm. 2018).
The MiReBooks was a successfully applied for project funded by the European Institute of
Innovation & Technology (EIT), within their Raw Materials Knowledge Innovation
Community. In 2019, the project commenced with the aim to create a new digital teaching
and learning experience which would enhance and optimise traditional paper-based
education by enriching it with virtual and augmented reality-based experience.
The MiReBooks project comprised a consortium of core HEIs, Industry entities and research
institutes and was led by the Montanuniversität Leoben in Austria. The other core HEI
partners were, Luleå University of Technology (LTU), Rheinisch-Westfaelische Technische
Hochschule Aachen (RWTH) in Germany, Tallinn University of Technology in Estonia and
the Technische Universität Bergakadamie Freiberg (TUBAF) in Germany, Technische
Universität Graz (TUGraz) in Austria, Università di Trento (University of Trento). Also,
research institutes and companies comprised of core and associated partners connected to the
project were, KGHM Cuprum sp. z o.o. Centrum Badawczo-Rozwojowe (KGHM Cuprum
Ltd. Research & Development Centre) in Poland, Teknologian tutkimuskeskus VTT
(Technical Research Centre of Finland Ltd. VTT) in Finland, Luossavaara-Kiirunavaara AB,
LKAB in Sweden and Epiroc AB also in Sweden (MiReBooks 2019).
1486
The main objectives of the MiReBooks project involved the following aspects:
- The development of an instructional ‘Didactical Handbook’ on using/integrating
Mixed Reality in lectures
- The development of a Virtual Reality lecturing tool
- The development of Mixed Reality integrated Mining book series
This paper will serve to demonstrate the abovementioned outcomes of the MiReBooks
project and provide insight into current/future activities. It will also present an account of the
didactical usage of ‘Mixed Reality’ in a Mining lecture, as taught at the Montanuniversität
Leoben, Austria within the academic year 2022.
2. Project Objective and Scope

The overall scope of the project was to develop a new digital learning experience that would
innovate the teaching, learning and subsequently applying, of mining engineering education
at a tertiary level. Combining state-of-the-art pedagogical methods with digitalisation of
teaching materials, students are able to maximise and take advantage of new ways of
classroom interaction that is more suitable for the needs of their generation as digital natives.
The tools and material outcomes from the project are correspondingly designed in such a way
that it is also attractive for professional industry application targeted at life-long learners to
bring existing employee’s competences in line with the latest developments.
2.1. MiReBooks – Didactical Handbook

One of education’s most important tools is pedagogy and subsequently, the didactics applied
for teaching/learning materials (OECD 2019/20). One of the pinnacle outcomes from the
project was to develop a didactical handbook for educators to understand and aid them in the
use of mixed reality (MR) in lessons. The book was developed and designed by pedagogues
in the English language and split into 4 chapters.
Chapter one provides an introduction to the applications of Mixed Reality in connection with
Raw Materials in lessons, Education 4.0. It provides deeper understanding and examples of
what, and how, MR aspects can benefit teaching and learning. Chapter two provides input on
Teaching 4.0. It specifically addresses didactics in education through the understanding and
application of Bloom’s Taxonomy to derive appropriate Learning Outcomes for lessons
integrating MR technology. It explains in detail Gamification and other methods for teaching
and how MR technologies can be applied in Raw Materials related education. Chapter three
provides a deeper account/understanding of teaching with Mixed Reality and the various
means that can be applied e.g. VR headsets, handheld devices e.g. mobile phone and possible
MR content. Finally, chapter four provides the reader with the practical set-up of integrating
1487
MR elements in domain teaching. It offers the reader a MR decision matrix to help guide the
lesson development to the appropriate MR tool for the purpose of instruction.
The MiReBooks Didactical Handbook is provided free of charge for all educators who would
like to apply these technologies in their lessons with the goal to improve and aid the
understanding and application of aspects in relation to e.g. mining engineering education.
2.2. MiReBooks – Virtual Reality Lecturing Tool

MiReBooks VR lecturing tool introduces the use of 3d models and 360° videos and images
to the lectures in the classroom as well as online for remote teaching. The MR media
contained in the book can be accessed for use in lectures and to enhance comprehension of
complex topics. The tool takes over the role of electronic presentation software e.g.
PowerPoint, including VR media, while allowing lecturer/student interaction. The VR
lecturing tool requires the use of VR headsets, currently Oculus Quest 2 is being used for the
students and a tablet, or laptop, for the lecturer to convey the lesson. The lecturer can play
360° videos and images and/or 3D models as part of the presentation. When in a new 360°
exciting environment it is easy for the students to get distracted and lose focus on the teaching
activity and miss the learning outcomes. To avoid focus diversion, the lecturer has some
features available to keep the student's focus. These features allow real-time (RT) drawing
on the 360° video/image or 3D model and field of view blocking. The lecturer has also an
overview of the connected student's status, to detect any connection issues and even loss of
focus from students. The students also have access to features, such as question placement
and 3D model manipulation. All these features ensure enough control over the VR
environment so that the learning goals can be achieved more efficiently than with traditional
methods.
2.3. MiReBooks – Mixed Reality (MR) Integrated Mining Book Series

One other important outcome from the project is the development of a mining book series
with integrated MR material. The development of this is a cross collaboration of several
Universities in Europe. The overall focus is on sustainability in mining and a total of 7 books
are in the series. These have been written by leading experts in the topic field from the
Universities’ and each book covers the following topics: 1. Sustainability and Mine Planning
2. Safety in Mining 3. Rock Fragmentation 4. Surface Mining 5. Underground Mining 6.
Rocks Mechanics and lastly, 7. Mine Closure.
Each mining handbook includes QR codes (Links) which trigger extra learning content. This
enhances comprehension of complex topics and provides more material to extend the
competence of the students. The platform is based on a webpage. The content linked to the
QR codes have different natures:
1488
1. Study cases: are for the student to put into practice what they learn in the books. So,
competence on the topic can be tested by solving problems of performing
calculations.
2. AR 3D models: can be explored on the platform and also as AR with the help of a
smartphone or tablet. The platform has an AR 3D model delivering system that
bridges the need for an APP. From the platform, the camera from the student’s
smartphone is used to show the 3D modes as AR. 3D visualization helps to
understand the spatial distribution of operations and promotes self-exploration as
well as technical discussions in the classroom.
3. 360° videos and photos: can be explored from the browser or with the help of
smartphones/tablets and VR headsets (e.g. Oculus Quest 2). By using VR headsets,
the student is immersed in a VR environment which helps with understanding the
scale and distribution of operations.
4. Links to more content: VR and AR are not always the best tool to enhance
comprehension of every topic. Conventional videos, as well as content on other web
pages, are shared to promote the students further reading and researching.
While the platform is conceived as a tool to share multimedia content triggered by the books,
users must find value in it even if they do not own any of the MiReBooks handbooks. All
content shared on the platform will include enough explanation so that it can be understood
and useful for all platform users.
3. Learning Pathway
The following section will provide insight into a university lecture, the use of virtual reality
to support the understanding and application of knowledge in the classroom and the
continuous feedback from students which allows insight for improving the user experience
from both the teaching and technological sides.
3.1. Continuous Mining Methods – Surface Mining (held on November 29, 2022)
A total of 15 students studying mining engineering attended a two-hour lecture about surface
mining. The didactical set-up was the integration of a MR environment whereby, each student
was provided with a VR headset and given instructions on its functionality. This method is
used specifically to aid the teaching and learning knowledge transfer. The learning outcomes
for the lesson where: upon completion of the lecture the student will be able to 1. Understand
the basics of continuous surface mining methods and 2. Explain the different surface mining
operations. During the lesson the students used MiReBooks programmed VR headsets and
were transported virtually onsite to two running open coal pit mines located in Germany
(Hambach and Jänschwalde). This allows the student full access to all operational aspects of
the mine that otherwise would not be accessible or difficult to arrange as a site visit. Ergo,
1489
the knowledge transfer goes hand-in-hand with the practical insight into the material being
taught and the lecturer is able to explain in detail what they are seeing, how it works and the
different surface mining methods that are applied.
Directly after the lesson, both the students and the lecturer are provided with an extensive
online feedback form which they immediately complete. The questions asked provide
answers regarding the software usability and lecture didactics, which also includes an
assessment of the learning outcomes. This feedback loop is indispensable in providing input
to develop the software further and to create a better user experience. In each iteration of the
software, bugs are corrected, and also, new features are built in to directly address the user's
feedback.
3.2. Lesson Feedback, ‘Continuous Mining Methods – Surface Mining’

A total of 15 students attended the lecture, whereby only 10 fully completed the feedback
form and could therefore be used. The gender division of the students were 4 females and 6
males. The feedback form is divided into several sections: general information, user ability
(VR), system usability, learning outcomes and lastly, open feedback questions to help
improve the experience.
The feedback (self-assessment) provided by the students in direct relation to the Learning
Outcomes of the lesson, was addressed by the following two questions and a 5-point Likert
scale was used whereby, 1 is strongly agree and 5 is strongly disagree. A total of two
questions in this section are provided to the students: 1. ‘Post lesson, I believe my competence
of the course/lecture subject matter has increased’ and 2. ‘The use of VR helped me to better
understand and apply the technical knowledge of the course/lecture.
The results of the 10 completed forms will now be presented in relation to the feedback form
questions on the lesson LO’s: LO 1. Upon completion of the lecture the student will be able
to: Understand the basics of continuous surface mining methods. LO 2. Upon completion of
the lecture the student will be able to: Explain the different surface mining operations.
Graph 1. Student Feedback Responses.
1490
In relation to the question: ‘Post lesson, I believe my competence of the course/lecture subject
matter has increased’, 70%, respectively 7 of the students indicated that they strongly agreed
that their competence(s) of the lecture in relation to surface mining methods had increased,
while 30%, respectively 3 agreed with the statement.
Graph 2. Student Feedback Responses.
In relation to the question: ‘The use of VR helped me to better understand and apply the
technical knowledge of the course/lecture’, 90%, respectively 9 of the students indicated that
they strongly agreed that the use of VR helped then to better understand and apply the
technical knowledge of the lesson, while 10%, respectively 1 agreed with the statement.
The above results give reason to believe that integrating MR within domain curriculum can
improve and enhance the students’ ability to not only comprehend the lesson material but
also in applying it. The Learning Outcomes are reached and tested immediately by the teacher
who is not only relaying information but has the possibility to demonstrate the application of
this within a classroom environment.
4. Conclusion/Outlook
Digitalisation in education must be part of the future learning pathway to ensure a competent
workforce for the mining engineers of tomorrow in Europe. The MiReBooks project has
changed the way traditional teaching and learning in mining education takes place. Through
the continuous feedback loop the various applications of MR, the platform and the book
series are constantly being improved. The feedback from students and teachers in direct
relation to the course/lesson Learning Outcomes demonstrate the transfer of knowledge and
competence of the participants can be enhanced. As Education 4.0 integrates technologies it
also addresses not only the necessary hard skills taught as part of the lessons but also the soft
skills needed within the field of digitalisation necessary for Industry 4.0.
1491
Acknowledgement
The MiReBooks project (2019-2021) and activities had received funding from the European
Institute of Innovation and Technology (EIT), a body of the European Union, under the
Horizon 2020, the EU Framework Programme for Research and Innovation.
References
Benjamin James Ralph, Manuel Woschank, Corina Pacher & Mariaelena Murphy. (2022).
Evidence-based redesign of engineering education lectures: theoretical framework and
preliminary empirical evidence. European Journal of Engineering Education, 47(4),636-
663. doi: 10.1080/03043797.2021.2025341
European Commission. (2018). Communication from the Commission to the European
Parliament, The Council, The European Economic and Social Committee and the
Committee of the Regions on the Digital Action Plan. url: https://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=CELEX:52018DC0022&from=EN
MiReBooks. (2019). Project Official Website. Retrieved January 19, 2023, from
https//mirebooks.com
Organisation for Economic Co-operation and Development (OECD). (2019/20). The
relevance of general pedagogical knowledge for successful teaching: Systematic review
and meta-analysis of the international evidence from primary to tertiary education. url:
https://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=EDU/WKP(
2019)20&docLanguage=En
Teo, T., S. Unwin, R. Scherer, and V. Gerdiner. (2021). Initial Teacher Training for Twenty-
First Century Skills in the Fourth Industrial Revolution (IR 4.0): A Scoping Review.
Computers & Education 170: 104223. doi: 10.1016/j.compedu.2021.104223
1492

Conference

Uploaded by

Copyright:

Available Formats

Conference

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Conference

Uploaded by

Copyright:

Available Formats

Congress UPV

9th International Conference on Higher Education Advances (HEAd’23)

Cover design by Gaia Leandri

9th International Conference on Higher Education Advances (HEAd’23)

This work is licensed under a Creative Commons License CC BY-NC-SA 4.0

Scientific committee chair

Local arrangements chair

3. Sponsors and Supporters

4. Scientific Program Committee

M. Ángeles Alcaide, University of Valencia, Spain

Diego Víctor de Mingo-López, Universitat Jaume I, Spain

Libor Juhaňák, Masaryk university, Czech Republic

Cesar Ortega-Sanchez, Curtin University, Australia

Fátima Suleman, ISCTE – Instituto Universitário de Lisboa, Portugal

Special Needs Pre-Service Teachers Digital Competencies: An Exploratory Study at

Nurturing the human connection: Increasing student engagement and personal

Interdisciplinary Week in Game Design: A Learning Experience ..................................... 263

Non-formal Faculty Development. Conceptual considerations and implementation in

The explicit teaching of vocabulary in French L2: theoretical models, teaching

Threshold Tests as a way to encourage long-term, self-regulated learners in

Peer coaching in a leadership development program: The role of developmental

Experiencing Third Spaces in between University and Society: Transdisciplinary

Enhancing Language Self-Efficacy of EFL University Students through Experiential

Multiple-Value Governance in (German) Higher Education: a New Paradigm? ............. 1297

Ideas on Digitally Supported Individualization of Teaching and Learning for

Examining university student satisfaction and barriers to taking

This work is licensed under a Creative Commons License CC BY-NC-SA 4.0

2.1. Development of survey instruments

2.2. Data collection

Table 1. Demographics of survey respondents

Background All With online exams experience

2.3. Data analysis

3.1. Satisfaction with online exams

3.2. Barriers to taking online exams

confidence in setting up and using technologies (13%), access to an internet connection

Almossa, S. Y. (2021). University students’ perspectives toward learning and assessment

Measuring student development using points

This work is licensed under a Creative Commons License CC BY-NC-SA 4.0

Involvement includes traditional academic activities such as studying - learning to do - as

3. Measuring student development

4. A field experiment in a living lab

Year 1 Year 2 Year 3 Year 4 Year 5

Figure 1: Development activity

Table 2 summarizes DD scores of 623 graduates participating in 6,474 activities totaling

University sustainability assessment and reporting: preliminary

This work is licensed under a Creative Commons License CC BY-NC-SA 4.0

4. Results and Discussion

Example of a University of University of University of University of

Influence on Relevance for Relevance for Relevance for Relevance for

Lozano, R. (2006). A tool for a Graphical Assessment of Sustainability in Universities

Creating the “examination kiosk”: blended assessment to foster

This work is licensed under a Creative Commons License CC BY-NC-SA 4.0

2. Comparison of Portfolio Work and Portfolio Examinations

Aspect Portfolio Work Portfolio Examination

3. Didactic Development of Portfolio Examinations

4. Creating the “Examination Kiosk”

Figure 1. “Examination Kiosk”

Objectives: The examination kiosk is used as a form of examination to check the

Pro-f-quiz: increasing the PROductivity of feedback through

This work is licensed under a Creative Commons License CC BY-NC-SA 4.0

$&% '!&% %%&% "&# !%)% &"!'!&*%&''+"()+"(% ' " #'!)&%

" " *$ '####%##% #) %&%

*$ - )! %"''!#%&"!'!#%)+