Medical Physics International

MEDICAL PHYSICS INTERNATIONAL Journal, vol.6, No.2, 2018 THE VERTTM PHYSICS ENVIRONMENT FOR TEACHING RADIOTHERAPY PHYSICS CONCEPTS – UPDATE OF FOUR YEARS’ EXPERIENCE M C Kirby, PhD Directorate of Radiotherapy, School of Health Sciences, University of Liverpool, Liverpool, UK Abstract—Radiotherapy Physics is a challenging subject – especially when teaching across disciplines. The primary role for therapy radiography students is entirely patient focused requiring clinical, empathetic, technical and other skills for successful treatment. Finding ways, therefore, of teaching fundamental Physics concepts, in a new and engaging manner, helps establish deep learning for enhancing excellent clinical practice and solid interprofessional working for advancing cancer treatments. Using a Virtual Environment for Radiotherapy (e.g. VERTTM) as a specific form of eLearning is one way we’ve found that helps students engage better in learning and understanding key Radiotherapy Physics principles, in an interactive and dynamic manner, with all the benefits of the environment. We have successfully used VERTTM Physics, a specialized module within VERTTM, for over four years now at the University of Liverpool in both 2D and 3D immersive modes to teach fundamental concepts to undergraduate and postgraduate radiotherapy students. First formats used small group sessions blending lecture and practical use for teaching concepts like consequences of FSD set-up error; beam quality indices and the derivation of field size factors. For each subject area, workbooks were provided with subgroups performing, alternately, calculations and virtual measurements using VERTTM Physics. Evaluation and feedback were excellent, especially regarding the small group methods; the results of which have been described previously. This paper details the rationale and results of the evolution of this format over four academic years – now bringing in interactive demonstrations of the measurement and characteristics of PDD Curves. Students predict photon curves and compare them with VERTTM Physics measurements, and consider electron and proton modalities too, with peer-to-peer and expert tuition. Evaluations have again been very positive, with students appreciating the small groups and focused tuition, and showing potential improvement in assessment results since PDD characteristics have been taught supplemented by our VERTTM Physics workshop sessions. delivery. The intention is an informed viewpoint and understanding of concepts to better aid clinical work and the patient experience through the radiotherapy pathway. Perhaps for this reason, blended learning and teaching methods bring real, positive results – by integrating more creative teaching and learning methods with the traditional, didactic ones in order to aid engagement and promote necessary deeper learning [1, 2]. These are continually our aims with both our undergraduate and postgraduate therapeutic radiography students at the University of Liverpool, for most of the modules on the radiotherapy programmes; complementing teaching methods by the use of real (clinical) world technologies which can simulate the full clinical world extremely well [3, 4]. The Virtual Environment for Radiotherapy Training (VERTTM) (www.vertual.co.uk) is one such environment we’ve found which, as a virtual one, brings a creative edge to teaching, enabling students to learn in an extremely engaging and interactive manner, using a number of different eLearning components and styles, whilst at the same time providing extra resources to complement the highly pressured real clinical equipment; with safety and freedom of risk at the centre of its design [410] VERTTM has been a key component for our institution and many others both nationally and internationally for many years [3, 11, 12]. Its origins and original design features are well covered in the literature [5-8]. Its use for student radiographer training has been well noted, with recent extensions reported for students of radiotherapy physics too [13-20]. Staff training and competency is part of its use [11, 12, 21-23], as is also as a method for helping patients themselves understand the treatment they are about to undergo [23-25]. Our own use for teaching radiotherapy physics concepts has been documented [16-18, 26, 27], but VERTTM Physics has been found to be highly adaptable and our methods have evolved over the last four academic years. This paper examines that evolution – the changes in and the rationale behind their development; and the continuing results obtained in terms of feedback and response from our students and, most recently, in terms of assessment marks – as an indication of the students ability to demonstrate the depth of their learning and understanding in concepts which are extremely important for their clinical work. Here is described the nature of our use of VERTTM Physics, beyond its design for clinical simulation [15], to one which still simulates the radiotherapy physics environment; but always with a focus on learning to aid clinical work and patient Keywords— Simulation, radiotherapy physics, radiographers, eLearning, VR. I. INTRODUCTION Teaching radiotherapy physics and technology to student therapeutic radiographers (radiation therapists) is challenging for the student – not necessarily because of the level of complexity required for their ultimate clinical task, but because of the range of skills which the radiographer needs to have for effective and safe clinical treatment 247 MEDICAL PHYSICS INTERNATIONAL Journal, vol.6, No.2, 2018 benefit, in a highly interactive, engaging and kinesthetic manner. The work reported here has continued to be run with second year undergraduates and both first and second year postgraduate radiotherapy students for the last four academic years. The main subject matter extension for the latter two years has been aimed at improving knowledge and understanding of radiation beam characteristics – for different energies and parameters, and comparisons with different modalities of electrons and protons. clinical placements – to further expand their experience [1618]. The group was split up into two, so that one smaller group (of about 3 or 4) could perform the virtual experiments using VERTTM Physics, whilst the other group worked together to perform the calculations associated with each experiment. Three practical experiments were devised and used; these were (a) an experiment using the ion chamber block to investigate the dosimetric effects on the patient of incorrect SSD set-up (whilst the calculation group used the inverse square law to predict the dosimetric error); (b) an experiment to simulate measuring quality indices for different photon beam energies using a fixed SSD and two depths in the ion chamber block (Whilst the calculation group considered how to calculate the quality index, compare it with a baseline value and determine whether it was within a 1% tolerance for routine quality control); (c) an experiment to measure the fieldsize factors, using a fixed FSD and depth for the ion chamber block and different fieldsizes - whilst the calculation group considered how the fieldsize factor data would be derived from each of the data points, normalized to a factor of unity for the reference field size of 10 x 10 cm. In every case, experiments and calculations were performed for each available photon energy (6 and 15 MV), with the two smaller groups swapping roles (calculation and experimental) between each energy [16-18, 26, 27]. II. MATERIALS AND METHODS A. Methods A.1 First Iteration of Teaching Methods (2014). The first iteration of the rationale and teaching methods using VERTTM Physics have been communicated previously [1618]. For the purpose of illustrating the evolution of the methods and continuity, they are described briefly here. Year groups (approx. 20 – 30 in number) were divided into smaller groups of approx. 6 – 10 students for each session. This was done to make feasible a more interactive and kinesthetic approach for all of the students. Because of the timing of the teaching of theoretical concepts and this practical approach within the semester (the theoretical concepts having been taught and discussed some weeks before), a 2 hour slot was devised, with the first hour being dedicated to a formal, refresher lecture on the appropriate radiotherapy Physics concepts which would be used in the practical session with ‘virtual’ Linac experiments. The recap highlighted the concepts of (a) inverse square law, particularly with respect to its use in calculating dosimetric errors when the wrong FSD is used for treatment fields; (b) central axis percentage depth dose curves as a characteristic of beam energy (especially with regarding to quality control and the measurement of quality indices); (c) the measurement of field size factors, so showing the origins of the data which the students had used for manual MU calculations. It also included elements of dosimetry which had been taught in the semester, mainly the use of ion chambers for photon measurements, dosemeter calibration (cross-comparison against a secondary standard) and the practicalities of independent, definitive calibration [28]. The lecture was 1 hour, followed by 1 hour of practical experiments. For the practical experiments, students were given detailed (verbal) instructions and shown how to use the VERTTM Physics software to make virtual measurements using the Linac. These included choosing and setting up the ion chamber block, changing depth of the ion chamber, and making measurements with the dosimetry panel for photon energies of 6 and 15 MV. Students were encouraged to use the hand pendant for the virtual machine to adjust set-up parameters, as per a real patient, and were invited to work with a machine type they were unfamiliar with from their A.2 Second iteration (2015 and 2016): Most feedback from the first iteration of this work was extremely positive [18]. However, in response to some of the slightly less positive comments, a key change was made for the second iteration and the way the class was run for 2015 and 2016. A number commented that the revision lecture at the beginning made the session feel overly long, difficult to focus upon, and difficult to appreciate the practical aspects with VERTTM Physics. These were possibly linked with those responses which also looked for more time for the calculations and for the session as a whole. In essence, the students wished to be engaged and interactive with VERTTM Physics much quicker and to have more time working together in the small groups and with the tutor, which was their overwhelmingly most reported comment [18]. So for the second iteration, the refresher lecture at the beginning was omitted. The VERTTM Physics session was scheduled closer to the subject matter pertinent to these Physics aspects and the clinical work which they were meant to help with understanding (i.e. the consequence of FSD set-up error), was timetabled, so only a small brief, introduction was used, together with the same tutoring and instructions for the use of VERTTM Physics as before, prior to going straight into the three main practical experiments described in A.1 above. As previously, the group was split into two smaller groups; one starting with calculations, the other with the virtual experiments. At the end of the experiment for a 248 MEDICAL PHYSICS INTERNATIONAL Journal, vol.6, No.2, 2018 particular beam energy, the groups swapped over; again employing, as previously, a change of all set-up parameters – so the new group doing the practical experiments would perform the set-up ‘from scratch’, in a similar style to that used on a real Linac in a definitive calibration [18, 28] for independence of measurement and confirmation of Linac calibration – as taught in theoretical classes for dosimetry. Identical peer-to-peer teaching was encouraged for the calculations and also in the practical groups, especially for those students unfamiliar with the hand pendants. Another identical feature, preserved because of the positive feedback, was the use of workbooks and the whiteboard space – so students discussed and performed calculations on the whiteboards, with the use of workbooks detailing the experimental work instructions needed, providing extra workspace and allowing notes to be made and kept for future learning and revision for assessments. Once again, the sessions were evaluated anonymously and these results have been reported previously [16-18]. Fig. 1 Start of the interactive demo for teaching and learning about depth dose curves for different radiation beams – introduction to the plotting tank and the output (dosimetry panel) of the virtual measurements using VERTTM Physics A.3 Third iteration (2017 and 2018): For the most recent two years, further changes were made to the sessions, partly in response to the continuing very positive comments (where students were asking for a greater use of VERTTM within the semester for teaching), but also in a desire to see if VERTTM Physics could supplement and improve upon teaching used for other aspects of Radiotherapy Physics necessary for clinical practice – most notably in improving understanding of radiation beam depth dose properties for different energies, different field sizes and in comparison with other modalities like electrons and protons in clinical treatments. Given the positive feedback in the use of VERTTM Physics and small group work, an extension was added to the sessions for the third and most recent iteration. The workbooks were also modified, with sections added in advance of the practical measurements, for students to predict percentage depth dose characteristics for photon beams (of different energies and different fieldsizes), electron and proton beams. Students discussed ideas in twos and threes during prediction, used the whiteboard to share their predictions and reasoning with the rest of the class and discussed the confirmation of results when measured with VERTTM Physics on the large, immersive screen. Different modalities were also examined interactively, with students again making predictions of similarities and dissimilarities between modalities in their workbooks and on the whiteboards. Concepts of changes because of phantom scatter and head scatter were examined for photons within the VERTTM environment, using the large, wall-wide VERTTM screen and immersive environment; with students encouraged to point out and discuss reasons for changes with energy and fieldsizes whilst gathered around the VERTTM screen (Figure 2). They were encouraged to make energy and fieldsize changes themselves, and dosimetric measurements using the virtual plotting tank in the VERTTM Physics software. Similarly, students made predictions for electrons and protons, noting commonality of (e.g.) depth of maximum dose for electrons and photons. This was done again both in their workbooks after discussion with one another and on the whiteboards, before final expert, tutor-led versions were drawn on the whiteboard in summary of the main similarities and differences. A.3.1 Interactive Demonstration: The engaging practice of the large screen (4m wide by 2m high, back-projected) and immersive style of work was used to introduce an interactive demonstration at the start of each session. Once again, VERTTM Physics was used to illustrate Radiotherapy Physics concepts and equipment – the extension to previous years now being the use of the plotting tank; firstly as a very brief demonstration of how depth dose data was collected in reality in clinic, for manual MU calculation data charts and MU programmes, and also for data to verify TPS models for photons (Figure 1). 249 MEDICAL PHYSICS INTERNATIONAL Journal, vol.6, No.2, 2018 virtual experiments, and the concepts of independent set-up using the hand-pendants and software for each energy change. Fig. 2 Image of the full-size, 3D room screen set-up for the interactive demonstration and plotting tank measurements. Tutor and students would use mouse and machine specific hand pendant controls for set-up and measurements Fig. 4 The whiteboard workspace used by the ‘calculation group’ for the virtual experiments of using inverse square law to determine the dosimetric error involved with incorrect SSD in patient set-up (left hand side) and beam energy specification (quality index) (right hand side) A.3.2 Practical experiments: The second part of each session then proceeded with virtual Linac practical experiments in the same way as the previous two iterations. A very short introduction was given about the dosemeter block (see Figure 3) so students were aware of how actual measurements were conducted in the clinic, and also to continue their instruction in making virtual dose measurements themselves using the VERTTM Physics software. B. Evaluation and Analysis B.1 Evaluations post session: For the first two iterations of the work, these have been reported previously [16-18] and were achieved using short, anonymized evaluation sheets given to each group member after the session. The same approach was maintained for the third iteration, inviting students to freely give feedback immediately after the full session (the interactive demo and the virtual practical experiments). The sheets used the same approach as previously, asking for open and honest opinions on the most positive aspects of the VERTTM Physics session; the least positive aspects and any suggested changes for future sessions. All responses were qualitatively coded and organized into descriptive, common themes and responses. B.2 Exam results analysis: Since part of the intention for making the changes for the third iteration was to see if VERTTM Physics might potentially improve understanding in the assessment setting, the results of four consecutive years of unseen, written examinations were analyzed. These were for the 2nd year undergraduate students – for the postgraduates, this was not attempted, since their assessment was primarily by essay-style, written assignment, without the necessary sub-division of applied marks which could be analyzed. For the undergraduates, focus was maintained on the marks of parts of long answer questions which were posed to allow students to show their knowledge and understanding of the depth dose characteristics radiation beams of different energies, fieldsizes, FSDs and modalities. Fig. 3 Short introduction and instruction given in the use of VERTTM Physics for the virtual practical experiments using the ion chamber block The previous approaches of dividing the group into two to enable peer-to-peer and individualized expert tuition were maintained; as were the work instructions and workspace in the workbooks and on the whiteboards (see Figure 4); and the swap around between performing calculations and 250 MEDICAL PHYSICS INTERNATIONAL Journal, vol.6, No.2, 2018 III. RESULTS liked the final, expert, tutor-led summary of characteristics drawn on the whiteboard, which they could use for their learning and revision for assessments (see figure 6). The key responses from the first two iterations have been reported upon previously [16-18]; and key points following those publications and communications are shown in figure 5. The students enjoyed the ease of use of the software and were able to perform the virtual experiments extremely quickly. The blended learning approach made the sessions ‘come alive’ compared to the more didactic, but discussion led lectures. They enjoyed the safety of the virtual environment, but appreciated that the virtual experiments were conducted as if on a real Linac, with the same professional approach to independence of measurements and minimizing of risk for systematic errors (i.e. by way of independently setting up the virtual Linac). From both calculations and virtual experiments, they were able to appreciate the dosimetric consequences of a few cm of setup error in FSD; and use their knowledge of legislation to determine whether such errors might be reportable to outside bodies under such directives. Fig. 6 The whiteboard workspace used for interactive work; predicting and comparing students’ own knowledge and understanding with ‘measurements’ from the virtual VERTTM Physics environment. Final expert, tutor-led summary of characteristics is shown. Note, only photon measurements are possible through VERTTM Physics. In terms of the anonymized evaluations and feedback from the students (figures 7 and 8), like the previous iterations, the responses are overall extremely positive. In terms of the good points listed, most felt that the sessions were well taught and explained and it made a difference in the use of VERTTM for this. The virtual environment was found to be very useful for explaining concepts and helping understanding. As with previous evaluations, the students appreciated the small groups, and working together within them, the interactive nature of the sessions, the workbooks for personalized working and the different way of learning enabled by the interaction, the whiteboards, the predictive nature of both the demonstration and the calculations, and the virtual environment. More sessions were called for like these ones. In terms of points for improvement, they felt the session could have been longer, so that various elements (like the practical work) were not felt to be rushed, although some appreciated the time constraints within the timetable. As an illustration of different abilities, some felt that the session could have actually included more work, whilst some struggled a little with understanding the calculations within the available time. Some commented under this banner that there were no bad points, and they would like more opportunities like these. Fig. 5 Key results from the first and second iterations of the work with VERTTM Physics and therapeutic radiography students (UG and PG) They commented highly and positively on the small group aspects, peer-to-peer teaching and individualized attention of the tutor for teaching and discussing concepts, particularly in relation to the calculations. So too the opportunity to perform calculations in predicting results which were then confirmed through the virtual practical measurements. For the third iteration, the whiteboard final output is shown in figure 6, and the summarized and themed responses are shown in figures 7 and 8. Students engaged very well with the interactive nature of VERTTM Physics, and engaged very well with peer-to-peer discussion and prediction of depth dose characteristics in their workbooks. Some members of the group found the session a safe space to share their predictions with the class on the whiteboard for different energies and modalities. Students particularly 251 MEDICAL PHYSICS INTERNATIONAL Journal, vol.6, No.2, 2018 Fig. 7 Bar charts summarizing the key responses regarding ‘good’ and ‘not Fig. 8 Key responses from the third iteration of the work – where an interactive demo, followed by the virtual practical experiments, was used. so good points’ for the third iteration of the work. The written responses for future suggestions are also shown. Regarding their suggestions for the future and things to try the next time, it was notable that many did not comment here – which may indicate an overall satisfaction with the session as it was. Those that did, re-iterated their desire to have longer and more sessions like this. By far the most popular response was for longer sessions, so that the smaller groups (calculation and practical) could swap around more. There was again appreciation for the workbooks, although some would prefer an enhancement here by providing more diagrams to explain the experiments and the clinical analogy being investigated for the simulation of FSD errors in set-up. In terms of the analysis of examination results, the data is shown in figure 9. Exam scripts were analyzed for the maximum, mean and minimum marks, for the four years of assessments undertaken since VERTTM was introduced into the department. Mean class size was 26, with a range of 2230. Because of the timetabling of the sessions within the academic years, the data points for 2017 and 2018 shown in figure 9 correspond to results obtained after the introduction of the third iteration of the VERTTM Physics sessions. We found that the range of maximum marks changed from between 71-80% to 86-89%; the range of mean marks from 46-47% to 58-61%....a full grade boundary (10%) change. Minimum marks are not really applicable, because they are weighted by the occasional student who did not answer the questions, and therefore scored zero for that question or part thereof. Fig. 9 Analysis of summative assessment components (exam results) which focus on depth dose curves for different energies and modalities. A modest improvement for both mean and maximum marks is noted for the third iteration (2017 and 2018) IV. DISCUSSION The reasons for the evolution of this type of learning and teaching, in this very interactive and engaging way, have been explained earlier – but this was still quite a considerable risk; given the highly positive evaluations especially from the second iteration. However, as illustrated, changes were made for specific reasons (in response to the feedback) and only to parts of the sessions – thereby minimizing the risk to students own learning and to the engagement which the virtual environment engenders. The results have shown that the latest evaluations have been just as positive as the first two – with students finding the sessions useful and a great way to help understanding; for a number, they found the virtual environment and the interaction made it easier to understand the necessary 252 MEDICAL PHYSICS INTERNATIONAL Journal, vol.6, No.2, 2018 concepts – better than just using theoretical, didactic classes on their own. They appreciated the clinical relevance of the concepts and the chance to use their knowledge and try to explain things first, before being confirmed by the software and the tutor. The workbooks are very much appreciated too – similarly to the responses for workbooks used in other, more familiar clinical uses of VERTTM [14, 18, 19] and in other clinical modules within the programme (e.g. for studying anatomy and physiology). In an age of electronic, digital media being readily available (e.g. through tablets and phones), the students still value the tactile nature of the workbooks to perform calculations, share viewpoints and then to use for a revision resource. The not so good responses all focused on ‘more’ – more sessions, more time, more opportunity to use this valuable resource and to have more sessions with this blend of interaction and engagement. This was particularly so for the virtual, practical experiments which followed the interactive demonstration. From a tutor’s perspective, the time constraints on sessions were more difficult for the practical and calculation parts; for those students finding the calculations more challenging, this would naturally increase pressure and the feeling of being rushed. The increase in pressure was an aspect which was the antithesis of the desire of the sessions in the first place and is something to be addressed in the future – in order to hold a safe space for the students, with an environment to easily ask questions and gain from the individual tuition offered through the small groups. Also from the tutor’s perspective, the sessions in this format were extremely easy to devise and to run; an aspect which has been identified by other educational groups in the university [26, 29] when our experiences and results have been shared in general learning and teaching conferences and active workshops. The blended nature of the learning strategy, the work with small groups and the highly active and interactive nature of the work are common elements which can be applied across disciplines – and indeed is being shared across the University for innovation in educational methods and developing the university’s curriculum across the board [29]. The analysis of the examination results shows some interesting trends and potential. Since the third iteration, both the maximum and mean marks have improved with changes of the order of a whole grade point (i.e. 10%). This could indicate the improved learning from these interactive and blended methods – but the exam questions used and considered are not always exactly the same format; so there are some potential difficulties in performing the comparisons. But the indicative direction is an improvement in results; which, for the best design of assessment, should mirror students being able to demonstrate an improved understanding in these subject areas. As with previous reports in other sectors [30, 31], the virtual environment simulates the physical world extremely well – for us, it is in its use beyond its original design (i.e. mainly as a clinical tool), to one which VERTTM Physics was designed for (for simulating radiotherapy physics equipment and principles), to a further one which is simulating the real use of the Linac for performing dosimetric experiments and demonstrations for highlighting important physics concepts needed for clinical work, and confirming theoretical knowledge acquired, in a highly practical way. V. CONCLUSIONS In conclusion, the VERTTM Physics virtual environment has proven to be one which is useful and highly engaging for student learning. It is easily adaptable to different paradigms of learning and has continued, through different iterations, to work extremely well as a teaching tool – as evidenced by anonymized evaluations and feedback, and through the potential increase in assessment marks. Students continue to find it useful, helpful and interactive – enabling a more ready way for understanding these concepts. Students enjoy the sessions, especially the small group structure, with combined peer-to-peer and expert tuition; something which is transferable to other disciplines and subjects in education and learning. The results show they can undertake the virtual experiments very easily, and are more ready to try and discuss calculations in this style of environment – which they find safe and relaxed. However, longer sessions are necessary (and are being planned for in future semesters) in order to allow more and longer sessions, to maintain the relaxed and less-stressful environment originally designed. One might cautiously hope that the continued upward trend in assessment results continues, demonstrating a better and potentially deeper understanding of these important topics, for the good of the clinical service. ACKNOWLEDGMENT The author would like to thank the undergraduate and postgraduate students in radiotherapy at the University of Liverpool for their engagement, hard work and enthusiasm within the sessions, and their open and honest feedback through the evaluation questionnaires. The support of colleagues within the directorate, School of Health Sciences and wider University of Liverpool in supporting new and innovative teaching and learning methods within all our programmes is noted with great appreciation. 253 MEDICAL PHYSICS INTERNATIONAL Journal, vol.6, No.2, 2018 REFERENCES 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. Jimenez Y, Ronn Hansen C, Juneja P and Thwaites DI. (2017). Successful implementation of Virtual Environment for Radiotherapy Training (VERT) in Medical Physics education: The University of Sydney’s initial experience and recommendations. Australas. Phys. Eng. Sci. Med. 40:909-916. 20. Jimenez Y, Thwaites DI, Juneja P and Lewis SJ. (2018). Interprofessional education: evaluation of a radiation therapy and medical physics student simulation workshop. J. Med. Radiat. Sci. 65:106-113. 21. James S, Dumbleton C (2013) An evaluation of the utilisation of the virtual environment for radiotherapy training (VERT) in clinical radiotherapy centres across the UK. Radiography, 19(2):142-150 22. Shah, U. (2016). VERTTM and staff education, BIR Meeting abstracts, BIR Meeting on VERT: an (online) educational study day for present and new users, London, UK, 2016. Published abstract available at https://www.bir.org.uk/education-and-events/vert-ondemand-content/ 23. BIR (2016). VERT: An (Online) educational study day for present and new users. Available at https://www.bir.org.uk/education-andevents/vert-on-demand-content/ 24. Stewart-Lord A (2016) From education to research: a journey of utilising virtual training. J. Radiother. Practice 15(1):58–90. https://doi.org/10.1017/S1460396916000030 25. Brown, M. (2016). Patient preparation for treatment, BIR Meeting abstracts, BIR Meeting on VERT: an (online) educational study day for present and new users, London, UK, 2016. Published abstract available at https://www.bir.org.uk/education-and-events/vert-ondemand-content/ 26. Kirby MC (2018). Virtual environment for teaching radiotherapy physics – a four year experience. University of Liverpool 16th Learning and Teaching Conference, Liverpool, UK 2018. 27. Kirby MC (2018) Teaching radiotherapy physics to student radiographers using VERTTM Physics – an update of four years’ experience, MPEC 2018 Abstracts, Medical Physics and Engineering Conference 2018, York, UK, p70. Published abstract available at https://www.ipem.ac.uk/Portals/0/Documents/Conferences/2018/MPE C18/MPEC%202018%20abstract%20book%20v2.pdf?ver=2018-0830-143755-707 28. Patel I, Weston S, Palmer AL et al. (Eds) (2018). Physics aspects of quality control in radiotherapy: IPEM Report 81, 2nd Edition. York, UK: Institute of Physics and Engineering in Medicine. 29. Kirby MC (2018). A blended approach to teaching therapeutic radiography students physics. University of Liverpool Centre for Innovation in Education Curriculum 2021 Framework and Resources. Available at https://www.liverpool.ac.uk/centre-for-innovation-ineducation/curriculum-resources/case-study/teaching-therapeuticradiography-students/ 30. Gaba DM. (2004). The future vision of simulation in health care. Qual. Saf. Health Care 13(Suppl 1):i2–i10. doi: 10.1136/qshc.2004.009878 31. Salleh S, Thokala P, Brennan A et al. (2017). Simulation modelling in Healthcare: and umbrella review of systematic literature reviews. PharmacoEconomics 35:937-949. Brandt BF, Quake-Rapp C, Shanedling J et al. (2010). Blended learning: emerging best practices in allied health workforce development. J. Allied Health 39:e167-e172 Butcher C, Davies C, Highton M (2006). Designing learning: from module outline to effective teaching. Routledge, London Kirby MC, Pennington H, Al-Samarraie F et al. (2014) Clinical technology in 21st century radiotherapy education – towards greater alignment with clinical competencies. Radiother. Oncol 111(S1):738 Kirby MC, Al-Samarraie F, Ball B et al. (2014) Radiography education programme development, BIR Meeting abstracts, BIR Meeting on Radiotherapy – meeting the current and future workforce challenges for patient care in a changing context, London, UK, 2014. Published abstract available at http://issuu.com/bir_publishing/docs/radiotherapy_workforce_challen ges_p/0 Phillips R, Ward JW, Beavis A (2005). Immersive visualization training of radiotherapy treatment. Studies in Health Technology and Informatics 111:390-396 Bridge P, Appleyard RM, Ward JW et al. (2007). The development and evaluation of a virtual radiotherapy treatment machine using an immersive visualisation environment. Computers & Education 49(2):481-494 Phillips R, Ward JW, Page L et al. (2008). Virtual reality training for radiotherapy becomes a reality. Studies in Health Tech. & Informatics 132:366-371 Boejen A, Beavis A, Nielsen K et al. (2007) Training of radiation therapists using a 3D virtual environment. Radiother. Oncol. 84:S275 Stewart-Lord A, Brown M, Noor S et al. (2016) The utilisation of virtual images in patient information giving sessions for prostate cancer patients prior to radiotherapy. Radiography 22:269–273. https://doi.org/10.1016/j.radi.2016.05.002 Powell, P. (2016). VERTTM and therapeutic radiography student education, BIR Meeting abstracts, BIR Meeting on VERT: an (online) educational study day for present and new users, London, UK, 2016. Published abstract available at https://www.bir.org.uk/education-andevents/vert-on-demand-content/ Bridge P, Giles E, Williams A et al. (2017). International Audit of Virtual Environment for Radiotherapy Training usage. J. Radiother. Pract. 16:375-382. Kane P (2018). Simulation-based education: a narrative review of the use of VERT in radiation therapy education. J. Med. Radiat. Sci. 65:131-136. Green D, Appleyard A (2011). The influence of VERT characteristics on the development of skills in skin apposition techniques. Radiography 17(3):178-182 Nisbet H, Matthews S (2011). The educational theory underpinning a clinical workbook for VERT. Radiography 17(1):72-75 Beavis A, Ward J (2012) The Development of a Virtual Reality Dosimetry Training Platform for Physics Training. Med. Phys. 39:3969 Kirby MC (2015) Teaching physics using simulation, UKRO 2015 proceedings, UKRO 2015 ‘Innovation and inspiration – national UK radiation oncology conference, Coventry, UK, 2015. Published presentation available at http://www.ukro.org.uk/2015-presentation Kirby MC (2015) Teaching radiotherapy physics using simulation, MPEC 2015 Abstracts, Medical Physics and Engineering Conference 2015, Liverpool, UK, 2015, p11. Published abstract available at http://www.ipem.ac.uk/Portals/0/Documents/Conferences/2015/1%20 MPEC%202015/ABSTRACT%20BOOK%20MPEC%202015.pdf Kirby, M. C. (2015). Teaching radiotherapy physics concepts using simulation: experience with student radiographers in Liverpool, UK. Medical Physics International Journal 3(2):87-93. Available at: http://www.mpijournal.org/MPI-v03i02.aspx Contacts of the corresponding author: Author: Institute: Street: City: Country: Email: 254 Revd Dr Mike Kirby University of Liverpool Brownlow Hill Liverpool UK [email protected]