Stepping Stones: Capacity Building In Engineering Education

Stepping Stones: Capacity Building in Engineering Education Arnold N. Pears1 , and Mats Daniels1 1 IT Department, Uppsala University October 1, 2008 Abstract The CeTUSS (www.CeTUSS.se) engineering education center was established by the Swedish Council for Renewal of Higher Education in 2004. During 2006/2007 CeTUSS funded ”Stepping Stones”, a multiphase (project based) initiative to build a community of engineering educators and to raise awareness of methods for assessing the impact of classroom interventions. Stepping Stones was based on earlier US, UK and Australian initiatives; the Scaffolding, Bootstrapping and BRACE programmes. The approach uses a joint, multi-method, research study to raise awareness of relevant theory. Community building is realised through joint work and shared experiences which promote convergence on a common set of values and ideals in relation to scholarship of teaching and learning. Investigative ”capacity” was enhanced by drawing together a Swedish pool of engineering education expertise. Stepping Stones was conducted in three phases. The first phase was a week long workshop examining relevant theory and empirical study design in engineering education research. This workshop introduced an ”experiment kit”, a protocol detailing experimental design of the project that participants jointly implemented in phase two. During phase two the participants gathered data in their own classrooms, contributing to a joint corpus of material for analysis in phase three. During the data collection process participants administered and validated a variety of instruments; surveys and interviews (including photo elicitation), and concept map collection using Explanograms (a tool for automating collection of handwritten data developed by CeTUSS). The final phase was a week-long workshop where participants analyzed the aggregated data and produced a written report, ’What is the Word for ”Engineering” in Swedish: Swedish Students Conceptions of their Discipline’ (http://www.it.uu.se/research/publications/reports/2007018). Introduction Enabling collaboration and supporting and enhancing collaborative development of educational competence is a key aspect of dissemination and capacity building initiatives. One aspect of capacity building is assisting people in locating and communicating with one another in order to collaborate and build 1 Utvecklingskonferensen 2008 www.CeTUSS.se Figure 1: Applications to study in Engineering Programmes on prior knowledge. Stepping Stones utilizes a proven project based model for community/capacity development as part of a larger initiative to promote diversity and move towards a more scholary and informed approach to teaching practices in Engineering. In the work described here we target two aspects of capacity building for Swedish Engineering Education. How do we combine and build upon national engineering education research competence, reinforcing and diversifying the network of active academics working in Engineering Education in Sweden? Can we collect broad ranging data about views of Engineering that provide insight useful when discussing the Swedish student recruitment and retention issue? The remainder of the paper is structured as follows. In the next section we discuss the motivations for this work, and identify some related theoretical and study results. The implications and uses of the study data for engineering teaching at University level are then discussed and a number of investigative directions are described. We conclude with some observations about the possible uses of the data and outline future work on the development of recruitment and retention strategies for Engineering Education in Sweden. Background Engineering education in Sweden, as in the rest of the world, is experiencing a decline in student interest (see Figure 1). There are concerns about the ways in which students think about engineering education, why they join an academic programme in engineering, and why they persist in their studies. In this context one of the aims of CeTUSS is to investigate the Swedish student experience and to identify and support a continuing programme of research leading to changes in higher education for engineers. Supporting this long term goal demands initiatives to expand the National capacity for disciplinary pedagogic investigation. Stepping Stones addresses these concerns in a multi-researcher, multi-institutional study that spans eight Swedish universities and a range of engineering programmes. As the study is situated in a uniquely Swedish education setting, it permits us to explore ”a Swedish perspective” on conceptions of engineering. 2 Utvecklingskonferensen 2008 www.CeTUSS.se Figure 2: Overview of Sepping Stones studies Stepping Stones was based on a model of research capacity-building previously instantiated in the USA and Australia [9]. Related work Stepping Stones draws on a variety of literature to define its research context and focus. Persistence in tertiary education has been investigated in a number of areas. Perceptions among school students influence their choice of career path and motivation to study engineering [8]. Students’ university experiences in Science, Technology and Mathematics have also been investigate in relation to academic persistance [2, 3, 10, 13]. Evidence of a disconnection between engineering practice and student experiences of tertiary engineering education has been advanced by a number of researchers. Studies of practicing engineers show a shift from a purely technical focus towards an increasing appreciation of social and environmental factors. See also Jonassen et al. [11] and Lethbridge [12] for discussions of the current dielectic between education and practice in engineering. Issues related to ancillary skills which might be acquired ”accidentally” are discussed by Walther and Radcliffe and Berggren et al. [1, 15] There is a small but growing body of literature on college students’ understanding of engineering and engineering practice. For example, ethnographic investigations into college students’ perceptions of engineering reveal the predominance of technical knowledge and mathematical problem solving which may be related to students’ limited experience with formulating and defining complex and ambiguous problems [4]. Turns et al. [14] used a word association task to study graduating civil engineering students’ schemas of civil engineering and found that technical knowledge predominated significantly over such issues as communication, multidisciplinary teams, and global and societal context issues. Downey and Lucena [5] also examined how students involved in design experiences perceive the distinction between science and design. Their findings suggest that students perceive design as a lesser subset of the engineering method of mathematical problem solving, and as such are ill prepared for dealing with the ambiguities and subjectivity inherent in engineering design problems. Similarly, Newstetter and McCracken (2001) found that freshmen engineering undergraduates’ early conceptions of design tend to conceptualize design as an artistic, creative process - ”a blaze of creative light that strikes some and not others” (pg. 70). In a related study, Mosborg et al. (2005) found that advanced practicing engineers ranked as most important among a list of 23 design activities ”problem formulation” and ”communication” ”Building” was 3 Utvecklingskonferensen 2008 www.CeTUSS.se Figure 3: Sepping Stones data demographics by institution. ranked among the least important activities and ”creativity” was included in neither the highest nor lowest ranked groups of terms. Our study drew on this work when formulating the focal questions for the research. Many of these questions evolved as we delved deeper into the dataset. These changes are documented in the individual studies we overview later in the paper. Data collection Stepping Stones data collection is based on five central design decisions. Firstly, we did not want to impose an existing framework of conceptions of engineering on study participants. Using elicitation techniques; such as interviews and concept maps seeks to reveal participant’s underlying conceptions of engineering. We sought to allow important aspects of the experience of engineering education to emerge during the investigation. Secondly, we wanted the study to be domain independent so that comparisons could be made across engineering domains. This was done by using general ”engineering” terms in the study instruments rather than terms that are specific to a sub-discipline (e.g., mechanical engineering or computing). Given the complexity inherent in our research questions we wanted triangulation, this is achieved by a data collection approach designed to increase the likelihood of contradiction or corroboration of other study findings. This was achieved by combining several different approaches and collecting both qualitative and quantitative data. Finally, we wanted the study to be of sufficient scale to allow generalisation. Simultaneously we wanted to leverage the power of existing studies reporting results from validated data collection instruments. These last objectives were achieved using a web-based survey [6,7] which collected data from many participants at many institutions in Sweden. The complete study comprises four data collection instruments: a web-based survey, the construction of a concept map, a critical incident interview and a photo elicitation interview. Data was gathered over the academic year 2006/07 from ten Swedish institutions (although not all institutions gathered all forms of data). For the concept map task and interviews, data were given a unique identifier of the form: AF01 where the first letter represents a unique institution code, the second letter represent experience level, of the form F (First Year Student 01), G (Graduating student), A (Alumni) or E (Educator) and the number represents a unique participant. Participants may have contributed to all parts of the study, to just the 4 Utvecklingskonferensen 2008 www.CeTUSS.se Figure 4: Frequency of student/instructor interaction distributed by Institution survey or the just the concept map task and interviews. Study Outcomes This section summarises the preliminary results for the five preliminary studies described in Figure 2 (Study E is presented last, as it builds on some of the earlier studies). As is apparent from the table some studies focus on a single data source, while others combine and compare information from several data sources. The unit of analysis varies in the different studies - some focus on institutional level characteristics, some on participant level characteristics, and some focus on both institutional and participant level characteristics. Study A In this part of the data analysis, the validity and implications of the constructs 1 of the survey data were compiled, plotted and analyzed per institution to explore students’ perceptions about engineering education at their respective university. Overall the results from this part of the data analysis reveal little that is surprising. In other words, there were few instances of noticeable differences across the institutions. With the exception of Construct 12 (”Frequency of interaction with instructors” (see Figure 4), there are few constructs where there is an observable difference between the universities surveyed. This is also true for constructs 11c, 13a and 13b. Therefore, it seems that institution is not an interesting unit of analysis. Instead, it suggests that this data, or the constructs thereof, may be more usefully analyzed with respect to other factors, such as gender or disciplinary program. Another implication of this analysis is that it suggests that there is considerable consistency in what Swedish engineering students think about engineering education at their universities. In this respect there appear to be no significant differences among the subjects of the survey sample for most of the constructs. The only case, Construct 12, which characterizes the frequency of interaction between students and instructors, is hardly surprising due to such institutionally variable factors such as the availability of resources, pedagogical approach, and the number of students. 1 See the Stepping Stones technical report, Appendix G 5 Utvecklingskonferensen 2008 www.CeTUSS.se Figure 5: Common terms in Concept Maps To facilitate subsequent analyses, we identified a set of categories of engineering education disciplines based on the educational programs that occur in the survey sample. We propose that this should be consistently applied in all areas of the data where educational programs manifest themselves2 . Study B This study draws on three of the Stepping Stones data sources; namely the web based questionnaire, the concept maps (CM) recorded using explanogram technology, and the interviews. The primary goal of the study was to explore concept map data, trying to find interesting patterns that could be further analyzed. A secondary goal has been to explore different approaches to analyzing concept maps. Our overarching research question is: What can we learn about students’ and educators’ views of engineering from their Concept Maps (CMs); how do they relate engineering to other concepts and which seem to be the most and least important concepts in their CMs? To operationalize this question we developed a set of more specific questions: • Which are the most or least important concepts in the drawings? • Can the structure or appearance of the drawings be categorized in some way? • Are particular concepts ”closer” to engineering than others? • Are there any differences for different groups of subjects? One contribution of this analysis is a classification of which concepts were central in the drawings. We defined ”central” with reference to the following properties: central in a spatial sense (in the ”middle” on the top of the drawing serving as a header, having many links, drawn as frame that includes other concepts, or as explained by additional text. Several concepts can simultaneously be considered as central. Which of the concepts appeared as central most often or rarely? To find the central concepts three researchers visually inspected all CMs and categorized them until they reached full agreement. All CMs with engineering as one of the central concepts were sorted out in a first pass. We then 2 The table is available in the Stepping Stones technical report, page 13 6 Utvecklingskonferensen 2008 www.CeTUSS.se Figure 6: Response Categorisation Q1 inspected the remaining CMs a second time and noted down all central concepts. One CM did not have any concept that could be interpreted as central; it was drawn as a sequence of concepts in a single line. The central concepts are summarised in Figure 5 Study C One of the focal questions of the Stepping Stones project deals with conceptions of engineering in Sweden. We wanted to know what students and educators think are the elements of engineering. We focused on three of the interview questions which ask specifically about engineering and what engineering ”is”. Questions 1 and 2 are at the beginning of the 30 minute interview, while Question 5 occurs right at the end of the interview. Q1: In a few words, what would you say real engineering is? Q2: Can you give me some examples of engineering in the world? Q5: After everything we’ve talked about, what would you say ”engineering” is for you? All responses to these questions were digitally clipped from the transcripts and printed out. Keywords and phrases in Q1 were first marked and then grouped into ten different categories by one person (see Figure 6). Another person verified the categories. The same process was used for Q5 where we found that the same categories could be used to group responses. The responses to Q2 were also treated in the same way, but since these were mostly nouns that left little room for interpretation only one person coded these. For Q2 we identified 11 different categories (see Figure 7). There are two noticeable changes in the way the participants characterize engineering between Q1 and Q5. When answering Q1 at the beginning of the interview 17% of participants describe engineering using examples of different academic subjects (e.g. mathematics, physics). This should be compared with the comments made to Q5 towards the end of the interview and critical incident exercise, where that number had decreased to 8%. We also note that the proportion of participants that mentioned the impact of engineering on society increased from 12% to 28% between questions 1 and 5 during the interview. 7 Utvecklingskonferensen 2008 www.CeTUSS.se Figure 7: Response Categorisation Q2 Figure 8: Frequency of student/instructor interaction distributed by Institution Study D Study D uses the data from the photo elicitation interviews (where participants were asked about the associations of engineering they had with three different images). There was also a final interview question regarding how the participant’s ideas about engineering might have changed. During the interview the participants were asked ”What associations of engineering does image A (B, C) have for you?” At the same time the corresponding photo was shown to the subject. The section of the interview dealing with photo elicitation has been extracted and analysed for specific words and concepts. The associations have been classified using a content based coding scheme . Only Image A has currently been analyzed (the relevant image is shown in Figure 8). We intend to analyse the Data for the other pictures in a future study. Six classes of association are identified by our analysis. Plan : Participants associate the image to planning a project, analysing something, interpretation of results, developing something new or solving a problem. Graduating students seemed to make this association to a higher 8 Utvecklingskonferensen 2008 www.CeTUSS.se extent then first year students; however the difference was not significant. Also 40% of the male participants had this association compared to 13% of the female participants. Male : Participants that observe and associate that there are only men in this picture are classified into this group. Female participants made this association to higher degree. It is also worth noting that educators have this association. 50% of females and 50% of all educators made this association compared to 7% amongst all male participants. Team : Participants that associate teamwork and people working together with this image are classified into this group. The only noticeable difference is that educators are slightly more represented. Sci : Participants that associate picture A with mathematics, physics or scientific research fell into this group. In this case there is no noticeable difference. Eco : Participants that associate the image with economics, the stock market, project economy or statistics are classified into this group. First year students and male participants are more represented. 36% of the first year students made this association compared to 23% of graduating students. 30% of male participants made this association compared to 6% of the females. Many participants with this association said the picture could equally well represent economists and didn’t have anything at all to do with engineering. Old : Many participants associated this picture with ”old ways” of engineering or made some comment that it was old, maybe from the fifties. Graduating students and educators are represented to higher degree in this group. 44% of the educators made this association compared to 19% of first year students and 27% graduating students. Also 44% of female participants compared to 24% of male participants made this association. Study E Study E examines the data collected from survey participants looking for gender differences. Such differences are also checked by triangulation with other data we collected in the project. Of 521 survey participants, 108 were females, 383 males and 30 did not state their gender. The study first looks for obvious gender differences in the answers given by the females as compared to the males, and then if differences could also be seen in the concept maps and the results from the interviews and the concept map debriefs. We chose to focus on a few aspects, such as why do students choose to study engineering, how do they rate their own skills as compared to their classmates and what traits do they believe are important for a working engineer. 9 Utvecklingskonferensen 2008 www.CeTUSS.se In question ten, the students were asked to rank ten different statements about why they chose to study engineering. The alternatives contained statements like ”Technology plays an important role in society”, ”Engineers make more money than most other professionals”, and ”My parents want me to be an engineer”. For the complete list of statements see the Stepping Stones Technical Report, Appendix A. Options for each question were; ”not a reason” ”minimal reason” ”moderate reason” and ”major reason” There were no systematic differences between the male and female responses except in the two statements ”My parents want me to be an engineer” and ”An engineering degree will guarantee me a job when I graduate”3 . The males did not identify the influence of their parents as a strong motivation for choosing engineering, while the females did. The females indicated, to a larger extent than the males, an expectation that an engineering degree would guarantee them a job after graduation. With regard to perceptions of professional practice, there were no differences attributable to gender. In question 13 of the survey, participants were asked to rate how important they considered different skills and abilities for a becoming a successful engineer. The rating was ’not important’, ’somewhat important’, ’very important’ and ’crucial’. There no observable differences in the female and male answers, other than a larger spread in the male answers regarding ’self confidence’ and ’communication skills’. Males rated communication skills as slightly more important than females. Conclusions and Future Work The Stepping Stones project has gathered a rich data set for exploring conceptions of engineering from a Swedish perspective. Initial analyses of a variety of factors has been summarised here in studies A-E reporting on experiences of engineering study and their impact on decisions to continute to study engineering at Swedish Universities, as well as perceptions of which concepts/tasks are central to engineering. We conclude that the experience of engineering studies is quite uniform in the Swedish education system. The manner in which students conceive engineering, and the elements of engineering practice that they perceive as important, are worth considering in an international context and contrasting with studies in other countries. This may help to identify areas where perceptions of Swedish engineering are different to those in other countries and educational systems. Finally we have considered gender differences in relating to engineering and the perception of engineering as a ”male” discipline. This aspect of the data deserves further investigation given the low representation of women in swedish tertiary engineering education. In terms of continuing work we intend to undertake the following analyses: Surveys : An analysis by discipline. Concept maps: Analysing how the concept maps develop over time, using the explanogram representations. For example, examining the sequences graphically and reformulating them as stories or narratives that could be further analyzed. For example, could we then tell the categories’ different stories? Some of the participants forgot or excluded certain words. Is there a pattern to this or is it just neglect to check that all concepts were used? How can we relate this to the debrief 3 These results are shown in figure 13 and 14 of the Stepping Stones report, respectively. 10 Utvecklingskonferensen 2008 www.CeTUSS.se information? Especially interesting would be relating these sequences with questions one and three from the concept map debrief, that ask students to relate the terms used in the mapping task to their university courses. Interview and survey data combined : An analysis of question 38 of the survey with respect to questions 1, 2 and 5 in the interview. For example, it should be possible to create a list of terms students associate with engineering and then compare these with the responses in the interview. Also, at this stage, no aggregation of the terms from the survey has been performed. Early analyses of the survey, reported here, indicate that ”problem solving” and ”mathematics” are terms that participants frequently use for describing engineering. In addition, it would be useful to continue the analysis of the photo elicitation interviews. Gender : What perceptions can be identified related to the ”gendered-ness” of engineering in the Swedish context? How might this affect our ability to recruit and retain women in engineering programmes in Sweden, and what similarities and differences are there compared to other major studies? Acknowledgements The Stepping Stones project was funded by Rådet för högre utbildning (the Swedish Council for Renewal of Higher Education) and by Myndigheten för nätverk och samarbete inom högre utbildning (the Swedish Agency for Networks and Cooperation in Higher Education). It was run within the context of the Nationellt ämnesdidaktiskt Centrum för Teknikutbildning i Studenternas Sammanhang project (CeTUSS), a national centre for pedagogical development in technology education in a societal and student oriented context. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any of these supporting bodies. We wish to thank the Stepping Stones course leaders Robin Adams and Sally Fincher, and the project participants, Jonas Boustedt, Jürgen Börstler, Peter Dalenius, Gunilla Eken, Tim Heyer, Andreas Jakobsson, Vanja Lindberg, Bengt Molin, Jan Erik Moström, and Mattias Wiggberg for their contributions to this project. References [1] K.-F Beggeren, D. Brodeur, E.F. Crawley, I. Ingemarsson, W.T.G. Litant, J. Malmqvist, and S. Ustlund. CDIO: An international initiative for reforming engineering education. World Transactions on Engineering and Technical Education, 2(1), 2003. [2] M.E. Besterfield-Sacre, C. Atman, and L.J. Shuman. Engineering student attitudes assessment. Journal of Engineering Education, 87(2):133–141, April 1998. [3] M.E. Besterfield-Sacre, M.E. Moreno, , L.J. Shuman, and C. Atman. Gender and ethnicity differences in freshmen engineering student attitudes: A 11 Utvecklingskonferensen 2008 www.CeTUSS.se cross- institutional study. Journal of Engineering Education, 90(4):477– 489, October 2001. [4] G. Downey and J. Lucena. Engineering selves: Hiring into a contested field of education. G.L. Downey and J. Dumit (eds), Cyborgs and Citadels: Anthropological Interventions in Emerging Sciences and Technologies, pages 117–142, 1997. [5] G. Downey and J. Lucena. When students resist: Ethnography of a senior design experience in engineering education. International Journal of Engineering Education, 19(1):168–176, 2003. [6] O. Eris, H. Chen, T. Bailey, K. Engerman, H.G. Loshbaugh, A. Griffin, G. Lichenstein, and Cole. A development of the persistence in engineering (PIE) survey instrument. In Proceedings of ASEE conference, 2005. [7] O. Eris, H. Chen, T. Bailey, K. Engerman, H.G. Loshbaugh, A. Griffin, G. Lichenstein, and Cole. A preliminary analysis of correlates of engineering persistence: Results form a longitudinal study. In Proceedings of ASEE conference, Oahu, Hawaii, 2007. [8] Extraordinary Women Engineers Coalition (EWEC). women engineers final report, 2005. Extraordinary [9] Sally Fincher and Josh Tenenberg. Using theory to inform capacitybuilding: Bootstrapping communities of practice in computer science education research. Journal of Engineering Education, 95(4):265–278, 2006. [10] M.A. Hutchinson, D.K. Follman, and M. Sumpter. Factors influencing the self-efficacy beliefs of first-year engineering students. Journal of Engineering Education, 95(1):39–47, 2006. [11] D. Jonassen, J. Strobel, and C. Beng Lee. Everyday problem solving in engineering: Lessons for engineering education. Journal of Engineering Education, 95:139–151, April 2006. [12] T.C. Lethbridge. What knowledge is important to software professionals? IEEE Computer, 33(5):44–50, 2000. [13] E. Seymour and N. Hewitt. Talking about Leaving. Westview Press, 1997. [14] J. Turns, J. Temple, and C. J. Atman. Students¿ conceptions of their engineering discipline: A word association study. In Proceedings of ASEE conference, 2000. [15] J. Walther and D. Radcliffe. Engineering education: Targeted learning outcomes or accidental competencies? In Proceedings of ASEE Conference, June, Chicago, 2006. 12