Cognitive engagement in the problem-based learning classroom

Manuscript Enkem"jgtg"vq"fqypnqcf"Ocpwuetkrv<"T(U"4232"Tgxkukqpu0fqe Enkem"jgtg"vq"xkgy"nkpmgf"Tghgtgpegu COGNITIVE ENGAGEMENT IN PBL Running head: COGNITIVE ENGAGEMENT AND ACHIEVEMENT Cognitive Engagement in the Problem-Based Learning Classroom Jerome I. Rotgans* Republic Polytechnic, Singapore Henk G. Schmidt Erasmus University Rotterdam, The Netherlands *Republic Polytechnic, Centre for Educational Development, 9 Woodlands Avenue 9, Singapore 738964. Phone: +65 91725213, Fax: +65 64151310, e-mail: [email protected] 1 COGNITIVE ENGAGEMENT IN PBL Abstract The objective of the present study was to examine to what extent autonomy in problem-based learning (PBL) results in cognitive engagement with the topic at hand. To that end, a short self-report instrument was devised and validated. Moreover, it was examined how cognitive engagement develops as a function of the learning process and the extent to which cognitive engagement determines subsequent levels of cognitive engagement during a one-day PBL event. Data were analyzed by means of confirmatory factor analysis, repeated measures ANOVA, and path analysis. The results showed that the new measure of situational cognitive engagement is valid and reliable. Furthermore, the results revealed that students’ cognitive engagement significantly increased as a function of the learning event. Implications of these findings for PBL are discussed. Keywords: Autonomy; cognitive engagement; confirmatory factor analysis; path analysis; problem-based learning. 2 COGNITIVE ENGAGEMENT IN PBL Cognitive Engagement in the Problem-Based Learning Classroom Cognitive engagement in the classroom can be characterized as a psychological state in which students put in a lot of effort to truly understand a topic and in which students persist studying over a long period of time. The present study is about this kind of cognitive engagement and how it emerges in the problem-based learning (PBL) classroom. PBL is an approach to higher education that has the following characteristics. Small groups of students discuss a problem guided by a tutor. Based on the discussion about the problem, students generate learning goals for subsequent self-directed learning. As such, students have a choice in deciding which learning goals they would pursue in order to adequately deal with the problem. After a period of self-directed learning, students share what they have learned about the topic and test whether their new understanding of the problem is now more accurate and elaborate than before. Once students are satisfied with their learning outcomes, they engage with a new problem and the cycle starts all over again (Hmelo-Silver, 2004; Schmidt, 1993). PBL can be interpreted as a form of cognitive-constructivist learning, based on at least three assumptions (Schmidt, Van der Molen, Te Winkel, & Wijnen, 2009). The first assumption is that in PBL students engage in theory construction. With the help of their peers they develop an initial theory about the phenomena described in the problem. Subsequently, self-directed learning activities (e.g. reading books or consulting internet resources) serve to test the initial theory against the literature thereby elaborating and changing and deepening their understanding of the topic. The second assumption is that the use of authentic problems or real-life problems encourage students to become interested in the topic at hand and as a consequence helps them gaining a deeper understanding of the principles or processes underlying 3 COGNITIVE ENGAGEMENT IN PBL the problem. The third assumption is that being in the position to identify one’s own learning goals in collaboration with peers fosters a feeling of autonomy, agency, and empowerment. Being autonomous from the direct intervention of a teacher and feeling in charge of one’s own learning is supposed to result in increased cognitive engagement with the topic to be learned, which eventually encourages deeper understanding of it. There is some empirical evidence suggesting that what students do in the tutorial group is indeed attempting to construct a mental model or “theory” that explains the phenomena described in the problem. For instance, De Grave, Schmidt, and Boshuizen (2001) showed students videos of their own contributions to a tutorial discussion and asked them to recall what they were thinking. This stimulated recall procedure in combination with verbatim transcripts of the verbal interaction in the group suggested that indeed theory building, and to a lesser extent, data exploration and hypothesis evaluation were central to the thoughts and verbal utterances of the students. Support for the second assumption, i.e. that the authentic character of the problems results in higher levels of student interest, can be found in a study by Rotgans, Lai, Ong, Choo and Schmidt (2010). In their study they examined whether there are differences in students’ interest between a problem-based learning and a conventional, direct-instruction, primary school mathematics classroom. The results showed that the problem-based group, which worked on an authentic problem, was significantly more interested in a particular subject than the direct-instruction group that worked on more abstract mathematical examples and definitions of the same subject. 4 COGNITIVE ENGAGEMENT IN PBL The third assumption, namely that autonomy in learning leads to more extensive cognitive engagement, has not been studied to the same extent as the previous ones. Following suggestions in the self-determination literature, Deci (1992) has proposed that classrooms that promote student autonomy and choice increase student’s engagement with the task at hand (see also Cordova & Lepper, 1996). Deci, Vallerand, Pelletier, and Ryan (1991) pointed out that choice has a positive effect on interest and engagement because people have an innate psychological need for competence, belonging, and autonomy. In self-determination research, having a choice is a means to satisfy that need for autonomy. In the PBL classroom students have the choice to determine what they wish to study (i.e. select their own learning goals and conduct their individual self-study), which, according to self-determination theory, should lead to a feeling of autonomy. Feeling autonomous and empowered in the classroom is expected to have a motivating effect encouraging students to engage themselves cognitively with the task at hand. Following this line of thought, we hypothesized that when students feel autonomous (from the tutor and the team members) they would display more cognitive engagement with the task. We expected that this would most likely happen when students are in charge of their own learning during individual self-study. Cognitive engagement is defined as the extent to which students’ are willing and able to take on the learning task at hand. This includes the amount of effort students are willing to invest in working on the task (Corno & Mandinach, 1983), and how long they persist (Richardson & Newby, 2006; Walker, Greene, & Mansell, 2006). Cognitive engagement has traditionally been operationalized by measuring the extent of students’ homework completion, class attendance, extra-curricular participation in activities, or their general interactions with the teachers, and how 5 COGNITIVE ENGAGEMENT IN PBL motivated they seem while engaging in classroom discussions (Appleton, Christenson, Kim, & Reschly, 2006). This description of cognitive engagement suggests that it is considered by most authors a more or less stable trait of students, independent of the context. We suggest that cognitive engagement is more or less dependent on the task at hand because the task determines the extent of students’ autonomy. For instance, working with groups and engaging in discussions, searching for information on the internet, or listening to a lecture is likely to result in different levels of cognitive engagement because of different levels of autonomy. Listening to a lecture is arguably the least cognitively engaging since under such circumstances there is little to no student autonomy. On the other hand, when students independently search for information on the internet – that is, when students engage in self-initiated information-seeking behaviors – the level of autonomy should be relatively high and thus lead to more cognitive engagement. Working in groups and engaging in discussions could result in either high or low feelings of autonomy, depending on the group dynamics. For example, if there are domineering peers in the group, a student may feel less autonomous and engages less cognitively as opposed to a group that works well together. In short, we suggest that the level of autonomy is inherently related to an activity or task and largely determines the degree to which students engage cognitively with that activity or task. As a consequence, if the task parameters change during a learning event – as it is the case in PBL – one would expect that students perceive different levels of autonomy and consequently engage differently. For instance, in PBL during the initial phase of defining the problem, students have to work in teams under the guidance of a tutor. During this phase one could expect that students’ autonomy would be relatively low. However after this, students undertake independent self-study, during which one 6 COGNITIVE ENGAGEMENT IN PBL could expect that their autonomy would be relatively higher and thus they would be more cognitively engaged. After this, students converge and share their findings, which would result again in a decrease of autonomy and cognitive engagement. To test whether this is indeed the case, one has to be able to measure changes in cognitive engagement over a learning event. Existing instruments are designed to measure generally more stable, trail-like cognitive engagement, which means that the grain size of these instruments is too large to pick up fairly small contextual variations. For instance, Appleton and colleagues developed the Student Engagement Instrument or SEI, (Appleton, et al., 2006) to measure students’ cognitive engagement (perceived relevance of school) and psychological engagement (perceived connection with others at school). Although the SEI goes beyond the conventionally used broad indictors of engagement, such as homework completion, attendance, and extracurricular participation, the instrument’s grain size is still too large to adequately measure contextual changes in engagement over a learning event. This becomes apparent when examining the items of the SEI, which are rather broad and related to school engagement in general (e.g. “I enjoy talking to the teachers here”, “Most of what is important to know you learn in school”, “Learning is fun because I get better at something”, “What I’m learning in my class will be important in my future”, and “I feel like I have a say about what happens to me at school”). The same applies to a study conducted by Ahlfeldt, Mehta, and Sellnow (2005) in which they used various elements of the National Survey of Student Engagement (Carini, Kuh, & Klein, 2006; Kuh, 2001) to measure student engagement in the classroom. Although the authors only selected items from the National Survey of Student Engagement that in their view measured student engagement at the classroom level with relation to collaborative learning, cognitive development, and 7 COGNITIVE ENGAGEMENT IN PBL personal skills development, the grain-size of the items is again too large to pick up context-dependent variations during the lesson itself (e.g. “I worked with classmates outside of class to complete class assignments”, “To what extent has this course emphasized the mental activities listed below?” “Memorizing facts, ideas or methods from your course and readings so you can repeat them in almost the same form”) Due to the absence of more specific cognitive engagement measures, researchers also resorted to using scales and sub-scales from existing instruments, which to some degree resemble cognitive engagement. In numerous studies, researchers adapted self-regulated learning strategy scales (e.g. DeBacker & Crowson, 2006; Dupeyrat & Mariné, 2005; Meece, Blumenfeld, & Hoyle, 1988; Metallidou & Vlachou, 2007). For instance, Metallidou and Vlachou, (2007) used various self-regulatory learning sub-scales from the Motivated Strategies of Learning Questionnaire (Pintrich, Smith, Garcia, & McKeachie, 1991) such as rehearsal, elaboration, and organizational strategies, as a measure of students’ cognitive engagement. Although we agree that learning strategies manifest themselves in different forms of cognitive engagement, we have reservations whether it is admissible to simply rename self-regulated learning constructs and use them as measures of cognitive engagement. Considering that there seem to be no suitable instruments available to fit the objectives of the present study to measure cognitive engagement as it happens in the classroom (in real time), we saw the need to construct and validate a short self-report questionnaire to determine students’ “situational cognitive engagement”. To differentiate this new measure from more general traditional measures of cognitive engagement, as discussed above, we added the designation “situational” to stress the contextual dependence of this measure. The new situational cognitive engagement 8 COGNITIVE ENGAGEMENT IN PBL measure is composed of three overlapping facets: (1) how students perceive their present engagement with the task, (2) how they rate their effort and persistence while working on the task, and (3) how much they feel absorbed by the learning task, for instance, whether it makes them forget everything around them. It is important to note that all three facets measure ongoing cognitive engagement and try to capture the activity of being engaged. This is conceptually different from existing measures of cognitive engagement, which are typically administrated at the end of the course or a semester and require students to make summative judgments of how engaged they generally were during a particular course spanning several weeks. To capture the dynamic aspect of engagement during class, we generated items that measure students’ effort (Blumenfeld, Kempler, & Krajcik, 2006; Corno & Mandinach, 1983; Volet, 1997; Wolters, 1999) and how willing they are to persist on the task at hand (Ainley, Hidi, & Berndorff, 2002; Pintrich & De Groot, 1990; Prenzel, 1992; Richardson & Newby, 2006; Walker, et al., 2006). As an ultimate form of being engaged in learning we added an item measuring flow, that is, being fully emerged in learning and forgetting everything around oneself (Csikszentmihalyi, 1975; Csikszentmihalyi & Csikszentmihalyi, 1988). The first objective of the present study was to establish the construct validity of this new measure (Study 1). To test whether students’ cognitive engagement changes during PBL, we administered the situational cognitive engagement measure five times during a one-day PBL event at a polytechnic in Singapore (Study 2). Eleven applied-science classes participated in the study. Students worked on one problem during the course of one day. At the polytechnic the PBL day is divided into five distinct learning phases: (1) the problem definition phase (i.e. initial theory construction and identification of learning goals); (2) Initial self-study (i.e. students 9 COGNITIVE ENGAGEMENT IN PBL 10 do a preliminary search, or self-study, to verify their hypothesized theory to explain the problem and to verify the adequacy of identified learning goals); (3) Initial findings sharing phase (i.e. student share the insights gained during the preliminary self-study and the tutor may contribute by asking questions to help students further structure their learning goals); (4) Self-study phase (i.e. student engage in further individual self-study to look for answers to the identified learning goals); and (5) Presentation and elaboration phase (i.e. student share their insights gained during selfstudy, synthesize their findings, and evaluate whether they have addressed all learning goals adequately). After each phase we administered the situational cognitive engagement measure. We expected the following pattern to emerge: During the phases in which students come together as a team, their cognitive engagement would be fairly low because contributions of peers and their tutor may constrain the level of autonomy experienced. On the other hand, if students do individual self-study, their autonomy would be higher and so does their level of cognitive engagement. Therefore, we expected to observe a wave-like pattern for cognitive engagement during the day; depending on the particular phase of the learning process engagement would start at a fairly low level, subsequently increase and then decrease, followed by another wave of increase and decrease. Study 1 The objective of the first study was to establish the reliability and construct validity of a short measure of situational cognitive engagement. To that end, we administered a four item self-report measure after students had completed the first phase of the PBL-cycle: the problem-definition phase. The construct validity was established by means of confirmatory factor analysis of two samples: (1) an exploration sample, and (2) a cross-validation sample. This approach is in line with COGNITIVE ENGAGEMENT IN PBL 11 common practices in SEM (Byrne, 2001). We computed the reliability of the scale by calculating the coefficient H (Hancock & Mueller, 2001). Method Participants Two samples were used in the validation study: (1) a smaller exploration sample (N = 61) for instrument construction and (2) a larger confirmation sample (N = 312) for cross-validation of the instrument. The average age of the participants for the exploration sample (62% female) was 20 years (SD = .98). The average age of the participants for the cross-validation sample (52% female) was 20 years (SD = 1.45). All students were enrolled in science-related modules at a polytechnic in Singapore. Educational context In this polytechnic, the instructional method is problem-based learning (PBL) for all its modules and programs. In this approach five students work together in one team under the guidance of a tutor. Each class comprises four to five teams. Unique to this polytechnic’s approach to PBL is that students work on one problem during the course of each day (Alwis & O'Grady, 2002). This means that students deal with one problem each day in all modules. A typical day starts with the presentation of a problem. Students discuss in their teams what they know, do not know, and what they need to find out. By doing so, students activate their prior knowledge, come up with tentative explanations for the problem, and formulate their own learning goals. Subsequently, a period of self-study follows in which students individually and collaboratively try to find information to address the learning goals (Hmelo-Silver, 2004; Schmidt, 1983, 1993; Schmidt, et al., 2009). At the end of the day the five teams come together to present, elaborate, and synthesize their findings. Measure COGNITIVE ENGAGEMENT IN PBL 12 A measure of students’ situational cognitive engagement was devised which consisted of three elements, measured by four items: (1) engagement with the task at hand (item: “I was engaged with the topic at hand”), (2) effort and persistence (item: “I put in a lot of effort”; “I wish we could still continue with the work for a while”), and (3) experience of flow or having been totally absorbed by the activity (item: “I was so involved that I forgot everything around me”) (Csikszentmihalyi, 1975; Krapp & Lewalter, 2001; Prenzel, 1992; Schraw, Flowerday, & Lehman, 2001). The items were scored on a 5-point Likert scale: 1 (not true at al for me), 2 (not true for me), 3 (neutral), 4 (true for me), and 5 (very true for me). Procedure The situational cognitive engagement measure was administered after the problem-definition phase, that is, after students had engaged in theory construction and after they had generated learning goals. The problem-definition phase took about 20 minutes. Responding to the questionnaire took about 20 seconds. After we had completed the validation with the exploration data, we repeated the validation procedure for a different sample, the cross-validation sample. Cross-validation is a necessary step in establishing the construct validity of new measure because it allows the researcher to test how stable the measure is in different contexts and samples. Analysis The construct validity of the situational cognitive engagement measure was established by means of confirmatory factor analysis (Byrne, 2001). With the present study we did not intend to study consequential or predictive validity. The assumption was that all four items were manifestations of one underlying factor. Parameter estimates were generated using maximum likelihood and tests of goodness of fit. Chisquare accompanied by degrees of freedom, sample size, p-value and the root mean COGNITIVE ENGAGEMENT IN PBL 13 square error of approximation (RMSEA) were used as indices of absolute fit between the models and the data. The Chi-square is a statistical measure to test the closeness of fit between the observed and predicted covariance matrix. A small Chi-square value, relative to the degrees of freedom, indicates a good fit (Byrne, 2001). A Chisquare/df ratio of less than 3 is considered to be indicative of a good fit. RMSEA is sensitive to model specification and is minimally influenced by sample size and not overly affected by estimation method (Fan, Thompson, & Wang, 1999). The lower the RMSEA value, the better the fit. A commonly reported cut-off value is .06 (Hu & Bentler, 1999). In addition to these absolute fit indices, the comparative fit index (CFI) was calculated. The CFI value ranges from zero to one and a value greater than .95 is conventionally considered a good model fit (Bentler, 1990; Byrne, 2001).The reliability of the measure was determined by calculating Hancock’s coefficient H. The coefficient H is a construct reliability measure for latent variable systems that represents a relevant alternative to the conventional Cronbach’s alpha. According to Hancock and Mueller (2001) the usefulness of Cronbach’s alpha and related reliability measures is limited to assessing composite scales formed from a construct’s indicators, rather than assessing the reliability of the latent construct itself as reflected by its indicators. The coefficient H is the squared correlation between a latent construct and the optimum linear composite formed by its indicators. Unlike other reliability measures the coefficient H is never less than the best indicator’s reliability. In other words, a factor inferred from multiple indicator variables should never be less reliable than the best single indicator alone. Hancock recommended a cut-off value for the coefficient H of .70. Results and Discussion COGNITIVE ENGAGEMENT IN PBL 14 First, the exploration sample was analyzed. The model fit statistics showed that the data fitted the hypothesized model very well. The Chi-square/df ratio was .17, p = .84, CFI = 1.00 and RMSEA = .00. All factor loadings were statistically significant and ranged from .51 to .96, with an average of .70. See Table 1 for an overview of the results. ---------------------------------------------Insert Table 1 about here ---------------------------------------------We then cross-validated the model with a larger sample. The results revealed that the data fitted the model equally well. The Chi-square/df ratio was .02, p = .94, CFI = 1.00 and RMSEA = .00. All factor loadings were statistically significant and ranged from .38 to .85, with an average of .58. The reliability for both samples was determined by calculating Hancock’s coefficient H. The coefficient H for the exploration sample was .93 and for the crossvalidation sample .78. Overall, the results demonstrate that for both independent samples the psychometric characteristics of the situational cognitive engagement measure are adequate. Study 2 The first objective of Study 2 was to examine the degree to which students are cognitively engaged during PBL across the five PBL phases: (1) the problem definition phase; (2) Initial self-study; (3) Initial findings sharing phase; (4) Self-study phase; and (5) Presentation and elaboration phase. We expected that cognitive engagement during the first phase would be relatively low; then it would increase during the first self-study phase; decrease again during group discussion; increase during the longer self-study phase; and eventually decrease again during the COGNITIVE ENGAGEMENT IN PBL 15 presentation and elaboration phase. This hypothesis was based on the assumption that when students come together in the team and with the tutor, their autonomy would be relatively lower because of the constraints on choice provided by group and tutor. Under this hypothesis, during self-directed learning their level of cognitive engagement would be relatively higher. The second objective of this study was to examine the extent to which situational cognitive engagement in one phase determines a students’ cognitive engagement during a next phase. We hoped to find answers to the questions: does it matter how cognitively engaged student are during the problem definition phase in predicting their subsequent engagement during self-study? Or, does the initial selfstudy phase predict students’ cognitive engagement during the second, more elaborate self-study phase? Method Participants The sample consisted of 208 participants (51% female) from an applied science module at the same polytechnic as in Study 1. The participants’ average age was 20 years (SD = 1.45). Measures Situational Cognitive Engagement. The measure for students’ situational cognitive engagement validated in Study 1 was administered for this study. All items were scored on a 5-point Likert scale: 1 (not true at all), 2 (not true for me), 3 (neutral), 4 (true for me), and 5 (very true for me). The coefficient H for all five cognitive engagement measurements ranged from .70 to .79 (average = .77). COGNITIVE ENGAGEMENT IN PBL 16 Academic Achievement. Academic achievement was determined by means of students’ course grades. The course grades were based on the results of written achievement tests and class performance. Procedure The situational cognitive engagement measure was administered during the five phases in the one-day PBL process (see Figure 1 for an overview of the administration during the day). ---------------------------------------------Insert Figure 1 about here ---------------------------------------------The first administration took place after the problem-definition phase in which they generated learning goals. The second administration was after the initial selfstudy phase. In this phase students did an initial search in the learning resources or the internet to verify whether their initial theories about the problem were correct. The students then converged and had a quick discussion with the group to share their initial findings. A tutor was present during this session and engaged in questioning the students about their findings and whether their learning goals adequately address the problem. Subsequently, the situational cognitive engagement measure was administered for the third time. Student then went out for two-hour self-study after which the fourth situational cognitive engagement measure was administered. Students shared their findings and engaged in elaboration about the problem and whether they have adequately addressed all learning goals. After this, the engagement measure was administered for the fifth and last time. Analysis COGNITIVE ENGAGEMENT IN PBL 17 As a first step in the analysis, we generated zero-order correlations to inspect how the five repeated measurements of situational cognitive engagement are related. Moreover, we calculated mean values of all five measurements. Potential mean level differences between the measurements were determined by means of a one-way repeated measures ANOVA with LSD comparisons of the means. Subsequently, the relationships between the five situational cognitive engagement measurement occasions were analyzed using path analysis. In the path analysis, we tested a sequential causal model (see Figure 1). This entails that one measurement leads to the next: situational cognitive engagement measure 1 leads to situational cognitive engagement measure 2, situational cognitive engagement measure 2 leads to 3, and so on. To allow for the possibility that the relationships are not entirely sequential, e.g. early engagement may lead to engagement later on during the day, we first tested an explanatory model which allowed for all possible relationships (see Figure 1, dotted lines). In the final model reported in the Results and Discussion section we only retained the relationships, which were statistically significant; all non-significant relationships were removed. For the model, Chi-square accompanied with degrees of freedom, p-value, and the root mean square error of approximation (RMSEA) were used as indices of absolute fit between the models and the data. Results and Discussion First, intercorrelations between the five measurement occasions were calculated (see Table 2). All correlation coefficients were statistically significant and were medium to strong, ranging from .15 to .88, suggesting that the level of cognitive engagement in one situation is associated with another during the one-day PBL process as well as students’ academic achievement (i.e. course grades at the end of the semester). COGNITIVE ENGAGEMENT IN PBL 18 ---------------------------------------------Insert Table 2 about here ---------------------------------------------The repeated measures one-way ANOVA revealed that there were significant differences between the five situational cognitive engagement measurements in absolute sense: F (4,207) = 47.53, p < .01 (eta-squared = .19). The pairwise LSD comparisons revealed that the first two measurements were not significantly different (see Figure 2: M(1) = 3.33 vs. M(2) = 3.37, p =.24). ---------------------------------------------Insert Figure 2 about here ---------------------------------------------This outcome suggests that there is not a significant difference in the cognitive engagement phases when students generate theories and learning goals and do an initial search in the resources to verify their initial theories to explain what is going on in the problem. However, after that, when the students come together and discuss what they have found during the initial self-study phase, their level of cognitive engagement increased significantly (M(1) = 3.33 vs. M(3) = 3.59, p < .01 and M(2) = 3.37 vs. M(3) = 3.59, p < .01). The results suggest that students’ level of cognitive engagement increases not so much during the initial self-study phase, as we had expected, but when they come together and share their findings with each other and the tutor. After this, students engaged in self-study leading to another significant increase in cognitive engagement (M(3) = 3.59 vs. M(4) = 3.66, p = .01). After selfstudy students seemed to have reached a peak in cognitive engagement; during the presentation and elaboration phase their levels of cognitive engagement remained the same (M(4) = 3.66 and M(5) = 3.66, p = .99). Overall, the data suggests that during COGNITIVE ENGAGEMENT IN PBL 19 the PBL day cognitive engagement increases gradually and does not develop in a wave-like pattern as we had hypothesized. As a next step in the analysis we investigated how the five cognitive engagement measurements were related to each other. For instance, does one level of cognitive engagement influences the next or does an earlier measurement predict a measurement during a later phase of the learning process. Figure 3 depicts the model with the path coefficients indicating the strength of relation between the measurements. The model fitted the data well: Chi-square/df ratio = 2.06, p = .10, CFI = 1.00, and RMSEA = .07. ---------------------------------------------Insert Figure 3 about here ---------------------------------------------The results revealed that the cognitive engagement measurements are strongly related to each adjacent measure in time. That is, if a student is cognitively engaged during the problem definition phases he or she is likely to be engaged during the next phases as well. There were also some weaker non-sequential relationships, for instance, between the first measure and the second and the second and the fourth and fifth. Overall, 81% of the variance in the last situational cognitive engagement measure could be explained by the preceding ones. General Discussion The objective of the present study was to examine the underlying assumption that students in PBL have a large degree of autonomy (i.e. when students engage in individual self-study), which is expected to result in cognitive engagement with the topic at hand. We hypothesized that when students experience a feeling of being autonomous from the tutor or the group, they would engage more cognitively with the COGNITIVE ENGAGEMENT IN PBL 20 problem (Deci, 1992; Deci & Ryan, 2004; Deci, et al., 1991). To test the extent to which this is the case we first devised and validated a short self-report instrument to measure students’ situational cognitive engagement in the classroom. Subsequently, we examined how situational cognitive engagement develops as a function of the learning process in PBL and the extent to which situational cognitive engagement during the learning process determines subsequent levels of cognitive engagement. Data were collected from applied-science courses at a polytechnic in Singapore. The results of the construct validation and cross-validation study suggest that the four-item instrument is a reliable and valid measure to determine students’ situational cognitive engagement in the classroom. As such, we used it for the subsequent analyses. Following Self-determination theory (Deci & Ryan, 2004) under which autonomy is defined as the degree to which individuals feel volitional and responsible for their initiation of their behavior (Williams, 2004), we hypothesized that students would have the highest feelings of autonomy (and thus would engage more) during self-study because during self-study they are expected to feel most volitional in their actions and are most responsible for their learning (Assor, Kaplan, & Roth, 2002; Deci, 1992; Flowerday & Schraw, 2003; Ryan & Deci, 2000). Following this point of view we expected that when students meet with the group and the tutor their feeling of autonomy would be reduced (relative to self-study), leading to less cognitive engagement with the problem. Because students first meet with the group, then break away for initial self-study, meet again with the group and the tutor, break away for self-study, and finally meet again with the group and the tutor, we expected to observe a wave-like pattern of cognitive engagement to emerge during the one-day PBL event. COGNITIVE ENGAGEMENT IN PBL 21 The results of our analyses did however not support this hypothesis. Students’ cognitive engagement did not progress in a wave-like pattern, but it increased significantly and consistently over the day. Strongest evidence against our hypothesis is that students’ situational cognitive engagement increased significantly not during the first self-study phase, but when students met with the group thereafter to discuss their finings. Situational cognitive engagement increased significantly again during the second self-study phase. Our data suggest that students’ situational cognitive engagement is not influenced by changes in task demand and associated feelings of autonomy, but situational cognitive engagement is more a function of the learning event itself: if students progress with their learning in PBL, their situational cognitive engagement increases. Considering this outcome, we offer an alternative hypothesis. We propose that students’ feelings of autonomy and situational cognitive engagement are a direct function of a students’ knowledge construction. During the early stages of the problem day (i.e. during the problem-definition phase), students struggle to come up with adequate theories to explain the phenomena described in the problem. Struggling to explain the problem is expected (and intended) because students lack relevant knowledge as they are supposed to engage in theory construction. At this stage students largely depend on the elaborations with the other team members and the questioning of the tutor. As such, choice and autonomy are expected to be generally low. However, as students gain a deeper understanding of the topic, they gradually depend less on the support of their peers and the tutor, because they have gained more knowledge to direct their own learning. With increasing knowledge, the knowledge of possible (learning) choices also increases, which translates into a feeling of autonomy and consequently higher levels of situational cognitive engagement. COGNITIVE ENGAGEMENT IN PBL 22 In conclusion, we propose that indeed, autonomy plays a significant role in students’ situational cognitive engagement during PBL. However, unlike we hypothesized, students’ autonomy seems less dependent on the phases during which they alternatively meet the group and engage in self-study, but on the knowledge students gain during their learning. The progression of knowledge and understanding of the topic seems to determine students’ autonomy and thus their increased situational cognitive engagement. In simple terms: more knowledge, more autonomy, more self-determination, more situational cognitive engagement. This finding opens up new areas of research for self-determination theory. Reeve (2004) has proposed that there is empirical evidence to support two conclusions about self-determination theory and its significance for education: (1) autonomously-motivated students thrive in educational settings; and (2) students benefit when teachers support their autonomy. Indeed, there is considerable evidence linking these two factors to positive educational outcome, such as higher academic achievement (Boggiano, Flink, Shields, Seelbach, & Barrett, 1993; Miserandino, 1996), higher rates of retention (Vallerand & Blssonnette, 1992; Vallerand, Fortier, & Guay, 1997), higher perceived competence (Ryan & Grolnick, 1986), greater conceptual understanding (Boggiano, et al., 1993; Flink, Boggiano, & Barrett, 1990), greater creativity (Amabile, 1985; Koestner, Ryan, Bernieri, & Holt, 1984), and higher self-esteem (Deci, Nezlek, & Sheinman, 1981; Ryan & Deci, 2000). However, largely missing from the current research agenda is the consideration of the significant role knowledge may play in autonomy and autonomy-supportive behavior. Needless to say, further research needs to be carried out to empirically test whether and how autonomy and knowledge (development) are interlinked. We suggest a fruitful approach would be to include, besides the situational cognitive COGNITIVE ENGAGEMENT IN PBL 23 engagement measure, a measure of students’ autonomy and their factual knowledge in the investigation to examine how these three factors are related and how they develop during the stages of student learning in PBL. COGNITIVE ENGAGEMENT IN PBL 24 References Ahlfeldt, S., Mehta, S., & Sellnow, T. (2005). Measurement and analysis of student engagement in university classes where varying levels of PBL methods of instruction are in use. Higher Education Research & Development, 24(1), 520. Ainley, M., Hidi, S., & Berndorff, D. (2002). Interest, learning, and the psychological processes that mediate their relationship. Journal of Educational Psychology, 94(3), 545-561. Alwis, W. A. M., & O'Grady, G. (2002). One day-one problem at Republic Polytechnic. Paper presented at the 4th Asia-Pacific Conference on PBL. Amabile, T. M. (1985). Motivation and creativity: Effects of motivational orientation on creative writers. Journal of Personality and Social Psychology, 48(2), 393399. Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44(5), 427-445. Assor, A., Kaplan, H., & Roth, G. (2002). Choice is good, but relevance is excellent: Autonomy-enhancing and suppressing teacher behaviours predicting students' engagement in schoolwork. British Journal of Educational Psychology, 72(2), 261-278. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238-246. Blumenfeld, P. C., Kempler, T. M., & Krajcik, J. S. (2006). Motivation and cognitive engagement in learning environments. In R. K. Sawyer (Ed.), The Cambridge handbook of the Learning Sciences. New York: Cambridge University Press. COGNITIVE ENGAGEMENT IN PBL 25 Boggiano, A. K., Flink, C., Shields, A., Seelbach, A., & Barrett, M. (1993). Use of techniques promoting students' self-determination: Effects on students' analytic problem-solving skills. Motivation and Emotion, 17(4), 319-336. Byrne, B. M. (2001). Structural Equation Modeling With Amos: Basic Concepts, Applications and Programming: Lawrence Erlbaum Assoc Inc. Carini, R., Kuh, G., & Klein, S. (2006). Student Engagement and Student Learning: Testing the Linkages. Research in Higher Education, 47(1), 1-32. Cordova, D. I., & Lepper, M. R. (1996). Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. Journal of Educational Psychology, 88(4), 715-730. Corno, L., & Mandinach, E. (1983). The role of cognitive engagement in classroom learning and motivation. Educational Psychologist, 18(2), 88-108. Csikszentmihalyi, M. (1975). Beyond Boredom and Anxiety. San Francisco: JosseyBass. Csikszentmihalyi, M., & Csikszentmihalyi, I. S. (1988). Flow: The psychology of optimal experience. New York: Cmbridge University Press. De Grave, W. S., Schmidt, H. G., & Boshuizen, H. P. A. (2001). Effects of problembased discussion on studying a subsequent text: A randomized trial among first year medical students. Instructional Science, 29(1), 33-44. DeBacker, T. K., & Crowson, M. (2006). Influences on cognitive engagement: Epistemological beliefs and need for closure. British Journal of Educational Psychology, 76(3), 535-551. Deci, E. L. (1992). The relation of interest to the motivation of behavior: A selfdetermination theory perspective. In K. A. Renninger, S. Hidi & A. Krapp COGNITIVE ENGAGEMENT IN PBL 26 (Eds.), The role of interest in learning and development (pp. 43-70). Hillsdale, NJ: Lawrence Erlbaum Associates. Deci, E. L., Nezlek, J., & Sheinman, L. (1981). Characteristics of the rewarder and intrinsic motivation of the rewardee. Journal of Personality and Social Psychology, 40(1), 1-10. Deci, E. L., & Ryan, R. M. (2004). Handbook of self-determination research: University of Rochester Press. Deci, E. L., Vallerand, R. J., Pelletier, L. G., & Ryan, R. M. (1991). Motivation and education: The self-determination perspective. Educational Psychologist, 26(3&4), 325-346. Dupeyrat, C., & Mariné, C. (2005). Implicit theories of intelligence, goal orientation, cognitive engagement, and achievement: A test of Dweck’s model with returning to school adults. Contemporary Educational Psychology, 30(1), 4359. Fan, X., Thompson, B., & Wang, L. (1999). Effects of sample size, estimation methods, and model specification on structural equation. Structural Equation Modeling, 6(1), 56-83. Flink, C., Boggiano, A. K., & Barrett, M. (1990). Controlling teaching strategies: Undermining children's self-determination and performance. Journal of Personality and Social Psychology, 59(5), 916-924. Flowerday, T., & Schraw, G. (2003). Effect of choice on cognitive and affective engagement. Journal of Educational Research, 96(4), 207-215. Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent systems. In R. Cudeck, S. d. Toit & D. Sörbom (Eds.), Structural COGNITIVE ENGAGEMENT IN PBL 27 equation modeling: Present and future - A festschrift in honor of Karl Jöreskog (pp. 195-216). Lincolnwood, IL: Scientific Software International. Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235-266. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55. Koestner, R., Ryan, R. M., Bernieri, F., & Holt, K. (1984). Setting limits on children's behavior: The differential effects of controlling vs. informational styles on intrinsic motivation and creativity. Journal of Personality, 52(3), 233-248. Krapp, A., & Lewalter, D. (Eds.). (2001). Development of interests and interest-based motivational orientations: A longitudinal study in vocational school and work settings. Amsterdam: Pergamon. Kuh, G. (2001). The National Survey of Student Engagement: Conceptual framework and overview of psychometric properties. Indiana University Center for Postsecondary Research & Planning, IN: Bloomington. Meece, J., Blumenfeld, P. C., & Hoyle, R. H. (1988). Students' Goal Orientations and Cognitive Engagement in Classroom Activities. Journal of Educational Psychology, 80(4), 514-523. Metallidou, P., & Vlachou, A. (2007). Motivational beliefs, cognitive engagement, and achievement in language and mathematics in elementary school children. International Journal of Psychology, 42(1), 2-15. Miserandino, M. (1996). Children Who Do Well in School: Individual Differences in Perceived Competence and Autonomy in Above-Average Children. Journal of Educational Psychology, 88(2), 203-214. COGNITIVE ENGAGEMENT IN PBL 28 Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33-40. Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A Manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor, MI: National Center for Research to Improve Postsecondary Teaching and Learning. Prenzel, M. (1992). The selective persistence of interest. In K. A. Renninger, S. Hidi & A. Krapp (Eds.), The role of interest in learning and development (pp. 7198). Hillsdale, NJ: Lawrence Erlbaum Associates. Reeve, J. (2004). Self-determination theory applied to educational settings. In E. L. Deci & R. M. Ryan (Eds.), Handbook of self-determination research. Rochester: University of Rochester Press. Richardson, J. C., & Newby, T. (2006). The Role of Students Cognitive Engagement in Online Learning. American Journal of Distance Education, 20(1), 23-37. Rotgans, J. I., Lai, K. C., Ong, H. L. C., Choo, H. K., & Schmidt, H. G. (2010). Situational Interest in Mathematics: A Microanalytical Comparison of Problem-Based Learning vs. Direct Instruction. Instructional Science, Submitted. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68-78. Ryan, R. M., & Grolnick, W. S. (1986). Origins and pawns in the classroom: Selfreport and projective assessments of individual differences in children's perceptions. Journal of Personality and Social Psychology, 50(3), 550-558. COGNITIVE ENGAGEMENT IN PBL 29 Schmidt, H. G. (1983). Problem-based learning: Rationale and description. Medical Education, 17(1), 11-16. Schmidt, H. G. (1993). Foundations of problem-based learning: Some explanatory notes. Medical Education, 27(5), 422-432. Schmidt, H. G., Van der Molen, H. T., Te Winkel, W. W. R., & Wijnen, W. H. F. W. (2009). Constructivist, problem-based, learning does work: A meta-analysis of curricular comparisons involving a single medical school. Educational Psychologist, 44(4), 227-249. Schraw, G., Flowerday, T., & Lehman, S. (2001). Increasing situational interest in the classroom. Educational Psychology Review, 13(3), 211-224. Vallerand, R. J., & Blssonnette, R. (1992). Intrinsic, extrinsic, and amotivational styles as predictors of behavior: A prospective study. Journal of Personality, 60(3), 599-620. Vallerand, R. J., Fortier, M. S., & Guay, F. (1997). Self-determination and persistence in a real-life setting: Toward a motivational model of high school dropout. Journal of Personality and Social Psychology, 72(3), 1161-1176. Volet, S. (1997). Cognitive and affective variables in academic learning: The significance of direction and effort in students' goals, Learn. Instr. (Vol. 7, pp. 235). Walker, C. O., Greene, B. A., & Mansell, R. A. (2006). Identification with academics, intrinsic/extrinsic motivation, and self-efficacy as predictors of cognitive engagement. Learning and Individual Differences, 16(1), 1-12. Williams, G. C. (2004). Improving patients' health through supporting the autonomy of pations and providers. In E. L. Deci & R. M. Ryan (Eds.), Handbook of self-determination research. Rochester: University of Rochester Press. COGNITIVE ENGAGEMENT IN PBL 30 Wolters, C. A. (1999). The relation between high school students' motivational regulation and their use of learning strategies, effort, and classroom performance. Learning and Individual Differences, 11(3), 281-299. table1 Enkem"jgtg"vq"fqypnqcf"vcdng<"Vcdng"30fqe COGNITIVE ENGAGEMENT IN PBL Table 1 Model fit Statistics of the Situational Cognitive Engagement Measure for the Exploration and Cross-Validation Samples. Statistics Exploration sample Cross-validation sample Chi-square/df 0.17 0.02 p-value 0.84 0.94 CFI 1.00 1.00 RMSEA 0.00 0.00 .73; .96; .51; .61 .57; .85; .38; .53 0.93 0.78 Standardized betas Coefficient H table2 Enkem"jgtg"vq"fqypnqcf"vcdng<"Vcdng"40fqe COGNITIVE ENGAGEMENT IN PBL Table 2 Intercorrelations Between the Situational Cognitive Engagement Measurements and Academic Achievement During One-day Problem-Based Learning. Measurements (1) Situational Cognitive Engagement Measure 1 (2) Situational Cognitive Engagement Measure 2 (3) Situational Cognitive Engagement Measure 3 (4) Situational Cognitive Engagement Measure 4 (5) Situational Cognitive Engagement Measure 5 (1) (2) (3) (4) (5) (6) M (SD) - .68** .58** .52** .57** .15* 3.33 (.53) - .64** .60** .68** .17* 3.37 (.54) - .79** .78** .25** 3.59 (.61) - .88** .28** 3.66 (.62) - .21** 3.66 (.61) (6) Academic 3.14 (.66) achievement Note: ** statistically significant at the 1% level, * statistically at the 5% level 1 line figure1 Enkem"jgtg"vq"fqypnqcf"nkpg"hkiwtg<"Hkiwtg"30fqe COGNITIVE ENGAGEMENT IN PBL Figure 1: Measurement occasions of the situational cognitive engagement measure during a one-day problem-based learning process. Situational Cognitive Engagement Measure 1 Situational Cognitive Engagement Measure 2 Situational Cognitive Engagement Measure 3 Situational Cognitive Engagement Measure 4 Situational Cognitive Engagement Measure 5 Problemdefinition phase: theory construction, identification learning goals. Initial self-study phase: searching for resources to verify theory and learning goals. Initial findings sharing phase: sharing of initial findings and adjustment learning goals. Self-study phase: individual selfstudy. Presentation and elaboration phase: presentation and elaboration of findings. 1 line figure2 Enkem"jgtg"vq"fqypnqcf"nkpg"hkiwtg<"Hkiwtg"40fqe COGNITIVE ENGAGEMENT IN PBL Figure 2: Mean values of the situational cognitive engagement measurements (SCEM) during a one-day problem-based learning event. 3.7 3.66 3.66 SCEM 4 SCEM 5 3.59 3.6 3.5 3.37 3.4 3.33 3.3 SCEM 1 Problemdefinition phase: theory construction, identification learning goals. SCEM 2 Initial self-study phase: searching for resources to verify theory and learning goals. SCEM 3 Initial findings Self-study phase: sharing phase: individual selfsharing of initial study. findings and adjustment learning goals. Presentation and elaboration phase: presentation and elaboration of findings. line figure3 Enkem"jgtg"vq"fqypnqcf"nkpg"hkiwtg<"Hkiwtg"50fqe COGNITIVE ENGAGEMENT IN PBL Figure 3 Causal relationships between the five measurement occasions of situational cognitive engagement during a one-day PBL event. Note: numbers above the arrows represent standardized regression weights. All standardized regression weights are statistically significant at the 1% level. 1