Reflective Judgment

DOCUMENT RESUME ED 457 747 AUTHOR TITLE PUB DATE NOTE PUB TYPE EDRS PRICE DESCRIPTORS IDENTIFIERS HE 034 400 Bauer, Karen W. The Effect of Participation in Undergraduate Research on Critical Thinking and Reflective Judgment. AIR 2001 Annual Forum Paper. 2001-06-00 22p.; Paper presented at the Annual Meeting of the Association for Institutional Research (41st, Long Beach, CA, June 3-6, 2001). Speeches/Meeting Papers (150) Reports Research (143) MF01/PC01 Plus Postage. *Critical Thinking; Higher Education; *Participation; *Personality Traits; *Research; *Undergraduate Students *Reflective Judgment ABSTRACT This study examined differences in personality type for college freshmen and the effects of participation in undergraduate research on critical thinking and reflective judgment scores. Participants were 266 undergraduate students. While correlations between personality and the thinking measures were low and of little practical significance, a repeated measures analysis revealed a two-way interaction between research participation and major for change in critical thinking score. A second repeated measures analysis with simple contrasts revealed a two-way interaction between research participation and gender for change in reflective judgment score. Findings thus indicate that participation in undergraduate research affects critical thinking and reflective judgment for some students. The paper also discuses implications and limitations of the study. (Contains 5 figures, 4 tables, and 39 references.) (SLD) Reproductions supplied by EDRS are the best that can be made from the original document. The Effect of Participation in Undergraduate Research on Critical Thinking and Reflective Judgment Karen W. Bauer The University of Delaware PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) ler: document has been reproduced as received from the person or organization Vordk originating it. 0 Minor changes have been made to improve reproduction quality. TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. 1 2 REST COPY AVAILABLE The Effect of Participation in Undergraduate Research on Critical Thinking and Reflective Judgment Karen W. Bauer The University of Delaware Abstract The study examined differences in freshman year personality type and the effects of participation in undergraduate research on critical thinking and reflective judgment scores. While correlations between personality and the thinking measures were low and of little practical significance, a repeated measures analysis revealed a two-way interaction between research participation and major for change in critical thinking score. A second repeated measures analysis with simple contrasts revealed a two-way interaction between research participation and gender for change in reflective judgment score. Findings thus indicate that participation in undergraduate research effects critical thinking and reflective judgment for some students. Implications and limitations are discussed. In recent years, there have been numerous calls for focused attention and reform in the research university to bolster student achievement, and in particular, critical thinking skills (National Commission on Excellence in Education, 1983; American Association of Colleges, 1985). The authors of Reinventing Undergraduate Education (Boyer Commission, 1998), urged research universities to "restructure the pedagogical and integrative aspects of the research university experience" (p. 1), in part, by making research-based learning the standard. These notions of inquiry-based learning enable faculty to guide students by mentoring and acknowledge that learning is a reciprocal process whereby both students and faculty learn from each other. Reinventing Undergraduate Education, was designed to be not another document deploring the state of undergraduate education but rather a guide, a "blueprint," for the way forward into integrating research in education; the way toward "reinvention," the report says, is to "make research-based learning the standard" for undergraduate Effect of Research on Critical Thinking and Reflective Judgment 2 education at research universities (Boyer, 1998, pp. 15-16). Historically, research and education have been closely integrated for undergraduates who collaborate with faculty to conduct research, and this practice underlies the "model" that the Boyer Commission proposes, with "scholar-teachers" who "would treat the sites of their research as seminar rooms in which not only graduate students but undergraduates observe and participate in the process of both discovery and communication of knowledge" (Boyer, 1998, p. 18). Among college outcomes, achievement has been one of the most frequently researched topics in higher education (Astin, 1977, 1993). Similarly, critical thinking skills are regarded as one of the major outcomes of college education (Pascarella, 1989; Pascarella & Terenzini, 1991). Pascarella, et al. (1989) found that college experiences were modestly associated with higher critical thinking skills, while the composite college experience scale (e.g., type of course work, non-classroom interaction with faculty, study time, and extra-curricular activities) showed significant partial correlation with overall critical thinking skills. Facione, Sanchez, Facione, and Gainen (1995) also argue that critical thinking dispositions are an important outcome of higher education and The American Association of Colleges and Universities commission report discusses the importance of reasoning about real-world issues as a central goal of higher education in the publication, The Challenge of Connecting Learning: "In the final analysis, the real challenge of college students and faculty members alike is empowering individuals to know that the world is far more complex than it first appears and that they must make interpretive arguments and decision- judgments that entail real consequences for which they must take responsibility and from which they may not flee by disclaiming expertise." (AAC, 1990) Effect of Research on Critical Thinking and Reflective Judgment 3 Cognitive Development Based on Perry's (1970) theory of ethical and intellectual development, ICing and Kitchener (1994) Wood (1997a), Knefelkamp (1974) and Widick (1975), BaxterMagolda (1985), have applied cognitive development theory to college students. In a cross sectional data analysis of The KneWi, a measure of intellectual development, Heidke and Omahan (1981) found seniors to score higher than freshmen. With a semistructured interview, the Measure of Epistemological Reflection (MER), Baxter-Magolda (1987) examined six domains of thinking related to learning. Each domain is measured by a series of questions that follow a general question intended to focus the respondents' thinking on the domain. While Baxter-Magolda (1988) found that both men and women "demonstrate similar cognitive structures regardless of learning style" (p. 533), she also reported (1987; Baxter-Magolda and Porterfield (1985) significant increases in MER score from freshmen to senior years. In a series of studies, Kitchener and King (1981) and colleagues King and Parker (1978, in Welfel, 1982), Lawson (1980), Strange and King (1982) and King, Kitchener, and Wood (1994) have examined the seven-stage theory of reflective judgment. The semi-structured interview, the Reflective Judgment Interview (RJI), and the paper-pencil instrument, Reasoning About Current Issues (RCI; Wood, 1997a) have established a positive correlation between education level and reflective judgment score. Consistent with findings of King, Kitchener, Davison, Parker, and Wood (1983), Welfel and Davison (1986) found a significant increase in reflective judgment stage for a sample of 25 students from the freshman to senior years (about one-half of one stage). Further, Welfel and Davison reported no significant difference in RJI score by gender, or time by 5 Effect of Research on Critical Thinking and Reflective Judgment 4 major interaction. Related to the developmental nature reflective judgment, Welfel (1982) notes two important points: 1). higher stages of reflective judgment cannot be explained by maturation alone; and 2). measures of critical thinking and verbal ability are associated but do not account for differences between groups in reflective judgment. Although research has revealed a significant relationship between education level and reflective judgment score, it is not known which factors within the educational institution influence this relationship. For example, Kitchener and King (1981) suggested that academic major may be related to reflective judgment. Similarly, co-curricular activities such as major-related internships and undergraduate research may be positively affecting one's reflective judgment score. Thus, the value in further investigation of reflective judgment is warranted. Use Of Personality Type to Help Assess Undergraduate Success While personality type has been used to predict occupational performance (Mount & Barrick, 1989; Hough, 1992), the use of personality type to assist in college student success has recently received more attention as an additional factor to help assess collegiate success. Entwistle's (1972) review of studies involving Cattell's 16 Personality Factors and Eysenck's Personality Inventory concluded that college success is associated with introversion, and Entwistle & Entwistle (1970) partly attribute introverts' higher academic achievement to their good study habits. Additionally, Digman & Takemoto-Chock (1981) found that conscientious students are well-organized, purposeful, and persistent, and that these characteristics are highly related to academic achievement (e.g., GPA). In a 45-year longitudinal study of 63 men, Soldz and Vaillant (1999) found that openness to experiences was strongly related to creativity and that Effect of Research on Critical Thinking and Reflective Judgment 5 conscientiousness in college was the best predictor of what happened to men in the future. Tross, Harper, Osher, and Kneidinger (2000) found that conscientiousness was related to retention and contributed significantly to the prediction of college GPA. Participation in Undergraduate Research Student involvement in research is an effective way to heighten inquiry-based learning. This involvement in research typically manifests itself in the colleagial and synergystic relationships that develop between undergraduate student, graduate students, and faculty members. Recognizing this benefit, the National Science Foundation Report (1989) stated "it is clear that the academic community regards the involvement of undergraduate student majors in meaningful research ... with faculty members as one of the most powerful instruction tools" (p. 6). According to Hakim (1998), there are four features inherent in undergraduate research (UR) experiences: mentorship, originality, acceptability, and dissemination. These features assume that students will be involved in meaningful contributions and will be using methods consistent with current practices within the discipline. UR experiences also include interactions between the faculty and student researcher focus on student learning, and experiences that will culminate in a tangible product that is critiqued by others in the discipline. Thus, undergraduate research students get the opportunity to develop both practical and complex thinking and problem-solving, technical competence, and leadership skills. These are important tasks that strongly contribute to the undergraduate experience and successful attainment of the baccalaureate degree. Effect of Research on Critical Thinking and Reflective Judgment 6 Research Questions for Current Study In an attempt to better understand factors that contribute to high quality undergraduate education, this researcher sought to understand the effect of participation in UR on critical thinking and reflective judgment for undergraduates at one Mid-Atlantic doctoral-extensive university. The research questions are: 1) What role does participation in undergraduate research play in the change in critical thinking scores (as measured through the Watson Glaser Critical Thinking Appraisal and the Reasoning About Current Issues test)? 2). Are there differences in these outcomes by major or gender? and 3). Does personality (as measured by NEO-FFI in the freshman year) differ for research versus non-research students? Participants Participants were 266 undergraduate students at a Research-Extensive state university who were part of a four-year longitudinal study funded by the National Science Foundation to the benefits of participation in undergraduate research. Students were enrolled in 47 majors, which were grouped into three categories: sciences/math/nursing; chemistry and engineering; and psychology and other majors. Among them were 155 females (58%) and 111 males (42%), 202 White (76%) and 64 non-Whites (24%). Instruments Information about these students were collected from the university's student records database and by responses obtained from the following three published measures: the Watson-Glaser Critical Thinking Appraisal (WGCTA; Watson & Glaser, 1994), the 8 Effect of Research on Critical Thinking and Reflective Judgment 7 Reasoning About Current Issues Test (Wood, 1987), and the NEO-Five Factor Inventoty (NEO-FFI; Costa, Jr.& McCrae, 1991). Watson Glaser Critical Thinking Appraisal (WGCTA). The WGCTA is a composite measure that examines attitudes of inquiry, knowledge of the nature of valid inferences, abstractions, and generalizations, and skills using attitudes and knowledge (Watson & Glaser, 1994). The WGCTA Form S consists of 40 items measuring five subtests of critical thinking: 1) inference--discriminating the truth or falsity of inferences drawn from given data; 2) recognition of assumption--recognizing unstated assumptions or presuppositions in given statements or assertions; 3) deduction--determining whether certain conclusions necessarily follow from information in given statements or premises; 4) interpretation--weighing evidence and deciding if generalizations or conclusions based on the given data are warranted; and 5) evaluation of arguments--distinguishing between arguments that are strong and relevant and those that are weak or irrelevant to a particular question at are strong and relevant and those that are weak or irrelevant to a particular question at issue (Watson & Glaser, 1994, pp. 9-10). The WGCTA was completed in the freshman and senior year, and has a reliability of 0.77 for this sample. Reasoning About Current Issues (RCI). The RCI is a paper-pencil measure designed to measure the epistemologic construct, reflective judgment. Reflective judgment research seeks to document progressive sophistication in the way that individuals reason about a particular type of problem termed an "ill-structured problem." The RCI is composed of two parts, each scored separately. These two parts are: 1). Student Essay Discrimination Measure. The first half of the RCI asks 9 Effect of Research on Critical Thinking and Reflective Judgment 8 students to play the role of an instructor who has assigned students "minute essays" about a real world topic. Students are presented with pairs of such essays and are asked to judge which student is reasoning about the issue in a more complex or sophisticated fashion. This part of the instrument is designed to assess whether students can identify which of a pair of statements is a more sophisticated justification for a position. 2). RCI Dilemmas: Student Endorsement of Justifications. The second half of the RCI presents students with five real-world issues as well as statements written to reflect arguments at varying levels of sophistication according to the Reflective Judgment Model. Students are asked to indicate how similar each of these statements are to their own thinking about the issue, and are then asked to rank the three statements which are most similar to their own reasoning about the issue. NEO-Five Factor Inventory (NEO-FFI). This instrument measures the most basic dimensions underlying human traits (Costa & McCrae, 1991). There are five subtests in this inventory, each composed of twelve items. The five subtests are: 1) NE0Neuroticism which measures an individual's level of adjustment and emotional stability (coefficient alpha= 0.86); 2) NEO-Extroversion measures level of sociability and consequent behaviors that occur as a result of interactions with others (coefficient alpha= 0.81); 3) NEO-Openness to Experiences measures imagination, aesthetic sensitivity, attentiveness to inner feelings, preference for variety, intellectual curiosity, and independence of judgment (coefficient alpha= 0.73); 4) NEO-Agreeableness measures level of sympathy and altruism toward others, eagerness to help (coefficient alpha= 0.77); and 5) NEO-Conscientiousness measures ability to manage impulses and desires and the process of planning, organizing, and carrying out tasks (coefficient alpha= 0.83). Effect of Research on Critical Thinking and Reflective Judgment 9 Procedure After receiving approval from the University's Human Subjects Committee, researchers sent a letter in March, 1997 to second semester freshmen students majoring in science, math, and psychology requesting their participation in this four-year longitudinal study of their academic experiences. Two hundred sixty-six students agreed to participate and met with the researchers to complete several questionnaires including the WGCTA and NEO-FFI. Students were also asked to complete the Reasoning About Current Issues (RCI) beginning in Spring 1998. Each survey took approximately 15 to 30 minutes to complete. A signed consent form also enabled the researchers to obtain demographic data from the university's student record system (i.e., high school GPA, sex, ethnic classification, SAT, and cumulative GPA). During each survey completion, students were offered a small monetary incentive ($5 in Spring 1997, $10 in 1998; $15 in 1999; and $20 in Spring 2000). Characteristics of the Undergraduate Research Program and Definition of Participation in Undergraduate Research Undergraduate Research Program opportunities, which are open to all interested undergraduates, include research collaborations that occur during the academic year as well as summer research, senior thesis, and oral presentations of research-in-progress to other students and faculty. Over 66 percent of the University's approximately 900member faculty regularly participate inUR, including over 90 percent of the faculty in the sciences and engineering. From 600 to 700 students are estimated to be engaging in UR at any one time. Students can begin research at any point in their undergraduate careers; however, normal starting time is mid-sophomore year. Student involvement generally Effect of Research on Critical Thinking and Reflective Judgment 10 lasts for one and one-half to two or more years and usually includes one or more summer or winter terms in addition to the academic semesters. Each spring, students reported the number of hours per week they had been involved in a variety of campus activities, including undergraduate research. Student self-reports were verified with faculty and then summed to reveal the total number of hours over the four baccalaureate years each student had participated in UR. The total number of hours for participation in research ranged from 0 to 3,342; the mean number of hours for those who participated in research was 626 hours, SD = 596.5. Based on a scatterplot of research hours, students were then grouped into three categories: 1). no research; 2). total research less than 700 hours; and 3). total research 700 or more hours. Results Table 1 shows descriptive statistics and Table 2 highlights the correlation matrix for the primary measures in this study. Shown in Table 2 and as expected, SAT is strongly correlated with first year cumulative grade point average. Also, WGCTA score is significantly correlated with SAT, but RCI score is not highly correlated with SAT. Although the correlation coefficients are significant, the values between .21 and .29 do not reveal strong relationships. An independent samples T-test revealed significant differences between research and non-research students for NEO-N (t=-2.81, p =.005) and NEO-E (t=1.98, p = .048). These findings indicate that students who later participated in UR. scored higher on the neuroticism and lower on the extroversion scales as freshmen than peers who did not participate in UR. 12 Effect of Research on Critical Thinking and Reflective Judgment 11 Table 1 Descriptive Statistics for WGCTA, RCI, and NEO FFI M Year 1 M Year 2 M Year 3 M Year 4 WGCTA RCI 29.47 (5.33) NEO N NEO E 21.90 (8.22) 30.51 30.37 (6.54) NEO NEO A NEO C 30.12 5.26 (.62) 5.31 (.66) (5.22) 5.28 (.67) (6.13) 32.33 (6.12) 32.72 (6.72) Changes in Critical Thinking Repeated measures analyses were performed to examine the effect of research participation on changes in critical thinking and reflective judgment. Table 3 illustrates the effects of research, major and gender on the outcome variable, WGCTA. Because of the high correlation between SAT and WGCTA, SAT was included as a covariate in the analysis for WGCTA. As shown in Table 3, major (F=4.129, p <.01) and the interaction and major and research (F=3.094, p <.02) had an effect on the WGCTA change score. Changes in WGCTA were not effected by gender or interactions of gender with major or gender with research. Figures 1 to 3 illustrate the interaction of research and major on WGCTA. As shown, WGCTA change is greatest for students who participated in 800 or more hours of research, but this change differs by major. As shown in the figures, freshman to senior year WGCTA scores increase for science/math/nursing and chemistry/engineering majors, but decreases for psychology/other majors. 13 Effect of Research on Critical Thinking and Reflective Judgment 12 Table 3 Repeated Measures ANOVA for Watson-Glaser Critical Thinking Appraisal Measure WGCTA WG x Research WG x SAT WG x Gender WG x Research x Gender WG x Major WG x Research x Major Error df Sum Sq 5.960 5.270 2.970 28.806 75.130 136.081 203.929 3163.578 MS 5.960 2.635 2.970 28.806 37.565 68.041 50.982 16.477 1 2 1 1 2 2 4 192 .362 .160 .180 1.748 2.280 4.129 * 3.094 * *p<.02 Figures 1-3 Estimated Marginal Means for WGCTA by Research Participation and Major Estimated Marginal Means of Watson-Glaser Estimated Marginal Means of Watson-Glasei At major= science, nurs, & math At major = chem & engin 33 32 31 2 30 RSRCHB no research 3 13 29 reaearch 28 small nech big rsrch RSRCHB -g XI small reach 27 ° big rarch 2 2 WG WG Estimated Marginal Means of Watson-Glaser At major= psych & othr 32 31 30 29 28 RSRCHB 27 O 26 no research 25 small rarch 24 bp rarch 1 2 WG 14 Effect of Research on Critical Thinking and Reflective Judgment 13 Table 2 Correlation Matrix Yrl gpa Total SAT Pearson Corr. 0.42 Sig. (2-tailed) 0.00 N Yrl WGCTA Yr2 RCI Yr3 RCI 0.25 Sig. (2-tailed) 0.00 0.62 0.00 264 258 0.22 0.66 Sig. (2-tailed) 0.00 N 0.00 212 207 0.67 0.00 210 Pearson COM 0.09 0.23 0.22 0.25 Sig. (2-tailed) 0.18 0.00 0.00 0.00 N 230 225 228 196 Pearson Corr. 0.09 0.18 0.29 0.00 0.34 0.00 0.35 0.00 N Pearson Corr. Sig. (2-tailed) NEO-N 201 181 0.21 0.23 0.00 0.27 0.00 0.46 0.00 0.00 Yr4 RCI NEO-N NEO-E NE0-0 NEO-A 205 206 202 195 180 -0.01 -0.08 -0.07 0.86 0.21 0.35 263 257 -0.08 0.17 263 209 227 200 205 Pearson Com -0.17 -0.18 -0.17 -0.16 0.06 0.01 -0.39 Sig. (2-tailed) 0.00 0.00 0.01 0.02 0.38 0.86 0.00 263 257 263 209 227 0.06 0.38 200 205 263 Pearson Com -0.02 0.15 0.04 0.01 0.02 -0.05 0.40 N 263 257 263 209 0.29 0.00 227 0.17 0.80 0.15 0.02 0.07 Sig. (2-tailed) 0.03 0.68 200 205 263 263 Pearson Corr. 0.06 0.25 0.00 -0.17 0.32 0.09 0.12 0.22 0.00 0.19 0.34 0.09 0.16 0.11 Sig. (2-tailed) 0.01 0.01 0.00 0.16 263 -0.07 0.27 257 263 209 227 200 205 263 263 263 0.22 0.00 -0.14 0.02 -0.23 0.00 -0:30 0.00 -0.01 -0.04 0.53 -0.25 0.00 0.23 0.00 -0.01 0.14 0.89 -0.09 0.18 0.92 0.02 263 257 263 209 227 200 205 263 263 263 263 Sig. (2-tailed) N NEO-C 0.70 0.00 -0.11 Pearson Corr. 15 198 0.06 0.39 208 N NEO-C 203 0.52 0.00 203 0.00 0.94 N NEO-A Yr3 RCI Pearson Corr. N NE0-0 Yr2 RCI N Sig. (2-tailed) NEO-E Yr 2 WG Pearson Corr. Sig. (2-tailed) Yr4 RCI Yrl WG 260 Pearson COM N Yr2 WGCTA total SAT 0.08 0.33 -0.13 0.07 0.84 16 Effect of Research on Critical Thinking and Reflective Judgment 14 Change in Reflective Judgment Score Table 4 shows the effects of research, major and gender on the outcome variable for reflective judgment, Reasoning About Current Issues. A separate model including NEO-N and NEO-E as covariates was run; no substantial differences were shown, thus the analysis presented herein does not include NEO scores as covariates. As shown in Table 4 using a simple comparisons contrast, participation in research had a main effect on RCI change when comparing change from the sophomore to senior year (F=3.818, p=.05). Similarly, the interaction of gender and research had an effect on RCI change from the sophomore to senior year (F=3.063, p=.04). Figures 4 and 5 illustrate the interaction between RCI, research participation and gender. Table 4 Repeated Measures ANOVA for Reasoning About Current Issues Test Measure SS RCI .743 1.626 .085 .604 .196 .850 1.673 level 2 vs 1 level 3 vs 1 RCI x Rsrc 2 vs 1 3 vs 1 RCI x Gendr 2 vs 1 3 vs 1 RCI x Rsrc x Gndr 2 vs 1 3 vs 1 RCI x Major 2 vs 1 3 vs 1 RCI x Rsrc x Maj 2 vs 1 3 vs 1 Error 2 vs 1 3 vs 1 df 1 1 2 2 1 1 2.608 .434 2.272 .553 1.220 64.178 71.530 *p<.05 17 2 2 2 2 4 4 168 168 MS .743 1.626 .042 .302 .196 .850 .837 1.304 .217 1.136 .138 .305 .382 .426 1.945 3.818 * .111 .709 .513 1.997 2.190 3.063 * .568 2.668 (p=.07) .362 .716 Effect of Research on Critical Thinking and Reflective Judgment 15 Figures 4 and 5 Estimated Marginal Means of RCI by Gender and Research Estimated Marginal Means of RCI Estimated Marginal Means of RCI At GENDER OF STUDENT = MALE At GENDER OF STUDENT = FEMALE 5.7 5.6 5.5 50 W 2 :2 5.3 3 Research 0 5.21 no rsseerch 5.11 ID rsrch ° big rerch 5.0 2 RC! RC' Discussion This study examined the effects of participation in undergraduate research on two college outcomes, critical thinking and reflective judgment. Based on findings from this sample of 266 students, results indicate that involvement in research does affect critical thinking score and, to a lesser degree, reflective judgment. Students who participated in research scored higher on NEO-Neuroticism and lower on NEO-Extroversion as freshmen than peers who did not participate in research. However, correlations between NEO scales and the outcomes measures were less than .30 and determined to be of little practical significance. Results from repeated measures analyses indicated that the change in WGCTA score from freshman to senior year was greatest for science/math/nursing and chemistry/engineering majors who participated in 800 or more hours of undergraduate research. The interaction of research participation and gender had an effect on RCI change, thus indicating that females who participated in 800 or more hours of research and males who participated in 1-799 hours of research showed the greatest increases in RCI score over the three year period. 18 Effect of Research on Critical Thinking and Reflective Judgment 16 Participation in undergraduate research can benefit undergraduate students by contributing to the development of complex thinking, evaluation, and communication abilities, time management skills, and enhancing interpersonal independence. Increases in these skills will likely increase the students' chances for success as well as increase level of satisfaction with the undergraduate experience. Limitations of this study included the relatively small size of the sample and selective inclusion of only certain majors. If the sample were larger, it is possible that additional relationships may emerge. Like King, et al, (1983) we found some of our students to regress in RCI score. Additional study of the pattern of epistemological development is warranted. Due to the selective sample, findings from the study are limited in generalizability to science and engineering majors. Findings from this study are also unique to the set of academic and social experiences, including undergraduate research, that are available at this institution. Additional studies with a larger, random sample may be helpful in examining the effects of academic experiences on critical thinking and reasoning outcomes. 19 Effect of Research on Critical Thinking and Reflective Judgment 17 References American Association for Colleges. (1990). The challenge of connecting learning: Liberal learning and the arts and science major. Washington, DC: AAC. American Association for Colleges. (1985). Integrity in the college classroom. Washington, DC: AAC. Astin, A. W. (1977). Four critical years. San Francisco: Jossey-Bass. Astin, A. W. (1993). What matters in college: Four critical years revisited. San Francisco: Jossey-Bass. Baxter Magolda, M. B. (1992). Cocurricular Influences on College Students' Intellectual Development. Journal of College Student Development, 33 (3). Baxter-Magolda, M.B. (1987). Comparing Open-Ended Interviews and Standard Measures of Intellectual Development. Journal of College Student Development, 28, 443-448. Baxter-Magolda, M.B, & Porterfield, (1985). A New Approach to Assessing Intellectual Development on the Perry Scheme. Journal of College Student Development, 26, 343-351. Boyer, E.L., (1987). The undergraduate experience in america. The Carnegie Foundation for the Advancement of Teaching. New York: Harper & Row. Boyer Commission on Education Undergraduates in the Research University (1998). Reinventing undergraduate education: A blueprint for america's research universities. New York: Author. Costa, P. Jr., and McCrae, R. R., (1991). The NEO Five Factor Inventory, Odessa, FL: Psychological Assessment Resources, Inc. Digman, J. M. & Takemoto-Chock, N. K. (1981). Factors in the natural language of personality: Re-analysis, comparison and interpretation of six major studies. Multivariate Behavioral Research, 16 (2), 149-70. Entwistle, N. J. (1972). Personality and academic attainment. British Journal of Educational Psychology, 42 (2), 137-51. Entwistle, N. J., & Entwistle, D. (1970). The relationships between personality, study methods, and performance. British Journal of Educational Psychology, 40 (2), 132143. Eynsenck, H., & Cookson, D. (1969). Personality in Primary School Children. British Journal of Educational Psychology, 39, 109-122. Effect of Research on Critical Thinking and Reflective Judgment 18 Facione, P. A., Sanchez, C. A., Facione, N.C. & Gainen. (1995). The disposition toward critical thinking. The Journal of General Education, 44 (1), 1-25. Hakim, T. (1998). Soft assessment of undergraduate research: Reactions and student perspectives. Council on Undergraduate Research Quarterly, 18, 189-192. Hough, L.M. (1992). The "Big Five" personality variables construct confusion: Description versus prediction. Human-Performance, 5(1-2): 139-155. Kitchener, K. S. & King, P. M. (1981). Reflective judgment: Concepts of justification and their relationship to age and education. Journal of Applied Developmental Psychology, 2 , 89-116. King, P. M., & Kitchener, K. S. (1994). The development of Reflective Judgment in adolescence and adulthood. Jossey-Bass: San Francisco. King, P.M., Kitchener, K.S., Davison, M.L., Parker, C.A., & Wood, P.M. (1983). The justification of beliefs in young adults: A longitudinal study. Human Development. 26, 106-116. King, P. M., Kitchener, K. S. & Wood, P. K. (1994). Research using the Reflective Judgment model. in P. M. King & K. S. Kitchener, The development of Reflective Judgment in adolescence and adulthood . Jossey Bass: San Francisco. Knefelkamp, L. (1974). Developmental instruction: Fostering intellectual and personal growth of college students. Dissertation Abstracts International, 36, 1271A, University Microfilms No. 75-21,059. Lawson, J.M. (1980). The relationship between graduate education and development of reflective judgment: A function of age or educational experience. Dissertation Abstracts International, 41, 4655A. Mount, M.K. & Barrick, M.R. (1989). Five reasons why the "Big Five" article has been frequently cited. Personnel Psychology, 51(4), 849-857. National Commission on Excellence in Education (1983). A nation at risk: The imperative for educational reform. Washington, DC: US Department of Education. National Science Foundation. (1989). Report on the National Science Foundation disciplinary workshops on undergraduate education. Washington, DC: NSF. Pascarella, E. T. (1989). The development of critical thinking: Does college make a difference. Journal of College Student Development, 30 (1), 19-26. Pascarella, E. T., Whitt, E. J., Edison, M., Nora, A., Hagedorn, L.S., Yeager, P. M., & Terenzini, P. T. (1996). What have we learned from the first year of the national study of student learning? Journal of College Student Development, 37 (2), 182-192. Effect of Research on Critical Thinking and Reflective Judgment 19 Pascarella, E.T., & Terenzini, P.T. (1991). How college effects students. San Francisco: Jossey-Bass. Perry, Jr., W.G. (1970). Forms of intellectual and ethical development in the college years. New York: Holt, Rhinehart, & Winston. Soldz, S. & Vaillant, G.E. (1999). The Big Five personality traits and the life course: A 45-year longitudinal study. Journal of Research in Personality, 33(2): 208-232. Strange, C.C. & King, P.M. (1982). Intellectual Development and its relationship to maturation during the college years. Journal of Applied Developmental Psychology, 2, 281-295. Tross, S. A., Harper, J. P, Osher, L. W. & Kneidinger, L. M. (2000). Not just the usual cast of characteristics: Using personality to predict college performance and retention. Journal of College Student Development, 41 (3), 323-334. Watson, G. B., & Glaser, E. M. (1994). The Watson-Glaser Critical Thinking Appraisal. Form S. San Antonio, TX: Psychological Corporation. Welfel, E.R. (1982). How students make judgments: Do educational level and academic major make a difference? Journal of College Student Development, 23, 49097. Welfel, E.R. & Davison, M.L (1986). The Development of Reflective Judgment During the College Years: A Four-Year Longitudinal Study. Journal of College Student Development, 27, 209-216. Widick, C. (1975). An evaluation of developmental instruction in a university setting. Dissertation Abstracts International, 36, 2041A, University Microfilms No. 7521,101. Wood, P. K. (1997). Development of assessment measures of epistemic cognition: How do students think about real-world ill-structured problems? Assessment Update, 9, 11-13, 16. Wood, P. K. (1997). A secondary analysis of claims regarding the Reflective Judgment Interview: Internal consistency, sequentiality and intra-individual differences in ill-structured problem solving. in J. Smart (Ed.) Higher Education: Handbook of Theory and Research. Vol. 12. (pp. 243-312). Bronx, NY: Agathon. 22 U.S. Department of Education Office of Educational Research and Improvement (OERI) National Library of Education (NLE) Educational Resources Information Center (ERIC) NOTICE Reproduction Basis El This document is covered by a signed "Reproduction Release (Blanket)" form (on file within the ERIC system), encompassing all or classes of documents from its source organization and, therefore, does not require a "Specific Document" Release form. This document is Federally-funded, or carries its own permission to reproduce, or is otherwise in the public domain and, therefore, may be reproduced by ERIC without a signed Reproduction Release form (either "Specific Document" or "Blanket"). EFF-089 (3/2000) IC