Handbook of complementary methods in education research

1 Final Draft of Introduction to Handbook of Complementary Methods in Education Research Green, J. L., Camilli, G., & Elmore, P. (Eds) Routledge, 2006 Acknowledgements The editors would like to acknowledge the intellectual debt owed to Richard Jaeger, editor, and both the authors and members of the Professional Development & Training committee of AERA who conceptualized and published the first and second editions of Complementary Methods for Research in Education (CMRE). They provided the grounding and framework for making visible multiple research traditions within AERA. We would also like to acknowledge the more than 200 AERA members, officers, and central office staff who supported the work on and contributed to the third edition. The process of producing this book began with AERA President Lorrie Shepard’s suggestion, during the first AERA Coordinated Committee Meetings in 2000, that the Professional Development and Training Committee (PT&D) revise the second edition of Complementary Methods for Research in Education (CMRE), which was last published in 1997. Over the next two years, members of the PT&D committees in 2000-2002 (Gregory Camilli, Patricia Elmore, Michael Garet, Judith Green, Judit Moschkovich, Barbara Scott Nelson and Iris Weiss) deliberated about ways in which in which to conceptualize the new volume, given the development of new theoretical and methodological perspectives since the publication of the second edition. During his presidency, Andy Porter supported the work of the PD&T in carrying out its charge by taking the concept of the new volume to the AERA Council in 2001. Once the Council expressed approval, PD&T members conferred with a range of committees at the October 2001 AERA coordinated meetings in Chicago to obtain guidance on the new edition. The Publications Committee approved the editorial team of Camilli, Elmore and Green. Andy Porter formalized the Advisory Panel for the new edition, which consisted of members from AERA committees: Publications (Christopher Clark and Marilyn Cochran-Smith), Research Advisory (Mary Elizabeth Brenner), Professional Development & Training (Iris Weiss and Michael Garet), and Scholars of Color (Maria Carlo) and Special Interest Groups (Richard Lomax). At the 2002 AERA annual meeting, Felice Levine, the Executive Director of AERA, and Linda Dziobek, Director of Publications, met with the PD&T Committee on the CMRE project, and a development and production timeline was established, along with procedures for working with AERA staff. In the process of revising CMRE, a number of other individuals also made important contributions to conceptualizing and facilitating our progress including William Russell, Suzanne Lane, Catherine Snow, Robert Linn, and Hilda Borko. In addition, we have benefited from the deliberations of the Publications Committee, Research Advisory Committee, and AERA Council. AERA Presidents, Marilyn CochranSmith and Hilda Borko have provided support for the volume by facilitating Presidential paper and poster sessions at the Annual Meeting of AERA in 2004 and 2005. We express special thanks to Richard Duran, Chair of Publications for his guidance in the final review process. We are also grateful to those AERA members who nominated colleagues 2 for positions of chapter authors and developmental reviewers. Last, but not least, we would like to acknowledge the authors and the developmental reviewers who engaged in respectful and intellectually important dialogues about the content of the chapters, leading to this volume. Authors and developmental reviewers are given at the beginning of each chapter, yet many others contributed to the work required for such a large and complex project. We are grateful to all. President’s Preface Complementary Methods for Research in Education originated nearly a quarter century ago as part of AERA’s commitment to the professional development of its community. In 1978, the Committee on Research Training commissioned well-known experts such as Robert Stake, Gene Glass, and Michael Scriven to prepare audio tapes of alternative research methods unlikely to be found in graduate texts. In 1999 as president, I asked Gregory Camilli and Judith Green, then members of AERA’s Professional Development and Training Committee, if they had considered reviewing AERA’s somewhat outdated though still popular best-seller. Ultimately, they presented to Council a compelling case for a major revision of the volume. The project to launch a revision involved several presidents: myself and Catherine Snow, Alan Schoenfeld as chair of the newly formed Research Advisory Committee, and Hilda Borko and Marilyn CochranSmith in their roles sequentially as chairs of the Publications Committee. The vetting process ensured that the volume would reflect the range and depth of research traditions represented within the association. Having survived the labyrinth of AERA committees, Green and Camilli agreed to serve as editors, along with Patricia Elmore, who chaired the Professional Development Committee through the negotiations process. Complementary Methods has a special place in the intellectual history of AERA. The time period, between production of the audio tapes and publication of the first volume in 1987, represented the height of the so-called quantitative-qualitative divide. Although philosophers of science had long since laid to rest positivism’s claims of objectivity, definitive knowledge, and value-free science, quantitative methods and quantitative perspectives of the world still dominated the field. Significantly, Richard Jaeger, a well-known statistician and psychometrician, edited the first Complementary Methods. His intention was to redress the imbalance found in typical textbooks, which emphasized experimental, quasi-experimental, and correlational methods “to the exclusion or neglect of methods that emphasize verbal portrayals of findings or observational techniques, and methods based on naturalistic inquiry” (p. i). In addition to traditional research methods, he invited chapters on historical and philosophical inquiry and case study methods. Harry Wolcott, an anthropologist, wrote the chapter on ethnographic research in education. Since the opening up of possibilities reflected in the first volume, the growth and development of new perspectives and methods of inquiry has been remarkable. Researchers in education work at the crossroads of multiple disciplines. Because of this interdisciplinarity, we are more aware than most social scientists of the ways in which narrow, disciplinary perspectives shape scholars’ understanding of substantive problems. To the extent that we can become adept at thinking about how we would conceptualize a problem if we approached it as a psychologist, a sociologist, or an anthropologist, we will be more insightful than the psychologist, sociologist, or anthropologist each studying 3 exclusively within their own tradition. We appreciate the need to study significant issues at micro and macro levels of analysis and to synthesize research findings across methods and contexts. This new, greatly expanded volume of Complementary Methods was undertaken to capture the wide range of research methods used to study education and to make the logic of inquiry for each method clear and accessible. It is intended as a teaching tool to help graduate students understand the kinds of questions that can be addressed by each method and to help seasoned researchers learn new perspectives and skills. The publication of Complementary Methods, we hope, puts researchers’ attention on substantive problems with the understanding that research methods are an important means to that end. AERA is indebted to the editors and authors of Complementary Methods for Research in Education for their contributions to the preparation of the next generation of researchers in education. Lorrie A. Shepard University of Colorado at Boulder Introduction to the Third Edition AERA through its advocacy and support of the past volumes has demonstrated its commitment to (1) training for students, (2) expanding professional knowledge of practitioners, and (3) promoting understandings about the epistemologies, techniques, and results of diverse research methods and programs of research. Accordingly, CMRE was designed to expose graduate students and researchers to a broad range of research methods and the kinds of questions these methods address. This new edition continues this tradition, while acknowledging the vibrant growth and evolution of new epistemologies, perspectives, and methods for research in the field of education today. What’s Complementary About Complementary Methods? -- From Past to Present Richard Jaeger, editor for both the first and second editions of Complementary Methods for Research in Education, placed the origins of the volume with AERA’s production of an audiotape series (1978), entitled Alternative Methodologies in Research in Education (Jaeger, 1988, p. i). He characterized the rationale for the tapes, and the resulting volume, in the following way: ...the tape series was intended to address the overemphasis on quantitative research methods to the exclusion or neglect of methods that emphasize verbal portrayals of findings or observational techniques and methods based on naturalistic inquiry, that was common in textbooks on educational research methods at the time. The Committee concluded that a series of audio tapes, each devoted to a single method of disciplined inquiry in education and accompanied by supplementary materials in print form, might significantly enlarge the set of resources available to instructors and students in introductory educational research courses. Jaeger describes the first edition of Complementary Methods for Research in Education (CMRE) as a response to “requests from members of the Association for a textbook that was based on the tape scripts” (p. i). The authors of the tapes were invited to revise and update their scripts, transforming each into a chapter, and “to introduce readings from the 4 educational research literature that exemplified sound application of the research methods in the chapter” (p. i). This edition, published in 1988 was widely used to introduce graduate students to educational research methods. The second edition, published in 1997, included the articles in the first edition and a set of new articles that were responsive to the field and its development in the previous decade. The introduction to this edition described the evolution as follows: In the near-decade since the first edition of this book was published, the landscape of educational research methods has changed materially. Although quantitative approaches to disciplined inquiry in education are still employed quite frequently, the publication of research that is totally devoid of quantitative summarization and analysis is now quite common. The American Educational Research Association’s journals now abound with reports on case studies, inquiry that incorporates ethnographic procedures, historical accounts, hermeneutic analyses, “thick” description, and narrative accounts—often with excerpts from transcripts of interviews or of discourse among research subjects. Nonetheless, the distinctions among research methods that were identified for the original tape series are, for the most part, still relevant and useful today, and most are present in this second edition (pp. i-ii). In the second edition, two new chapters were added along with one new section. New chapters were added to the Philosophic Inquiry Methods in Education (Maxine Greene, 1997). Maxine Green’s chapter traced the evolution of thought that “has given warrant to a far more inclusive body of researchers” (p. ii). A chapter by Lee Shulman (1997), on the nature of disciplined inquiry in education, was added to the first section. This chapter advanced “the readily confirmed hypothesis that choice of a method of inquiry involves more than determination of the way in which a given research question will be answered. To a greater degree, it also determines the nature of the research question that will be asked” (p. iii). The new section introduced an approach to aesthetics in education research (Barone & Eisner, 1997), designed to “identify and describe the qualities of narrative products of inquiry that made them “arts-based” and to “grapple with the critical but necessarily sticky problem of legitimacy and propose criteria by which products of aesthetic inquiry on education should be evaluated and judged” (pp. ii-iii). The introduction to the second edition goes on to argue that “[i]n a period when inquiry in education lends itself to a richly diverse collection of methods and techniques, it is virtually impossible for a single individual to develop expertise in each” (p. iii). The volume, therefore, served as a basis for introducing a range of perspectives to a broader audience in order to expand the repertoires of, and resources available to, AERA members. What is complementary about complementary methods: Issues and Challenges In this, the third edition, we take up the goal of introducing a range of perspectives to AERA members. However, rather than add chapters to the cumulative text of the first and second editions, the Professional Development Committee elected to re-envision the content and approach to identifying and representing the richly diverse range of research methods and perspectives currently available to and used by AERA 5 members. The committee also confirmed the view that “it is virtually impossible for a single individual to develop expertise in each” (p. iii). In the decade since the publication of the second edition, the landscape of educational research methods continues to undergo extraordinary change. A panoply of new quantitative approaches for addressing complex educational phenomena has been developed (e.g., in the areas of multilevel and structural equation modeling). These approaches parallel progress in new theoretical and philosophical traditions that represent different ways of knowing (e.g., western, indigenous, and gendered ways of knowing). Today, it is virtually impossible for any one approach to be used to address the complex issues being explored through research in education. Further, no longer is it a question of alternative research traditions (the concern of Jaeger in 1988) but of which approaches are appropriate to the questions under study and which can be productively combined within a program of research. The principles guiding the process of bringing different theoretical perspectives and research approaches together, however, are not well developed. This argument is represented in a recent National Forum on Applying Multiple Social Science Research Methods to Educational Problems, co-sponsored by the Center for Education of the National Academies, the American Educational Research Association, the American Psychological Association, and the National Science Foundation. In the announcement of this forum, it was argued that “The application, fit, and articulation of different scientific research methods to tackle major issues relevant to educational policy and practice is a largely undeveloped area that is ripe for sustained inquiry and knowledge accumulation” (AERA, November, 2004). In framing this forum around these issues, leaders of these major research organizations, each with interests and areas of research in education, suggested that the issue of bringing different traditions together is not solely the task of the individual researcher or research team. It is a problem for the fields that engage in research in and on education. Today, there are no commonly agreed to guidelines for use of multiple methodologies in particular projects, although some disciplines (e.g., health science) have begun to publish handbooks and other guidelines for mixed methods. What is evident, however, is that the challenge facing those seeking to use different theoretical, not merely alternative methods, is to identify which can be productively brought together--for what purpose(s), in what ways, and on what scale--to explore which phenomena. In this edition of CMRE, these issues are raised in a number of chapters, including one specifically addressing the design of mixed methods by M.L. Smith. However, when the chapters focusing on programs of research are examined, the issue of different approaches becomes more visible, when the phenomenon under study becomes the focus, not the method itself. Even in these chapters, however, the methods are most often used to address different questions and less frequently used in complementary ways. This observation suggests that the answer to what constitutes complementarity among methods or what relationships need to exist for methods to be defined as complementary, rather than merely mixed, is an area that needs careful discussion and debate. This issue is not a problem for education alone. Across disciplines, issues of complementarity have been explored for varying lengths of time. A search for definitions of the terms complementary, complementary methods, and complementarity across disciplines shows that these issues have been, and are currently being debated in a broad 6 range of disciplines in the natural and social sciences, including physics, linguistics, allied medicine, law, environmental studies, psychology, neuroscience, sociology, genetics, organizational studies in business, economics, mathematics, history, and education, among others. What became evident in our exploration of these concepts across disciplines is that central to most definitions is the concept of relationships between two or more phenomena. These debates focus on both the complementarity of phenomena understudy and methods used to study them, supporting the argument that theory-method relationships are critical to understanding and defining what is complementary about complementary methods. In the first two editions, as suggested previously, complementary methods referred primarily to alternative methods. As education researchers, we need to explore which of the complex phenomena that we are examining or assessing is complementary, and if they are related, what the nature of that relationship is. From this perspective, we need to consider how the phenomena under study are conceptualized, if we are to select appropriate sets of methods to mix and/or use in complementary ways. The chapters in this volume provide historical and conceptual information about what each method is designed to explore, the nature of the phenomena involved, and the questions each can address. As such, they lay the foundation for exploring which approaches might be productively brought together to study a common phenomenon, and which might be juxtaposed to make visible similarities, differences and complementarities between phenomena. Further, they provide a basis for understanding the level of scale for which the method was designed, and how one level of scale sheds light on particular aspects of the phenomena, while masking others. The inclusion of chapters on Philosophic Issues in Educational Research (Bredo), Epistemology and Educational Research (Kelly), and The Ethics of Educational Research (Strike) lays a foundation for the discussion and debates about what it means to claim that two methods are complementary, and how we can understand and study complementary phenomena within complex educational phenomena. These chapters identify ways in which such dialogues can be undertaken and the theoretical issues that need to be addressed in order to shape new and dynamic understandings of the purposes, goals, approaches, claims, and outcomes of research into the complex educational processes facing students and their teachers, school systems, families and communities in the 21st Century. Designing the editorial process In this section, we describe the ways in which we engaged a broad group of AERA members who represented different divisions, committees, constituencies, and perspectives in the process of identifying authors and developmental reviewers for the third edition. The process for creating this volume began with extended deliberations by multiple groups within AERA regarding the potential range of approaches that might be included in the third edition. Given the tremendous diversity of approaches, however, we realized that not all could be the subjects of chapters Rather, the perspectives and methods to be included were selected using the following set of principles: a method is recognized or employed broadly in the divisions of AERA; there exists a rich body of epistemological work associated with the method or tradition; and the topics were appropriate for first and second year graduate students who want to devise, select and 7 adapt research methods for their dissertation work. In conjunction with our Advisory Panel, the full range of chapters was examined for coherence. The next step in the process was to identify potential authors. For each chapter, we planned to invite a primary author as well as two other scholars in the area to serve as developmental reviewers, using a process similar to the one used for the Review of Research in Education. Peer review is a critically important aspect for strengthening scholarly works, and is a standard procedure for all AERA products. For this purpose, at least two reviewers were assigned to each chapter. To identify potential authors and to insure broad representation of AERA membership, we invited nominations from Division Vice Presidents (12), SIG chairs (about 150), members of AERA standing committees (about 200), and the CMRE Advisory Panel. This call for nominations was distributed in the middle of May 2002. We received nominations by email, and additional names from nominees (for reviewers or primary authors if the nominee declined). An initial roster of authors and developmental reviewers was then finalized. With this roster, we began contacting nominees. In many cases, the primary authors accepted the invitation. However, it was necessary to select alternates--as can be expected in a volume of 46 chapters. The principle we used in selecting alternates was to maintain the original intent of the chapter in terms of both concepts and content. By early 2003, we requested that primary authors (or primary teams) send chapter outlines to the reviewers for comment, and that by the end of the summer, first drafts were for the most part completed for review. We asked for final drafts by early spring 2004, and for the most part, these documents were received by mid-summer. To provide further comments to authors, we proposed a session to Marilyn Cochran-Smith in which authors would meet with AERA members at the meeting in San Diego (2004). She invited the innovative session and more than 900 AERA members attended one or more of the three-part Presidential Session. Authors then had an opportunity to make changes based on the questions and discussions at these sessions. Final versions of the chapters were submitted by September 2004. The volume was then submitted for “volume” review to the publications committee, who sent it to three scholars in the field for review. Comments from those reviewers were used to complete the volume. Using the Volume This third edition following Richard Jaeger’s second edition of Complementary Methods for Research in Education has not only more authors and chapters but also more diversity in perspective, approach and thought regarding education research methodology. The third edition is divided into four major sections: Foundations, Design and Analysis, Describing Regularities [we renamed this section Measurement, Pattern Identification, and Classification], and Illustrative Research Programs. The Foundations section has three chapters on epistemology, philosophy, and ethics and provides the framework for the entire volume. Design and Analysis is the largest section with 28 chapters covering a broad range of topics from arts-based educational research to individual and multiple case studies, discourse, statistical graphics, data modeling, and narrative and philosophical inquiry. This section also includes chapters focusing on data collection approaches, including interviewing, survey methods, and survey sampling, 8 with observation and field methods included in qualitative and ethnographic research chapters. Measurement, Pattern Identification, and Classification includes six chapters on classical measurement theory, item response theory, and generalizability theory from a quantitative perspective and analysis of data from videotape, cross-case analysis, and finding patterns from field notes from a qualitative perspective. The section on Research Programs: Illustrative examples includes classroom interaction; language research; issues of race, culture, and difference; policy analysis from an institutional and practice perspective; program evaluation; student learning; and, teacher education. A question most professors, who have used the first two editions, will invariably ask is “How is this edition different and will I be able to use it in the same way for my research methods course?” As we have already indicated, the third edition contains many more chapters representing diverse perspectives and research methodologies. In “A Note to Authors” Richard Jaeger said, “All of these materials were designed for use in graduate-level courses on the methodology of educational research. Because the materials are modular, they can be used in many ways, and their usefulness transcends courses devoted solely to the study of research methodology. Beyond the introductory section on the nature of disciplined inquiry in education, the remaining sections of this book are virtually independent. Therefore, selected portions of the book could be used in courses devoted to specific methods of research or to specific research perspectives, such as qualitative inquiry methods.” We believe that the above instructions provided by Richard Jaeger remain appropriate for this edition as well. Keep in mind that the chapters are not equally accessible to beginning graduate students and some chapters will require fairly extensive previous knowledge on the part of students and “heavy lifting” on the part of professors to facilitate understanding and application. Further Richard Jaeger stated “This book and the associated audio tape series could be used either to structure an entire course, or to enrich units devoted to particular methods contained in a course with some other principal focus—such as a graduate course in educational psychology or a graduate course on the sociology of education.” Certainly this book can be used for a research methods course but may be more appropriate for a one-year seminar at the beginning of the doctoral program in education to introduce students to the methodologies they will encounter throughout their program of study and to encourage students to begin their own research program and scholarly productivity. Another important use of this volume is as a reference for established researchers with an interest in changing directions, learning a new technique, or becoming more expert in a methodology already known and used. This book was conceptualized and its production realized through the efforts of the AERA Professional Development and Training Committee. The intent of that committee is to use the volume as a roadmap for selecting and encouraging professional development sessions on topics covered in this new edition. 9 Note to students from students: on reading and using the volume Audra Skukauskaite & Elizabeth Grace As editorial assistants, who had the privilege of working with the editors on this volume, we were asked to write a note to our fellow graduate students -- a primary audience for this third edition of Complementary Methods for Research in Education (CMRE). Throughout our teaching and graduate careers, we have come to understand that like teaching, research is a situated and contextualized learning process, requiring careful consideration of the traditions underlying research methodologies. This volume was designed to provide an introduction to the history, theory and practice guiding each of the traditions represented in the third edition of CMRE. We see this volume as both supporting and extending our work as graduate students, and as opening possibilities for new research directions throughout our careers. The third edition of CMRE is a resource that can help us conceptualize, design and expand our research programs, and can help us create and foster dialogues within and across educational research traditions. The need for dialogue was captured by Lorrie Sheppard, in her introduction to the Presidential Invited Session for CMRE (AERA 2004). The vision of the editors, AERA presidents and various committees involved in the publication of this volume was to provide access to issues of epistemology associated with the broad range of research traditions currently used by members of AERA. The richness of traditions represented brings unique challenges, since, as the current and previous editors state, no one person can know and do research from all traditions. We found that the dialogues among the editorial team from different disciplinary backgrounds were instrumental in bridging the challenge and making visible to us potential links between and among different research methodologies and their underlying theories. The need for such dialogues in the field is captured by Kelly (this volume), who describes guidelines for respectful and productive dialogue and debate within and across research communities in education. As you read, you will find that like the first two editions, this volume is by no means a “cook book,” containing easy-to-follow recipes of how to do research using a particular method. Though initially designed for beginning graduate students, the volume in fact presents a broad range of “difficulty levels.” While all of the chapters are a form of introduction to the field, some will require more background knowledge than others. The volume is a resource and a reference that can help us explore what is available and what is possible in educational research. It will require us to go to additional resources and to engage in conversations with our professors, mentors, and colleagues within and across institutions, as discussed above. In the remaining section we share some of the approaches we found helpful in working with this volume as well as the questions we asked in trying to understand the arguments and methodological frameworks presented by the authors. We hope that these will also be useful to you. o We found it useful to first browse through the whole volume to see what is available, i.e., to identify the landscape of research represented. The preface and the introduction were particularly helpful in orienting us to the scope of research traditions in the volume. 10 o The first three foundational chapters provided a lens through which to read any particular chapter and across chapters. These chapters guided us in exploring the questions authors asked, phenomena they studied, and epistemological and ontological traditions underlying theoretical and methodological bases of their research. o As we read, we found it helpful to keep in mind the overall structure of each of the design and analysis chapters to understand the goals and procedures of each of the methods. All authors were asked to provide a brief historical overview of the research tradition, list the questions that can be addressed through this perspective/method, and present an example of how this approach is used. o We found the last section, Illustrative Programs of Research in Education, helpful in making visible the historical developments of particular areas of study in education. These chapters, rather than focusing on one method or study, look across different ways of studying educational phenomena and provide conceptualizations of the current state-of-the-art in each of these areas. o As we read a variety of chapters in the volume, including chapters that initially seemed not to relate to our research, each of us found the references particularly useful in helping us (re)formulate our own research questions and make our research designs stronger. o Perhaps one of the most important things that we learned as we worked on this volume is that we had to consider the theory-method-practice relationships underlying each research design and explore how the questions being asked by this particular tradition fit within the larger field of educational research. To examine the theory-method-practice relationships, we use the following questions, derived from the ethnographic framework of our research community (Green, Dixon & Zaharlick, 2003) to make visible the underlying thinking and logic of each tradition from the perspective of the authors. o What is the author (authors) arguing? o What are key assumptions guiding the construction of the argument? o What are the key terms? How are they defined? o What kinds of questions is the method designed to address? What kind of questions does it not address? o What data need to be collected to address these questions? o What kinds of analyses need to be undertaken? o What are theoretical assumptions guiding question-asking? o How do these theoretical assumptions guide research design, including data collection, analysis, interpretation and representation? o What kinds of claims can be made by using this method? (Consider the scope and level of scale for the claims.) o What are the historical roots of the method? o How have developments in the field shaped and reshaped the methodology? o How does this method contribute to researching educational phenomena? o What are other methods that ask similar questions? How do these methods differ and how are they similar? 11 As graduate students, currently completing our dissertations, we hope that this note can serve as a “cultural guide” for those entering the field. We also believe we need to continue asking these questions as we interact with our colleagues across academic disciplines, theoretical traditions, and institutions. In order for us to use complementary methods, as called for by editors across the three editions of the Complementary Methods for Research in Education, we will need to continue exploring the multifaceted, dynamic field of educational research to which this volume is a guide. We look forward to these conversations. 12 Reference Green, J. L., Dixon, C. N. & Zaharlick, A. (2003) Ethnography as a logic of inquiry. In Flood, J., Lapp, D., Squire, J. R., & Jensen, J. M. (Eds.) Handbook of research on the teaching of the English language arts. 2nd ed. Lawrence Erlbaum associates, publishers. Pp. 201-224.