Comparison in Qualitative Research

Oxford Research Encyclopedia of Education reached a major milestone this month by publishing our 500th article! For more information visit our News page. Comparison in Qualitative Research Lesley Bartlett and Frances Vavrus Subject: Education, Change, and Development, Educational Politics and Policy, Research and Assessment Methods, Educational Theories and Philosophies, Education and Society Online Publication Date: Jun 2020 DOI: 10.1093/acrefore/9780190264093.013.621 Summary and Keywords Comparison is a valuable and widely touted analytical technique in social research, but different disciplines and fields have markedly different notions of comparison. There are at least two important logics for comparison. The first, the logic of juxtaposition, is guided by a neopositivist orientation. It uses a regularity theory of causation; it structures the study by defining cases, variables, and units of analysis a priori; and it decontextualizes knowledge. The second, the logic of tracing, engages a realist theory of causation and examines how processes unfold, influenced by actors and the meanings they make, over time, in different locations, and at different scales. These two logics of comparison lead to distinct methodological techniques. However, with either logic of comparison, three dangers merit attention: decontextualization, commensurability, and ethnocentrism. One promising research heuristic that attends to different logics of comparison while avoiding these dangers is the comparative case study (CCS) approach. CCS entails three axes of comparison. The horizontal axis encourages comparison of how similar policies and practices unfold across sites at roughly the same level or scale, for example across a set of schools or across home, school, religious institution, and community organization. The vertical axis urges comparison across micro-, meso-, and macro-levels or scales. For example, a study of bilingual education in the United States should attend not only to homes, communities, classroom, and school dynamics (the micro-level), but also to mesolevel district, state, and federal policies, as well as to factors influencing international mobility at the macro-level. Finally, the transversal axis, which emphasizes change over time, urges scholars to situate historically the processes or relations under consideration. Keywords: comparison, qualitative methods, comparative case studies, epistemology, horizontal comparison, vertical comparison, transversal comparison The Plurality of Comparative Methods Comparison, all would likely agree, is a key analytical move across the social sciences and humanities. But beyond this recognition, accordance fails. Different fields define comparison in distinct and occasionally dogmatic ways, often ignoring the traditions of other disciplines. And while it’s called “the comparative method,” as though there is only one, few agree on what that “method” entails. In fact, there are a plurality of comparative approaches, with distinct epistemologies, disciplinary affiliations, and methodological techniques. For some fields, comparison generally means cross-national or cross-cultural comparison and embraces the a priori identification of units of analysis and variables. For example, the dominant model of comparison in political science promotes many cases and comparison across units (e.g., national or subnational units) that are presumed, prima facie, to be the same (e.g., Snyder, 2001), and it requires the identification of hypotheses about relationships between independent and dependent variables (Landman & Robinson, 2009; see critique in Simmons & Smith, 2015). The “controlled” or “paired comparison” method, which often relies on case selection strategies that minimize or maximize differences in presumed independent and dependent variables, gives variables a central role (see, e.g., Gisselquist, 2014; Slater & Ziblatt, 2013; Tarrow, 2010). Area studies in comparative politics, which are more likely to feature qualitative research, still generally privilege comparison across nation-states and judge qualitative research by positivist notions of validity, reliability, and generalizability (Simmons & Smith, 2015). Process tracing, one of the most processual approaches in comparative politics, aims to lay out the mechanisms that connect independent and dependent variables and “document whether the sequence of events or processes within the case fits those predicted by alternative explanations of the case” (Bennett, 2008, p. 705). Even advocates of “contextualized comparisons” that select cases which are “analytically parallel” and focus on complex dynamics, rather than attending only or primarily to independent variables, nonetheless emphasize outcome over process (e.g., Locke & Thelen, 1995, p. 344). In sociology, comparative methodologist Charles Ragin argued that quantitative studies tend to distort data, become vague and abstract in a search for maximum generalizability, and overlook important questions. He valued the conjunctural and complex vision of causation possible with qualitative comparative analysis (Ragin, 2014). Ragin (1987) developed a Boolean logic based on set theory, which used binary scores (0, 1) to code elements of cases and look for set relations. For example, an analyst might consider all the countries that experienced mass protest in the 1980s against the austerity measures imposed by the International Monetary Fund (IMF) and list causal conditions; the researcher might then consider the negative cases and similarly identify the presence or absence of possible causes. The resulting table allows the researcher to consider various combinations of factors and outcomes. Later, Ragin explored the use of “fuzzy sets,” which used an ordinal scheme to allow for consideration of phenomena that vary by level or degree. For example, in a study of fragile states (which might, at the risk of oversimplifying, be glossed as a state with weak capacity and legitimacy), 0 might mean “fully out of the set”; lower than 0.5 means “more out than in”; higher than 0.5 but under 1 signals “almost fully in”; and 1.0 means “fully in the set” (Ragin, 2000). While Ragin envisions his approach as blurring qualitative and quantitative approaches, his method notably requires the analyst to impose variables on the data and determine the strength of a variable’s presence. These types of approaches tend to identify the relevant variables or factors and units of analysis from the outset, avoiding any temptation to vary them as the study unfolds. From a very different disciplinary perspective, anthropologists generally consider their work to be inherently comparative; as Margaret Mead famously wrote, “every single statement that an anthropologist makes is a comparative statement” (1955, p. 9). This is assumed even when the scholar is focused on only one site, given that the ethnographer as the instrument of research is constantly comparing his or her experiences and assumptions to those of study participants. Ethnographers are expected to be responsive to participants and to the study as it unfolds, eschewing a study that is fully pre-structured. Interpretivists insist that objects of study and analytical categories must be determined “in semiotically mediated exchanges” between researchers and participants, taking into account their logics and the meanings they ascribe to processes (Handler, 2013, p. 272). Further, they more self-consciously interrogate their own research practices, recognizing that units of analysis are shaped by volatile historical and social processes (Kuper, 2002). The advent of “multi-sited ethnography” (Marcus, 1998) has intensified (but also complicated) anthropology’s comparative commitment to examine linkages across space and time. Logics of Comparison and Methodological Implications Drawing on Mohr’s distinction between variance and process methods (1982), as developed by Maxwell (2013), we argue that there are (at least) two primary logics of comparison: a variance- (or variable-)oriented logic of juxtaposition and a processual logic of tracing (Bartlett & Vavrus, 2017). The logic of juxtaposition is what is usually imagined by the phrase “compare and contrast.” It is guided by a neopositivist orientation and engages what Maxwell (2012) calls a regularity theory of causation, in which causation is the regular association of events or variables, with no other cause. It defines, a priori, cases, variables, and units of analysis. It assumes that a study bounds cases and units and holds them constant while varying other factors or variables to test hypotheses. They isolate the entity being analyzed from the context, rather than looking at connections. This positivist logic is consistent with only some varieties of qualitative research. Most good qualitative research is rather more responsive and iterative. As Becker (2009) stated, generally, qualitative scholars don’t fully specify methods, theory, or data when they begin their research. They start out with ideas, orienting perspectives, or even specific hypotheses [or propositions], but once they begin, they investigate new leads; apply useful theoretical ideas to the (sometimes unexpected) evidence they gather; and, in other ways, conduct a systematic and rigorous scientific investigation. Each interview and each day’s observations produce ideas tested against relevant data. Not fully prespecifying these ideas and procedures, as well as being ready to change them when their findings require it, are not flaws, but rather two of the great strengths of qualitative research … (p. 548; emphasis added) Thus, we argue, some qualitative research calls primarily for a processual logic of connection that, instead of contrasting, traces across individuals, groups, sites, or states (see Bartlett & Vavrus, 2017). This approach engages what Maxwell (2012) calls a realist approach to causation, which assumes that causation involves real entities and mechanisms, including meanings, which act and interact to produce the result. Studies may look at how policies or processes unfold (Anderson & Scott, 2012), influenced by actors and events over time, in different locations, and at different scales, including transnationally. Much contemporary anthropological research seeks to engage in multisited ethnography, which exemplifies this logic of tracing. Multisited ethnography does not contrast places assumed to be unrelated; instead, it looks at linkages across place, space, and time (Falzon, 2009; Marcus, 1998; see also Bartlett & Vavrus, 2017). Such a perspective requires a multisited, multiscalar approach. It is important to note that it is possible to combine these two logics—juxtaposition and connection—in a single study, using a mixed-methods approach to data collection and analysis, or as different phases of a study (see Bartlett & Vavrus, 2017, for examples). For all researchers, we argue that it is essential to recognize the heterogeneity of comparative approaches, the logic informing them, and the epistemological assumptions and commitments of the techniques (see also Wiseman & Popov, 2015). Different questions and forms of analysis and argumentation will be better suited to one logic than another (Guba & Lincoln, 1994). The Dangers of Comparison There are some dangers of comparison that merit attention. Here we discuss three: decontextualization, commensurability, and ethnocentrism. Decontextualization Some methodologists have specifically questioned whether a concern for context can be blended with comparison. For example, case study methodologist Robert Stake warned that “direct comparison diminishes the opportunity to learn from [the case]” (1994, p. 240). He continued: “I see comparison as an epistemological function competing with learning about and from the particular case. Comparison is a powerful conceptual mechanism, fixing attention upon the few attributes being compared and obscuring other knowledge about the case” (1994, p. 242). A neopositivist epistemology, which focuses on variables and ignores context, certainly runs this risk. However, a processual epistemology and an iterative methodology can remain responsive to the interests and perspectives expressed by participants and to what is learned as the study unfolds. Furthermore, some discussions of “context” (and thus of decontextualization) reify this term by relying on a rather static, confined, and deterministic sense of context as place or time. But no “place” is unaffected by history and politics; any specific location is influenced by economic, political, and social processes well beyond its physical and temporal boundaries. Our notion of context must attend to power relations and the critical theories of place and space put forward by critical geographers and anthropologists. Massey (1991, 1994, 2005) argued explicitly against the romantic idea that a place has a single, essential identity based on a limited history of territory. She instead promoted a notion of places as “articulated moments in networks of social relations and understandings,” with much of that “constructed on a far larger scale than what we happen to define for that moment as the place itself” (Massey, 1991, p. 28). So-called local contexts, she argues, are quite heterogeneous and produced from the intersection of social, economic, and cultural relations linked to various scales. Finally, it is important to note that context is not found but produced (processually) by scholars as the study unfolds. As Sobe and Kowalczyk have argued, context should not be conceptualized as a “stable and prefigured analytic category” (2012, p. 60), a container, setting, or background that stands apart from (and causes) the object of interest (p. 71). Establishing the context, they insist, should not be treated as a “preparatory task” that “precedes the real work to be done,” but should instead become a continuous part of the study itself. Context is produced by the scholar (and her or his theoretical framework) in the emergent and ongoing process of research. It is, as Sobe and Kowalczyk put it, “the artifacts of epistemological structures, or ways of knowing the world” (2012, pp. 63–64), which are never removed from broader relations of power. From this perspective, comparison does not “decontextualize” the phenomenon of interest so much as require scholars to “recontextualize” it. Commensurability Comparison begs the question of commensurability, or the notion that the instances or phenomena share a similar measure (Handler, 2013). In some ways, this assumption relies on a notion of universalism. Can all cases be compared for similarities? Must some only be contrasted? Furthermore, when comparisons draw upon categories or boundaries that are determined a priori and not revisited or revised in light of accumulating evidence, they face the challenge of commensurability. Are the categories truly commensurable for multiple cases or sites, or do they impose categories and assumptions from one onto another? Ethnocentrism Related to questions of commensurability are questions of ethnocentrism. In his 2013 Lewis Henry Morgan lecture “The Value of Comparison,” anthropologist Peter van der Veer criticized the “astonishing … pervasiveness of ethnocentrism in the social sciences”: “One of the greatest flaws in the development of a comparative perspective seems to be the almost universal comparison of any existing society with an ideal-typical and totally self-sufficient Euro-American modernity” (2013, p. 3). Ethnocentrism is reinstated through the problem of translation of other languages and conceptual traditions into English and the translation of participants’ concerns and views into theory. “Western” concepts may not fit the social reality of participants. However, recognition of this fact may generate the exaggerated and othering claim that other people or societies cannot be understood in Western terms—and can lead to an often nationalistic, misleading, and fruitless effort to demarcate “Western” and “non-Western,” or “insider” and “outsider.” Reflexivity To address these concerns requires a resolute commitment to reflexivity and self-awareness on the part of the researcher. It requires that scholars who bound a study consider how those boundaries themselves have resulted from historical processes and local, national, and international politics. It demands that scholars remain aware of the histories attached to categories or concepts invoked in the work, as well as the sociohistorical backgrounds and political and economic roots of 2020 processes and phenomena. Comparison requires attention to how methods are entrenched in their linguistic, national, and disciplinary boundaries. Ultimately, comparison, as van der Veer argues, “is not a relatively simple juxtaposition and comparison of two or more different societies but a complex reflection on the network of concepts that underlie our study of society as well as the formation of those societies themselves. It is always a double act of reflection” (2016, p. 29). The Value of Comparison and the Comparative Case Study Approach Comparison is an underrated tool in qualitative studies. It offers scholars the opportunity to refine theoretical insights and to extend the transfer of insights from one site to other times and places. As van der Veer argued in his 2013 Lewis Henry Morgan lecture “The Value of Comparison,” comparison offers a way forward, a clever middle path between irrelevant specificities and empty universalisms. In our 2017 work, we have promoted a comparative case study (CCS) approach that attends to three axes of comparison: • a horizontal comparison that not only contrasts one case with another, but also traces social actors, documents, or other influences across these cases; • a vertical comparison of influences at different levels, from the international to the national to regional and local scales; and • and a transversal comparison over time (Bartlett & Vavrus, 2017). The three axes allow scholars to compare over time and over sites (as socially produced), and they provide for the incorporation of multiple logics of comparison. The horizontal axis encourages comparison of how similar policies and practices unfold across sites, often with distinctly different consequences. A 2017 example comes from research conducted by Pekol (2017), who studied the phenomenon of faculty engagement through a CCS of a human rights education partnership between one U.S. and four Colombian universities. She considered the troubled historical relationship between the two countries as a critical element in the effort to build trust, and she engaged in a vertical analysis through interviews with the staff at the two U.S.-based funding agencies as well as interviews with faculty at the Colombian and U.S. universities. Her primary focus, however, was the horizontal axis that allowed her to compare and contrast faculty views on the concept of engagement and the distinctly different social, political, and economic constraints on the work of faculty at public and private institutions and in these two countries. Another way the horizontal axis can be utilized is through the logic of tracing across sites. For example, Usma Wilches (2015) engaged a CCS approach to examine how universalizing language policies, in this case the importation of English language instruction, are appropriated as they are remade at national, district, and school scales in Colombia. He traced Colombia’s national government policy for adopting English as an “international” language (and its equation by some to “quality and competitiveness”), the adaptation and indigenization of the policy in Medellín, and the ways in which various teachers and leaders in public and private schools, officials, language “experts,” and transnational agencies appropriated the policy. The vertical axis urges comparison across micro-, meso-, and macro-levels or scales. For example, a study of education and migration in the United States should attend not only to homes, communities, classroom, and school dynamics (the micro-level) but also to meso-level district, state, and federal policies as well as factors influencing international mobility at the macro-level. By employing this axis, researchers methodologically avoid the assumption that the “local” has any more “geographic actuality” than other scales (Sobe & Kowalczyk, 2012, p. 56). Indeed, work at each level or scale might entail multiple sites. For example, at the federal level, a study of migration and education could involve thinking about federal education, migration, and labor policies; foreign relations and trade relations; deportation orders; and immigration and customs enforcement offices and officers, among other actors. Finally, the transversal axis, which emphasizes change over time, urges scholars to situate historically the processes or relations under consideration. In her analysis of contemporary bilingual education policy in Peru, Valdiviezo (2013) traced the shift from castellanización, or instruction in Spanish, as official language education policy that aimed to “civilize” indigenous populations throughout the 20th century to bilingual intercultural education (BIE). BIE was an approach pushed by international donors as well as indigenous social movements, and it was adopted as official policy in the early 1990s. BIE could not have come into being without a vast assemblage of actors and institutions whose position vis-à-vis this policy reflected vestigial colonial and neocolonial relations of power. Thus, we find that: (a) the “contemporary context” is contingent upon an historical past, and (b) the phenomenon of interest—2020 BIE policy—cannot be cleaved from these historical antecedents. Valdiviezo’s exploration of how the BIE policy was appropriated and implemented by six teachers in three rural, predominantly indigenous schools in southern Peru, exemplifies the conjoining of axes in the CCS approach. These examples of the use of the CCS approach, and the many more we cite in our book (Bartlett & Vavrus, 2017), demonstrate that comparison in qualitative research necessitates an expansion of the logics of comparison we engage in our research and the axes of comparison to which we attend in our analysis. Conclusions Comparison is an essential but frequently misconstrued analytical move in qualitative research. To better understand comparison in qualitative studies, we need to be clear about the epistemologies and the different methods of comparison, the logics of comparison, the challenges of comparison, and the value of comparison. The CCS approach recommends an emergent and reflexive notion of comparison constructed as the phenomenon of interest becomes increasingly specified over the course of study. The three axes of CCS remind us to consider whether there are potential elements in an assemblage that may be relevant but have thus far been unexplored or insufficiently considered. Thus, the axes promote reflexivity as the qualitative study emerges. References Anderson, G. L., & Scott, J. (2012). Toward an intersectional understanding of process causality and social context. Qualitative Inquiry, 18, 674–685. Find this resource: Bartlett , L., & Vavrus, F. (2017). Rethinking case study research: A comparative approach. New York, NY: Routledge. Find this resource: Becker, H. S. (2009). How to find out how to do qualitative research. International Journal of Communication, 3, 545–553. Find this resource: Bennett, A. (2008). Process tracing: A Bayesian perspective. In J. M. Box-Steffensmeier, H. E. Brady, & D. Collier (Eds.), The Oxford handbook of political methodology (pp. 702–721). Oxford, UK: Oxford University Press. Find this resource: Falzon, M. A. (2009). Introduction. In M. Falzon (Ed.), Multi-sited ethnography: Theory, praxis and locality in contemporary research (pp. 1–24). Aldershot, UK: Ashgate. Find this resource: Gisselquist, R. M. (2014). Paired comparison and theory development: Considerations for case selection. PS: Political Science & Politics, 47(2), 477–484. Find this resource: Guba, E., & Lincoln, Y. (1994). Competing paradigms in qualitative research. In N. K. Denzin & U. Lincoln (Eds.), Handbook of qualitative research (pp. 105–117). Thousand Oaks, CA: SAGE. Find this resource: Handler, B. (2013). The uses of incommensurability in anthropology. In R. Felski & S. Stanford Friedman (Eds.), Comparison: Theories, approaches, uses (pp. 272–292). Baltimore, MD: Johns Hopkins Press. Find this resource: Kuper, A. (2002). Comparison and contextualization. In A. Gingrich & R. G. Fox (Eds.), Anthropology by comparison (pp. 143–163). New York, NY: Routledge. Find this resource: Landman, T., & Robinson, N. (Eds.). (2009). SAGE handbook of comparative politics. Thousand Oaks, CA: SAGE. Find this resource: Locke, R. M., & Thelen, K. (1995). Apples and oranges revisited: Contextualized comparisons and the study of comparative labor politics. Politics and Society, 23, 337–368. Find this resource: Marcus, G. (1998). Ethnography through thick and thin. Princeton, NJ: Princeton University Press. Find this resource: Maxwell, J. A. (2012). A realist approach for qualitative research. Thousand Oaks, CA: SAGE. Find this resource: Maxwell, J. A. (2013). Qualitative research design: An interactive approach. Thousand Oaks, CA: SAGE. Find this resource: Mead, M. (1955). Theoretical setting: 1954. In M. Mead & M. Wolfenstein (Eds.), Childhood in contemporary cultures. Chicago, IL: University of Chicago Press. Find this resource: Mohr, L. B. (1982). Explaining organizational behavior. San Francisco, CA: Jossey-Bass. Find this resource: Pekol, A. (2017). Faculty agency in a U.S.–Colombia university development partnership: Bending toward justice (Unpublished doctoral dissertation). University of Minnesota. Find this resource: Ragin, C. C. (1987). The comparative method: Moving beyond qualitative and quantitative methods. Berkeley: University of California. Find this resource: Ragin, C. C. (2000). Fuzzy set social science. Chicago, IL: University of Chicago Press. Find this resource: Ragin, C. C. (2008). Redesigning social inquiry: Set relations in social research. Chicago, IL: University of Chicago Press. Find this resource: Ragin, C. C. (2014). The comparative method: Moving beyond qualitative and quantitative strategies. Berkeley: University of California Press. Find this resource: Simmons, E. S., & Smith, N. R. (2015). The case for comparative ethnography. Qualitative and Multi-Method Research, 13(2), 13–18. Find this resource: Slater, D., & Ziblatt, D. (2013). The enduring indispensability of the controlled comparison. Comparative Political Studies, 46(10), 1301–1327. Find this resource: Snyder, R. (2001). Scaling down: The subnational comparative method. Studies in Comparative International Development, 36(1), 93–110. Find this resource: Sobe, N., & Kowalczyk, J. (2012). The problem of context in comparative education research. Journal for Educational, Cultural and Psychological Studies, 6, 55–74. Find this resource: Sobe, N., & Kowalczyk, J. (2014). Exploding the cube: Revisioning “context” in the field of comparative education. Current Issues in Comparative Education, 16(1), 6–12. Find this resource: Stake, R. (1994). Case studies. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 236–247). Thousand Oaks, CA: SAGE. Find this resource: Tarrow, S. (2010). The strategy of paired comparison: Toward a theory of practice. Comparative Political Studies, 43(2), 230–259. Find this resource: Usma Wilches, J. (2015). From transnational language policy transfer to local appropriation: The case of the national bilingual program in Medellín, Colombia. Blue Mounds, WI: Deep University Press. Find this resource: Valdiviezo, L. (2013). Vertical and horizontal approaches to ethnography of language policy in Peru. International Journal of Sociology of Language, 219, 23–36. Find this resource: van der Veer, P. (2013). The value of comparison. Transcript of the Lewis Henry Morgan Lecture, November 13. Find this resource: van der Veer, P. (2016). The value of comparison. Durham, NC: Duke University Press. Find this resource: Wiseman, A., & Popov, N. (2015). An introduction to comparing comparative methodologies: A framework for understanding pitfalls and operationalizing promises. In A. Wiseman and N. Popov (Eds.), Comparative sciences: Interdisciplinary approaches (pp. 1–11). Bingley, UK: Emerald Group. Find this resource: Yin, R. (2014). Case study research: Design and methods (5th ed.). Thousand Oaks, CA: SAGE. Find this resource: Lesley Bartlett University of Wisconsin–Madison Frances Vavrus University of Minnesota PRINTED FROM the OXFORD RESEARCH ENCYCLOPEDIA, EDUCATION (oxfordre.com/education). (c) Oxford University Press USA, 2020. All Rights Reserved. Personal use only; commercial use is strictly prohibited (for details see Privacy Policy and Legal Notice). Subscriber: OUP-Reference Gratis Access; date: 03 July 2020 Dismiss