Political science methodology: A plea for pluralism

Studies in History and Philosophy of Science xxx (xxxx) xxx–xxx Contents lists available at ScienceDirect Studies in History and Philosophy of Science journal homepage: www.elsevier.com/locate/shpsa Political science methodology: A plea for pluralism Sharon Crasnow∗ Distinguished Professor of Philosophy Emerita, Norco College, Norco, CA, USA HIG HL IG HTS of the merits of case study research in political science. • Discussion roles for case studies identified and explored. • Six • Argument for methodological pluralism in political science research. 1. Introduction: methods, methodology, and epistemology Case studies were once the primary mode of inquiry in political science. As this approach has fallen out of favor, political scientists interested in questions of methodology have debated whether case study research still has value. These methodological debates offer an opportunity to explore how methodological commitments are intertwined with epistemological commitments as well as value assumptions and implications. The claim that methodology is related to epistemology is not particularly controversial. Roughly, methodology is a normative view about how to produce knowledge. Epistemology is the study of knowledge and its relation to belief. It is unsurprising that there is a connection between how we think about knowledge and views about how to conduct the search for knowledge. The idea that values have a bearing on methodology or that methodologies incorporate value commitments might seem less obvious, however recent philosophical work challenging the value-free ideal of science raises questions about the value neutrality of methodology. As an aid to thinking about the value of case study methodology, I start by distinguishing method, methodology, and epistemology. As Sandra Harding describes them, methods are “techniques for gathering evidence” (Harding, 1987, p. 2). This description is not quite right since it is data that is gathered and that data then becomes evidence when it is analyzed and interpreted. Perhaps methods are better described as techniques for gathering and analyzing data, and then interpreting it so that it may serve as evidence. Methodology is “a theory and analysis of how research should proceed” (Harding, 1987, p. 2). This may include particular ideas about what are appropriate techniques to use when gathering, analyzing, or interpreting evidence, although methodology need not dictate method. Methodology includes views about how best ∗ to conduct research, what sorts of research questions are appropriate, when research should be considered complete, what counts as a result, and how such results should be reported. Finally, Harding describes epistemology as a “theory of knowledge or justificatory strategy” that underlies methodology (Harding, 1987, p. 2). Epistemology seeks to justify methodology. That is to say, an account of why a methodology is successful for knowledge production is given through a theory of knowledge – including a theory of evidence. An epistemological account tells us both that the methodology results in (better) knowledge and provides resources for understanding why such knowledge is better. Is case study research a method or a methodology? It is often described as method. The title of a recent book – Causal Case Study Methods: Foundations and Guidelines for Comparing, Matching, and Tracing (Beach & Pedersen, 2016) – suggests that case study research should be thought of as methods. However, this is perhaps because the authors are focused on various techniques for the use of case studies in causal inferences specifically. Beach and Pedersen distinguish between within-case inferences and cross-case or between-case (comparative) inferences, as do most political methodologists. Within-case inferences result from an analysis of the case using methods such as process tracing – a method in which the hypothesized causal mechanism is “traced” in the case from the hypothesized cause to the effect. Successful process tracing produces evidence that a particular causal mechanism is operating in the case. Process tracing within a case could be thought of as a case study method, however, approaching research through a case seems to be more properly described as a methodology involving a variety of techniques or methods, such as process tracing. Within-case research typically involves the use of a variety of methods – interviews, surveys, archival research, statistical analysis, 925 Archer Street. San Diego, CA, 92109, USA. E-mail address: [email protected]. https://doi.org/10.1016/j.shpsa.2018.11.004 Received 30 November 2017; Received in revised form 14 March 2018; Accepted 16 November 2018 0039-3681/ © 2018 Elsevier Ltd. All rights reserved. Please cite this article as: Crasnow, S., Studies in History and Philosophy of Science, https://doi.org/10.1016/j.shpsa.2018.11.004 Studies in History and Philosophy of Science xxx (xxxx) xxx–xxx S. Crasnow and others.1 Each of these methods might provide information through which a coherent account of the case under investigation can be constructed. The process of doing so is often iterative. While there may be some sense of what counts as “the case” when the research begins, the boundaries of the case only become clear (or clearer) in the research process. Charles Ragin refers to this iterative process as “casing” (Ragin, 1992). To see these various bits of information as part of some coherent whole – the case – involves a methodological commitment to telling the story of the case. Between-case comparison also involves different methods. Examples are qualitative comparative analysis (QCA) and Boolean analysis – both methods aimed at formalizing comparison. Case comparisons depend on prior within-case analysis – at least to the extent of individuating cases to be compared. Again these individuated cases are studied using a variety of methods. Approaching research through cases – either through within-case or between-case comparison – thus seems better understood as methodology rather than method both because such research makes use of a variety of methods and because it includes an understanding of how to best conduct research, what sorts of research questions are appropriate, when research should be considered complete, what counts as a result, and how such results should be reported. Mary Morgan makes a similar point when she describes case study research as an epistemic genre – as a way of doing (social) science (Morgan, 2014, p. 289). The epistemological issues addressed here are primarily concerned with what the methodology offers for knowledge production. Methodology may raise ontological issues as well. For example, when political scientists employ game theoretic models to explain events, they treat political actors as rational agents operating strategically. Similarly, when seeking causal explanations of particular sorts – using mechanisms or through reference to average effects – they may be referring to ontologically distinct types of causality. Julian Reiss distinguishes between conceptual and ontological causal pluralism where the former is merely a matter of thinking of causes differently whereas the latter involves the view that there are different kinds of causes operating in the world. For the purposes of the discussion I will not address ontological issues here. However, the methodological pluralism that I advocate minimally requires a conceptual pluralism about causality. The philosophical methodology that I employ for exploring case study research is naturalistic in the following sense. I begin with an examination of how political scientists use case studies and what they have to say about the methodology. In other words, I start with a descriptive examination of practice. But the examination of case study research methodology leads to a normative conclusion through what it reveals about the role of case studies in knowledge production. The current interest in using case study work together with statistical and experimental approaches – usually referred to as mixed methods research – reflects an awareness of these roles. Thus the way case study research is used in political science reveals the value of the methodology and in this way supports methodological pluralism – a pluralism of approaches to the investigation of political phenomena. Methodological pluralism in this sense is the view that using a variety of approaches – not just a variety of methods or techniques – will produce better knowledge. This is not – or not merely – the claim that different methods producing different sorts of evidence are likely to offer more secure grounds for knowledge claims. Advocates of mixed methods research sometimes do argue that “triangulation” or convergence of evidence in support of a single claim should increase confidence in the conclusion – the more evidence from a wider variety of sources the better. While this is an appealing idea, it is not always clear how evidence from different research traditions or methodologies can work together since the goals and concepts are not always commensurate (Goertz & Mahoney, 2013; Seawright, 2016). The notion of triangulation also fails to capture a number of valuable roles that case studies can play in knowledge production. I identify six such roles: the investigation of rare events; research design; providing evidence for (or against) hypothesized causal mechanisms; revealing the limits of general causal claims; informing policy; and teaching sensitivity to context. The first four speak to the epistemic value of case studies and the fifth raises both epistemic and ethical issues. The last of these – improving sensitivity to context – supports the successful use of cases in their other roles and thus is a particularly important aspect of case study research. The skills gained through case study research are important throughout the discipline. If the primary goal of research is seen as providing evidence that directly bears on causal inference the significance of the supporting processes through which such evidence is produced is likely to be undervalued. Knowledge of cases is a major contributor, both directly and indirectly, to producing such evidence and hence, in advocating for a methodological pluralism, I advocate for recognizing both the variety of activities that go into knowledge production and the broader epistemological framework that such recognition implies. Case study research serves as just one example – a particularly rich one – of what methodological pluralism offers. 2. Cases Case study research in political science involves the close examination of a case. Some examples are a particular country during a particular period, an election, and a political movement at a particular time and place. Sometimes case study research involves detailed study and comparison of a small number of such cases.2 Political science researchers seek explanations of a variety of political phenomena in case studies – civil war, revolution, the peaceful resolution of conflict, or regime change. In so doing, they seek causes and general theoretical principles through which explanations can be given and offer accounts through a narrative presentation of the details relevant to the event in which they are interested. The evidential materials in a case study are predominantly qualitative – interviews, archival materials, contemporaneous documents, and records of the relevant events – although rarely produced exclusively through qualitative methods. The details of the case can include quantitative data, statistical analysis of that data, or statistical analysis of qualitative data that can be treated quantitatively through being coded. Good case research is detail-oriented in order to develop a narrative. The case aims to be a coherent account of the relevant events – relevancy determined in part by what the case is a case of, which in turn depends on the aims of research. Cases, though particular, reference some general phenomenon or category of events. This point is important. While cases are sometimes studied because they are of interest in themselves, they are frequently studied with an eye to what they might reveal about events of a particular type. As John Gerring puts it, a case study is “[t]he intensive study of a single case for the purpose of understanding a larger class of 1 One might wonder if this automatically makes case studies research “mixed methods” research. Most loosely “mixed methods” could mean any use of multiple methods and so case studies would qualify in this sense. However, generally those who endorse mixed methods research mean something more specific. The fundamental idea behind mixed methods research is that through using multiple methods one can produce better evidence. The goal of using multiple methods within a case study is to get a better picture of what happened in the case. Mixed methods research is discussed in more detail in section 3. 2 There is a dispute among political methodologists about whether comparative case study research is worthwhile (cross-case analysis). Beach and Pedersen (2016) do not think so. Goertz (2017) has also raised questions about it. However, comparative case study work has been and continues to be used in the discipline. 2 Studies in History and Philosophy of Science xxx (xxxx) xxx–xxx S. Crasnow similar units (a population). … a case study research design may refer to a work that includes several case studies, for example, comparativehistorical analysis or the comparative method” (Gerring, 2012, pp. 410–411). Gerring signals the intent of political science case study research to move beyond the particular towards the general and, in so doing, he touches on one of the biggest difficulties of case study research: How can an inference from a single case, or even several cases, support the sort of general conclusion that the sciences (presumably including political science) seek? This worry becomes particularly pressing in the current disciplinary climate where the focus is on seeking evidence to support general causal claims. Whatever might appear to be causally linked within a case may be the result of the specific interactions among various factors at play in that case. These factors may not all be present in other cases – even in similar cases. Establishing that one can transport what is learned from a case elsewhere is thus highly problematic.3 Over the past several decades, statistical (large-N observational) research approaches and, more recently experimental approaches, have dominated political science research. While there are important differences between large-N observational and experimental approaches, they also share a number of methodological commitments. For the purposes of the following discussion, the most important of these is that they are both what Beach and Pedersen call “variance-based” approaches (Beach & Pedersen, 2016). Variance-based research designs identify causality through significant differences in average treatment effect between the treated (experimental) and untreated (control) group (Beach & Pedersen, 2016). Such variance-based approaches take their underlying rationale from Mill's Method of Difference in which a comparison is to be made between two circumstances differing only in one respect (the treatment). However, Mill did not think that experiments were possible in the social sciences primarily due to the impossibility of identifying potential confounders – unknown and possibly unknowable additional factors that might affect the outcome. In other words, Mill did not think it possible for an experimenter to ensure that the two circumstances differed only in regard to the treatment in the social world. Randomized controlled experimental design is thought to address this worry about confounders through randomized assignment of members to control or treatment groups. Probability theory predicts that given enough repeated trials (a large enough population) the results should converge to the mean thus ultimately eliminating the effects of confounders. If a significant difference in means (difference in average effects) is found, it provides evidence in support of the causal influence of the treatment.4 But if one is studying the effects of inequality on democracy, for example, it is hard to see how experimental methods might be used. How could researchers randomly assign members of the population to different economic strata? Given that much of political science research involves exploring these sorts of questions, experiments were relatively rare in political science until recently.5 Large-N observational studies depend on statistical techniques to control for confounders and thus can been seen as approximating experiments. They also seek a difference in the outcome variable (the dependent variable) depending on whether the suspected causal variable (independent variable) is present or not. Thus both approaches share a commitment to the fundamental idea underlying the method of difference – that there should be a difference in the outcome where the suspected cause is present from those circumstances where it is not – and more specifically that the difference will be exhibited as a difference (variance) in average effect. Methodological debates in political science have been strongly influenced by the idea that there is one “logic of inference” as King, Keohane, and Verba (1994) argued. According to their account, all methodological approaches should follow the same underlying pattern of reasoning. At the time that they wrote they were more focused on modeling qualitative research (as it might be used in case study research) on statistical reasoning, however, the general point speaks to a preference for variance-based reasoning as I have described it. This claim – that all methodological approaches should follow the same pattern of reasoning – encourages researchers to think of evidence in isolation from context. In fact, a claimed virtue of variance-based methods – both statistical and experimental – is that they allow researchers to isolate and identify causes. Thinking of causal inference in this way emphasizes a “moment of inference” – a point at which the evidence licenses the causal conclusion – and may lead to downplaying or even ignoring the relevance of background knowledge (beliefs and assumptions) and aims (practical implications). The result is an epistemological narrowness that both supports and is supported by a methodological imperialism – a view that knowledge is best produced through some one right methodology superior to and not dependent on others. In contemporary methodological discussions case study research is often contrasted with research using either large-N observational or experimental methods. Randomized controlled experiments are usually understood to be the gold standard of evidence for causal claims since randomization together with the law of large numbers is thought to control for confounders.6 However, given that it is difficult to randomize the assignment to control and experimental groups in most of the circumstances that political scientists seek to study, political science experiments make use of many of the same statistical techniques used in large-N observational research in order to approximate random assignment. Another way of putting this point is that the same sorts of assumptions required to make use of large-N statistical techniques in causal inference are often required to produce experimental results. For this reason, in what follows I mostly treat these together as variancebased approaches.7 Case study research offers a different methodology – not merely a different method. Consider again how methodology is characterized above. It includes views about how best to conduct research, what sorts of research questions are appropriate, when research should be considered complete, what counts as a result, and how such results should be reported. Statistical and experimental approaches offer sets of tools to use in conducting research, but these tools come with the evaluative understanding that they are the best tools for giving answers to the research questions that are and ought to be asked. The specific techniques within these traditions are best suited to answering questions about average effects in populations. If these are the only questions that are appropriate then that is an argument in support of the methodology. But it is not at all clear that there are no questions worth asking (and attempting to answer) that cannot be addressed through such methods. For example, suppose we want to know about a one-off or rare event where the population is too small for such techniques to be useful. If researchers believe that only these methods are appropriate, that is a methodological commitment – not just a preference for method. It carries with it a 3 I do not mean to suggest by this that establishing a causal claim within a case is straightforward. 4 A “significant difference” is one that could not be attributed to chance. 5 Experiments first appear in more psychologically oriented spheres of political science – for example, in the study of voting behavior. Experimental research design is now widely used, having traveled to the discipline via development economics. 6 The law of large numbers says roughly that repeated trials – in the case of randomized controlled experiments, a large enough population for control and experimental groups – the probability of the outcome will converge on the expected average outcome. 7 There are other important questions unique to experiments in political science, for example, worries about sample size and ethical concerns, particularly for field experiments. I do not discuss them here. 3 Studies in History and Philosophy of Science xxx (xxxx) xxx–xxx S. Crasnow conception of what counts as a legitimate research question, what counts as a result (an end) of research, and what sorts of things should be studied– populations and not individuals or singular event. Although the tradition of case study research has not vanished from political science, its importance is diminished. While most political methodologists acknowledge that case studies are useful for a variety of heuristic purposes, there is skepticism about their ability to provide evidence for causal claims since the move from a specific case to a general conclusion is not inductively supported. This skepticism is further reinforced by an average effects conception of causal claims. One result of these methodological preferences is that researchers who have continued to work in the case study tradition often find themselves on the defensive. Their methodology seems not to produce the most coveted prize – a significant result. Gary Goertz (2017) argues that mixed method research involves a commitment to searching for causal mechanisms – something that requires case study research. His reasoning is that neither large-N statistical methods nor experimental methods can provide evidence for a specific causal mechanism although they may provide evidence that some mechanism is operating. But, as Goertz notes, there are political scientists that reject the idea that knowing mechanisms depends on examining cases – King et al. (1994), for example, argue that evidence for mechanisms can be garnered through statistical methods by focusing on causal relations at each stage of the mechanism. Claims about mechanisms are thus reduced to a series of discrete causal claims. At its most extreme this approach would eliminate any need for appealing to mechanisms. I address these issues briefly. First, as noted in section 1, I put aside the question of whether the different conceptions of causality associated with different methodologies are ontologically distinct. It is not always clear whether evidence from case studies and large-N evidence can be combined to support the same causal claim – partly because it is not always clear what “the same” causal claim is.10 There are circumstances in which the use of cases to discover something about a causal mechanism seems to fit with the use of statistical method. Statistical evidence might be used to 1) identify that there is a causal connection between two types of events in the first place and 2) establish that there is or is not a correlation between tokens of those types operating in a way that conforms to some hypothesized mechanism through which events of that type are thought to be connected. The second of these may involve process tracing to show that the elements of a hypothesized causal mechanism are present on the path from the cause to the effect. Evidence that the mechanism is operating – bringing about the effect – would include specific types of events that conform to what we would expect if the mechanism were at work. I have argued elsewhere (Crasnow, 2012) that whether combining evidence from methods is successful depends on a variety of factors. When combining statistical research with investigation of singular events the notion of singular cause must be able to be integrated with the causal account underlying the statistical work.11 That is to say that the way singular causation is understood must be commensurate with the statistical understanding employed if the evidence is to be combined. One way of achieving this would be through showing how a singular cause can be treated within, or compatibly with, the statistical account. An argument that the same or compatible conceptions of causality are being used must be offered or must be generally accepted. Currently, such arguments are rarely given. Two possible explanations for this come to mind. First, researchers simply assume they are using a uniform sense of cause. Second, abstract questions about conceptions of “cause” are typically questions for philosophy of science and the variety of accounts of causality available there do not readily support a unified or integrated notion of causality (see Reiss, 2012). In addition, phenomena used in the hypothesis have to be understood and operationalized consistently in both the statistical work and in the hypothesis under investigation. Measures of latent variables or abstract concepts – such as democracy – need to be consistent across both the statistical work and the case study research.12 In summary, the different methodologies have to be shown to share both an understanding of causality and also of the phenomena under investigation. 3. Case studies and mixed methods Given the usefulness of large-N statistical studies and more recently of experimental work, methodologists who still value case study research increasingly advocate mixed or multi-method research (Beach & Pedersen, 2016; Goertz, 2017; Seawright, 2016). Mixed methods research is research that combines two or more methods in order to increase inferential leverage.8 To clarify, at least in political science, the term has come to mean not just the use of multiple methods – case studies have always used a variety of methods – but the use of evidence generated through different methods in order to support causal inference. Thus what is being claimed is that case study research can play an evidential role in causal inference. How mixed methods research designs accomplish this is debated and a variety of epistemological justifications are offered to support mixed methods research for causal inference. Two main justifications are 1) causal claims are better supported with evidence derived through a variety of sources and 2) since methods all have different strengths and weaknesses, combining methods will compensate for any weaknesses of individual methods. These justifications are speaking to a conception of mixed methods research in which the results of each method are thought to converge as support for a causal claim. The metaphor of “triangulation” is often used to capture this idea. Jason Seawright offers the following characterization of this intuition: “Simply put, triangulation designs involve asking the same question of causal inference using two different methods, and checking that the same substantive conclusions are produced by both” (Seawright, 2016, p. 4). A number of political methodologists have pointed out that such arguments fail to acknowledge that different methods are associated with different methodological approaches. Goertz and Mahoney (2013), for example, argue that these different approaches define “two cultures” differing in values, beliefs, and norms and resulting in different research procedures and practices and ultimately in different modes of causal inference. Beach and Pedersen (2016) appear to agree: “Debates about the nature of causation are important to understand because different conceptions of causation result in different types of causal claims, which then imply that different methodologies are needed to investigate them appropriately” (Beach & Pedersen, 2016, p. 15). Seawright also worries about the use of the metaphor noting that it is often hard to determine in what sense the results might be said to agree given that the results are, in some sense, of different kinds.9 (footnote continued) considered relative to conflict – to the logged percentage of mountainous area of Colombia in some way, it is not immediately obvious how to do so. These pieces of information appear to be answering quite different questions (Seawright, 2016, pp. 5–8). 10 Additionally there is the issue of what to do when methods yield conflicting results and areas in which there is no agreement (Seawright, 2016, p. 4). 11 An example of an attempt to give such an account is Woodward’s 2003 discussed in Crasnow, 2012. 12 This appears to be the issue that Seawright is focused on in the example referred to in footnote 9. 8 I will use “mixed methods” although “multi-method” is also widely used. Nothing discussed here depends on the terminology. 9 Seawright has an extended example in which he looks at quantitative and qualitative (case study) evidence for the role of mountainous geography for civil wars. Looking at the logged percentage of Colombia's mountainous area differs in kind from looking at the geographical layout of the particular mountain ranges in Colombia. While it might be possible to compare the information that comes from case analysis – some of which shows potentially relevant differences between some mountainous areas and others when 4 Studies in History and Philosophy of Science xxx (xxxx) xxx–xxx S. Crasnow In the previous section I raised concerns about the idea that mixed methods worked seamlessly as the metaphor of triangulation suggests. Seawright (2016) shares this concern and proposes an additional way of thinking about mixed methods research, distinguishing triangulation from what he refers to as “integrative” multi-method design: “Integrative designs are multi-method designs in which two or more methods are carefully combined to support a single, unified causal inference. With such a design, one method will produce the final inference, and the other is used to design, test, refine, or bolster the analysis producing that inference” (Seawright, 2016, p. 8). Seawright identifies four roles for the integrative use of case studies: 1) turning up evidence that helps to set scope conditions or identify causes in one or a few cases; 2) improving or validating measurement; 3) building or testing hypotheses; and 4) aiding in the identification of potential omitted variables (Seawright, 2016, p. 46). The six roles that I identify overlap to some extent – for example, I consider both building hypotheses and identifying confounders as aspects of research design. Determining limits for general causal claims is, at least in part, a matter of determining scope conditions. However, Seawright is concerned primarily with mixed methods and the integration of case study work specifically in order to support causal inference. He does not address the question of rare events. Additionally, he does not consider the aim of producing knowledge for policy – a goal that perhaps requires a different approach to evidence as Nancy Cartwright has argued (Cartwright, 2006; Cartwright & Hardie, 2012).13 Nor does he address the role that training in case study research has for knowledge produced by the discipline as a whole. Seawright's focus is mixed methods research – which is one benefit of methodological pluralism – but the argument that I make here is about methodologies. I am arguing that the political scientists should continue embracing different approaches. This includes embracing different goals, concepts, and frameworks. learn something about how to foresee and perhaps even control them. Airplane crashes are rare events and yet when they do occur they are studied in great detail for these reasons. Failures of expected regularities suggest areas where knowledge could be improved. Additionally, regularities might explain that an event was likely to occur or of a sort that might happen under a particular set of circumstances, but not why this event occurred at this particular time and in the particular way that it did. Such arguments conform with the notion that the study of the mechanisms through which a singular event happens support a recognition of the important heterogeneity of cases – a key insight of the mechanisms approach (Seawright, 2016, p. 30). Second, case study work serves a heuristic role through informing research design. Researchers need detailed knowledge of relevant cases in order to identify potential confounders, specify models, and determine alternative hypotheses. This works in a variety of ways. One is through supporting the counterfactual reasoning that can aid in each of these tasks. Identifying potential confounders, specifying models, and suggesting alternative hypotheses are all aided by counterfactual thinking – a consideration of most similar possible worlds in which other factors could play a causal role. The underlying counterfactual questions are whether the effect could have come about in some other way or whether it would have occurred had the suspected cause not been present. Knowing what is relevantly similar or relevantly different, determining which changes are the minimal and which not depends on knowing details of cases. Case studies inform specific research designs variably. Susan Hyde's natural experiment design depends on her knowledge of post-Soviet Armenia derived through her previous research of that country – previous case study research that she had engaged in. She uses what she knows in order to investigate the effect of independent observers on election fraud in the 2003 Armenian election (Hyde, 2007).15 Her knowledge of the case guides her in constructing the arguments that the research design is indeed a natural experiment through which to examine the impact of independent observers. She argues that because the incumbent president of Armenia was the only one with the power to carry out election fraud at the polls, detecting a difference between monitored and unmonitored sites through the number of votes for the incumbent is a proxy for how successful independent observers were at limiting election fraud. Fewer votes for the incumbent indicate fewer fraudulent ballots. Comparing monitored sites with unmonitored sites thus provides a way of assessing the effectiveness of monitors, if the monitors have been randomly assigned. She supports that they were through qualitative methods often used in case study research – for example, interviewing members of the international body that was responsible for distributing the observers.16 For experiments, case studies can be an aid in the identification of subpopulations in order to consider heterogeneous treatment effects (identifying potential confounders). A similar point can be made for observational methods. What is known about cases informs matching, sampling, and other techniques for adjusting observational data and correcting for problems with sampling. Third, case study research may play a role in producing evidence for or against hypothesized causal mechanisms. The specification of such mechanisms includes intermediary events, actions, the structure or behavior of institutions, or other facts that may be identified (or not) through process tracing. Although finding such evidence does not confirm a hypothesis, it may “infirm” a hypothesis (weaken its acceptance) (Campbell, 1975; Morgan, 2012) and under some circumstances – when all alternatives are eliminated – it may be possible to “clinch” 4. How is case study methodology valuable? As stated previously, there are at least six ways in which case study research contribute to the production of knowledge in political science: the investigation of rare events; research design; providing evidence for (or against) hypothesized causal mechanisms; identification of confounders or omitted variables; revealing the limits of general causal claims; informing decisions about implementing policy; and teaching sensitivity to context. In this section I explore each of these briefly. First, even if it is agreed that the best research practices are large-N observational and experimental, we often want to know about events that cannot be studied in these ways. Many of the phenomena that political scientists are interested in are rare events: wars, democratic transitions and reversions, civil wars, revolutions, and so on. Such events cannot be analyzed through large-N methods simply because there are typically too few of them for these methods to be effective. If there are enough events of a type, cross-case small- or medium- N comparisons could be useful together with cross-case comparative methods.14 Political science resembles history in some circumstances where a one-off case study of a singular event is what is required. The Cuban Missile Crisis and the collapse of the Soviet Union provide examples of this type. It might be argued that rare events are not a proper topic of science, which should instead seek regularities or laws. But this criticism would seem to ignore that we are in fact interested in these sorts of events, not only because we seek explanations of them but also because we hope to 13 15 Cartwright distinguishes evidence for use from evidence for warrant. One issue with the political methodology discussion is that it is primarily concerned with evidence for warrant. This is one aspect of what I refer to later in the paper as an epistemological narrowness. 14 Although the usefulness of such methods is debated as previously noted. In investigating this case, Hyde hopes to shed light on the effect of independent observers on election fraud more generally, using the results from the Armenian natural experiment to do so. 16 There is other evidence offered as well. The example is discussed in detail in Crasnow, 2015. 5 Studies in History and Philosophy of Science xxx (xxxx) xxx–xxx S. Crasnow the rejection of the hypotheses (Cartwright 2012). In addition, process tracing may produce a preponderance of events when competing hypotheses are evaluated relative to the details of the case. The “fit” with what is expected may increase the degree of belief in one hypothesis over others in this way. Bennett (2008, 2015) and Beach and Pedersen (2013, 2016) have argued that a Bayesian analysis of process tracing provides the epistemology underlying this method. Evan Lieberman's use of a mixed method – what he calls “nested analysis” – also provides an example (Lieberman 2005; 2015). Beginning with large-N research that indicates that there is a causal connection between a dependent and independent variable, Lieberman suggests that one can move to a selection of cases that are then “nested” into the analysis. Here one investigates within-case processes (mechanisms) through an examination of a variety of evidentiary materials. The nesting involves a coordination of the coding rules for the large-N research with the specifics of the cases examined – the sort of effort that could support an argument for triangulation. Also in carrying out this process the researcher may discover a number of other important factors that are integrative: the conformation (or not) of the case with the large-N findings, limitations and hence the scope of the findings, suggestions for alternative hypotheses or expansions of the original hypothesis. Another example of process tracing is Kenneth Schulz's analysis of the Fashoda Incident – a late 19th century confrontation between Great Britain and France in the upper Nile Valley that nearly resulted in war. Given the Democratic Peace hypothesis – the claim that democracies do not go to war with each other – Schultz seeks to use the case to test various hypotheses about the mechanisms through which democracies maintain peaceful relations with each other. He argues that the details of the case allow us to trace, and hence argue for, a peace sustaining mechanism by showing events to be consistent with his preferred hypothesis, while also inconsistent with the other proposed hypotheses (Schulz 2001).17 Fourth, case studies can indicate the limits of generalizing causal claims and point to specific factors that either serve as helping factors – previously invisible background conditions that must be met in order for a causal mechanism to operate – or disrupting factors – features of the environment that can prevent the mechanism from producing the usual effect. As Haggard and Kaufman put it in their mixed method research on democratic transitions and reversions: “The focus on average treatment effects masks the heterogeneity of actual transition paths; the variable in question may be significant or not significant across a population, but this does not necessarily provide useful information on the outcomes in a particular case” (Haggard & Kaufman, 2016, p. 23). Seawright notes that the search for mechanisms that is so often associated with case studies is “part of a thoroughgoing commitment to the proposition that causal relations in the social world are heterogeneous” (Seawright, 2016, p. 56). The revelation of these constraints can suggest multiple means of producing or preventing outcomes, a feature of case study research that may inform policy – a fifth role that case study research may play. Even when experimental or observational evidence for average treatment effects is strong, using such knowledge for policy is not straightforward, in part because to know that a general (probabilistic) causal principle applies does not ensure that it is operating or operating unobstructed in the particular case where the policy is being considered. Informing policy through an understanding and comparison of the contexts of the original research and that in which the policy would be implemented depends on knowledge of details – the sort of knowledge that case studies provides. Exploring differences among similar cases provides an understanding of contextual features relevant to the effective implementation of policy (Cartwright & Hardie, 2012). There are two considerations to be aware of here. First, although the evidence that both large-N observational and experimental designs provide is typically intended to support causal claims that are widely applicable in different contexts, the assumptions that are built into both the research and the analysis of the data are often empirically substantial in a way that could limit the transportability of such claims. Consequently, understanding which “supporting factors”, as Cartwright (2013) calls them, are needed for the effect to occur as hoped can be crucial. In addition to Cartwright's worries, there is a further ethical/social justice concern when using average effects causal evidence to inform policy. When evidence for evidence-based policy is constrained by an epistemological commitment to a methodology that only acknowledges average effects evidence for causal claims, then the methodology itself makes details of context irrelevant. Differences among individuals are these sorts of details. Methodological commitments include a conception of evidence – an epistemology – that may not be appropriate for all ends. Thus the commitment to average effects or variance-based conclusions carries with it a particular stance towards individuals affected by policy implementation. It is a stance that downplays difference and de-values the specificity of the individual. Such an approach threatens to have a disproportionate impact on those who are who are not “average” in the requisite sense – typically those who are underrepresented or marginalized.18 A possible corrective can be to acknowledge the heterogeneity of the population and do further research on subpopulations. This move addresses some aspects of the problem, but also further highlights the relevance of case research for research design. Which factors determine relevant subpopulations will depend on the sorts of details that case knowledge provides, specifically what further factors may be causally relevant (Crasnow, 2011). However, it is not clear how this strategy addresses interactive causal factors such as those that might arise when intersectional analysis is appropriate nor does it address the underlying conceptual framework in which the uniqueness of the individual does not register as relevant.19 Further partitioning the reference class may also decrease the sample size and thus undermine the power of the method. These are both epistemic and moral issues and these concerns raise questions about the relationship between them. It might be argued that factors relevant to social justice issues are not relevant to consideration when discussing issues of evidence for causal claims. Such an argument is based on a distinction between epistemic and non-epistemic values and the presumption that non-epistemic values cannot play an evidential role. Although it might be tempting to relegate social justice values to a role only in relation to evidence for use, the analysis above suggests that evidence for use and evidence for warrant may not be so easily disentangled. Of these roles, it could be argued that only the role for case studies in providing evidence for causal mechanisms is a direct evidential role. The others are indirect or, in the case of guiding public policy, evidential in the sense of evidence for use (evidence that supports policy recommendations) rather than evidence for warrant (evidence that supports the causal inference). Such an argument presupposes an understanding of evidence isolated from the context through which it becomes evidence – the supporting factors (Cartwright, 2013). While arguably some evidence may be “closer” to the conclusion than some background beliefs, all function as part of the evidential framework that supports the causal inference. The heuristic use of case studies occurs when they are used to produce evidence that supporting factors that enable causal mechanisms to operate are present. These supporting factors play a role both in the evidence for use and evidence for warrant. To argue that case studies provide evidence of supporting factors is to argue that such evidence is directly relevant to a causal conclusion 18 Donal Khosrowi, 2018 makes a similar point. Intersectional analysis approaches questions about multiple axes of oppression – that a black woman, for example, is subject to both racism and sexism and that this cannot be accounted for with an additive model (not racism plus sexism, but requires some more complex causal analysis). The term originates with Kimberlé Crenshaw (Crenshaw, 1989). 19 17 The role of process tracing in providing evidence for causal mechanisms is discussed in detail in Crasnow, 2017. 6 Studies in History and Philosophy of Science xxx (xxxx) xxx–xxx S. Crasnow since the causal claim is a claim not merely about an isolated causal factor but about the factor together with the other factors without which the effect would not have occurred. While we might focus on one factor and refer to it as a cause this is with the understanding that it is one necessary condition among many others.20 There are other accounts of evidence that emphasize the role of background beliefs and assumptions. Helen Longino's contextual empiricism is one example (Longino, 1990, 2001).21 Longino's account also speaks to the concern about the values as evidence as do several other approaches (Solomon, 2001; Anderson, 2004; Douglas, 2009, 2016). The main thrust of such contextualist understandings of evidence is that they weaken the distinction between direct and indirect evidence and treat evidence holistically. Evidence becomes evidence within a context and the context includes relevant factors of the sort that good case study work provides as well as the goals for which the knowledge is sought. Such accounts emphasize the importance of all elements of the process of knowledge production and not only causal inference. limits of generalization, and the consideration of alternative hypotheses all of which are important for the successful use of any research method. One might object that it is not clear that doing case study work is the only way in which sensitivity to context can be achieved. I do not claim that it is. My argument is that it is one way in which this can be accomplished, but this benefit together with the other roles that case studies can play support the claim that there should be continued training in and practice of case study methodology in the discipline. There are also reasons to think that case study research may be uniquely suited to encourage the awareness of context argued for here. First, a case is approached as a complex whole. To identify the case as a whole is not to say that the researcher knows what its boundaries are prior to the investigation, nor even that they know what it is a case of (remember Gerring's comment that the case refers to a larger class of similar cases). Researchers begin with a preliminary idea but it is in the process of studying the case that its limits emerge. These limits may be temporal – the start and end dates of an historical case for example – or geographical – the relevant borders of the region being investigated. As previously noted, this is the process that Ragin describes as “casing” (Ragin, 1992). Casing is an activity that brings the empirical and the theoretical together – the empirical details of the case that are relevant are delineated by the concepts through which the aspects of the case are conceived and the hypotheses through which it is investigated. Ragin is not focusing on the inferential role of cases – some way in which they would support or disconfirm a causal hypothesis – rather he is interested in the iterative process through which a researcher comes to understand the case as a whole. The notion of “casing” is a good one. It makes reference to the importance of the details but it is a valuable way of thinking about case study work in that it emphasizes process. Case study methodology differs from variance-based approaches in that they aim at a result – an inference to a causal claim – not the process through which a case is understood. Consequently, methods that support variance-based reasoning or average effects conclusions emphasize a different set of skills. Arguably we want to discover effects across populations, differences within populations, and understand what happens to individuals in those populations – although we may not always want to know all of these things at the same time or for the same purposes. But I have also argued that techniques for finding effects across populations frequently require information about details that distinguish members within those populations. If so, while it may not be necessary that all political science research involve case study research, the skills that it hones support a variety of different goals and so are good for the discipline as a whole. 5. Learning from cases I conclude with a sixth role for case study methodology. Producing case studies, and the research this requires, both increase awareness of the importance of context. I include as part of context the research questions asked, the beliefs and presuppositions that give rise to those questions, the aims of the research, the explicit and implicit background beliefs and presuppositions required to conduct the research (including those that are required by the methods used), how the objects of inquiry are understood (concept development), and the material conditions of the research. There are two ways in which case study research aids in revealing context. Researchers learn the needed particulars surrounding the event of interest – that is, they learn about the case. Additionally, those engaged in case study research learn what it is to pay attention to the details of the context. They learn how to do case study work through doing it.22 Ideally, they have both knowledge-that (knowledge about the case) and knowledge-how (how to study a case). I have argued that the details – the knowledge of the case – are important in a variety of ways, but I also want to claim that there is a skill developed in learning to examine the complexity of a case and ordering those details into a complex whole. Learning to do something – like learning to drive a car – involves discrete episodes of practice in different situations. While a new driver will not have encountered every situation that they will meet on the road, they will have firsthand experience of many and will know something of how to respond in those circumstances. But even when coming across new situations – situations not encountered during drivers’ training – new drivers are sensitized to what sorts of things to pay attention to, what sorts of results their actions might have, and thus what features of their surrounding, for example, are likely to matter for the purpose of arriving safely at their destination. Learning to do case study research functions in a similar way. Researchers learn not only what matters in the particular case studied but also what kinds of factors are likely to matter. This awareness has the potential to inform research design, hypothesis testing, awareness of the 6. Epistemological conclusions The epistemological framework underlying variance-based approaches focuses on the inference from evidence to causal claim in isolation rather than on the work that goes into producing and interpreting a result so that it can become evidence. An epistemology that acknowledges how evidence is embedded in a broader system of knowledge is one that offers a better home to case study research. When evidence is treated in isolation, it forces us to think of the role of evidence in cases through what Katherina Kinzel describes as a “confrontational model” – in which the case confronts the hypothesis and either challenges it or comports with it (Kinzel, 2015, p. 49). But even when cases are used to challenge or support hypothesized causal mechanisms this typically occurs through process tracing – a method that is more dependent on the details of the case and how they are embedded in the entire narrative of the case than a confrontational model would suggest. Political scientists have rightly welcomed large-N and experimental methods because they offer a clear way to garner support for causal claims. My arguments for methodological pluralism have played up the role of case study research because it is currently a methodological underdog, not because I think it should replace these other approaches. 20 This analysis is in keeping with Mackie's account of causality as INUS conditions. On this account a cause is an insufficient but necessary part of an unnecessary but sufficient set of conditions for bringing about the effect (Mackie, 1965). 21 Another example is Reiss's (2014, 2015) pragmatist/inferentialist and contextualist theory of causation and evidence. 22 Bent Flyvbjerg makes a similar point about learning in tying case study research to what we know about learning through cognitive science. “[T]he case study produces the type of context-dependent knowledge that research on learning shows to be necessary to allow people to develop from rule-based beginners to virtuoso experts (Flyvbjerg, 2006, p. 221). 7 Studies in History and Philosophy of Science xxx (xxxx) xxx–xxx S. Crasnow But the emphasis on causal inference as the culmination of knowledge production draws our attention away from the complications of the process and the phenomena under investigation. If causality in the social sciences is complex and systematic – as indeed it appears to be – it is misleading to seek to support causal claims without close attention to the context in which causality occurs. Knowledge production is iterative, and ongoing – local while referencing the general, making general claims with an awareness of the need to modify for specific circumstances, and never wholly transportable but with the potential to partially inform us in new circumstances. The methodological pluralism that I argue for here supports the turn to mixed method research, however, not for the reasons that are usually proffered. The triangulation arguments neither fully develop the metaphor of triangulation nor do they address the problem that different methods sometimes produce conflicting evidence. Even when evidence appears to converge, further argument is required to support the claim that the compatibility of both the understanding of the causal claim and the understanding of the key abstract concepts are compatible. The integrative role of mixed method research acknowledges that evidence produced through one method frequently requires additional support. These supporting roles are additional aims for knowledge production and thus incorporate an understanding of knowledge production that includes additional goals other than direct support of causal inference. Even if one were to take inferential support for causal claims as the primary goal of political science, it is still necessary to search for background knowledge, to consider ethical presuppositions of methodology and ethical implications of applications, as well as to investigate context in ways that are not available through the currently preferred variancebased methods. Consequently examining the ways in which case study research improves knowledge production speaks not only for methodological pluralism but also for an epistemology that is broad enough to recognize the multiple aims for which knowledge is sought, the ways in which knowledge is shaped by those aims, and how what we know is embedded in our networks of beliefs and values that result from and are best formed through a variety of ways of investigating the world. Bennett, A. (2008). Process tracing: A bayesian perspective. In J. M. Steffenmeier, H. E. Brady, & D. Collier (Eds.). The Oxford handbook of political methodology (pp. 702– 721). Oxford: Oxford University Press. Bennett, A. (2015). Appendix: Disciplining our conjectures: Systematizing process tracing with Bayesian analysis. In A. Bennett, & J. T. Checkel (Eds.). Process tracing: From metaphor to analytic tool. Cambridge, UK: Cambridge University Press. Campbell, D. (1975). Degrees of freedom’ and the case study. Comparative Political Studies, 8(2), 178–193. Cartwright, N. (2006). Well-ordered science: Evidence for use. Philosophy of Science, 73(5), 981–990. Cartwright, N. (2013). Evidence, argument and prediction. In V. Karakostas, & D. Dieks (Eds.). EPSA11 Perspectives and foundational problems in philosophy of science, the European Philosophy of Science Association Proceedings 2. Cartwright, N., & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better. Oxford: Oxford University Press. Crasnow, S. (2011). Evidence for use: Causal pluralism and the role of case studies in political science. Philosophy of the Social Sciences, 41(1), 26–49. Crasnow, S. (2012). The role of case study research in political science: Evidence for causal claims. Philosophy of Science, 79(5), 655–666. Crasnow, S. (2015). Natural experiments and pluralism in political science. Philosophy of the Social Sciences, 45(4–5), 424–441. Crasnow, S. (2017). Process tracing in political science: What's the story? Studies In History and Philosophy of Science Part A, 62, 6–13. Crenshaw, K. (1989). Demarginalizing the intersection of race and sex: A black feminist critique of anti-discrimination doctrine, feminist theory and anti-racist politics (pp. 139– 167). University of Chicago Legal Forum. Douglas, H. (2009). Science, policy, and the value-free ideal. Pittsburgh, PA: University of Pittsburgh Press. Douglas, H. (2016). Values in science. In H. Paul (Ed.). The oxford handbook of philosophy of science (pp. 609–630). Oxford: Oxford University Press. Flyvbjerg, B. (2006). Five misunderstandings about case-study research. Qualitative Inquiry, 12(2), 219–245. Gerring, J. (2012). Social science methodology: A unified framework. Cambridge, UK: Cambridge University Press. Goertz, G. (2017). Multimethod research, causal mechanisms, and case studies: An integrated approach. Princeton: Princeton University Press. Goertz, G., & Mahoney, J. (2013). A tale of two cultures: Qualitative and quantitative research in the social sciences. Princeton, NJ: Princeton University Press. Haggard, S., & Kaufman, R. (2016). Dictators and democrats: Masses, elites, and regime change. Princeton, NJ: Princeton University Press. Harding, S. (1987). Introduction: Is there a feminist method? In S. Harding (Ed.). Feminism and methodology (pp. 1–14). Bloomington: Indiana University Press. Hyde, S. D. (2007). The observer effect in international politics: Evidence from a natural experiment. World Politics, 60(1), 37–63. Khosrowi, D. (2018). Trade-offs between epistemic and moral values in evidence-based policy. Economics and Philosophy, 1–30. https://doi.org/10.1017/ S0266267118000159. King, G., Keohane, R. O., & Verba, S. (1994). Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press. Kinzel, K. (2015). Narrative and evidence. How can case studies from the history of science support claims in the philosophy of science? Studies in History and Philosophy of Science, 49, 48–57. Lieberman, E. S. (2005). Nested analysis as a mixed-method strategy for comparative research. American Political Science Review, 13, 37–59. Lieberman, E. S. (2015). Nested analysis: Towards an integration of comparative-historical analysis with other social science methods. In J. Mahoney, & K. Thelen (Eds.). Advances in comparative-historical analysis (pp. 240–263). Cambridge: Cambridge University Press. Longino, H. (1990). Science as social knowledge: Values and objectivity in social inquiry. Princeton, NJ: Princeton University Press. Longino, H. (2001). The fate of knowledge. Princeton: Princeton University Press. Mackie, J. L. (1965). Causes and conditions. American Philosophical Quarterly, 2(4), 245–264. Morgan, M. (2012). Case studies: One observation or many? Justification or discovery? Philosophy of Science, 79, 667–677. Morgan, M. S. (2014). Case studies. In N. Cartwright, & E. Montuschi (Eds.). Philosophy of social science: A new introduction (pp. 288–307). Oxford: Oxford University Press. Ragin, C. (1992). ‘Casing’ and the process of social inquiry. In C. Ragin, & H. S. Becker (Eds.). What is a case? Exploring the Foundations of social inquiry (pp. 217–226). New York, NY: Cambridge University Press. Reiss, J. (2012). Causation in the sciences: An inferentialist account. Studies in History and Philosophy of Biological and Biomedical Sciences, 43, 769–777. Reiss, J. (2014). What's wrong with our theories of evidence? THEORIA, 80, 283–306. Reiss, J. (2015). Causation, evidence, and inference. New York: Routledge. Schultz, K. (2001). Democracy and coercive diplomacy. New York: CambridgeUniversity Press. Seawright, J. (2016). Multimethod social science: Combining quantitative and qualitative tools. Cambridge: Cambridge University Press. Solomon, M. (2001). Social empiricism. Cambridge, MA: MIT Press. Woodward, J. (2003). Making things happen: A theory of causal explanation. New York: Oxford University Press. Acknowledgements Thanks to the Center for Philosophy of Science at the University of Pittsburgh where I was a Fellow in Spring 2017. The support and companionship were invaluable. Drafts of the paper were given at the Roundtable of Social Science Roundtable, University of British Columbia, Vancouver in May 2017 and the Centre for the Humanities Engaging Science and Society (CHESS) at Durham University, Durham, UK in October 2017. Thanks to both the Roundtable participants and especially to the graduate students and post-docs in Durham for suggestions on how to improve and clarify the paper. Finally, thanks to three reviewers. To all – I hope I have done your comments and suggestions justice. Appendix A. Supplementary data Supplementary data to this article can be found online at https:// doi.org/10.1016/j.shpsa.2018.11.004. References Anderson, E. (2004). Uses of value judgments in science: A general argument, with lessons from a case study of feminist research on divorce. Hypatia: A Journal of Feminist Philosophy, 19(1), 1–24. Beach, D., & Pedersen, R. B. (2013). Process-tracing methods: Foundations and guidelines. Ann Arbor: University of Michigan Press. Beach, D., & Pedersen, R. B. (2016). Causal case study methods: Foundations and guidelines for comparing, matching, and tracing. Ann Arbor: University of Michigan Press. 8