Adapting Design-Based Research As A Research Methodology in Educational Settings
Adapting Design-Based Research As A Research Methodology in Educational Settings
Adapting Design-Based Research As A Research Methodology in Educational Settings
10 October 2013
Li Li
Graduate School of Education, the University of Exeter, Exeter, United Kingdom
Email: [email protected]
Abstract
Recently, design-based research has received a significant amount of attention in educational
research literature. Accordingly, we aim from this paper at reviewing design-based research
literature in order to provide the educational researchers with different aspects of this research
methodology that might contribute to guiding them to use effectively this methodology to address
the critical issues that occur in their educational settings. These aspects involve (1) research
paradigms and design-based research, (2) design-based research in terms of its emergence, (3)
definition, (4) purposes and characteristics, (5) the procedure for conducting it, (6) ensuring the
required rigour in the findings, (7) the participants, and (8) data collection and analysis techniques.
Finally, a conclusion is included.
Knipe, 2006; White, 1999). A paradigm, according to Neuman (2007), is an “integrated set of
assumptions, beliefs, models of doing good research, and techniques for gathering and analysing
data” (p. 41). On the other hand, positivism (objectivism, and realism); interpretivism
(constructivism, naturalism, idealism, and rationalism); critical theory (transformativism, and
relativism); and pragmatism (functionalism) comprise the main educational research paradigms
(Grix, 2004; Henn, et al., 2006; Luo, 2011; Mackenzie & Knipe, 2006; Neuman, 2007; Schuh &
Barab, 2007; White, 1999).
Education research is conducted to achieve different purposes and functions, such as describing,
comparing, evaluating, explaining, designing, and developing elements of the teaching and learning
process (Plomp, 2007). These purposes and functions can be accomplished by using different
research methodologies, which refer generally to “principles, procedures, and practices that govern
research” (Marczyk, DeMatteo, & Festinger, 2005, p. 22). However, Plomp (2007) suggested the
following purposes that can be served by using the various research methodologies indicated:
Survey: to describe, to compare, to evaluate;
Case studies: to describe, to compare, to explain;
Experiments: to explain, to compare;
Action research: to design/ develop a solution to a practical problem;
Ethnography: to describe, to explain;
Correlational research: to describe, to compare;
Evaluation research: to determine the effectiveness of a programme; and
Design research: to design/develop an intervention (such as programmes, teaching-learning
strategies and materials, products and systems) with the aim to solve a complex educational
problem and to advance our knowledge about the characteristics of these interventions and the
processes to design and develop them. (p. 12)
The majority of writers in the literature on research methodologies agree that pragmatism is an
appropriate paradigm for underpinning design-based research (Barab & Squire, 2004; Juuti &
Lavonen, 2006). Furthermore, the term pragmatism “is derived from the Greek words ‘pragmein’
and ‘pragma’ (thing and fact) which literally mean ‘to do.’ The emphasis is on what is done; on
outcomes rather than ideas or ideals” (Mouton, 1996, p. 8). In addition, pragmatism “was first
introduced through the works of Charles Sanders Peirce (1839–1914), and then further developed
by William James (1842–1910), and John Dewey (1859–1952)” (Given, 2008, pp. 671-672). The
pragmatic paradigm was founded by the above philosophers “in order to provide an answer to the
mind-body-problem: how our immaterial mind can acquire knowledge of a material world” (Juuti &
Lavonen, 2006, p. 57). That indicates that the philosophy of pragmatism is concentrated on the
neutral of truth. Thus, the pragmatists believe that “truth is found in ‘what works’ and that truth is
relative to the current situation” (Given, 2008, p. 672).
Finally, the pragmatic paradigm has a set of characteristics that distinguish it from other
research paradigms; Creswell (2007) sums up these characteristics as follows:
Pragmatism is not committed to any one system of philosophy and reality.
Individual researchers have a freedom of choice. They are ‘free’ to choose the methods,
techniques, and procedures of research that best meet their needs and purposes.
Pragmatists do not see the world as an absolute unity.
Truth is what works at the time; it is not based in a dualism between reality independent of the
mind or within the mind.
2
International Journal of Education and Research Vol. 1 No. 10 October 2013
Pragmatist researchers look to the ‘what’ and ‘how’ to research based on its intended
consequences—where they want to go with it.
Pragmatists agree that research always occurs in social, historical, political, and other contexts.
Pragmatists have believed in an external world independent of the mind as well as those lodged
in the mind. But they believe that we need to stop asking questions about reality and the laws of
nature. (p. 23)
3
ISSN: 2201-6333 (Print) ISSN: 2201-6740 (Online) www.ijern.com
methodological challenges in creating complex interventions in classroom settings,” were the first
authors who introduced design-based research as a methodology in education research (O’Donnell,
2004). Moreover, Herrington, et al. (2007) suggested that, according to Collins and Brown, design-
based research involves:
addressing complex problems in real contexts in collaboration with practitioners;
integrating known and hypothetical design principles with technological affordances to render
plausible solutions to these complex problems; and
conducting rigorous and reflective inquiry to test and refine innovative learning environments as
well as to define new design principles. (p. 4090)
Finally, a wide range of definitions of design-based research have been presented in the
literature. For example, Shavelson, Phillips, Towne, and Feuer (2003) mentioned that design-based
research is “based strongly on prior research and theory and carried out in educational settings,
seeks to trace the evolution of learning in complex, messy classrooms and schools, test and build
theories of teaching and learning, and produce instructional tools that survive the challenges of
everyday practice” (p. 25). In addition, Barab and Squire (2004) stated that design-based research is
a “series of approaches, with the intent of producing new theories, artefacts, and practices that
account for and potentially impact learning and teaching in naturalistic setting” (p. 2). Further,
Wang and Hannafin (2005) defined it as “a systematic but flexible methodology aimed to improve
educational practices through iterative analysis, design, development, and implementation, based on
collaboration among researchers and practitioners in real-world settings, and leading to
contextually-sensitive design principles and theories” (pp. 6–7).
It appears from the above definitions of design-based research, that Wang and Hannafin’s
definition is most persuasive because it provides comprehensive details about design-based research
from different perspectives.
6). Accordingly, Edelson (2002) pointed to three types of theories that can be generated from
design-based research. These include:
Domain Theories: A domain theory is the generalisation of some portion of a problem analysis.
Thus, a domain theory might be about learners and how they learn, teachers and how they teach,
or learning environments and how they influence teaching and learning;
Design Frameworks: A design framework is a generalised design solution. Although design
theories are descriptive, design frameworks are prescriptive. They describe the characteristics
that a designed artefact must have to achieve a particular set of goals in a particular context;
Design Methodologies: A design methodology is a general design procedure. Like a design
framework, it is prescriptive. However, a design methodology provides guidelines for the
process rather than the product. (pp. 113–115)
Finally, design-based research has some similarities with other research methodologies such as
participatory action research, formative evaluation research, ethnography, and positivist
experiments (Hoadley, 2011; MacDonald, 2008). Yet, design-based research has a set of
characteristics that distinguish it from these methodologies. Thus, Wang and Hannafin (2005)
summarised these characteristics as shown in Table 1.
robust theory about learning and teaching, (d) conduct literature review, needs analysis, etc., to
generate research questions, (e) design an educational intervention, (f) develop, implement, and
revise the design intervention, (g) evaluate the impact of the intervention, (h) iterate the process,
and (i) reporting the findings of design-based research. Furthermore, the intervention in design-
based research could include “programmes, teaching-learning strategies and materials, products,
and systems” (Plomp, 2007, p. 13), and other similar types.
More specifically, McKenney (2001) suggested three phases for conducting design-based
research; these involve (a) needs and context analysis, (b) design, development, and formative
evaluation, and (c) semi-summative evaluation. In addition, Collins, Joseph, & Bielaczyc (2004)
developed guidelines for carrying out design-based research, which include (a) implementing a
design, (b) modifying a design, (c) identifying multiple ways of analysing the design, (d) measuring
dependent variables, (e) measuring independent variables, and (f) reporting on design research.
Moreover, there is another design for conducting design-based research that is commonly cited in
many studies that have used design-based research (e. g., Ashford-Rowe, 2008; Dix, 2007; Hood,
2008; Mantei, 2008; Parker, 2011; Sari & Lim, 2012). This design, however, was developed by
Reeves (2000; 2006) who suggested four phases for carrying out design-based research, as Figure 1
shows; these include (a) analysis of practical problems by researchers and practitioners, (b)
development of solutions with a theoretical framework, (c) evaluation and testing of solutions in
practice, and (d) documentation and reflection to produce ‘design principles.’
Finally, Plomp (2007) examined the above designs for conducting design-based research and
concluded that the majority of writers cited in the literature agree that design-based research should
comprise the following three phases:
Preliminary research: needs and context analysis, review of literature, development of a
conceptual or theoretical framework for the study;
Prototyping phase: iterative design phase consisting of iterations, each being micro-cycle of
research with formative evaluation as the most important research activity aimed at improving
and refining the intervention;
Assessment phase: (semi-) summative evaluation to conclude whether the solution or
intervention meets the pre-determined specifications. As also this phase often results in
recommendations for improvement of the intervention, we call this phase semi-summative. (p.
15)
From the guidelines discussed above for conducting design-based research, Plomp’s three
phases constituted an appropriate design because it seemed that these phases covered all different
activities and iterative cycles that a study based on design-based research principles should take into
account.
6
International Journal of Education and Research Vol. 1 No. 10 October 2013
More specifically, with regard to achieving objectivity, which “deal[s] with the idea of
neutrality or the extent to which the research is free of bias in the procedures and the interpretation
of results” (Ary, Jacobs, Sorensen, & Razavieh, 2010, p. 504). Nevertheless, achieving objectivity
in design-based research is not an easy task (Akilli, 2008; Design-Based Research Collective, 2003;
Hoadley, 2011; O’Donnell, 2004) because “researchers conducting design-based research usually, if
not always, need to immerse themselves in the research context and intensely interact with
participants. As a consequence, it is difficult to keep being objective and neutral” (Instructional
Technology Ph.D students at the University of Georgia, 2006c, para, 2). However, employing
triangulation by using multiple sources and kinds of data can maintain and increase the objectivity
in the findings of design-based research (Akilli, 2008; Design-Based Research Collective, 2003;
O’Donnell, 2004; Thurmond, 2001; Wang & Hannafin, 2005).
Validity, on the other hand, concentrates mainly on two aspects. The first aspect is external
validity, which refers to “the extent to which the findings of a study can apply to a wider
population. Research which is generalisable enables the results and implications of a study to be
brought into more general use” (Bloor & Wood, 2006, p. 93). However, design-based research
literature agrees completely that the findings of design-based research cannot be generalised from a
sample to a large population (Akilli, 2008; Barab & Squire, 2004; Hoadley, 2011; O’Donnell, 2004;
Plomp, 2007) because design-based research has a “highly contextualised research agenda and its
7
ISSN: 2201-6333 (Print) ISSN: 2201-6740 (Online) www.ijern.com
heavy reliance on thick description for data analysis”(Instructional Technology Ph.D students at the
University of Georgia, 2006c, para, 3). Therefore, the literature suggests that design-based research
should result in generalisable theory from its context to other contexts; as mentioned earlier, this
theory might include domain theories, design frameworks, and design methodologies (Bowler &
Large, 2008; Edelson, 2002; Plomp, 2007).
The second aspect is internal validity, which refers generally to “the degree to which the
investigator’s conclusions correctly portray the data collected” (Bloor & Wood, 2006, p. 148).
There are different factors that may threaten the validity of design-based research, so Hoadley
(2004) suggested that validity has a larger sense, involving “the likelihood that our interpretation of
the results accurately reflects the truth of the theory and hypotheses under examination” (p. 204).
Therefore, adopting many iterations of the phases of design-based research over time as well as
repeating the analysis through cycles of iterations can result in strengthening the internal validity of
the findings of design-based research (Bloor & Wood, 2006; Design-Based Research Collective,
2003; Dix, 2007; Instructional Technology Ph.D students at the University of Georgia, 2006c).
Reliability, refers generally to “the extent to which research produces the same results when
replicated” (Bloor & Wood, 2006, p. 147). Achieving reliability in design-based research also faces
challenges, so using triangulation through multiple data sources contributes to improved reliability
in the findings of design-based research (Design-Based Research Collective, 2003; Dix, 2007;
Instructional Technology Ph.D students at the University of Georgia, 2006c).
indicates that a researcher who uses design-based research can use any research approaches—
quantitative, qualitative, or a combination of the two.
Indeed, the majority of design-based research literature agrees that the mixed methods approach
is an appropriate one for collecting and analysing design-based research’s data because it can
maximise the validity as well as increase the objectivity, and reliability of ongoing research (Bell,
2004; Design-Based Research Collective, 2003; Wang & Hannafin, 2005). However, a mixed
methods approach, according to Migiro and Magangi (2011), is “research in which the researcher
uses the qualitative research [approach] for one phase of a research study and the quantitative
research [approach] for another in order to understand a research problem more completely” (p.
3757).
On the other hand, Amiel and Reeves (2008, p. 35) pointed out that collecting data in design-
based research should aim at achieving three purposes as follows: (a) re-define the problems, (b)
explore possible solutions, and (c) consider the principles that might best address them.
9. Conclusion
In conclusion, it seems that design-based research has emerged as a reaction against the failure
of some traditional research methodologies to link theory and practice within educational research,
and as a means of generating useful knowledge to guide educational practice.
According to the above discussion, we encourage all educational researchers to adopt design-
based research to investigate the different issues that face them in their educational settings because
it will contribute effectively to addressing these issues as well as bridging the gaps in research
between theory and practice.
References
Akilli, G. K. (2008). Design based research vs. mixed methods: The differences and commonalities.
1-10. Retrieved from
http://it.coe.uga.edu/itforum/paper110/Akilli_DBR_vs_MM_ITForum.pdf
Amiel, T., & Reeves, T. C. (2008). Design-based research and educational technology: Rethinking
technology and the research agenda. Educational Technology & Society, 11(4), 29-40.
Andriessen, D. (2007). Combining design-based research and action research to test management
solutions Paper presented at the 7th World Congress Action Learning, Action Research and
Process Management, Groningen.
Ary, D., Jacobs, L. C., Sorensen, C., & Razavieh, A. (2010). Introduction to research in education
(8th ed.): Cengage Learning.
Ashford-Rowe, K. (2008). Applying a design-based research approach to the determination and
application of the critical elements of an authentic assessment. Paper presented at the
Proceedings of the Second Emerging Technologies Conference, Wollongong: University of
Wollongong.
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of
the Learning Sciences, 13(1), 1-14.
Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational
Psychologist, 39(4), 243-253.
Bloor, M., & Wood, F. (2006). Keywords in qualitative methods: A vocabulary of research
concepts. London: SAGE.
9
ISSN: 2201-6333 (Print) ISSN: 2201-6740 (Online) www.ijern.com
Bowler, L., & Large, A. (2008). Design-based research for LIS. Library & Information Science
Research, 30, 39-46.
Brown, A. L. (1992). Design experiments: theoretical and methodological challenges in creating
complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2),
141-178.
Burrell, G., & Morgan, G. (1985). Sociological paradigms and organisational analysis: Elements of
the sociology of corporate life. London: Heinemann.
Collins, A. (1992). Toward a design science of education. New York: Center for Technology in
Education.
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological
issues. The Journal of the Learning Sciences, 13(1), 15-42.
Creswell, J. W. (2007). Qualitative inquiry and research design: Choosing among five approaches
(2nd ed.). Thousand Oaks: Sage Publications.
Crotty, M. (1998). The foundations of social research: Meaning and perspective in the research
process. London: SAGE.
Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for
educational inquiry. Educational Researcher, 32(1), 5-8.
Dills, C. R., & Romiszowski, A. J. (1997). The instructional development paradigm: An
introduction. In C. R. Dills & A. J. Romiszowski (Eds.), Instructional development
paradigms. New Jeresy: Educational Technology Publications.
Dix, K. L. (2007). DBRIEF: A research paradigm for ICT adoption. International Education
Journal, 8(2), 113-124.
Edelson, D. C. (2002). Design research: What we learn when we engage in design. The Journal of
the Learning Sciences, 11(1), 105-121.
Fraenkel, J. R., & Wallen, N. E. (2009). How to design and evaluate research in education (7th
ed.). New York, NY: McGraw-Hill.
Given, L. M. (2008). The Sage Encyclopedia of Qualitative Research Methods. Los Angeles;
London: SAGE.
Grix, J. (2004). The Foundations of Research. Houndmills: Palgrave Macmillan.
Guba, E. G. (1990). The paradigm dialog. Newbury Park, CA. ; London: Sage Publications.
Guba, E. G., & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. In N. K. Denzin
& Y. S. Lincoln (Eds.), Handbook of Qualitative Research (pp. 105-117): Thousand Oaks,
CA: Sage.
Henn, M., Weinstein, M., & Foard, N. (2006). A short introduction to social research. London:
SAGE.
Herrington, J., McKenney, S., Reeves, T., & Oliver, R. (2007). Design-based research and doctoral
students: Guidelines for preparing a dissertation proposal. Paper presented at the
Proceedings of World Conference on Educational Multimedia, Hypermedia and
Telecommunications 2007, Chesapeake, VA: AACE.
Hoadley, C. (2011). Design, research, and DBR: How can we improve and study how people live
and learn with their technology? , Retrieved from
http://www.kaputcenter.umassd.edu/events/cs/10-
11/8_hoadley_042011/hoadley_presentation.pdf
Hoadley, C. M. (2004). Methodological alignment in design-based research. Educational
Psychologist, 39(4), 203-212.
Hood, G. (2008, June). Using a design-based research paradigm to develop an online course aimed
at disseminating research findings and informing practice. Paper presented at the
10
International Journal of Education and Research Vol. 1 No. 10 October 2013
11
ISSN: 2201-6333 (Print) ISSN: 2201-6740 (Online) www.ijern.com
12