Evaluation MattersHe Take T Te Aromatawai 2: 2016

New Zealand Council for Educational Research 2016

http://dx.doi.org/10.18296/em.0012
http://www.nzcer.org.nz/nzcerpress/evaluation-matters

Developmental evaluation: A tool to

support innovation
Helen McDonald

The term developmental evaluation was coined by Michael Quinn

Patton in the mid-1990s to describe the role of evaluators in the
development of an innovative initiative. Core evaluation skills, such
as articulating the initiatives theory of change, asking evaluative
questions, and providing timely information on emerging results,
help the initiative to develop through iterative cycles of learning
and adaptation. A growing number of case studies provide valuable
insights into the specific situations where developmental evaluation
may be useful, the roles and skills required of the evaluator, and
the challenges this approach brings. This article provides a concise
summary of the insights from theory and case studies to help pro-
gramme developers and evaluators decide whether and how to use
developmental evaluation.

Introduction
There is significant demand across the world and across the pub-
lic, private, and non-profit sectors for innovation (Patton, 2016a).
Developmental evaluation supports the creation of innovative ini-
tiatives to address complex problems. In developmental evaluation

New Zealand Council for Educational Research 2016 79

Helen McDonald

the evaluator collaborates closely with the development team. The

evaluator uses skills such as asking evaluative questions, applying
programme logics, and data collection and analysis to inform the
development process. Developmental evaluations purpose is there-
fore quite distinct from the purpose of more traditional formative and
summative evaluations. While formative and summative approaches
are focused on improving or assessing an established initiative,
developmental evaluation aims to generate timely learning to assist
development before there is a complete model to improve or assess
(Dozois, Langlois & Blanchet-Cohen, 2010). The primary users of a
developmental evaluation are the individuals and organisations seek-
ing to achieve significant change (Patton, 2011). The success of the
evaluation is measured by the extent to which the knowledge and
lines of inquiry it helps produce contribute to the development pro-
cess (Lam, 2012).
Developmental evaluation is still a relatively new approach.
Wehipeihana and McKegg (as cited in Patton, 2011) have previously
observed that it is not well understood in New Zealand. First artic-
ulated by Patton in 1992, it has been further developed by contribu-
tions from Gamble (2008), Dozois et al. (2010), Patton (2011), and
Preskill and Beer (2012). Knowledge about when and how to use it
is gradually being built up through the publication of a number of
case studies discussing its use in New Zealand and overseas. These
examples provide valuable insights into the types of projects where
developmental evaluation may be useful, the roles the evaluator may
play, and the impact of developmental evaluation.
The purpose of this article is to provide a concise introduction to
the theory and the practice of developmental evaluation to assist New
Zealand programme developers and evaluators to determine whether
and how to use it. It looks at the specific situations where develop-
mental evaluation is of value, what an evaluator contributes to the

80 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

development process, the skills and qualities required for develop-

mental evaluation, and the challenges that arise.

Situations suitable for developmental evaluation

Developmental evaluation is promoted as a tool to aid the ongoing
development and adaptation of innovative initiatives in complex
situations (Patton, 2011). Key characteristics of complex situations
include:
unpredictabilityvariables are non-linear, with small actions
potentially having large effects (Walton, 2014)
high levels of uncertainty, with no known solutions and no clear
way forward (Dozois et al., 2010)
rapid and unpredictable change (Patton, 2011)
outcomes emerging as elements of the system adapt and evolve in
response to changing circumstances (Patton, 2011)
the need for collaboration between stakeholders, given many inter-
dependent variables (Patton, 2016a).
Innovative initiatives in this context are likely to be driven by vison
and values rather than a clear intended outcome, and they require
collaboration between multiple stakeholders (McKegg & Patton,
2014, as cited in McKegg, 2015). Dickson & Saunders (2014) note
that developmental evaluation is used in situations where the imple-
mentation approach is more akin to ready, fire, aim, rather than the
more traditional approach of ready, aim, fire (p. 178).
Patton (2011) identified five specific complex situations, likely to
overlap, as the niche for developmental evaluation:
when ongoing development is needed to adapt an initiative to
changing conditions (for example, in technology or demographics)
in a complex, dynamic environment

New Zealand Council for Educational Research 2016 81

Helen McDonald

when initiatives that have been effective in one setting are adapted
to a new environment
when there is a need to quickly respond to a crisis or sudden signif-
icant change, such as a natural disaster
during the early development of an innovation that has the scope
to be applied more broadly
during substantial systems change.
Two case studies considered in this article primarily concern the
development needed to adapt an initiative to a new environment.
The other case studies discussed here use developmental evaluation
to assist the early development of an innovation.

What an evaluator brings to the development process

When developing initiatives in highly uncertain, emergent situa-
tions, the skills of an evaluator help remove some of the uncertainty
that can be a barrier to innovation. These skills include:
helping to articulate the objectives and success measures of the
initiative
providing data as initial impacts emerge
recording the lessons learned
building stakeholders capacity to use data to manage the initiative.
Helping to articulate the objectives and success measures of
the initiative
One of the core roles for a developmental evaluator is to ask evaluative
questions to prompt discussion and help the project team progress its
thinking (Patton, 2011). These might be focused on several areas:
drawing out the nature of the problem; clarifying which aspects
of the initiative need to adapt; the underlying values and assump-
tions; and the criteria for success (Lam, 2012). The Mori and Pacific

82 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

Educational Initiative (MPEI) involved multiple different initiatives

over a 5-year period, in different communities, with different needs.
As part of the development team, the evaluators worked to develop
key success criteria with the trustees of the philanthropic trust who
were funding the initiative. These criteria were used to provide a
framework to help develop the detail of the initiative beyond its high-
level vision of improving Mori and Pacific educational achievement
(McKegg, Wehipeihana, Becroft & Gill, 2016). Lam (2012) noted
that in developing the Assessment Pilot Initiative (API), an inno-
vative approach to teacher education using Twitter at a Canadian
university, the development team and the evaluator collaboratively
used evaluative questioning to judge how well particular ideas fitted
or could be adapted to the proposed initiative.
Patton (2011) notes that the deep knowledge accumulated by an
experienced evaluator about the nature of what is likely to work is
very useful in helping a development team articulate an initiatives
programme theory. Programme theory was used to provide a frame-
work to inform the development of the Assessment Pilot Initiative
(Lam & Shulha, 2015). In New Zealand, McKegg, Wehipeihana,
Pipi and Thompson (2013) used a developmental evaluation approach
to support the implementation of a significant change in focus for He
Oranga Poutama. In 2009 the objectives of this Sport New Zealand
initiative changed from increasing the participation by Mori in
sport to participating and leading as Mori in sport. Wehipeihana
and McKegg (2009) consider that a developmental evaluation
approach was appropriate as there was little experience of how to
apply the concept of as Mori, and the change in focus was based on
a vision rather than clear, measurable goals. Further, they note that
diverse organisations were involved in its implementation in a com-
plex environment. This environment included a deteriorating econ-
omy and ongoing changes in Sport New Zealand. Developmental

New Zealand Council for Educational Research 2016 83

Helen McDonald

evaluation facilitated the development of a shared understanding of

the concept of as Mori in sport, and the development of a frame-
workTe Whetu Rhuawhich formed the basis for assessing
the fit of a range of programmes with the initiative (Wehipeihana,
McKegg, Thompson, & Pipi, 2016).
Providing data as initial impacts emerge
In a developmental evaluation data gathering and reporting is
required in real time to inform decisions in an ongoing and timely
way (McKegg et al., 2016). Such decisions include where future
attention should be focused or how the initiative should be altered
(Patton, 2011; Preskill & Beer, 2012). This contrasts with traditional
evaluations where data collection and reporting may follow a linear,
planned process. During the developmental evaluation of the Mori
and Pacific Educational Initiative, the evaluators found that instead
of an approach of interim and final reports, the needs of the trustees
of the philanthropic trust to be kept informed of progress were bet-
ter met by regular, shorter reports containing quantitative data as it
became available (McKegg et al., 2016).
Lam and Shulha (2015) consider that the nature of the data used in
the development of a teacher education tool was broader than what is
traditionally collected in a summative evaluation, such as data about
an initiatives activities or outcomes. For example, the data provided
included information from the social science and education literature
about the use of technology to assist learning, which provided the
development team with new ideas for the initiative.
The case studies highlight the impact of data collected during the
development of the initiative. For example, Lam (2012) notes that
the information gathered on the piloting of Twitter with a group of
student teachers highlighted the limitations of learning through a
short tweet. The development team was able to respond by including

84 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

a discussion board alongside Twitter. Fagen et al. (2011) discuss the

use of data in the developmental evaluation of the Illinois Caucus for
Adolescent Heath initiative to change school board sexual-education
policies. They note that information gathered through surveys, focus
groups, and interviews confirmed an initial view that the presence of
a core group advocating for change was a critical factor in achieving
change in a school districts policy.
Recording the lessons learned
A key role of the developmental evaluator is in recording what the
development is seeking to achieve, what decisions are taken and why,
what emerges, and what was learned (Patton, 2008). Lam (2012) notes
that the developers of the Assessment Pilot Initiative observed that,
given the fast changing nature of the development, the summaries he
produced of the development discussions and steps taken along the
way were very useful for helping them to understand and keep track
of where the development was at. Documenting the process also
makes the process more transparent, supports accountability, and
provides a valuable resource for future interventions (Gamble, 2008).
Building stakeholders capacity to use data to manage the
initiative
The close involvement of the evaluator in a developmental evaluation
can build stakeholders knowledge of evaluation and their capac-
ity and inclination to gather and use data to inform their practice.
McKegg et al. (2013) note that an important part of the evaluators
role was to work with the providers to co-construct a monitoring
framework for He Oranga Poutama. The objective was to provide an
evidence base so that providers and Sport New Zealand could know
the type and number of as Mori activities being funded. This moni-
toring framework was subject to ongoing testing and refining. While
the extent to which the providers understood evaluation at the end

New Zealand Council for Educational Research 2016 85

Helen McDonald

of the programme varied, they all reported learning something about

the key aspects of evaluation (McKegg et al., 2013).

The skills and qualities required for developmental

evaluation
To work collaboratively in the development process requires the eval-
uator to possess a broad range of skills and qualities. The following
section discusses four key qualities:
the ability to build trusting relationships
cultural fit and competence
strong methodological skills
an in-depth understanding of the initiative.
The ability to build trusting relationships
Given the collaborative learning essential for a developmental eval-
uation process, fundamental abilities are: to build strong, trusting
relationships with management, staff, and stakeholders; to actively
listen; and to facilitate participation (Langlois, Blanchet-Cohen &
Beer, 2013; McKegg & Patton, 2014, as cited in McKegg, 2015). The
evaluators also need to understand when, and how, to provide feed-
back in a way that will support development (Langlois et al., 2013).
The strong relationship focus of developmental evaluation lends
itself very well to development taking place in a Mori context.
McKegg et al. (2013) note that in a Mori community evaluation
has more credibility if the evaluator is known to the community and
is perceived as understanding the community and its priorities. They
consider that their role was made easier by their pre-existing relation-
ships with the manager of the He Oranga Poutama programme and
some providers.

86 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

Cultural fit and competence

An important influence on the ability of the evaluators to effectively
support the development of an initiative is the cultural fit between
the values, characteristics, and language of the culture(s) developing
the initiative and the evaluator (Goodwin, Sauni & Were, 2015). In
the New Zealand case studies considered here, Mori and Pasifika
evaluators were part of the evaluation teams. In the evaluation to
support the development of Whnau Ora for Youth Mental Health
services by Mori and Pasifika service providers, Goodwin et al.
(2015) observe that, particularly at the beginning of the evaluation,
the cultural fit of the evaluation teamconsisting of one Pasifika
and two Mori evaluatorswas important to the service providers.
With the implementation of as Mori in sport, the evaluators of He
Oranga Poutama deliberately placed Mori values, cultural concepts,
and te reo Mori at the heart of their evaluation approach (McKegg
et al., 2016). The evaluators chose methods to accurately and appro-
priately capture the views and experiences of Mori providers and
communities (McKegg et al., 2013). This included regular hui and
facilitation methods that included waiata (songs) and mahi toi (arts
and crafts) that would encourage everyone to share their experiences.
They noted that providers said they appreciated the emphasis that
the evaluators placed on face-to-face sharing and learning from one
another (McKegg et al., 2013).
Cultural competence is recognised as encompassing more than
an evaluation carried out in a culturally appropriate way, to also
encompass drawing on the values, needs, strengths and aspirations
of the culture of those a policy or programme is intended to benefit
to determine what successful design, delivery, and outcomes for the
initiative look like (Aotearoa New Zealand Evaluation Association,
2011, p.10). As already noted, McKegg et al. (2013) drew on Mori
values and perspectives to develop a shared understanding of the

New Zealand Council for Educational Research 2016 87

Helen McDonald

concept of as Mori in sport and the outcome framework, Te Whetu

Rhua (Wehipeihana et al. 2016). In the development of Whnau
Ora for Youth Mental Health services, discussions with the Pasifika
and Mori service providers highlighted the importance of cultural
fit to the initiative and this was incorporated in the success criteria
that the evaluators developed with each of the providers.
Strong methodological skills
Given the range of functions an evaluator may need to carry out, it is
valuable for developmental evaluators to have strong methodological
skills and experience with a wide range of methods (Wehipeihana &
McKegg, 2009). These include strong analytical and data-analysis
skills, and the ability to think strategically and see the big picture
(Dozois et al., 2010). Patton (2016b) also observes that knowledge
of systems and complexity theory is important for a developmental
evaluator in understanding the nature of the innovative initiative,
how it might emerge, and the impact of interactions with other parts
of the system.
The evaluator must also be flexible and able to adapt their methods
and areas of focus in response to emerging issues (Patton, 2011). Thus
Fagen et al. (2011) noted that in the initiative to change school board
sexual-education policies in Illinois, the evaluation team revised its
logic model at the beginning of Phase 2 using its findings from Phase
1. The team also revised its evaluation questions and methods.
An in-depth understanding of the initiative
Alongside these skills is the need for the evaluators to have an in-depth
understanding of the initiative (Wehipeihana, 2015). This includes
understanding the issues to be addressed, key stakeholders, the wider
system, and the resources available (time and financial) (Dozois et al.,
2010). Going a step further, specific subject knowledge can also be a
significant advantage (Dozois et al., 2010). Lam (2012) observes that

88 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

his knowledge of teacher education and communication technologies

such as Twitter was one of the reasons the developers selected him,
and he was able to significantly shape the development in a manner
that would not have been not possible without that knowledge.
Most case studies considered here used a team of evaluators, pro-
viding a broader knowledge and skill set and the ability to support
each other. Honadle, Zapata, Auffrey, vom Hofe and Looye (2014)
consider that a real strength was derived from the multidisciplinary
nature of the evaluation team involved in the Stronger Economies
Together initiative. The depth of relevant knowledge and credibility
it provided enabled the evaluation team to have a substantive input
into the initiative.

The challenges of developmental evaluation

While developmental evaluation seems like a common-sense approach
involving an ongoing cycle of learning and adapting, it is challenging
in practice and will not be the right tool for every complex situation.
Some of the challenges faced by developmental evaluators include:
achieving buy-in
defining the boundaries of the evaluators role
balancing the need for in-depth knowledge and objectivity
providing robust data in real-time.
Achieving buy-in
A crucial factor for a successful developmental evaluation is that key
stakeholders understand and demonstrably buy-into the develop-
mental evaluation approach (Patton, 2011). They must be open to
experimentation, reflection, and using evaluative data to adapt the
initiative as needed (Preskill & Beer, 2012). A lack of understanding
of evaluation in general, and of developmental evaluation in partic-
ular, reduces the likelihood of buy-in to developmental evaluation

New Zealand Council for Educational Research 2016 89

Helen McDonald

(Poth, Pinto & Howery, 2012). McKegg and Wehipeihana (2015)

note that they often find it necessary to explain the purpose and
process of evaluation before going on to explain developmental eval-
uation. Ramirez, Shephard and Kora (2015) described their use of
developmental evaluation in the continuing development of a social
and financial education programme aimed at youth. On-site face-to-
face engagement with the programme funder and the client facilitated
getting buy-in to the approach. The funders key priority was that the
evaluation produced useful knowledge for the programme. This pro-
vided the development team with the freedom needed to explore a
range of potential opportunities. Similarly, buy-in is impeded by the
concern that developmental evaluation will not provide the evidence
of the value of the initiative that is sought (McKegg & Wehipeihana,
2016). Regular reports about the projects progress and what has
been learnt can help to address this (Poth et al., 2012). This issue can
also be addressed through the development of trusting relationships,
along with the credibility built up from a track record of previous
successful evaluation work (Wehipeihana & McKegg, 2009).
Defining the boundaries of the evaluators role
Developmental evaluation is not prescriptive in the manner in which
it is approached and applied. The design and methods depend on
the nature of the situation and the key questions that need to be
answered to progress its development (Dozois et al., 2010; Patton,
2011). One challenge arising from this flexibility is the need to clearly
define roles and responsibilities to ensure all parties are clear what
will be undertaken by the evaluators and the time and resources that
will be needed (Wehipeihana, 2015). Poth et al. (2012) discuss the
developmental evaluation undertaken to support the development
and implementation of an educational technology leadership project
funded by the Alberta provincial government. They note that it was

90 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

challenging to define the boundary between the evaluation and proj-

ect activities. Because the responsibilities of the evaluator increased
over time it was agreed that the evaluation team would only deal with
tasks that would inform project development.
Balancing the need for in-depth knowledge and objectivity
The degree of power and control a development team is willing to
give the evaluator will vary and this affects the extent to which the
evaluator can influence the development (Lam & Shulha, 2015). A
developmental evaluator often has to strike a balance between the
detailed understanding and opportunities for influence that come
with being part of the team and being independent and objective
(Gamble, 2008). Dozois et al. (2010) note that an internal staff
member working as a developmental evaluator has existing relation-
ships and knowledge. However, they may be seen as less objective,
may struggle to keep their role clearly defined, and they may find
it difficult to provide frank feedback. External developmental eval-
uators may be more objective, may have insights from previous
projects from which to draw insights, and may be clearer on their
role. However they have less direct knowledge, they need to invest
time to establish relationships, and they may have less time due to
cost (Dozois et al., 2010).
Providing robust data in real-time
Some case studies note the challenges of data collection in a fast-mov-
ing development, where issues emerge quickly and the desire for
real-time feedback makes collecting reliable data difficult. Collecting
data on He Oranga Poutama was challenging given the variety of
different contexts and activities through which it was delivered and
the logistics of collecting individual and group data (McKegg et al.,
2013). In this instance, the quality of the data collected improved
in the second year as providers and Sport New Zealand became

New Zealand Council for Educational Research 2016 91

Helen McDonald

more experienced in their respective roles of collecting and analys-

ing the data. Data collection was assisted by basing the monitoring
framework on Te Whetu Rhua, the framework for implementing
the concept of as Mori in sport. Because the framework was based
on Mori cultural knowledge and practice, and developed with the
involvement of the providers, data collection was made more real and
important to providers. Data collection was also facilitated by He
Oranga Poutama reporting back on results at national and regional
levels. This enabled the providers to see that the data was useful both
to Sport New Zealand and to their own organisations. In the eval-
uation of an educational technology leadership programme, Poth et
al. (2012) note that the evaluators were responsible for ensuring data
was available on an ongoing basis to inform decisions. However, they
found that administrators only posted about their technology experi-
ences when they were successful, in part because they were uncertain
of how this information would be used in the development of the ini-
tiative. To address this, questions were used to guide online feedback
and on the evaluation teams suggestion, their client demonstrated
the type of information that was sought.
Conclusion
Complex situations often have no clear way forward, with outcomes
emerging as the elements of the system adapt and evolve in response
to changing circumstances in a non-linear fashion. Developmental
evaluation provides a potential tool for reducing the high level of
uncertainty. This is achieved by core evaluation skills, such as asking
evaluative questions and using data collection and analysis. These
skills help to articulate the objectives and success measures of an
initiative, and provide data on initial impacts as they emerge. The
recording of the process and the lessons learnt provides a valuable
resource for future developments. The close involvement of the

92 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

evaluator can build stakeholders knowledge of evaluation and their

capacity and inclination to gather and use data to inform their prac-
tice. Further, the use of developmental evaluation in New Zealand
shows that the strong relationship-based focus of this approach
lends itself well to development taking place in Mori and Pasifika
contexts.
However, developmental evaluation is not the right tool for every
complex situation. Before proceeding with a developmental evalua-
tion approach, the project commissioners, design team, and potential
evaluator should carefully consider the extent to which they are able
to respond positively to a number of key questions, which include the
following.
Do the project commissioners and development team understand
the developmental evaluation approach?
Are the project commissioners and development team open to an
iterative development process, using evaluative data to learn and
adapt the initiative as needed?
Have the key parties established, or are confident they are able to
establish, a strong, trusting relationship?
Is the evaluator comfortable with providing data in real-time and
willing to change evaluation methods in response to emerging
issues?
Does the evaluator have a deep understanding of the issues to be
addressed, the interests of key stakeholders, and how the different
elements of the system interact?
Have the parties considered the extent to which subject-specific
knowledge is required?
Have the roles and responsibilities of the developmental evaluator
been considered and agreed?

New Zealand Council for Educational Research 2016 93

Helen McDonald

The insights provided by the theoretical literature and the case stud-
ies considered in this article show that when project commissioners,
developers, and evaluators are willing and able to commit to the use
of evaluative skills to support the development process, developmen-
tal evaluation can help reduce the uncertainty inherent in developing
initiatives to address complex situations. The case studies considered
here identify the benefits and challenges of developmental evaluation
primarily from the perspective of the evaluator. While some of the
case studies touch on the feedback they received from providers, the
perspectives of the clients of developmental evaluationthe funders
and developers of innovative initiativeswould be an interesting
area for further research.

References
Aotearoa New Zealand Evaluation Association. (2011). Evaluator
competencies. Retrieved from http://www.anzea.org.nz/wp-content/
uploads/2013/05/110801_anzea_evaluator_competencies_final.pdf.
Dickson, R. & Saunders, M. (2014). Developmental evaluation: Lessons
for evaluative practice from the SEARCH program. Evaluation, 20(2),
176194. http://dx.doi.org/10.1177/1356389014527530
Dozois, E., Langlois, M., & Blanchet-Cohen, N. (2010). DE 201:
A practitioners guide to developmental evaluation. Retrieved from
http://www.mcconnellfoundation.ca/assets/Media%20Library/
Publications/DE%2 0201%20EN.pdf.
Fagen, M. C., Redman, S. D., Stacks, J., Barrett, V., Thullen, B., Altenor,
S. & Neiger, B. L. (2011). Developmental evaluation: Building
innovations in complex environments. Health Promotion Practice, 12(5),
645650. http://dx.doi.org/10.1177/1524839911412596
Gamble, J. A. (2008). A developmental evaluation primer. Montreal:
JW McConnell Family Foundation.
Goodwin, D., Sauni, P. & Were, L. (2015). Cultural fit: An important

94 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

criterion for effective interventions and evaluation work. Evaluation

MattersHe Take T Te Aromatawai, 1, 2546.
http://dx.doi.org/10.18296/em.0003
Honadle, B. W., Zapata, M. A., Auffrey, C., vom Hofe, R. & Looye,
J. (2014). Developmental evaluation and the Stronger Economies
Together initiative in the United States. Evaluation and Program Planning,
43, 6472. http://dx.doi.org/10.1016/j.evalprogplan.2013.11.004
Lam, C. Y. (2012). A case study on the use of developmental evaluation for
innovating: navigating uncertainty and unpacking complexity. (Masters
Thesis, Queens University, Ontario, Canada). Retrieved from
http://hdl.handle.net/1974/6935.
Lam, C. Y. & Shulha, L. M. (2015). Insights on using developmental
evaluation for innovating: A case study on the co-creation of an innovative
program. American Journal of Evaluation, 36(3) 358374.
http://dx.doi.org/10.1177/1098214014542100
Langlois, M., Blanchet-Cohen, N., & Beer, T. (2013). The art of the nudge:
Five practices for developmental evaluators. The Canadian Journal of
Program Evaluation, 27(2), 3959.
McKegg, K. (2015, 2 March). Developmental Eval Week: Kate McKegg
on Readiness for Developmental Evaluation [Online forum comment].
Retrieved from http://aea365.org/blog/developmental-eval-week-kate-
mckegg-on-readiness-for-developmental-evaluation/.
McKegg, K. & Wehipeihana, N. (2015, 14 April). Developmental
evaluation week: Kate McKegg and Nan Wehipeihana on talking to
clients and communities about developmental evaluation [Online forum
comment]. Retrieved from http://aea365.org/blog/de-week-kate-mckegg-
and-nan-wehipeihana-on-talking-to-clients-and-communities-about-
developmental-evaluation/.
McKegg, K. & Wehipeihana, N. (2016). Developmental evaluation in
synthesis. Practitioners perspectives. In M. Q. Patton, K. McKegg &
N. Wehipeihana (Eds.), Developmental evaluation exemplars: Principles in
practice (pp. 271288). New York, NY: Guildford Press.

New Zealand Council for Educational Research 2016 95

Helen McDonald

McKegg, K. & Wehipeihana, N., Pipi, K. & Thompson, V. (2013).

He Oranga Poutama: What we have learned. A report on the developmental
evaluation of He Oranga Poutama. Wellington, Sport New Zealand.
McKegg, K., Wehipeihana, N., Becroft, M., & Gill, J. (2016).
Developmental evaluations role in supporting community-led solutions
for Mori and Pacific young peoples educational success: The Foundation
North Mori and Pacific Education Initiative. In M. Q. Patton,
K. McKegg & N. Wehipeihana (Eds.), Developmental evaluation exemplars:
Principles in practice (pp. 125142). New York, NY: Guildford Press.
Patton, M. Q. (1992). Developmental evaluation: An approach for
empowerment-oriented and multi-cultural programs. New Directions for
Program Evaluation, 53, 1733.
Patton, M. Q. (2008). Utilization-focused evaluation (4th ed). Thousand Oaks,
CA: Sage.
Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts
to enhance innovation and use. New York, NY: The Guilford Press.
Patton, M. Q. (2016a). State of the art and practice of developmental
evaluation: Answers to common and recurring questions. In M. Q. Patton,
K. McKegg & N. Wehipeihana (Eds.), Developmental evaluation exemplars:
Principles in practice (pp. 124). New York, NY: Guildford Press.
Patton, M. Q. (2016b). The developmental evaluation mindset: Some guiding
principles. In M. Q. Patton, K. McKegg & N. Wehipeihana (Eds.),
Developmental evaluation exemplars: Principles in practice (pp. 289312).
New York, NY: Guildford Press.
Poth, C., Pinto, D. & Howery, K. (2012). Addressing the challenges
encountered during a developmental evaluation: Implications for evaluation
practice. The Canadian Journal of Program Evaluation, 26(1), 3948.
Preskill, H. & Beer, T. (2012). Evaluating social innovation. Retrieved from
http://guelph.ca/wp-content/uploads/contract_12-191_description.pdf.
Ramrez, R., Shephard, D. & Kora, G. (2015). Utilization focused
developmental evaluation: Learning through practice. Journal of

96 Evaluation MattersHe Take T Te Aromatawai 2 : 2016

 Developmental evaluation: A tool to support innovation

MultiDisciplinary Evaluation, 11(24), 3753.

Walton, M. (2014). Applying complexity theory: A review to inform
evaluation design. Evaluation and Program Planning, 45, 119126.
http://dx.doi.org/10.1016/j.evalprogplan.2014.04.002
Wehipeihana, N. & McKegg, K. (2009, November 14). Developmental
evaluation in an indigenous context: Reflections on the journey to date. Paper
presented at the American Evaluation Association Conference, Orlando, FL.
Wehipeihana, N. (2015, March 03).Developmental eval week: Nan
Wehipeihana on roles and responsibilities in developmental evaluation
[Online forum comment]. Retrieved from http://aea365.org/blog/
developmental-eval-week-nan-wehipeihana-on-roles-and-responsibilities-
in-developmental-evaluation/?utm_source=feedburner&utm_
medium=email&utm_campaign=Feed%3A+aea365+%28AEA365%29
Wehipeihana, N., McKegg, K., Thompson, V. & Pipi, K. (2016). Cultural
responsiveness through developmental evaluation: Indigenous innovations
in sport and traditional Mori recreation. In M. Q. Patton, K. McKegg &
N. Wehipeihana (Eds.), Developmental evaluation exemplars: Principles in
practice (pp. 2544). New York, NY: Guildford Press.

The author
Helen McDonald is a senior analyst at the New Zealand Treasury,
and a member of its Regulatory Quality Team. The views expressed
in this article are nonetheless those of the author and do not neces-
sarily reflect the views of the New Zealand Treasury. Helen is also
currently enrolled in the Postgraduate Diploma in Social Sector
Evaluation Research at Massey University.
Email: [email protected]

New Zealand Council for Educational Research 2016 97