Journal of Educational Evaluation for Health Professions

J Educ Eval Health Prof 2010, 7: 1 DOI: 10.3352/jeehp.2010.7.1

Open Access
BRIEF REPORT

Program Evaluation in Medical Education: An Overview of the

Utilization-focused Approach
Matt Vassar*, Denna L. Wheeler, Machelle Davison, Johnathan Franklin
Oklahoma State University, Center for Health Sciences, Tulsa, OK, USA

Abstract

Medical school administrators, educators, and other key personnel must often make difficult choices regard-
ing the creation, retention, modification, or termination of the various programs that take place at their insti-
tutions. Program evaluation is a data-driven strategy to aide decision-makers in determining the most appro-
priate outcome for programs within their purview. The purpose of this brief article is to describe one program
evaluation model, the utilization-focused approach. In particular, we address the focus of this model, the
personal factor, the role of the evaluator, and the evaluation process. Based on the flexibility of this model
as well as its focus on stakeholder involvement, we encourage readers to consider the utilization-focused
approach when evaluating programs.

Key Words : Program Evaluation; Medical Education; Utilization-Focused Evaluation

Medical education is a dynamic, rapidly-evolving enter- to know if trainees meet educational goals as well as ensure
prise. Technological advances such as simulation, virtual real- that their training programs are of sufficient quality. In each
ity, and personal digital assistants, to name a few, have con- of these scenarios, the effectiveness of the program is ques-
siderably altered the way in which medical students are edu- tioned. The means for answering such questions are accom-
cated and trained. By incorporating such technologies into plished through a systematic and comprehensive program
the learning experience, administrators at various levels of evaluation. Unfortunately, the majority of evaluations per-
medical training are faced with an important series of ques- formed often lack the proper structure or theoretical frame-
tions: Do these technologies sufficiently enhance student work to effectively guide the process.
learning, and are the costs associated with implementing According to Fitzpatrick, Sanders, and Worthen, program
these technologies justifiable? evaluation is defined as “the identification, clarification, and
Questions such as these extend well beyond technologi- application of defensible criteria to determine an evaluation
cally-driven learning experiences. Consider medical school object’s value (worth or merit) in relation to those criteria”
administrators attempting to determine whether an alterna- [1]. In broad terms, this process involves determining stan-
tive admissions program is effective or if a new standardized dards to assess the quality, collecting appropriate informa-
patient approach might increase pass rates on high stakes tion, and applying the standards to evaluate the effective-
exams. In the post-graduate years, residency directors need ness and value of the evaluation object.

*Corresponding email: [email protected]

Received: May 26, 2010, Accepted: Jun 11, 2010, Published: Jun 15, 2010
This article is available from: http://jeehp.org/

2010, National Health Personnel Licensing Examination Board of the Republic of Korea
cc This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use,
distribution, and reproduction in any medium, provided the original work is properly cited.

Page 1 of 3
(page number not for citation purposes)
 There are many program evaluation models. A few exam- ested in evaluating a standardized patient program. An objec-
ples include: objectives-oriented approaches, management- tives-oriented method would establish clearly defined objec-
oriented approaches, participant-oriented approaches, exper- tives that the program must meet in order to be effective,
tise-oriented approaches, and logic models. These models such as a pass rate percentage on a high stakes exam. In con-
have been adopted in various educational fields and have en- trast, a participant-oriented evaluator might conduct quali-
joyed widespread usage. Program evaluation models for med- tative interviews or focus groups of medical students or stan-
ical education also exist but are much more limited. Durn- dardized patients to understand their perspectives in greater
ing, Hemmer, and Pangaro rightly note that research relat- depth. Since utilization-focused evaluations are flexible and
ed to program evaluation in medical education is less devel- vary by context, they are able to incorporate elements from
oped than other educational fields [2]. Readers are referred any program evaluation model to best meet the stakehold-
to Gibson et al. [3], and Musick [4] for specific information ers’ needs.
related to evaluation models within the medical education Utilization-focused evaluation is also very personal. In fact,
domain. the personal factor is highly emphasized. Given that there
Given the limited exposure of program evaluation models may be an unlimited number of stakeholders, each with com-
in medical education, the purpose of the current article is to peting interests, it is the responsibility of the evaluator to
briefly describe one model of program evaluation, the uti- scale down potential stakeholders to a group of interested
lization-focused method. The authors feel that this model is individuals who actively participate and care about the eval-
well-suited for medical education, given its flexibility in uation and the findings it generates. The evaluation seeks to
approach as well as its focus on the active involvement of serve their interests primarily. The evaluator often develops
stakeholders. This active position promotes the stakehold- a strong working relationship with users and actively engages
ers’ investment in the process and increases the likelihood them throughout the evaluative process.
that they will see value in the results needed to affect change. Within the context of evaluation, the role of the evaluator
In the next sections we will address the focus of this model, is often a key element. For example, if the goal of the evalu-
the personal factor, the role of the evaluator, and the evalua- ation is to provide generalizable knowledge of cause-effect
tion process. For additional information regarding this eval- relationships between the program and a particular outcome
uation model, readers are referred to Patton, the source from variable, the evaluator serves as a methodologist with exper-
which the following information is based [5, 6]. tise in research design. Consider a second example in which
Utilization-focused evaluation begins with the assump- the purpose of an evaluation is to examine a program’s over-
tion that evaluations should be judged by their actual use all merit. In this case, the evaluator must become a judge.
and utility [5]. It is the responsibility of the evaluator to pay The primary role of the utilization-focused evaluator is that
careful attention to the design of the evaluation, ensuring of negotiator. In this function, the evaluator negotiates with
that each step taken from beginning to end affects intended users the roles in which he or she will take part. Patton also
use. The users (stakeholders) guide the evaluator and are res- expresses the role of the evaluator as active-reactive-adap-
ponsible for deriving goals or objectives of the evaluation tive. Utilization-focused evaluators are, first of all, active in
project. deliberately and calculatedly identifying intended users and
Utilization-focused evaluation is context specific. It is not focusing useful questions. They are reactive in listening to
limited to a particular approach or methodology. Rather, it intended users and responding to what they learn about the
is a means of aiding users in determining the approach and particular situation in which the evaluation unfolds. They
methodology that works best within the context of their par- are adaptive in altering evaluation questions and designs in
ticular program. Situational responsiveness facilitates this light of their increased understanding of the situation and
interactive process between the evaluator and the users. With changing conditions [5].
the expansive options available, users are able to choose the As with any program evaluation, utilization-focused eval-
purpose of the evaluation (formative or summative), the data uations are systematic and follow a logical sequence. First,
type (quantitative, qualitative or both), the type of research the primary users/stakeholders of the evaluation are identi-
design (experimental, quasi-experimental, or non-experimen- fied. Next, the evaluator and users commit to the purpose of
tal), and the evaluation focus (process, outcomes, costs, ben- the evaluation and derive a focus to address particular goals
efits, etc.). In addition, specific situational variables are always or to take into account the program’s theory of action. The
present and must be incorporated into the evaluation. Exam- third step of the process involves the selection of the research
ples might include the diversity of the stakeholders, avail- design and measurement considerations. Sampling issues,
able resources, political factors, and interests. To briefly de- the nature of the data, and the quality of assessment devices
monstrate a utilization-approach in contrast with other eval- are but a few of the decisions that must be made during this
uation models, consider the example of stakeholders inter- phase. Following data collection, users are brought together

Page 2 of 3
(page number not for citation purposes)
J Educ Eval Health Prof 2010, 7: 1

to assist in interpreting findings and making judgments REFERENCES

based on the data to derive recommendations. In the final
step, decisions are made about the dissemination of the eval- 1. Fitzpatrick JL, Sanders, JR, Worthen BR. Program evaluation: alter-
uation report. native approaches and practical guidelines. 3rd ed. Boston (MA): Pear-
Medical schools are required to continuously assess pro- son Education Inc.; 2004. p. 5.
gram effectiveness, especially in light of recent budget cuts 2. Durning SJ, Hemmer P, Pangaro LN. The structure of program eval-
and diminishing resources. Given that programs are under uation: an approach for evaluating a course, clerkship, or components
increased scrutiny, effective program evaluation has never of a residency or fellowship training program. Teach Learn Med. 2007;
been more vital. For these reasons, the previous description 19:308-18.
provides a glance into a program evaluation model that is 3. Gibson KA, Boyle P, Black DA, Cunningham M, Grimm MC, Mc-
useful in a wide array of contexts within medical education. Neil HP. Enhancing evaluation in an undergraduate medical educa-
One of its key benefits is that it is a flexible approach that tion program. Acad Med. 2008;83:787-93.
can be useful in answering a wide variety of programmatic 4. Musick DW. A conceptual model for program evaluation in graduate
questions. It also actively includes principal stakeholders in medical education. Acad Med. 2006;81:759-65.
the decision making process, increasing the likelihood that 5. Patton MQ. Utilization-focused evaluation. In: Stufflebeam DL, Ma-
results will be used to guide program development and deci- daus GF, Kellaghan T, editors. Evaluation models. Boston (MA):
sion making. For these reasons, we encourage those interest- Kluwer Academic Publishers; 2000. p. 425-38.
ed in evaluating a particular program to consider the utiliza- 6. Patton MQ. Utilization-focused evaluation. 4 ed. Thousand Oaks
tion-focused approach. (CA): Sage Publications; 2008.

Page 3 of 3
(page number not for citation purposes)