Syllabus | UAPP684-010

Performance Management and Program Evaluation

2017 Fall Semester
Wednesday 2:00 p.m. to 5:00 p.m. | 187 Graham Hall

Maria P. Aristigueta
Telephone: 302 831-4570
Email: [email protected]

Course Overview

The purpose of Performance Management and Evaluation is to familiarize students with the basic frameworks, concepts,
logic, techniques, methods and theories of performance measurement and program evaluation. It is expected that students
will obtain a sufficient understanding of these to be able to select among, interrelate, critique, and build upon individual
approaches when appropriate in a public context. This course should serve as a foundation for further understanding and
utilization of these approaches in areas related to the student’s specialization and interests. This course focuses on how to
measure the achievement of organizational goals and objectives and not how decisions actually are made; hence, the focus
is on normative not descriptive decision-making.

We will operate on the premise that even if an analyst in the various departments conducts the analysis or we contract out
for the evaluation, it is important to understand the logic and methodology in order to manage the organization.
Performance Management and Evaluation introduces some important modeling techniques by creating an environment in
which to use them self-consciously and experimentally. The major models that we will consider are strategic planning,
performance measurement, PERT/critical path method for project management, and a series of program evaluation
techniques.

The main elements of this environment are the following:

 books and articles that explain the models and techniques

 case studies and/or practical application of tools
 a computer with a spreadsheet program (i.e. Lotus 1-2-3 or Excel)
 the other members of the class and your work (study) group
 three hours a week in class to discuss what we did and what it was good for
 individual comments on your work
 canvas with power point presentations and assignments

Course-Based Competencies

1. Understand and apply basic frameworks, concepts, theories, techniques, and methods of normative, analytical
decision-making models and understand the underlying assumptions, requirements, and subtleties of these
models. Such models include PERT/critical path analysis, Logic Models, Performance Measures, Program
Evaluation techniques and Strategic planning.
2. Gain familiarity with the use of analytical models, their frameworks and applications, in varying situations, and
organizations which utilize these methods such as GAO and other government organizations, policy centers, and
university research centers.
3. Think carefully, precisely, analytically, and quantitatively, framing issues and decisions in analytical frameworks
leading to the generation and analysis of alternatives while understanding the requirements, terms and concepts,
limits, and advantages of analytical models and under what circumstances they are applicable and useful.
4. Learn how to work in groups and apply analytical models understanding the technical and non-technical issues in
group decision making and communicating utilizing the terms and concepts of the model.

1
Classes

Learning-by-doing is the underlying premise of the educational process to be utilized in this course. Thus, students will be
assigned examinations, group assignments, and computer work as appropriate.

Assignments and Grades

Homework and Class Assignments (10%)

Homework will be utilized to reinforced reading materials. Homework assignments will be due as stated on the syllabus. I
will grade your assignments on a pass/fail basis. These assignments will be worth 10% of your grade

Use of Methods found in literature (20%)

Each student will review the literature or technical reports for examples of use of methods discussed in class to examine a
policy issue or administrative problem. Examples of journals to research include: Journal of Policy Analysis and
Management, Public Administration Review, Journal of American Planning Association, Journal of Urban Analysis,
Public Performance and Management Review. Reports such as GAO are also a good source and website is included on
the syllabus. The purpose of this assignment is to reinforce the methods by expanding on how others are using the
method. This is an oral and written assignment (5-8 pages long) and will count 20 points and 20% of your grade. Please
come prepare to discuss your article in class the week after your method is discussed.

1. What is the policy issue or administrative problem the author is addressing?

2. What method(s) are utilized by the author to study the issue?
3. Would you have Chosen the same method(s)?
4. Is the data well presented? Would you have presented the information differently?
5. Does the author identify the client?
6. What recommendations are made? Are alternatives presented?
7. Do the findings support the recommendations? Would you have made similar recommendations?

Case Study (30%)

Doctoral students will be required to do a case study. Master’s students may choose to complete a case study assignment
instead of the freight project. You will be asked to present your case study to the class. The written assignments should be
typed, double-spaced. Information for the assignment will be distributed in class. This assignment is also worth 30% of
your grade. Sample case studies may be found at: https://aspacap.wordpress.com/cap-case-study-program/2014-cap-case-
studies/

Freight Project (30%)

All Master’s students will be required to work on the evaluation of the 2017 Delmarva Freight Summit, unless you choose
to do a case study. We will work on this project throughout the semester and will have the opportunity to use the tools
learned in class for the evaluation.

Exams (40%)
There will be two exams worth 20 points each and 40 percent of your grade. The first exam will be in-class covering
terminology and the second, will be a take-home exam.

Grades
Students will be graded on the following scale:
A Excellent, professional level work with major insights into the material, great detail in the work,
understanding of subtleties and issues in work, well-written
A- Excellent work with few problems or questions, almost an A
B+ Competent with insights beyond simple understanding
B Competent, understanding of basic material, acceptable level of work
B- or below Lack of comprehension of the material at an acceptable level

2
Hints
Hints for doing well in this course:
1. Keep up-to-date on readings and homework assignments. The work is cumulative.
2. Read the text twice—once before the class and once after the class. You are more likely to understand the subtleties if
you reread the Chapters after you have worked with the method for a while. You must read the assignments to
understand the class sessions.
3. Turn in assignments on time. Late assignments result in a lower letter grade.
4. Work as a team on group assignments to enhance decision making process.
5. Consider the information as cumulative and display your knowledge on assignments.

Readings
Textbooks

Theodore H. Poister, Maria P. Aristigueta*, and Jeremy Hall, Managing and Measuring Performance in Public and
Nonprofit Organizations (Jossey-Bass, 2015)

Kathryn E. Newcomer , Harry P. Hatry and Joseph S. Wholey, Handbook of Practical Program Evaluation (Jossey-Bass,
2015).

And other readings available on websites or Canvas and listed on the course schedule.

*My royalties for use of this book will be donated to the University of Delaware’s School of Public Policy and
Administration”.

Websites

American Evaluation Association: http://www.eval.org/

American Society for Public Administration:

http://www.aspanet.org/cap/index.html

U.S. Government Accountability Office: http://www.gao.gov/

The National Academy of Public Administration: http://www.napawash.org/pc_government_performance/index.html

APA Style: http://www.apastyle.org/ The American Psychological Association (APA) style guide provides some tips for
students on line at its website. You may find particularly helpful information about citation, collaboration and other
components of academic integrity on-line.

Free Management Library:

http://managementhelp.org/performancemanagement/index.htm

US Office of Management and Budget: www.omb.gov

Kellogg Foundation Logic Model Guide:

http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-
Guide.aspx

Balanced Scorecard: http://www.balancedscorecard.org/BSCResources/AbouttheBalancedScorecard

3
Citizen-Driven Performance Measurement Curricular Project: accessed through
http://newark.rutgers.edu/~ncpp/sloan/index.htm (this will be referred to in the syllabus as Rutgers website).

Harvard Family Research Project: accessed through http://www.gse.harvard.edu/hfrp/about.html#top (Harvard’s website).

How Delaware compares: accessed through http://www.howdelawarecompares.org/index.html

Recommended Readings

Wouter Van Dooren, Geert Bouckaert, and John Halligan, Performance Management in the Public Sector (Routledge
2010).

Christopher Pollitt and Geert Bouckaert, Public Management Reform: A comparative analysis (Oxford University Press,
2004)

Arie Halachmi, “Who Gets What When and How: Performance Measures for Accountability? For Improved
Performance? International Review of Public Administration, Vol. 7, No. 1, 2002.

Managing for Results: Progress in Linking Performance Plans with Budgets and Financial Statements. GAO-02-236
http://www.gao.gov/cgi-bin/getrpt?gao-02-236

GAO Tracks States' Progress Implementing the Recovery Act http://www.gao.gov/recovery/ and
http://www.gao.gov/cghome/d09952cg.pdf
How does your program measure up? By ELISE CASTELLI May 18, 2009
http://federaltimes.com/index.php?S=4092610

2010 Obama Budget: "Replacing PART with a New Performance

Improvement and Analysis Framework"
http://www.whitehouse.gov/omb/budget/Analytical_Perspectives/

“Research Based Evaluations of Agency Performance” By Elise Castelli

http://www.federaltimes.com/article/20090511/AGENCY02/905110305/-1/RSS

Harry Hatry, Performance Measurement: Getting Results (Urban Institute Press, 1999). A must if you have not had
Management Decision Making.

Paul Light, Tides of Reform (Yale University Press, 1997).

Elaine Morley, Elisa Vinson, and Harry P. Hatry, Outcome Measurement in Nonprofit Organizations: Current Practices
and Recommendations (Urban Institute Press and Independent Sector 2001).

Elaine Morley, Scott P. Bryant, and Harry P. Hatry. Comparative Performance Measurement (Urban Institute Press,
2001).

Kevin P. Kearns, Managing for Accountability (Jossey-Bass, 1996).

Mark G. Popovich Creating High-Performance Government Organizations (Alliance for Redesigning Government,
1998).

4
Course Schedule
August 30, 2017 Introduction and Use for Tools for Planning, Evaluation, and Policy Analysis

Readings: Newcomer, et al, Ch. 30; https://www.youtube.com/watch?v=EOFKR5c7-NE&list=PLwCA1bP2u-

kupasWKejeQdrgNkslTfwB9 on project management. http://www.mindtools.com/critpath.html and
http://www.netmba.com/operations/project/pert/ for CPM and PERT.
Human Subjects Review (http://www.suapp.udel.edu/content/protect-human-subjects)

 Introduction to the course

 Course Requirements
 Pros and Cons of Rational Analysis
 Strategic Planning (overview)
 Policy Analysis Process
 Evaluation
 Problem Definition/Study Questions
 Project Management
 Critical Path and PERT

Troy Mix to explain Freight Project

September 6, 2017 Strategic Planning and legislative requirements

Readings: Poister, et al, Ch. 8; http://www.gpo.gov/fdsys/pkg/PLAW-111publ352/pdf/PLAW-111publ352.pdf
http://www.businessofgovernment.org/blog/business-government/gpra-modernization-act-2010-explained-part-1
The Use of Strategic Planning:

 Framework
 Process
 Use
 Mission
 Values

Assignment: Examine GPRA and its update, Public Law 111/352. What are the requirements of the law? Are there
similar laws in States? What about Local Government? Please post on CANVAS by Tuesday, Sept 9 and be prepared to
discuss in class.
Sign-up for literature review and case study (unless you are working on the freight project)

September 13, 2017 Strategic Planning continues. Newcomer, et al, Ch. 2; Poister et al Ch. 13
Lit Review on use of Strategic Planning
Key Steps in the Process continue

 Identifying Stakeholders
 SWOC
 Identifying Strategic Issues
 Performance Analysis
 Strategies
 Vision

Using strategic planning to support performance measurement

Assignment: Please submit via CANVAS selection of agency, department, or program for case study and article for Use
of Method assignment. Do a SWOC analysis of Freight to Wilmington.

5
September 20, 2017 Establishing Evaluation Criteria and Logic Models
Readings: Newcomer et al, Ch. 3 and Chapter 4; GAO/GGD-98-26 – Performance Measurement and Evaluation:
Definitions and Relationships. Poister, et al, Chapter 3; logic model discussion:
(http://www.gse.harvard.edu/hfrp/projects/aftersChool/resources/learning_logic_models.html) Kellogg Foundation Logic
Model Guide, http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-
Development-Guide.aspx

 Study/Research Questions
 Outlining the Problem
 Goals
 Objectives
 Criteria
 Measures
 Logic Models
 Decision Trees
 PERT

Assignment: verify, define and detail a problem that you have identified in the Freight industry. Display your problem in a
logic model. Write goals, objectives, criteria, and measures for the problem.

September 27, 2017 Designing Performance Monitoring Systems

Readings: Newcomer, et al, Ch. 5 and Poister, et al, Chapters 1, 2, and 15
Free Management Library, browse the performance management content at:
(http://managementhelp.org/performancemanagement/index.htm); balance scorecard

 Implementation Evaluation
 Performance Monitoring

Assignment: Select a Kids Count report from a state of your choice. What types of measures are used? How useful is the
information? What changes might you make to increase usefulness? How might you use this information?

October 4, 2017 Analysis and Use of Performance Data

Readings: Poister, et al., Chapters 4, 5, 6, 7
http://www.nasact.org/nsaa/committees/performance/downloads/bestpractices/Contracting_for_Services.pdf

 Making Information Useful

 Analysis of Performance Information
 Reporting
 Uses Including Budgeting
 Quality Control
 Performance Contracts

Assignment: Select a government or nonprofit website of your choice and critique the display of performance measures:
How are these different from Delaware Compares? Is the information useful? How might it be used? What changes
would you make?

October 11, 2017 First Exam (in-class) and Designing Performance and Evaluation Studies
Readings: Newcomer et al., Ch. 1; Poister et al., Ch. 10
Evaluation--(http://www.gse.harvard.edu/hfrp/projects/aftersChool/resources/snapshot1.html#sample)

 Managing Employees
 Planning and Designing Evaluation
Assignment: Using the measures you identified last week, design an evaluation study to determine the causal relationship
of one or more of the measures.

6
October 18, 2017 Data Collection: Agency Records and Customer Surveys
Readings: Newcomer Ch. 13 and 14

 Information Obtainable
 Advantages and Disadvantages
 Customer Survey Questionnaires
 Survey Administration
 Ways to Reduce Costs

Assignment: Critique Freight survey on Canvas.

October 25, 2017 Data Collection continues: Role playing, trained observers, and field data collection.
Readings: Newcomer et al., Chapters 15, 16, 17;
(http://www.gse.harvard.edu/hfrp/projects/aftersChool/resources/snapshot5.html).

 Expert Judgment
 Role Playing
 Focus Groups
 Field Data Collection
 Trained Observers
 Internet

Assignment: Develop criteria for trained observer, role playing, or field data collection for Freight Conference

November 1, 2017 More Practical Considerations

Readings: Newcomer et al., Chapters 10, 11, 12, 21

 Multisite Evaluations
 Community Change
 Culturally Responsive
 Using Stories

November 8, 2017 Analyzing Evaluation Data and Writing for Impact

Readings: Newcomer et al., Ch. 22 and 25; “Research Based Evaluations of Agency Performance” By Elise Castelli
http://www.federaltimes.com/article/20090511/AGENCY02/905110305/-1/RSS

 Qualitative Data Analysis

 Systematic Review
 Evaluation Syntheses

November 15, 2017 Getting Evaluation Results Used

Readings: Newcomer et al., Ch. 27 and 28; GAO/GGD-00-204 – Program Evaluation

 Explain Program Performance

 Managing Evaluation Projects
 Reporting Results
 Maximizing Use of Evaluations

Presentation of Case Studies

Assignment: Analyze a GAO report of your choice and explain format utilized. Is there anything that is included in your
readings that is missing from the report? What might you do differently in writing up the Freight Report?

7
November 29, 2017 Other Issues in Evaluation and Performance Measurement
Readings: Poister et al., Ch. 11 and 15; Newcomer et al., Ch. 31; http://www.govexec.com/gbc/three-states-engaging-
data-driven-innovation/119558/?oref=sidebar-gbc-promo

 Pitfalls of Evaluation
 Contracting for Evaluation
 Performance Management in Grants and Contract Programs
 Big Data
 Evidence-Based
 Ethical Consideration

Assignment: What are some broader implications to alternative service delivery models? How can public administration
maintain public service values in alternative service delivery models?

December 6, 2017 Presentation of Freight Evaluation Project, Second Exam (take-home) Distributed

December 13, 2017 Second Exam Due