Performance evaluation

1
The fundamental purpose of evaluation:
Is the determination of the worth or merit of a program or
solution

The ultimate purpose, and value, of determining this worth is in

providing the information for making data-driven decisions that
lead to improved performance of programs and organizations.

The notion that evaluation’s most important purpose is not to

prove but to improve

Every component of an evaluation must be aligned with the

organization’s objectives and expectations and the decision will
have to be made as a result of the evolution findings.

These decisions are essentially concerned with

 How to improve performance at all levels of organization:

-internal deliverables
-organizational gain
 -public impact

At its core, evaluation is a concept:

 It compares results with expectation.
 It finds drivers and barriers to expected performance.
 It produces action plans for improving the program and
solutions.

Here are some general reasons for conducting an evaluation

 To see if a solution to a problem is working, that is,

delivering valued ends.
 To provide feedback as part of a continual monitoring,
revision, and improvement process.
 To provide feedback for future funding of initiatives.
 To confirm compliance with a mandate.
 To determine if value are added for all stakeholders.
 To justify decisions that has already been made.
 To hold power over resources.

Performance improvement as a systematic combination of

three fundamental processes:
 Performance analysis(or needs assessment)
 Cause analysis(the process that identifies the root
causes of gaps in performance)
 Intervention selection(selecting appropriate solutions
based on the root causes of the performance gaps)

Making evaluation happen:

Ensuring stakeholders’ Buy-In

Evaluators should be competent in some basic areas.

Sanders(1979)proposed that at a minimum, evaluator
should be able to:

 Accurately describe the object and contest of that

which is being evaluated.
 Conceptualize the purpose and framework of the
evaluation.
 Derive useful evaluating questions, date
requirements, and appropriate date sources.
 Select the means for collecting and analyzing data.
 Determine the value of the devalued (object).
 Effectively communicate results and
 recommendation to the audience.
 Manage the evaluation project
 Maintain ethical standards.
 Adjust to external factors influencing the
evaluation
 Evaluate the evaluation

Recommended text books:

1. ingrid J. Guerra-Lopez,2008. Performance evaluation.(San

Francisco:Jossey-Bass)

2. Gary Dessler, 2005.Human Resource Management.

(Prentice Hall Inc.)

3. Heinz Weihrick and Harold Koontz,1994.(New York:

McGraw-Hill)

Term paper topic:

“Application of performance evaluation system in your
preferred organization”

Organization of term paper

 Theoretical frame work about the performance
evaluation theory.
 Analysis of performance evaluation system in your
preferred organization
 How do you employ “performance evaluation theory”
to analyze what is good and what goes wrong and what
is your alternative solutions?
 What is your suggestions to change and
implementation
 References(your citation)

General evaluation orientations

 Formative evaluation

 Summative evaluation

Formative evaluation
 Typically occurs during the development stage of a program
and can be used to improve the program before it is formally
launched.

Summative evaluation

Occurs after the implementation of a program of solution and

usually requires some appropriate amount of time to have
transpired so that object of evaluation has the opportunity to
have the full impact required on performance at various levels
of the organization.

Challenges that evaluators face

 Convincing people that programs require evaluation (costs

and benefits).
 Getting people to use findings and recommendations.
 Lack of key stakeholder and consumer involvement.
 Lack of leadership support.
 Limited expertise.
 Failure to see improvement.
 Fear and cynicism( used for control and intimidation)
 Ensuring commitment
 To get ownership of evaluation.
 To get commitment and participation of all stakeholders.
 To obtain buy-in form those managers with a common
purpose of improving things.
 To involve partners in setting objectives and sharing with
them the results of any success and shortfalls.
 To have an open and honest relationship with stakeholders
based performance data, not just biased opinions and
perceptions.

Benefits of evaluation

 Provide relevant, reliable, and valid data to help make justifiable

decision.

 Provide the basis for design, development, and implementation

project management plans.

 Identify any adjustments that have to be made during and after

 development and implementation, so that resources are

maximized.

 Provide the means to document successes so that a merit of

decisions, department. Staff and solutions are recognized by all.

 To disseminate and market the organization’s successes to

internal and external partners.

Principles of performance evaluation

Six principles that guide high quality evaluation

Principle 1: ask the right questions

Principle 2: Understand between means and ends

Principle 3Examine the goals of organization as a function of its

valid needs

Principle 4: needs should be validated using a top-down external

approach

Principle 5: Evaluation standards should be based on a useful and

real strategic vision

Principle 6: The set of questions derived from the first five

principles drives the evaluation study

Principle1: evaluation is based on asking the right questions

 Societal string:

-What is the role of the organization in our society?

-What are the ultimate results sought?

-Are the results of benefit to society?

-what should be accomplished by the organization as a whole?

-etc.

 Process string:

-How will this program be implemented?

-What is the timeline for deliverables?

-what resources will be used?

-what are the limits of the influence of the evaluation?

-how will results be measured?

-etc.

 Decision string:

-which program will yield the best return?

-what is the least-cost method the meets the objectives and

criteria?
 -Should we look into different alternatives?

 System string:

-What is the ultimate purpose of our organizational system?

-are all subsystem working for the greater system?

-what is the communication flow?

-what is the resource flow?

-do all objectives add up to a coherent system goal and

overreaching objective?

-etc.

Principle 2: evaluation of process is a function of

obtained results

Two concepts that often get blurred in organization (and in life):

“What and how” or, stated another way,

“Ends and means”

Ends are:

Results, accomplishments, and consequences.

Means are:

Ways, methods, and resources

Therefore, useful evaluations focus on ends and not just on means.

Principle 3: goal and objectives of organizations should be based on

valid needs

Valid and useful goal and objectives come from a proper needs

assessment, which should never be confused with a wants or wish

list.

Needs are gaps in results (between what should be and what is), not

gaps in resources or methods. Gaps are collected at three levels of

results and consequences: societal, organizational, and departmental

or individual.

Principle 4: derive valid needs using a top-down approach

An organizational vision is the ultimate result and organization seeks
to obtain, it is tied to the strategic level of an organization.

The organizational mission is a more immediate and operational

result that the organizational commits to deliver in the next year or

two, this is tied to the tactical level of organization and it is basis for

the operational planning and processes that go on at the level of the

functions or departments.

Principle 5: every organization should aim for the best that society

can attain

Generally, customers figure out which organizations have their

benefits in mind and which care only about the bottom line.

Think for a moment about the type of organizations that earn your

loyalty and business.

Most consumers have the same criterion: organizations that provide

quality products and services that truly meet their needs, from the

most basic to the most specialized.

Principle 6: the set of evaluation questions drives the evaluation

study

The evaluation efforts should be aligned with the questions that

stakeholders must answer.

The evaluation team should develop evaluation questions related to

each objective and activity relevant to the program or solution under

evaluation.

The set of questions determines which activities will be undertaken

and which methods will be used to observe, gather, and record data.
一月 18 日

Models of evaluation

Overview of six classic evaluation models

 Objective-based evaluation: tyler(1949)

 Consumer- oriented evaluation: scriven(1967)
 Discrepancy model of evaluation: provus(1971)
 Goal-free evaluation: scriven(1974)
 Responsive/client-centered evaluation: stake(1975)
 Utilization-focused evaluation: patton(1977)

Objective-based evaluation: Tyler (1949)

Objective-based evaluation describes whether students have met
their goals, with the results informing how to handle a new
instructional strategy (revise, adopt, reject).

One weakness of this approach is that the evaluator may

overlook unexpected outcomes or benefits of instruction beyond
the original goals.
Consumer- oriented evaluation: Scriven (1967)
This approach is focused on meeting consumer needs and
societal ideals more than achieving the developer’s objectives
for a given program.

Discrepancy model of evaluation: Provus (1971)

This model has four basic phases:
- Establishing objectives
- Collecting evidence of compliance with standards
- Indentifying and discrepancies between pre-established
objectives and what was accomplished
- identifying and starting corrective actions

Goal-free evaluation: Scriven (1974)

The evaluator purposely remains uninformed of the program’s
predetermined goals and looks for all effects of a program
regardless of its developer’s intended objectives.
Goal-free evaluation provides important supplementary
information, expands the sources of evaluative information, is
effective for finding unexpected information, is cost efficient,
and is welcomed by clients.

Responsive/client-centered evaluation: stake (1975)

Based on Ralph Tyler’s conception that evaluators should
compare observed outcomes to intended outcomes.

Stake’s approach expanded evaluation focus to include

background, process, standards, judgment, and outcomes, and
calling for continuing communication between evaluator and
stakeholders for the purposes of discovering, investigation, and
addressing important issues.

Utilization-focused evaluation: Patton (1977)

Closely linked to decision-oriented theory, this approach to
evaluation is concerned with designing evaluation that inform
decision making.

You have just been asked to take the leadership in an evaluation

effort in your organization, but you do not know much about
evaluation. How will you go about figuring out which
evaluation models or conceptual framework (or combination of
models and framework) will fit your purpose?

1 月 26 日

Selecting a model

There is no perfect model of evaluation; there is only the most

useful one for a given situation. Frequently this will be a
combination of models and approaches.

Kirkpatrick’s four levels of evaluation

 Developed in 1959 by Donald Kirkpatrick as part of his

dissertation research.
 Is well accepted evaluation framework in the industrial and
organizational psychology arena (Cascio, 1987).
 It has been used primarily to evaluate traditional instructor-
led training programs.
 Kirkpatrick’s consists of evaluating four
distinct general areas in sequential order

 Reactions: focuses on participants’ opinions about the

training, process, and results.
 Learning: focuses on the degree to which learning
actually took place.
 Behavior: focused on the linkage between learning
workplace performance.
 Results: focuses on bottom-line results.

Kirkpatrick’s levels
Level 1 evaluation: reactions
- How participants react to a training program, with the driving
question being “did they like the program?”
This level is often referred to as a “smile sheet” because the
data are often gathered through a questionnaire.
- While some doubt the utility of such instruments, Kirkpatrick
(2006) argues that every program should be evaluated at least
at this level
Because it is a measure of customer satisfaction.
 The following sample questions can help maximize the utility
of a smile sheet:
 Is the information present relevant?
 Is the information present sufficient?
 Are the instructional materials complete?
 Is more background information required?
 Etc.
An instrument like this is particularly useful in the context of a
Formative Evaluation, where the participant responses will be
used to improve the development of the training program.

Where reactions are positive, so are the chances for learning.

And if learning is improved, then the chances or a change in
behavior are also improved.

Level 2 Evaluation: learning

- Focusing on the extent to which students have increased their
skills knowledge. Or desired attitudes.
- Often test participants before the training (pretest) and after
training has been completed (posttest).
- Requiring the evaluator to measure the right thing in the right
way.
Level 3 evaluation: behavior
- Focusing on the change that has occurred in the behavior of
learners due to the training program. Evaluation at this level
attempts to determine whether the skills, knowledge, or
attitudes obtained from the training program are being
applied to the job.
- Foxon’s transfer of training model(1993) outlines transfer as
a series of stages:
1. Intention to transfer
2. Initiation of transfer
3. Partial transfer
4. Conscious maintenance
5. Unconscious maintenance

Level 4 evaluation: results

- Measuring the success of the training program in terms of
organizational or departmental performance indicators, for
example, production quality, costs, reduced complaints, sales,
or profits.
These are the organizational results that the training program
ultimately is to affects.
 One important consideration here is to allow sufficient time
to transpire in order to see an impact on bottom-line results.

2月1日

Phillip’s return-on-investment methodology

 Jack Phillips (1997a, 1997b) popularized the return-on-

investment (ROI) evaluation process within the training and
performance improvement arenas.
 This approach is commonly known as “level 5” adding a new
dimension to Donald Kirkpatrick’s four levels of evaluation.

The main drive of Phillips’ methodology is to measure the

monetary benefits of training and development programs against
their costs.

One key attribute of the model is that it address and provides

guidance for the isolation of the effects of training on the bottom
line through a mathematical formula designed to calculate a
percentage return on the total costs of the program:
ROI (%) = net program benefits X 100 / program costs

Phillips (1997b) warns that for the ROI process to be useful, it

must balance feasibility, simplicity, credibility, and soundness.

He also identifies some common barriers to ROI

implementation: costs and time, lack of skills, faulty needs
assessment, fear, lack of discipline and planning, and false
assumptions.

Phillips’ ROI process

Begins with data collection after the program has been

implemented and end with calculation the ROI.
As with kirkpatrick’s model, data should be collected at levels 1,
2, 3, and 4 so that, accepting kirkpatrick’s sequential analysis,
the chain of impact occurs as participants learn and apply skills
and knowledge that affect business results.
 Collecting post-program data

Data collection is the heart of this process. These data should

include:

 Hard data(for example, production or cost)

 Soft data (such as attitudes).

These data can be collected with host methods:

 Questionnaires
 Observations
 Interviews
 Focus group
 Program assignment
 Action plan
 Performance contract
 Performance monitoring
 Isolating the effects of training

 A control group:
One group receives the training, and a similar group (the
control group, does not.
 Trend lines:
These are visual aids used to project the values of specific
indicators if training had occurred.
 Forecasting models:
If mathematical relationships between input and output
variables are known, this is an appropriate approach.
 Participant estimations
The participants are provided with the overall observed
improvement, on a pre-and post-program basis, and asked to
estimate the percentage of improvement that they believe is
related to the training program.
 Leadership estimations:
The same approach as participant estimates, but the
respondents here are supervisors or management.
 Expert estimations:
The same approach and logic as participant and leadership
 estimations.
 Accounting for other influencing factors:
The other variables are identified and their impact estimated
or calculated.
 Customer estimations:
In some cases, customers can provide their perceptions of the
extent to which training has influenced their decision to use a
product or service.

Converting data to monetary values

After level 4 data are estimated. They are converted to monetary
values and compared to program costs. To do this, a value must
be placed on each unit of data connected with the program.

Here are ways to make these conversions:

 Output data: converted to profit or costs savings.
 The cost of quality: estimated, and quality improvements
are directly converted to cost savings.
 Participant wages and benefits: be used as a value for time.
 Historical costs are used if they are available. In this case,
organizational costs data are used to establish the specific
 value of an improvement.
 Experts can be used to estimate the value of a improvement.
It depends on the credibility of the expertise. The expert can
be from outside or from within the organization.
 Database (reliable, valid, and timely) may be available to
estimate the value or costs of data items. The challenge is
finding databases that are relevant for the purpose at hand.
 Participants can estimate the perceived value of data items.
Provided they are qualified or capable of dong so.
 Leader can provide their estimation.
 HRD Staff can be a good source of estimate, provided they
remain relatively unbiased.

Tabulating the costs of the program

On the other side of the cost-benefit analysis equation are the

program costs. Estimating these costs depends on ongoing
monitoring. Here are some of the elements to consider:

 Costs to design and develop the program. Perhaps prorated

over the expected life of the program.
 Costs of all program materials. Including those provided to
 the instructor and participants.
 Costs of the instructor. Including preparation, facilitation,
and other time on task.
 Cost of the facility
 Travel, dodging, and meal costs for participants.
 Salaries and employee benefits of the participants who
attended the training.
 Administrative and overhead costs of the training function,
allocated efficiently and accurately.
 Costs associated with a needs assessment-if one was
conducted-as well as the evaluation.

Two considerations here are key:

 Consider the standard cost variables used by other

departments in the organization in their ROI estimations.
These comparisons should be as equitable as possible when
leadership compares the ROI’s of various programs across
different department.
 When in doubt, err on side of conservativeness. Include all
the fair costs you can, so that your efforts to perform an
honest ROI are obvious.
Calculating the return on investment of the program

The cost benefit ration (CBR) is the program benefits divided by

the cost:
CBR = program benefits / program costs

The ROI uses the net benefits divided by program cost, with the
net benefits essentially being the program benefits minus the
costs:
ROI (%) = net program benefits / program costs

Identifying the intangible benefits of the program

Most training programs have intangible, non-monetary benefits

that should also be considered-for example, increase job
satisfaction, increased organizational commitment, improved
teamwork, or reduced conflict. There too can be expanded into
measurable performance indicators and precisely quantified and
measured.
2009 年 2 月 15 日

The impact evaluation process

 One of the underlying ideas is that everything we do is

aligned to some greater purpose.
 The evaluation must judge whether achieving those goals
would contribute to the welfare of clients and consumers.
 The idea is that with a well-articulated plan. The actual
evaluation process will be simpler and more straightforward.

The elements of the process

The impact of evaluation process consists of seven elements.

1. Identifying stakeholders and expectations.
2. Determining key decision and objectives.
3. Deriving measurable indicators.
4. Identifying data sources.
5. Selecting data collection methods.
6. Selecting data analysis tools.
7. Communication of results and recommendations
Step1: Identifying stakeholders and expectations.
The stakeholder groups are those who will be making decisions
throughout the evaluation process or directly as a result of the
evaluation findings.
Including this group will make the implementation of the
evaluation plan easier, particularly during the data collection
stage.
Each stakeholder group will have its own set of expectations.

One common element that should tie these individual

expectations together is the organizational vision.

These processes presume that if the evaluator does not clearly

identify the expectations and requirements of stakeholders, it is
nearly impossible to meet those expectations and requirements.
Also critical is aligned stakeholder expectations with external
societal demands.

Step2: determining key decisions and objectives

 Asking the stakeholders to think carefully about these
decisions early in the process will help the evaluator focus on
 the important issues and lead to useful data.
 All organizations have objectives, both external and internal,
and everything within organization should contribute toward
those objectives.
 It is important that evaluator identify a comprehensive and
representative list of questions.
 What the most critical questions are, and thus what this
evaluation study commits to deliver.
 The ultimate purpose of the organization relates to long-term
success based on meeting the needs (not just want) of
external stakeholder and thus adding social value.

Step3: deriving measurable indicators

 Indicators are observable phenomena, and it can be linked to
something that is not directly observed.
 Indicators for customer service include referral, repeat
business, customer retention, length of accounts, and
satisfaction survey scores.

2 月 22 日
Step 4: identifying data sources
 With a list of specific indicators in hand about which data to
 collect, evaluators must determine where to find those data.
 The sources the evaluator selects depend on the type of data
sought. Other sources may be experts, employees, and
leadership.

Step 5: selecting data collection methods

 The right data collection methods and tools are a function of
the data required.
 If you seek hard data such as sales figures, for example, do
not use survey to get people’s opinion of these sales figures.
Rather, review sales reports. Conversely, if you are interested
in attitudes, there are a number of ways to get them
(interview, focus groups, and surveys, among others)
 The secret ingredient for successfully collecting valid and
reliable data is alignment of data type, data source, data
collection tools, and, later, data analysis procedures.
Step 6: selecting data analysis tools
 Selecting the appropriate technique depends on not only the
scale sued to measure the data but also the specific purpose
of our analysis.
 Statistics in evaluation enable us to sensibly and clearly
summarize large amounts of data. There are basic categories
 of summary statistics:
1. Central tendency (mean, mode, median)
2. Dispersion of variance (SD)[standard deviation]

 Another purpose statistics is that they can be used to

determine the relationship between two or more events.
 The tern used for this relationship in quantitative technique is
Correlation, which represents the degree to which the item is
related and is expressed in terms of a Coefficient (ranging
from -1 to +1).

Step 7: communication of results and recommendations

 The evaluation results will be worthless if it does not
communicate the importance and purpose of the evaluation
process and associated activities throughout and the data-
supported findings and action that must be taken as a result.
 If people are not moved to action as a result of the
evaluation, the main objective to create positive change-has
not been met.