Benchmarking as a Maintenance Performance

Measurement and Improvement Technique

By Sandy Dunn,
Director, Assetivity Pty Ltd
Benchmarking was a "hot" topic in the early 1990's. It was seen as the next "big
thing" in business improvement. In contrast to the Quality Improvement movement,
which offered small, frequent, incremental improvements in business performance,
Benchmarking offered the key to large-scale, step improvements in performance.
Unlike the labor intensive, high workforce involvement Quality Circle approach,
benchmarking was supposed to provide these gains with relatively little effort, as it
involved learning from the best organisations, and "copying" what they did. Everyone
was "doing" benchmarking, or talking about doing it. Governments were even
assisting private enterprises to do it. Like most over-hyped techniques, the bubble
soon burst, and benchmarking was in serious danger of becoming viewed as yet
another of those "flavour of the month" management consulting-driven fads that
would quickly sink into oblivion.
With the passage of time, however, benchmarking has not disappeared into the realms
of folklore. It is increasingly being viewed as one of a number of important business
improvement tools that any organisation should have in its kit bag. If anything, it is
going through something of a resurgence, as organisations attempt to become less
inward- and more outward-focused. In this section of the paper, we set out to discuss
benchmarking in an equipment maintenance context, and discuss a number of key
questions, including:

What is benchmarking?

Why benchmark?

Obtaining, analysing and understanding global best practice - The

benchmarking process

Determining what to benchmark

The importance of consistent terminology

Adopting and incorporating best practice

What is benchmarking?
There are a number of different definitions of benchmarking, which all have a similar
flavour, but a slightly different emphasis. The definition that I prefer, and which is
contained within the Maintenance Terminology article at www.plantmaintenance.com/terminology.shtml, is as follows:
The process of comparing performance with other organisations,
identifying comparatively high performance organisations, and learning
what it is they do that allows them to achieve that high level of
performance.

*a-se-tiv'-i-ti, n, the efficient and effective configuration, operation and maintenance of physical assets

PO Box 1315
Booragoon WA 6154
AUSTRALIA

www.assetivity.com.au
[email protected]
ACN: 092 112 288
ABN: 32 092 112 288

Phone: +61 8 9474 4044

Fax: +61 8 9474 4055

The key aspects of this definition are that:

Performance is measured - this implies the existence of quantitative,

identically defined performance measures

High performance organisations must be identified - this requires some

knowledge of those organisations

It is not sufficient to merely compare performance - to be truly effective,

benchmarking involves gaining an understanding into why those organisations do
well, and, as a result, what you need to do to achieve similar performance
The last point above is particularly important. Often, it is easy to get hung up on the
quantitative comparison of performance to the exclusion of gaining a true
understanding of why high performance organisations achieve the results they do, and
then applying that knowledge to improve your own organisation.

Why benchmark?
So, why benchmark? In theory, benchmarking gives benefits in the following areas:

Improved understanding of business processes

New ideas leading to improved business performance

The ability to "fast-track" improvement initiatives by utilising knowledge

already held by "best-practice" organisations

In practice, in my view (and there may be many that disagree), the benefits in these
areas are frequently oversold, although there is little doubt that, when performed
successfully, significant benefits can be achieved through benchmarking. However,
one benefit of benchmarking, and one which, in most organisations, is sufficient in
itself to warrant undertaking some form of benchmarking, is to generate broader
recognition of the need for any improvement at all.
In many organisations, particularly at lower levels within the organisation, and
especially so in those organisations where there has been a history of distrust between
management and "the workers", there is a general perception that "things are OK as
they are", and therefore if it aint broke, dont fix it. Comments that are heard
among shop floor and first line supervisory staff are often along the lines of "Sure
there are a few minor things that could be done better, but, for the most part, we do a
pretty good job". Benchmarking often shatters that illusion. For example in John
Campbell's book Uptime, he quotes the case of a European microchip manufacturer
which had set itself what it thought to be a daunting goal - to double a production
line's reliability from 24 to 48 hours. However, when it did some comparisons with
similar production lines in Japan, it discovered that reliability on those lines averaged
200 hours. Instantly, the goal of 48 hours became obsolete - the production line could
not possibly hope to be competitive, let alone a world leader, if it achieved such a low
level of performance.
The change process is often described as consisting of three phases - Unfreezing,
Changing and Refreezing. In order to "unfreeze" an organisation there needs to be a
common recognition that it needs to change. Benchmarking certainly has a vital role
to play in this area. Benchmarking can be used to provide the basis for an "Imperative
for Improvement" within the organisation. This is often its major value.
Copyright, Assetivity Pty Ltd., 2003
Page 2

Does this mean that all organisations that undertake benchmarking get some value
from it? Some interesting research from the American Quality Foundation in 1992
surveyed 580 service and manufacturing firms that had undertaken benchmarking.
They classified these firms into three classes - high performers, medium performers,
and low performers, based on their Return on Assets. What they found was that firms
that were already high performers showed particularly high and positive results from
their benchmarking activities. Firms that were already medium performers showed
"no compelling positive impact" from any of their benchmarking activities, while low
performance firms actually demonstrated negative impact from having conducted
benchmarking. The conclusion of the researchers was that low performing firms
probably needed to focus more on getting their core business under control, rather
than distracting themselves by trying to emulate the "best of the best".

Obtaining, analysing and understanding global best practice - the

benchmarking process
In its simplest form, the benchmarking process can be outlined as in the following
diagram:
Plan the Project
Form the Teams
Identify the data
Collect the data
Analyse the data
Take Action

The first step, Planning the Project, involves:

Establishing the objectives of the benchmarking project

Defining the boundaries of the benchmarking project which business

processes are included, which geographical areas etc.

Deciding whether to benchmark best in class or best in industry. For

example, if you were a mining company wishing to benchmark its spare parts
warehousing and logistics processes, then you could decide to benchmark
yourself only against other mining companies (best in industry) or you could
choose to benchmark yourself against any other companies which have spare
parts warehousing and logistics functions, such as the automotive industry
(best in class). Benchmarking best in class potentially will provide
greater benefits, in providing insights into good practices that may not yet
have filtered into your industry, but is also potentially more expensive. It can
also be more difficult to identify potential benchmarking partners, and some of
the practices that are identified may not be entirely appropriate for your
industry.

Deciding whether to communicate directly with your benchmarking partners,

or to benchmark through a third party. Particularly if you are seeking to
Copyright, Assetivity Pty Ltd., 2003
Page 3

benchmark within your own industry, many of your competitors may be

reluctant to invite you to examine their data (and you may be reluctant to give
too much away to them also!). In this situation, benchmarking through an
impartial third party may be an avenue to ensure that more reliable data is
made available, and that there is more widespread involvement in the
benchmarking project. This impartial third party could, potentially, be an
industry or professional body (such as the Society of Maintenance and
Reliability Professionals see below), or alternatively could be a respected
consulting organisation.
The second step, Forming the Teams, recognises that, to be effective, benchmarking
must result in concrete improvement activities. It is important, therefore, that those
people who will be responsible for, or potentially affected by, the resulting actions
should be identified early in the process, and become involved in performing and/or
overseeing the benchmarking activities. Performing formal stakeholder analysis to
identify the stakeholder groups or individuals who should be involved in the project is
extremely worthwhile at this stage.
The third step, Identifying the Data, involves confirming the business processes that
you wish to benchmark, and identifying the qualitative and quantitative data that you
wish to collect. This step also involves identifying potential data sources, and making
decisions regarding where the data will be collected from. A data collection plan
should be developed at this point which addresses the following points:

How will you organise the data that you collect?

How will you check that the data you collect is relevant, accurate and up-todate?

How will you arrange the responsibility for collection data?

What data will be used from external sources and/or third party publications
and databases, and what data will be collected directly from site visits?

What tools will you use to collect the data and make sure that it is complete?

How much data do you need?

There are many possible third party sources of benchmarking data some of these are
available via the internet, but almost all benchmarking data costs money.
Some of the benchmarking data is industry-specific, and maintenance benchmarks are
only one part of the total benchmarking package. Examples of these include:

Solomon Associates (www.solomononline.com), for those in the petroleum

refining or petrochemicals industry

Electric Utility Benchmarking Association (www.euba.com) the International

Electricity Generation Benchmarking Association (www.iegba.com), or the
Generation Knowledge Service (www.beyondbenchmarking.com) for those in
the Power Generation, Transmission or Distribution businesses.

The American Productivity and Quality Center (www.apqc.org), has taken

over the International Benchmarking Clearing House, and offers a Knowledge
Sharing Network and Best Practices database covering most industries, and

Copyright, Assetivity Pty Ltd., 2003

Page 4

several processes, but there is not a maintenance-specific benchmarking study.

Other benchmarking databases are specific to maintenance, and are generally crossindustry databases. There are two, that I am aware of, that are free of charge, but
these are of limited value, due to concerns about the quality of the data contained in
their databases. The two free maintenance benchmarking resources are:

MaintenanceBenchmarking.com (www. maintenancebenchmarking.com), This

site contains a small number of specific questionnaires/surveys, free of charge,
on topics relating to maintenance, such as use of infrared thermography,
CMMS benchmarking, Electric Motor Testing etc. These are of limited value,
as there is no data cleansing, and they use someones preconceived assessment
of what represents best practice as the benchmark. While these
preconceived notions may be accurate, they do not enhance the learning
process.

Plant Maintenance Resource Center (www.plantmaintenance.com/benchmarking.shtml) - a quick, free, survey tool, which
collects, by industry, information regarding maintenance costs as a % of
estimated equipment replacement value, and maintenance costs as a % of total
site costs. Again, this has limited value, as there is no quality assurance on the
data that is collected, and there are only two measures collected.

Other, pay for service, benchmarking services and databases that I am aware of are
listed below:

SIRF Roundtables (www.sirfrt.com.au) offers an excellent benchmarking

service, consisting of a site visit to collect and validate data, and then
comparison with their excellent database of over 100 maintenance
organisations. This database has been collected over the last 6 or 7 years, with
a significant proportion of the organisations being in the mining industry, but a
wide range of other industries are included also, including petrochemical
organisations, other continuous manufacturing process organisations, and
discrete manufacturing organisations.

The Society for Maintenance and Reliability Professionals (www.smrp.org)

offers its Executive Company members a free annual maintenance
benchmarking service. Other non-member organisations can also participate
for a very moderate fee. The benchmarking study consists of a detailed
questionnaire, in eleven sections, but there is no independent verification of
the accuracy of the data submitted.

The Australian Graduate School of Engineering Innovation, in association

with the Maintenance Engineering Society of Australia
(www.agsei.edu.au/benchmarking) also offers a benchmarking service, but
this is relatively in its infancy.

There are also many other consulting organisations that offer Maintenance
Benchmarking services, including Assetivity, and others.

The bottom line is that you will pay more for a quality database the free services
simply wont meet the needs of anyone looking to perform serious benchmarking.
The fourth step in the benchmarking process, Collecting the Data, involves actually
Copyright, Assetivity Pty Ltd., 2003
Page 5

collecting the data in the format that you have previously determined, using the tools
that you have selected. If you are using third-party data sources, then you are likely to
have more time available to browse the data that is available, and manipulate this into
the format that you prefer for later analysis. However, if your data is being collected
during site visits, then time is likely to be more limited, and you must, therefore, be
highly organised to make sure that you collect all of the data that you have planned to
collect. You will need to ensure that the data collection roles of every member of
your site visit team are well defined and clearly understood. You should also
establish, beforehand, which data collection methods your benchmarking partner is
happy for you to employ for example, at some sites, you may be permitted to take
photographs and/or video, while at others, you may not even be permitted to take
notes.
The fifth step in the benchmarking process, Analysing the Data, involves comparing
and contrasting the data that has been collected, and sifting out the important
observations from the merely interesting. There are a number of analytical tools that
may assist here at the most sophisticated end (and only possible with large
quantities of reliable data) are quantitative statistical analysis tools such as multiple
regression analysis, which permit you to identify those factors that appear to have the
greatest impact on organisational performance. On the other hand, you may be able to
identify significant improvement opportunities by using simple tools such as
comparison charts.
When performing the analysis, what you are seeking to identify are the key business
process enablers. The quantitative or qualitative data that you have collected will
assist you to identify those organisations that are performing well, but you must focus
on identifying those organisational factors that have enabled this high level of
performance to be achieved. In essence, the performance measures tell you how far
you may be able to improve, but the enablers tell you how to achieve that
improvement.
The final step in the benchmarking process, Take Action, involves developing
concrete improvement actions, assigning responsibilities for these actions, ensuring
that adequate resources are made available for implementing these actions,
determining the timing of actions, and then making it happen. This is easy to write in
one sentence, but far more difficult to actually achieve! In essence, all improvements
need to be effectively project managed, and the most effective organisations simply
do not take their eye off the ball until the improvements are in place.
Determining what to benchmark
Benchmarking Metrics vs Processes
As mentioned previously, it is often easy to put undue emphasis on the quantitative
portion of the benchmarking process - the benchmarking metrics. However, knowing
the size of the gap you have to bridge in order to achieve "world class" performance is
only of limited value if you have no idea how you are going to bridge it. It really is
vitally important to understand the reasons why the good organisations are as good as
they are. This involves getting in behind the numbers and looking at all of the things
which they do which might be leading to their excellent performance, and learn from
them.
For example, you may find that the best Maintenance organisations have a lower ratio
Copyright, Assetivity Pty Ltd., 2003
Page 6

of Maintenance planners to craftsmen than your organisation has. Does this mean that
you should reduce the number of planners in your organisation? Not necessarily.
Digging in behind the numbers, you may find that world class Maintenance
organisations have computerized information systems that permit the more speedy
development of Maintenance plans; they may also have built up, over time, a
comprehensive library of job plans, based on previous work that they have done,
which means that whenever one of those jobs arises again, then they simply need to
pull the job plan out of the library, make a few adjustments, and then issue it (whereas
you may need to develop each job plan from a blank sheet of paper); you may also
find that the world class organisation has progressively eliminated causes of failure,
thereby leading to a lower requirement to plan corrective work, because there is less
corrective work arising requiring planning; and so on.
So there are some dangers in simply measuring quantitative benchmarks of
performance the best way to benchmark is to use these quantitative benchmarks as a
starting point for further investigation into the underlying reasons for the differences,
which will inevitably lead you into consideration of such things as maintenance
management processes, maintenance work practices, organisation structure, skill
levels, organisational culture, the use of information and other technologies, and so
on.
Which Metrics to Benchmark?
Having decided to benchmark some metrics, the next key question to be answered is
Which metrics should we benchmark?
As discussed earlier in this paper, Kaplan and Norton have introduced the concept of
the Balanced Scorecard for performance measurement, and while this concept is
more commonly used to establish the appropriate performance measures within an
organisation, I believe that it is equally applicable in identifying benchmarking
metrics.
To recap, the Balanced Scorecard suggests identifying performance measures based
on four key perspectives:

The Owner/Shareholders view of your business

The Customers view of your business

Internal Processes

The Learning Organisation

In my view, the first three of these are likely to be highly relevant for benchmarking.
The performance measures that are relevant in the final view the Learning
Organisation are more likely to be company-specific, and therefore less relevant for
benchmarking.
Kaplan and Norton also suggested that there should be relatively few performance
measures that are reported in each of these dimensions (but, as discussed earlier, these
performance measures could differ as you move down through the organisation the
performance measures that are of interest and relevance to a craftsman or production
operator are likely to be different to those that are of interest to the Maintenance
Manager or Production Manager, for example).

Copyright, Assetivity Pty Ltd., 2003

Page 7

Lets re-examine the balanced scorecard from the point of view of the Maintenance
process, but the principles could be applied to any particular business process that you
may be benchmarking.
The Shareholders View Shareholders are most likely going to be interested in
financial and risk performance. The measures that should be developed here are
likely, therefore, to include measures of costs, and risks, such as safety,
environmental, asset integrity risks.
The Customers View While I am hesitant to refer to Production as Maintenances
customer, in this case, lets consider the Maintenance process from a Production
perspective. What are they looking for? In this case, the measures are likely to be
physical output measures from the maintenance process. These could include
measures such as Equipment Availability, Equipment Reliability, Equipment
Productivity/Efficiency, Product Quality etc. They may also be interested in other
performance measures such as breakdown response times, maintenance workmanship
quality etc.
The Internal Process View In this case, we are interested in measuring the
performance of key internal maintenance processes which lead to high levels of
performance in the Shareholders and Customers eyes. For example, we may wish to
measure the proportion of work that is performed which has been effectively planned
and scheduled in advance, or we may wish to measure our labour productivity, the
nature of on-the-job delays that have occurred, or the number of unpredicted
failures.
In each of these areas, there are many performance measures that could be used.
Some of these could be more applicable than others. How should we select the most
useful performance measures from the rest?
I would suggest a process of selecting benchmark measures that uses the following
process:
Brainstorm potential performance measures identify as many measures as you
can that could measure performance in the three balanced scorecard views discussed
above. You may wish to supplement this with other measures that are commonly
used in industry refer to other benchmarking studies that you may be aware of, or
other publications. Terry Wiremans book Developing Performance Indicators for
Managing Maintenance (Industrial Press, ISBN 083113080) contains a
comprehensive list of over 80 Maintenance performance indicators that may also be
useful to consider.
As was discussed earlier, select the most appropriate measures by assessing their:

Relevance

Reliability

Understanding

Availability of Data

Timeliness

Controllability

Copyright, Assetivity Pty Ltd., 2003

Page 8

The importance of consistent terminology

When benchmarking quantitative measures, it is extremely important to ensure that
the definitions of the measures used are consistent across all the organisations being
compared. For example, take a fairly common performance measure that is used
across many maintenance organisations equipment availability.
What is the definition for equipment availability that is used in your organisation?
There are a number of areas where there may be differences in the way that this is
calculated in different organisations. For example:

If your normal production schedule calls for your equipment to operate

between 6am and 6pm, 7 days per week, and you choose to perform a service
of that equipment after hours for example, between 6pm and 6am, is this
downtime included, or excluded from your calculation of availability?

What about if a maintenance task is started at noon one day, and is not
completed until noon the following day is the downtime for this equipment
(and therefore its impact on equipment availability) 12 hours, or 24 hours?

What if, for operational reasons (say, for example, a product change),
production did not require the equipment for two hours between noon and
2pm, and you took the opportunity to perform some maintenance on the
equipment during that time would the equipment downtime be included in
your calculation of availability, or not?

What if you are operating a fleet of mobile equipment, such as a truck fleet. Is
the time required to transport the truck from its normal operating area to the
workshop, for routine maintenance included, or excluded from the availability
calculation? What about the time that the truck is waiting outside the
workshop for a suitable service bay to become available?

You can see that, depending on what we include, or exclude from our availability
calculations, the recorded figure for availability may vary significantly from
organisation to organisation. It is vital, therefore, to ensure that, for each metric that
you are using for benchmarking, that you have a common definition that is to be used
across all organisations, and that this definition is defined in sufficient detail, to
enable valid comparisons to be made.
And this is just one possible benchmarking metric, that, you may think, has a fairly
universal definition. There are many other benchmarking metrics that are used that
have even less common definitions across organisations, and across industries. For
example:

Reliability (Mean Time Between Failures what is defined as a Failure?),

% Planned Work (by jobs, by labour hours? What is the definition of a

planned job?)

Schedule Compliance (based on what timeframe does a job that is done

within a schedule period, but not on the specific day that it was scheduled,
comply?)

On many, if not most, occasions, the benchmarking definition of a metric will vary
from the definition that you use within your organisation. In order to be able to

Copyright, Assetivity Pty Ltd., 2003

Page 9

perform meaningful comparisons, your data will need to be cleaned and adjusted to
take into account the benchmarking definition. This can sometimes be a lengthy and
labour intensive, but entirely necessary, exercise.

Adopting and incorporating best practice

The key aspect of benchmarking to remember is that it is a change process you can
measure and compare yourself with as many organisations as you like, but if, after
having done so, you have not implemented any improvements, then ultimately the
process has been a waste of time, albeit most likely an interesting one!
While benchmarking, if performed properly, can provide useful insights regarding
what your organisation should do to make step change improvements in performance,
it does not tell you how to make those changes. Implementing significant change
within organisations is a topic, indeed probably a conference, in its own right, and
there is insufficient time or space within this paper to discuss it in detail. However
some of the fundamental principles of successful change that I have learnt, through
experience include:

To be successful, there must be a powerful mandate for change within the

organisation the case for change should be clear and compelling, and not
changing should not be an option. This is easiest to achieve when the
organisation is in significant financial difficulties, but benchmarking can also
assist here in providing an indication of the gap between current and best
practice. Combining this with appropriate analysis of your competitive
marketplace can provide incentive for change.

Implementing significant change requires commitment from higher levels in

the organisation just how high depends on the nature of the change, but for
the largest scale changes, commitment may be required from the CEO and/or
the board

Successful change initiatives proactively manage all stakeholder interests.

Stakeholders can have both positive and negative influences on change these
must be identified and proactively managed throughout the change process.

Successful change requires adequate resourcing many good ideas have

floundered due to a lack of time and/or money being dedicated to their
implementation

What gets measured gets managed make sure that your change projects are
able to visibly demonstrate progress use visual charts and performance
reports extensively to focus attention on what needs to be done.

Look for quick wins nothing breeds success like success. Getting some
runs on the board early in the project helps to generate support for further
improvements.

Overcome the resistance from conventional wisdom. How many excuses

can you come up with for why changes cannot be made? Your challenge is to
replace those excuses with reasons why the status quo is not a realistic option.

Copyright, Assetivity Pty Ltd., 2003

Page 10

Conclusion
So in summarising, there are a few key points to take note of if you wish to effectively
perform benchmarking. These are:

Plan your benchmarking effort carefully

Perform benchmarking in teams

Select your benchmarking data sources with care

Benchmark metrics and processes

Ensure that you effectively implement changes

Copyright, Assetivity Pty Ltd., 2003

Page 11