Control of Major Accident Hazards

COMAH Competent Authority Workstream 2e

Process safety performance indicators

(Operational Delivery Guide)

the Competent Authority

 Control of Major Accident Hazards

Purpose 3

Justification 3

Topic Scope 3

Resource 3

Stages of development 4

Competent authority expectation 4

Main actions 4

Links to sector performance indicators 5

Supporting information 5

Success criteria 6

Judging success and moving on 6

COIN IRF Scoring 6

Enforcement 7

COIN time reporting 7

Further information 8

Annex 1: Core intervention issues 9

Annex 2: Example bulk tank filling 14

2 of 15 pages
 Control of Major Accident Hazards

Purpose

The COMAH Competent Authority’s strategic aim for Process Safety Performance Measurement is that:

■ By the end of 2015 all major hazard establishments and duty holders will measure their performance on the
control of major hazard risks by way of key leading and lagging performance indicators.

The strategic inspection topic covered by this guide is designed to continue the promotion and development of
site level process safety performance indicators (PSPIs) as part of the monitoring arrangements for an effective
process safety management system at major hazard sites.

Justification
Recent major accidents in both the UK and abroad highlighted a serious weakness in how some major hazard
plants were operated. Namely, that executive decisions in the board room were often made without those in
charge understanding the risks of their plant or the implications of business decisions on the continued safe
operation of the plant.

The BP Grangemouth major incident investigation report concluded that one of the key lessons for major
hazards sites was: ‘Companies should develop key performance indicators for major hazards and ensure
process safety performance is monitored and reported against these parameters’.

There is a need for a robust and systematic process safety management system supported by strong process
safety leadership from board level down throughout the organisation in order to achieve the right culture and
enable key decision makers to understand all the dimensions of process safety risk when making business and
operational decisions. Good management decisions can only be made on the basis of sound information and
PSPIs provide the means to highlight key issues of process safety management performance.

Key process safety control systems deteriorate in use, therefore operators need to monitor their continuing
effectiveness. Auditing may be too infrequent to detect and rectify critical deterioration of RCS and audit
protocols frequently are not outcome based.

Topic Scope

All COMAH Site Operators should develop, implement and use PSPIs for their process safety management
systems for major hazard operations. Organisations do not need to monitor everything but rather focus on the
vulnerable aspects of their process safety management system.

PSPIs should be the subject of planned interventions at major hazard establishments in the context of COMAH
Schedule 2, paragraph 4(f) ‘Monitoring performance’, one of the key elements of a Safety Management System.

Resource

Interventions should be undertaken by Regulatory Inspectors. It is anticipated that Regulatory resource will be
0.5 days per intervention.

Input may be required from process safety specialists should particular problems with a dutyholder’s
management of process safety be found. Involvement should be arranged and discussed locally.

3 of 15 pages
 Control of Major Accident Hazards

For companies within the Lead Unit System, the lead inspector should coordinate the intervention. We would
expect each site to have developed their own suite of indicators unless the sites are very similar in their
business and have the same risks, controls in place, when a generic suite of indicators would suffice.

Stages of development

The development of process safety performance indicators on a site can be considered to involve 4 stages:

■ Stage 1 – Introduction to the topic and assimilation – general understanding of need for, nature of and
methods for developing PSPIs usually informed by HSG254 and/or visit or workshop by HSE inspector, may
take up to 3 months.
■ Stage 2 – Development phase – should follow as soon as possible from stage 1. Involves creation of
suitable team to develop an initial set of indicators, (probably for one area or process) along with data
collection and reporting systems, can take around 6 months.
■ Stage 3 – Trial period – normally need to run with the initial set of indicators for 6 – 12 months to generate
sufficient data to judge suitability & quality. Further indicator development for other areas/processes can also
be taking place.
■ Stage 4 – Roll out and operation – implementation of full suite of developed and tested indicators across
the site, could take a further 6-9 months.

From this it follows that the full implementation of a site wide PSPI programme could take between 18 months
and 2 years 6 months from when the initial introduction takes place.

The actual time taken will depend on the size and nature of the site along with the level of commitment and
resources that are applied. Local judgement should be exercised when discussing timetables with dutyholders.

Competent authority expectation

■ The practice and knowledge on how to develop PSPIs is now well established and understood and so the
CA expectation for COMAH sites is that they:
– Have a programme for PSPIs – as one of the main ways the adequacy and performance of PS risk is
managed,
– PSPI’s are linked to effective Process Safety Leadership and should be used to inform high level
decision making
– KPIs should be set according to risk profile of the activities undertaken
– Adopt both Outcome (lagging) and Activity (leading) indicators,
– Use the findings from the KPI programme to drive improvements on site, and
– share sector performance data with the CA as part of a dialogue about performance.

Main actions

On site interventions:

■ Inspectors should assess which stage sites are at within the development of PSPIs and input this into the IRF.
■ Inspectors should check progress made towards the development of PSPIs has been based on sound
decisions/information. See Annex 1.

4 of 15 pages
 Control of Major Accident Hazards

Monitoring progress:

■ It is expected that inspectors should assess a dutyholder’s performance approximately every 12 months and
input a rating into COIN. This may be part of a planned intervention, but if no site visit is made, could be by
telephone or letter. Although a ‘physical’ check of indicators cannot be made each year – this allows us to
keep a year on year picture of progression.

Links to sector performance indicators

To support the development of PSPIs, HSE is working with trade associations to support them to develop and
implement sector based indicators and guidance will be produced for their member/sector companies.

Primarily, sector indicators should be viewed as the means by which a trade association demonstrates and
benchmarks the performance of their members. Sector indicators can also, where the member companies
undertake similar activities that present similar risks, form a common set of indicators that provide individual
operators with the range of information needed to monitor process safety on site. When operators rely on
sector indicators to also monitor site performance, inspectors should check that the indicators used are
appropriate for the activities undertaken on site (see what we look for in PSPIs at Annex 1) and also that
operators have the necessary understanding to able to use the information effectively.

Supporting information (with links)

1 Texas City report.

2 BP Grangemouth major incident investigation report.

3 Section 11 of the safety report assessment manual (Major accident prevention policy and safety
management systems aspects) includes criteria for measuring performance and arrangements for their review:
Criteria 11.23, 11.23, 11.25 and 11.27 (P59-61) make explicit reference to setting carefully selected and
targeted leading and lagging performance indicators.

4 HSG254 - Developing process safety performance indicators: a Step-by-step guide for chemical and
major hazard industries.

This publication sets out:

■ the difference between leading and lagging (performance) indicators;

■ a 6-step process to implementing a process safety measurement system; and
■ a case study (chemical bulk storage facility) with examples of indicators across a range of relevant risk
control systems.

5 OECD Guidelines on safety performance indicators. This publication shows how companies can select
indicators against the key elements of a process safety management system. The methodology described
mirrors HSG254 but includes useful additional guidance and examples.

6 Process Safety Leadership Group (PSLG) Final Report.

7 Scottish Power Case Study highlighting how the company successfully implemented an integrated process
safety management system and PSPI programme.

5 of 15 pages
 Control of Major Accident Hazards

Success criteria

By end March 2013:

■ verified that all ‘Buncefield’ type sites have effective monitoring of process safety performance in place and
that site specific leading and lagging performance indicators have been developed as required by PSLG (link
to new DG).
■ that PSPI progress is monitored in accordance with 2012/13 intervention plans; and
■ an updated rating on sites’ progress will be input on the COIN IRF.

Judging success and moving on

The COIN Inspection Performance Rating (IRF) system is used measure a site’s progress with this Strategic
Topic.

This is not an absolute measure of site ‘performance’ for this Topic but will be used by the CA to monitor the
progress made by COMAH sites in establishing PSPIs, and auditing their effectiveness.

COIN IRF scoring

The COIN Inspection Rating Form (IRF) tab on the COMAH intervention Plan Service Order should be used to
record the operator’s progress on PSPIs in the “Key Performance Indicators” line.

6 of 15 pages
 Control of Major Accident Hazards

Progress should be recorded following each PSPI intervention on the following basis:

Progress Rating Description CA action required Score

Stage 4 A site has had indicators in place for No action required. 10
over 12 months and are content they
are providing them with assurance
their process safety risks are
controlled
Stage 3 A site has recently developed a suite Monitor progress 20
of indicators and are trialling them
Stage 2 A site is in the process of developing Agree timetable for move 30
a suite of indicators to Stage 3
Stage 1 A site has just had their introductory Agree timetable to at least 40
visit and is considering the use of Stage 3
indicators
N/A N/A N/A (50)
Not Started A site has not had an introductory Ensure topic covered on 60
visit and is not known to use PSPIs site asap

The data will be used by the CA to monitor the progress made by COMAH sites in establishing PSPIs and
auditing their effectiveness.

Comments about progress and the intervention should also be made in the Notes field of the Service Order
as normal, with the Summary line to include the Key-Word somewhere in the field to PSPI1, PSPI2,
PSPI3 or PSPI4 (no spaces)

Enforcement

Where a COMAH site operator currently does not have PSPIs in place and where the nature of the major hazard
control systems and supporting management arrangements to prevent a major accident are complex and likely
to deteriorate without careful and routine monitoring to detect change. Then the expectation is that the duty
holder should be required to implement a programme of PSPIs in line with good practice (note: HSG 254 is just
one method of developing PSPIs). This DG sets the anticipated timescales for the stages of implementation of
a PSPI programme and inspectors should be mindful in taking enforcement action requiring a programme to be
implemented that such programmes typical take many years to develop, test and fully implement.

COIN time recording

In order that the Competent Authority can demonstrate the deployment of resources to Strategic Topics, there
has been a slight alteration to the way we record time spent on these inspections on COIN:

■ For Strategic Topics there should be an entry made in the ‘Category’ box of the Time Line via a drop down menu
■ The category selection will be restricted to the Strategic Topics and the relevant one should be selected when
■ All other time line instructions are as before

7 of 15 pages
 Control of Major Accident Hazards

Business unit Work Desc Activity Category Work context Type

(BU)
COMAH Company INSP_COMAH StratTop emer ELSE/ Case/Service
onsite OFFCO/ Order
StratTop emer OFFNC/
off-site OSITE
StratTop PSPI
KPI
StratTop Ageing
Plant
StratTop
Buncefield PSLG
StratTop
Containment

Further Information

This guidance was reviewed and revised in February 2012. Any comments or queries on content should be
addressed to Viki Beckett (VPN 523 5807).

8 of 15 pages
 Control of Major Accident Hazards

Annex 1 – Core intervention issues

Competent Authority Expectations

COMAH operators should monitor the effectiveness of their process safety management systems and
demonstrate its effectiveness by setting process safety performance indicators.

Key actions for inspections

■ Inspectors should assess which stage sites have reached within the development of PSPIs and input this
into the IRF.
■ Inspectors should ensure any progress sites have made reflects the key objectives set out in this Delivery Guide.

The information below has been split under the four key stages of a KPI development programme and the
associated line of questioning/advice required from inspectors.

Stage 1 – yet to commence development of PSPIs

If a company has yet to commence the development of PSPIs, inspectors should introduce the concept, how
PSPIs support effective risk management and the basic steps to develop them (based on HSG 254). For COMAH
sites it is the CA’s expectation that the company should have least started to develop PSPIs and be in a position to
have fully implemented leading and lagging PSPIs by 2015. Inspectors should explore why the company has yet to
start on this programme.

A very thorough PowerPoint presentation covering all the above information can be found in TRIM folder 4.6.91,
doc no 2008/651392. This will be a useful starting point for any presentations required for a site visit or indeed
just for your own use, as background knowledge.

Stage 2 - development phase

Ask questions on the process followed so far:

1 Who has been involved?

A wide range of staff should be involved in the development process:

■ Safety professionals and engineers for knowledge of process safety management.

■ Workforce involved in the operational procedures – for their knowledge of the process ‘in practice’, also to
encourage shared understanding of the risks and controls, why the information is being collected.
■ Senior management – they are the key customers for the information.

How have they decided what they are measuring/monitoring?

We would expect the site to have:

■ Identified and focussed on the main vulnerabilities within the PSM – what’s likely to go wrong quickly and/or
with the greatest consequence? All process safety management systems deteriorate over time but especially
those where the process involves or relies upon human intervention.

9 of 15 pages
 Control of Major Accident Hazards

■ From this risk/vulnerability profile, apply James Reason’s model (Managing the Risks of Organizational Accidents
- James Reason 1997, Ashgate Publishing Limited) to identify barriers or key control systems in place to prevent
loss of containment – it is usual for companies to discover gaps in the processes/controls at this stage
■ Started to set indicators to monitor these key control systems.

Depending on how well companies are doing, it may be a good idea to run through the above using a specific
company example. You don’t need to know the system in any detail, just prompting the company to think this
process through is really helpful. The example at Annex 2 outlines the main factors to consider when setting
KPIs for bulk tank filling.

Tolerances

If tolerances haven’t been set for each PSPI, explain why they are important, ie it is the point at which the
information is brought to senior management attention and so senior managers should be involved in deciding
the tolerance point.

Stage 3 – PSPIs developed and being trialled by a company

It is useful to review the decisions made around the indicators to use and how they have been determined,
particularly whether they match the risk profile identified at Stage 2.

For each KPI the company should know:

■ what it is monitoring;
■ why this is key;
■ what does this information tell them; and
■ if the outcome is ‘poor’’ - what does it mean and what improvements to risk control are needed?

System for collating/presenting information

Useful questions to ask are:

■ Is there someone responsible for collating the data?

■ Is the data easy to collect?
■ How often is the information collected? In a trial period it is good to collect it as often as possible.
■ Who is considering the information? It should be a good mixture of people, health and safety professionals,
process operators, senior management.
■ Are there systems in place to highlight when the dta shows that key risk controls are not working?

Presentation

Inspectors should enquire whether the site is keeping the presentation of the results simple and structuring the
information so that only relevant information is presented, eg senior managers may wish to have only high level
information, whereas plant managers, supervisors and operators may need to have more detail.

There are various ways of presenting information, graphs, traffic lights, smiley faces etc. Whichever is chosen
should clearly show where the company is doing well/badly. Being able to show the link between the leading and
lagging indicators against a particular process is advantageous as it can show if control systems are failing to
stop adverse safety outcomes.

10 of 15 pages
 Control of Major Accident Hazards

Leading Indicator

Lagging Indicator

This may mean the control systems are ineffective!

Stage 4 – Company has been using an agreed suite of indicators for over 12 months

Unless you have been involved in the development process from the very start, it is difficult to just look at a
suite of indicators and be able to make an assessment of their quality. Try to ask the questions that will make
the site assess whether or not the PSPIs are working for them.

Example questions:

■ Has the information received given you more assurance that the key parts of your control systems are
operating as intended?
■ Have you received any surprising data?
■ Did you identify any gaps in your control systems through the development process?
■ Have senior managers taken any action as a result of the information received so far?
■ Have you modified the PSPIs to ensure they are appropriate?

Process Control vs Programme Indicators

To give the most accurate picture of how well process safety risks are being managed it is always best to
set indicators against direct operational controls and against the safe operational envelope rather than to use
indicators simply to monitor progress with particular programmes of work. The diagram below helps illustrate
this. In practice however, duty holders will have a range of indicators and it is important to check whether the
right balance has been struck between these types of indicators.

Operational RCS examples Generic RCS examples Programme Indicator examples

Process/operating envelope. PTW Audits done to time
Control of: Management of Change Audit actions closed out
■ Overfill Inspections completed to time Staff trained to specified
■ Overpressure Instrumentation and alarms competence
■ Corrosion/ageing Safety tours / toolbox talks
■ Overtemperature completed
■ Flow rate
■ Accidental leakage

11 of 15 pages
 Control of Major Accident Hazards

The examples below show how the two different types of indicator can be set against a risk control topic.

Operational Control Indicator Programme Indicators

% Inspection of process control instruments and % statutory inspections completed to time
safety critical alarms completed to time
% Critical operational control – competence % staff trained to specified competence
monitoring completed

Operational control - leading/lagging indicators

Lagging Indicators typically give rise to an outcome that shows whether the process condition is being met.
Indicators should be set at a point before a loss of containment occurs by selecting the lowest detectable event
(eg excursion beyond or a breach of the process control envelope) and setting a lagging indicator at that point.

Leading Indicators typically relate to two aspects – the functioning of instrumentation and sensors that monitor
the process condition and correct completion of critical actions by personnel involved in that stage of the process.

Dutyholders should not get too concerned about whether an indicator is leading or lagging – if it works for
them, that is the main thing.

Reviewing indicators

The management of process safety risk is not static but a dynamic activity and process safety risk control is
never fixed and should constantly be reviewed to take into account:

The performance of the indicators:

■ is the information needed being provided?

■ What does the data reveal?

The scope of indicators:

■ Do the indicators still reflect the main process risks? For instance has there been significant:
– changes in processes/new processes introduced;
– changes in staff levels;
– any loss of competence; or
– plant design alterations.

12 of 15 pages
 Control of Major Accident Hazards

Is the data suggesting the site indicators may need to be reviewed?

Leading Indicators

Lagging Indicators

Potential review issue Leading indicators may be too far Tolerance may be set in the
removed from critical control wrong place – too lenient
Tolerance may be set in the wrong
place – too stringent

In summary, if dutyholders are:

■ monitoring the critical aspects of their process safety management systems and this reflects the main site
risks, and
■ using PSPI information to better control process safety.

Rate a company as having reached stage 4 - PSPIs have been implemented and are being successfully used
in practice.

13 of 15 pages
 Control of Major Accident Hazards

Annex 2
Example: Bulk tank filling

The risk to a loss of containment is – failure to correctly manage the liquid level in the tank.

When setting ‘outcome’ indicators use the guide words/stages

■ Is there agreement on the desired successful outcome from the activity?

■ Can the outcome be readily detected? If not, it can’t be measured
■ Is the information/data on the outcome already recorded, captured somewhere? This makes its adoption as
a KPI easier

The key point about outcome metrics is that they reveal no direct information on the cause of an outcome.
Adverse outcomes therefore need to be reported and investigated to find out why, and what went wrong. This
may touch on cultural issues within an organisation as reporting failure may not be welcomed or rewarded.

Process control indicators

■ The outcome (lagging) indicator is whether the desired/intended level is always achieved when filling (this
level can of course vary but will always be predetermined before the tank is filled (hopefully!)
■ The metric becomes – the number or percentage of tanks that were filled beyond the intended level. The
period can be any length of time eg per week, month, year, etc.

The safety success parameter should preferably be set the same as the process/business success parameter,
as too much in a tank for whatever reason always indicates a process problem/ error and provides an
opportunity to learn about what gave rise to that error.

■ The metric should not be ‘the number of times the tank is overfilled until a loss of containment occurs’ (eg
over topping) as this is way too late in terms of picking up on systematic failure.
■ The metric could be ‘number/percentage of tanks filled to the high level alarm’ as these will be less frequent
events and perhaps easier to identify when such an adverse or unintended outcome has occurred. But this
metric is not advantageous of ‘filled beyond the intended level’.
■ The first activity (leading) indicators would be ‘whether the instrument, in this case the tank level gauge,
sensor and control room readout system is inspected and maintained to the appropriate schedule’.

Note: many organisations simply focus on capturing inspection or maintenance of alarms or shut down
systems as being safety critical, whereas the real critical item, as with Buncefield, is the functioning of the tank
level gauge.

■ Taken across an organisation as a whole if level control is critical (as it is at tank farms) the metric would be
‘the percentage of tank gauge instrumentation systems inspected and maintained to schedule’.

If pressure control was the dominant risk then it would be ‘pressure sensors/indicators’ etc.

A second and just as important activity KPI relates to the completion of the correct action by the process
operators to achieve the desired successful outcome. There will be lots of operator actions, probably all
recorded in operating procedures, SOPs. Some analysis is needed to work out which are more important.

14 of 15 pages
 Control of Major Accident Hazards

Again we use guide words to help:

■ Which actions are done frequently, and perhaps needed every time the task is undertaken?
■ Which are closest in time to achieving the desired outcome?
■ Which allow for some degree of variation or personal judgement?

So for the tank filling example these may include:

i. Select the tank with the right design specification for the product
ii. Work out the tank head space (ullage) before filing the tank
iii. Design the flow route, valve sequence to get the product to the selected tank,
iv. Set the tank alarm levels on the control system
v. Set the correct flow route valves/ connections
vi. Open the valves in the right sequence, start the pumps
vii. Monitor the change in level
viii. Close off the pumps when at the desired level

So the most critical would be ii, iii, v and vi.

■ But actually the metric would be ‘when checked (by observation ) the percentage of critical tank level control
actions completed correctly’.

Collection of this metric data could be by a period sample check for each key operator undertaking these
activities. So for instance every operator is checked at least annually – more frequently if new or the tasks
are very critical. This KPI is much more useful than measuring how many operators have been trained in the
process or indeed how many tasks have appropriate SOPs (up to date etc).

Examples of Generic Risk Control Indicators

Generic risk control indicators typically relate to site-wide control systems rather than those covering specific
operational processes. Typically the outcome indicator could be whether there was a loss of containment or
adverse consequence relating to the activity.

■ For instance for Plant Change’ it would be to detect whether the desired improvement achieved, plant
process decommissioned without incident etc.

The same would apply for a permit to work system eg:

■ to detect whether there are any unplanned loss of containment or energy releases associated with the work.

The activity measures (leading) again based on critical task analysis, typically would be a retrospective check (audit)
to see that the correct authority was given to the change or for the PTW maintenance work to go ahead, and or
was the change implemented according to the approved design, or for PTW were the correct isolations made.

Sometimes, it is not possible to set an outcome measure (lagging KPI) for a generic control system, for
example emergency arrangements. This is because a successful outcome for a mitigatory measure is always
that less harm was done in the event of an emergency than would have been the case had the measure not
been in place. In such circumstances activity measures become much more critical.

Once again, the activity (leading) indicators would typically be measured during drills and tests of emergency
arrangements eg whether critical actions within the emergency plan where undertaken correctly.

15 of 15 pages