HI KPI Guidelines

Guidance on developing Key Performance Indicators and Minimum Data Sets to Monitor Healthcare Quality
Health Information and Quality Authority
Guidance on Developing
Key Performance Indicators and
Minimum Data Sets to Monitor
Healthcare Quality
September 2010
Safer Better Care
About the Health Information and Quality Authority

The Health Information and Quality Authority is the independent Authority which has
been established to drive continuous improvement in Irelands health and social care
services. The Authority was established as part of the Governments overall Health
Service Reform Programme.
The Authoritys mandate extends across the quality and safety of the public,
private (within its social care function) and voluntary sectors. Reporting directly to the
Minister for Health and Children, the Health Information and Quality Authority has
statutory responsibility for:
Setting Standards for Health and Social Services Developing person centred
standards, based on evidence and best international practice, for health and social care
services in Ireland (except mental health services)
Social Services Inspectorate Registration and inspection of residential
homes for children, older people and people with disabilities. Inspecting children
detention schools and foster care services. Monitoring day and pre-school facilities. 1
Monitoring Healthcare Quality Monitoring standards of quality and
safety in our health services and implementing continuous quality assurance
programmes to promote improvements in quality and safety standards in
health. As deemed necessary, undertaking investigations into suspected
serious service failure in healthcare
Health Technology Assessment Ensuring the best outcome for the
service user by evaluating the clinical and economic effectiveness of drugs,
equipment, diagnostic techniques and health promotion activities
Health Information Advising on the collection and sharing of
information across the services, evaluating information and publishing
information about the delivery and performance of Irelands health and
social care services
Not all parts of the relevant legislation, the Health Act 2007, have been commenced. Those parts that
apply to childrens services are likely to be commenced in 2010.
Table of Contents
Executive Summary
Part 1:
Overview of performance monitoring
1 Introduction
1.1
Overview of the Health Information function
1.2
Background
1.3
The role of KPIs
2 Quality
2.1
Structure/Process/Outcome
2.2
Quality improvement
10
2.3
Domains of quality
12
2.4
Conceptual frameworks
13
2.5
Methods of monitoring and improving quality
14
3 Key performance indicators for healthcare quality assessment
16
3.1
Types of indicators
16
3.2
Benefits
18
3.3
Considerations
20
Part 2:
Development of Key Performance Indicators and Minimum Data Sets
4 Development of KPIs
24
4.1
Define the audience and use for measurement
24
4.2
Consult with stakeholders and advisory group
24
4.3
Choose the area to measure
27
4.4
Achieve a balance in measurement
28
4.5
Determine selection criteria
29
4.6
Define the indicator
33
5 Develop the Minimum Data Set (MDS)
36
5.1
Define the level of health information
36
5.2
Define the frequency of collection
37
5.3
Document the data collection process
37
5.4
Identify data sources
38
5.5
Identify data for development
38
5.6
Assess compliance with Information Governance
39
5.7
Plan data quality checks
39
6 Data reporting to stakeholders
40
6.1
Determine frequency of processing and analysis
41
6.2
Define method of analysis
41
6.3
Determine level of aggregation
42
6.4
Develop risk-adjustment strategy
42
7 Pilot test KPIs

7.1
Determine review frequency
43
43
8 Conclusion
44
Reference List
45
Glossary of terms
50
Appendices
Appendix 1: HCQI Framework
53
Appendix 2: Examples of consensus techniques
54
Appendix 3: Delphi study example of brief assessment instrument
55
Appendix 4: Example of a clinical KPI
56
Table of Figures
Figure 1: Quality Assurance Triangle(12)
10
Figure 2: Types of KPIs
17
Figure 3: Examples of types of KPIs
18
Figure 4: KPI development process
25
Table of Tables
Table 1: Selection Criteria
30
Table 2: Example of data element attributes
39
ii
Executive Summary
The primary mandate of the Health Information and Quality Authority (the Authority) is
to drive patient safety in health and social care in Ireland. A key component of this
work relates to effectively monitoring the performance of healthcare services. Key
performance indicators (KPIs) are an essential tool in this process as they enable the
public, service users and healthcare providers alike to have reliable information on
current and desired standards in healthcare services. KPIs are used to identify where
performance is good and meeting desired standards, and where performance requires
improvement.
KPIs promote accountability to service users by facilitating comparisons with other
organisations and to stated objectives or targets of an organisation. Further, they
promote accountability to central Government for the efficient use of resources with
other comparable organisations
Reflecting an increased awareness of the importance of quality and safety in healthcare,
quality assessment has become increasingly critical - unless we actually measure the
quality and safety of care, we cannot determine if improvements are being made. This
is one of the key ways in which key performance indicators can have a positive impact
for patients and service users.
Performance monitoring is a continuous process that involves collecting data to
determine if a service is meeting desired standards or targets. It is dependent on good
quality information on health and social care which can only be achieved by having a
systematic process to ensure that data is collected consistently, both within, and across
organisations. One tool that is frequently used to assist in performance monitoring and
which can subsequently contribute to performance improvement in quality and safety, is
the development and monitoring of key performance indicators (KPIs).
KPIs, which are specific and measurable elements of health and social care, can be used
to assess the quality of care. They are measures of performance, based on standards
determined through evidence-based academic literature or through the consensus of
experts when evidence is unavailable.
The purpose of this document is to provide guidance for the development of KPIs and
associated minimum data sets (MDSs) to monitor healthcare quality. Minimum data sets
refer to the minimum amount of information required for the purpose of monitoring
quality and safety through KPIs.
The guidance outlined in this document is based on an analysis of evidence from an
extensive literature review. It is intended as a resource for all stakeholders, including
the public and service users, but more specifically, policy makers and frontline
practitioners, with responsibility for the development and implementation of KPIs and
associated MDSs.
Part 1 of this document provides an overview of relevant literature and outlines the
importance of performance monitoring in contributing to the safety and quality of health
and social care. It introduces key performance indicators (KPIs) and their role in
performance monitoring, including benefits and risks.
Part 2 of this document examines best practice and provides specific guidance on the
development of KPIs and minimum data sets (MDSs). It identifies important factors that
should be taken into consideration when developing and evaluating KPIs for
performance monitoring.
Part 1:
Overview of performance monitoring
1 Introduction
1.1 Overview of the Health Information function
Health is information-intensive, generating huge volumes of data every day. It is
estimated that up to 30% of the total health budget may be spent one way or another
on handling information, collecting it, looking for it, storing it. It is therefore imperative
that information is managed in the most effective way possible in order to ensure a high
quality, safe service.
Safe, reliable, healthcare depends on access to, and the use of, information that is
accurate, valid, reliable, timely, relevant, legible and complete. For example, when
giving a patient a drug, a nurse needs to be sure that they are administering the
appropriate dose of the correct drug to the right patient and that the patient is not
allergic to it. Similarly, lack of up-to-date information can lead to the unnecessary
duplication of tests if critical diagnostic results are missing or overlooked, tests have to
be repeated unnecessarily and, at best, appropriate treatment is delayed or at worst not
given.
In addition, health information has a key role to play in healthcare planning decisions where to locate a new service, whether or not to introduce a new national screening
programme and decisions on best value for money in health and social care provision.
The Health Information and Quality Authority was established under the Health Act,
2007 with the primary objective of promoting safety and quality in the provision of
health and personal social services for the benefit of the health and welfare of the
public.
Under section (8) (1) (k) the Health Act, 2007 the Authority has responsibility for setting
standards for all aspects of health information and monitoring compliance with those
standards. In addition, under section 8 (1) (j) the Authority is charged with evaluating
the quality of the information available on health and social care and making
recommendations in relation to improving the quality and filling in gaps where
information is needed but is not currently available.
Information and Communications Technology (ICT) has a critical role to play in ensuring
that information to drive quality and safety in health and social care settings is available
when and where it is required. For example, it can generate alerts in the event that a
patient is prescribed medication to which they are allergic. Further to this, it can support
a much faster, more reliable and safer referral system between the patients general
practitioner (GP) and hospitals.
Although there are a number of examples of good practice, the current ICT
infrastructure in Irelands health and social care sector, is highly fragmented with major
gaps and silos of information which prevents the safe, effective, transfer of information.
This results in service users being asked to provide the same information on multiple
occasions.
Information can be lost, documentation is poor, and there is over-reliance on memory.
Equally, those responsible for planning our services experience great difficulty in
bringing together information in order to make informed decisions. Variability in practice
leads to variability in outcomes and cost of care. Furthermore, we are all being
encouraged to take more responsibility for our own health and well-being, yet it can be
very difficult to find consistent, understandable and trustworthy information on which to
base our decisions.
As a result of these deficiencies, there is a clear and pressing need to develop a
coherent and integrated approach to health information, based on standards and
international best practice. A robust health information environment will allow all
stakeholders the general public, patients and service users, health professionals and
policy makers to make choices or decisions based on the best available information.
This is a fundamental requirement for a high reliability healthcare system.
Through its health information function, the Authority is addressing these issues and
working to ensure that high quality health and social care information is available to
support the delivery, planning and monitoring of services. One of the areas currently
being addressed through this work programme is the need to provide guidance on the
development of key performance indicators (KPIs).
1.2 Background
Information plays a pivotal role in promoting improvements in the safety and quality of
patient care. Performance measurement promotes accountability to all stakeholders
including the public, service users, clinicians and the Government by facilitating
informed decision-making and safe, high quality and reliable care through monitoring,
analysing and communicating the degree to which healthcare organisations meet key
goals(1). Accurate performance measurement is dependent on information that is of
good quality, comparable and can be shared within the health sector.
KPIs play an important role in the performance measurement process by helping to
identify and appropriately measure levels of service performance. In and of themselves,
KPIs cannot improve quality however, they effectively act as flags or alerts to identify
good practice, provide comparability within and between similar services, where there
are opportunities for improvement and where a more detailed investigation of standards
is warranted. The ultimate goal of KPIs is to contribute to the provision of a high quality,
safe and effective service that meets the needs of service users.
Data used to support KPIs should be standardised, with uniform definitions, to ensure
that it is collected consistently and that it supports the measurement process, facilitating
meaningful comparison. This can be achieved through the development of a minimum

data set (MDS) containing a list of standardised data to support performance
measurement with KPIs.
The purpose of this document is to provide guidance for the development of KPIs and
associated MDSs to monitor healthcare quality. The guidance is based on an analysis of
the evidence from an extensive literature review and is intended to be a resource for
stakeholders in the development of KPIs and MDSs.
1.3 The role of KPIs
Healthcare providers are constantly striving to improve the quality and safety of the care
they provide, and service users are becoming increasingly interested in the quality of
care provided by various organisations and individuals. A seminal report published by
the United States (US) Institute of Medicine, To Err is Human(2) identified deficiencies in
the quality and safety of healthcare in the US and led towards the worldwide realisation
that there was an urgent need to monitor the quality and safety of the care provided
and increase efforts at improvement.
There are three distinct drivers that can encourage organisations to improve the quality
and safety of the care they provide - professionalism, regulation and market forces(3).
With regard to professionalism, members of a profession establish and maintain
standards for its membership through a system of governance. In regulation, the
government and independent regulators such as the Authority, establish standards to
which everyone must comply resulting in an overall increase in the quality of services.
Finally, through market forces, consumers influence improvement in quality and safety
by selecting those organisations that have desirable quality and safety records.
Assessing the quality and safety of care has become increasingly important because,
unless we actually measure the quality and safety of care, we cannot determine if
improvements are being made(4). Although it is a contributing factor, the measurement
of quality alone does not lead to improved performance(3). However, performance
measurement contributes to improving quality in a number of ways(5). Firstly, it drives
improvement by enabling service users to make choices based on quality measures,
which in turn creates an incentive for providers to improve performance so as to attract
more service users.
Secondly, professionals have an intrinsic desire to improve performance when they are
made aware, through performance measurement, that there is potential for
improvement.
Finally, performance measurement drives improvement through comparing the
performance of individuals, teams or organisations resulting in a desire to improve or
maintain performance relative to others and the reliability of the quality and safety of
services that they provide.
The idea of monitoring healthcare quality has been in existence for many years
however, it is only in recent years that it has received extensive attention in published
literature. In order to monitor the quality of the healthcare system it is essential to
determine what aspects need to be measured.
Performance monitoring is dependent on good quality information which can only be
achieved by having a systematic process to ensure that data is collected consistently
both within, and across, organisations. One tool that is frequently used to assist in
performance monitoring and which can subsequently contribute to performance
improvement is key performance indicators (KPIs).
KPIs are an invaluable tool that contribute immensely to the performance monitoring
process. However, for KPIs to be effective, they need to have clear definitions to ensure
that the data collected is of high quality (that is, consistent, reliable and in keeping with
shared definitions) and to enhance their validity and reliability. Valid KPIs measure what
they are intended to measure and reliable KPIs will consistently produce the same result
regardless of who performs the measurement.
Using KPIs can lead to improvements in quality and safety when they are used for
learning at organisational level, facilitating improvements in local service delivery rather
than solely being used as a tool to evaluate providers(6) at a national, system, level.
Using performance indicators at a local level assists organisations develop an insight into
safe and effective care processes.
This guidance has been developed to assist individuals and organisations identify
develop or select KPIs and associated minimum data sets for the purpose of monitoring
quality and safety in health and social care.
The delivery of health and social care is dependent on both clinical and administrative
staff, with a variety of information needs. This guidance is intended as a resource for all
staff and identifies important factors to be considered in order to deliver a balanced
suite of good quality KPIs.
2 Quality
Quality involves meeting and exceeding an acceptable level of performance through the
provision of a safe and effective service. It is a broad and complex concept which is
neither simple to define nor measure but is nonetheless central to effective, modern,
healthcare services. For this reason, improving quality has become an integral
component of effective healthcare delivery and is mandatory in some countries where
there are obligations to comply with standards for healthcare.
In healthcare, concerns about quality usually revolve around the ability of organisations
to achieve desired outcomes using processes that have been demonstrated to achieve
those outcomes(7). Even though quality can be improved without measuring it, for
example through the use of clinical practice guidelines and specialist education, it is only
through measurement that we can be sure that improvements are being made.
Measurement is therefore critically important both in identifying where quality and
safety is compromised and in monitoring quality improvement processes.
2.1 Structure/Process/Outcome
One of the most significant developments in relation to performance monitoring in the
last 30 years has been Avedis Donabedians(8) division of healthcare into structure,
process and outcome, for the purpose of defining and measuring quality. Donabedian
has contributed significantly to improvements in the quality and safety of health and
social care through his lifelong commitment to the use of performance measures.
According to Donabedian(9), healthcare quality can be assessed using a three-part model
based on the structures, processes and outcomes of the healthcare system. This division
of healthcare has allowed the identification of data across the full spectrum of
healthcare that contributes to monitoring the quality of the various constituents of
healthcare delivery.
Structure relates to the resources of the healthcare system that contribute to its
ability to meet the healthcare needs of the population. Structural indicators refer to
the resources used by an organisation to deliver healthcare and include buildings,
equipment, the availability of specialist personnel and available finances.
Process relates to what is actually done for the service user and how well it is done.
Process indicators measure the activities carried out in the assessment and
treatment of service users and are often used to measure compliance with
recommended practice, based on evidence or the consensus of experts.
Outcome relates to the state of health of the individual or population resulting from
their interaction with the healthcare system. It can include lifestyle improvements,
emotional responses to illness or its care, alterations in levels of pain, morbidity and
mortality rates, and increased level of knowledge(10).
Donabedian also stated that each part of the model is interdependent and that good
structures promote good processes and, in turn, good processes promote good
outcomes. The healthcare quality measurement process can be assisted through the use
of KPIs to capture a variety of selected factors and trends of both health and the
healthcare system(11).
2.2 Quality improvement
Improving quality is a continuous cycle involving defining quality, monitoring quality and
improving quality (Figure 1).
Figure 1: Quality Assurance Triangle(12)
2.2.1 Defining quality

Defining quality involves setting and following standards for an acceptable level of
performance. According to vretveit:(13)
A quality health service provides the range of services which meet the most
important health needs of the population (including preventative services) in a
safe and effective way, without waste and within higher-level regulations.
10
In other words, a quality healthcare service provides care based on the assessed needs
of the population, using finite resources efficiently to attain optimum outcomes and
minimise the risks associated with healthcare delivery.
According to Donabedian(8) healthcare quality is the combination of the science and
technology of healthcare and the application of that science and technology in actual
practice. Providing quality healthcare involves providing care that is accepted as best
practice at the time of delivery using available technology and resources.
The most common and most widely accepted definition for quality in healthcare has
been proposed by the US Institute of Medicine(14) as:
the degree to which services for individuals and populations increase the
likelihood of desired outcomes and are consistent with current professional
knowledge.
McGlynn(15) explains that this definition recognises a scale of performance which can
theoretically range from poor to excellent, identifies that monitoring can involve both
individual and population perspectives and that efforts to improve health outcomes must
be based on scientific evidence or on the consensus of experts in the absence of
research.
The variety of definitions of quality found in the literature reaffirms the view that quality
is a complex concept and also highlights the importance of having a shared
understanding of quality prior to commencing the process of monitoring.
2.2.2 Monitoring quality
As a result of the complexity of quality, monitoring quality can pose many challenges.
Monitoring quality involves evaluating current performance, including service-user
perspectives, against a standard or expected level of performance. This consists of
defining indicators, developing information systems and the analysis and evaluation of
results(12).
It is important that we are clear on the reasons for monitoring and that we are not
monitoring merely for the sake of it. The main reason for monitoring health and social
care quality is to identify opportunities to improve performance where it has been
highlighted that performance is not at the desired standard(8). Sub-standard
performance in the delivery of health and social care compromises the safety of service
users and contributes to undesirable outcomes.
The ability to monitor and report on quality is accepted as a basis for the improvement
in the delivery of healthcare. Monitoring and reporting on quality assists healthcare
providers improve performance through benchmarking, empowers consumers to make
11
informed decisions and facilitates system-wide quality improvement by informing

national policies(16;17). KPIs can be used for monitoring and reporting on performance
through benchmarking, to identify areas for detailed attention in the assessment
process and may even prompt risk-based assessments.
2.2.3 Improving quality
Improving quality involves closing the gap between current and expected level of
performance. This can be achieved by analysing the results of the monitoring process to
recognise and address shortcomings and enhance identified strengths(12) in
performance.
2.3 Domains of quality
In order to effectively monitor healthcare quality and safety, it is essential that we
clearly define and agree those aspects of healthcare delivery that should be measured.
The Organisation for Economic Cooperation and Development (OECD) is an intergovernmental economic research institute established in 1961 and has membership of
30 developed countries, including Ireland. The organisation launched the Health Care
Quality Indicator (HCQI) project in 2003 to further develop on previous work to identify
quality indicators for international comparison and to set priority areas for additional KPI
development.
The OECD HCQI project has identified the most common domains of healthcare quality
assessed in a number of countries including Australia, Denmark, Canada, Netherlands,
United States and United Kingdom(18). According to Arah et al(19), domains of healthcare
quality are:
those definable, preferably measurable and actionable, attributes of the
system that are related to its functioning to maintain, restore or improve health.
A review of the literature and evidence has identified five key domains that can be used
to describe healthcare quality:
12
Safe: the service protects the health and welfare of service users; it minimises the
risk associated with delivering care; it prevents adverse events, minimises their
impact when they occur and learns when things go wrong
Effective: care that delivers the best achievable outcomes through the evaluation
and use of available evidence
Person-centred: care that centres on the needs and rights of service users,
respects their values and preferences and actively involves them in the provision of
their care
Equitable: the service enables fair access to care which is delivered based on need.
It also addresses identified health inequalities of the population served
Efficient: the service manages and develops its available resources sustainably to
deliver and maintain the best possible quality of care.
2.4 Conceptual frameworks

Research indicates that the health status of individuals and populations is the result of a
combination of factors and is not solely dependent on access to and the use of
healthcare services but it is influenced by factors such as genetics, environment,
education, income and interpersonal relationships.
The determinants of health model(20) recognises that while people have control over
some factors that influence their health status, other factors are outside of their control.
Similarly, the quality of health services can impact on the health status of individuals
and populations, but there are additional factors that contribute to health that are
beyond the scope of the health service. A determinants of health model incorporates(20):
income and socioeconomic status higher income and social class are associated
with better health
education lower levels of education are associated with poorer health
environment pollution, working environments, accommodation all contribute to
health status
employment unemployment is associated with poorer health status
genetics some people are more likely to develop illness based on their family
history
personal behaviour people can influence their health status by food choices,
physical activity levels, alcohol/drug consumption and smoking status
gender men and women are prone to developing different illnesses
health services access to and use of health services can influence the prevention
and treatment of illness.
The OECD HCQI project has developed a conceptual framework (see Appendix 1) to
recognise that health is determined by a number of interdependent factors, one of
which is healthcare. A conceptual framework provides a structure to guide the process
of developing KPIs. The OECD framework consists of four interconnected levels
representing(19):
13
health to capture the health status of the population

non-healthcare determinants of health to capture non-healthcare factors that
influence health
healthcare system performance to capture the performance of the healthcare
system
health system design and context to capture contextual information regarding

individual countries.
Healthcare systems should develop a structure for performance measurement within

which individual performance measures can be identified and developed(19). A
conceptual framework can incorporate the domains of healthcare such as safety and
person-centred care that have been identified by individual healthcare systems. It is also
sufficiently broad to incorporate all the factors that influence healthcare outcomes
including healthcare system performance. It facilitates the development of performance
measures that can be used for comparison internationally while taking into account the
non-healthcare determinants of health relevant to individual countries.
2.5 Methods of monitoring and improving quality
Monitoring performance is a key element of quality improvement. The activity of
monitoring assists organisations to benchmark performance against identified targets or
expectations in order to identify where there is room for improvement. There are a
number of methods through which the performance and quality of healthcare
organisations can be monitored and improved and in practice monitoring is often a
combination of methods, including(21):
regulatory inspection
surveys of consumer experiences
third-party assessments
key performance indicators.
2.5.1 Regulatory inspection

This involves the inspection of organisations by regulatory authorities to assess
compliance with licensing regulations. It has been described as the sustained and
focused control exercised by a public agency over activities which are valued by a
community(22). The standards against which organisations are inspected are often
based on minimum legal requirements to safely care for service users.
2.5.2 Surveys of consumer experiences
As healthcare delivery increasingly focuses on empowering service users through health
education and increasing their understanding and awareness of the expected level of
performance of healthcare services, consumer surveys are increasingly being used as a
means of assessment. The benefits of this approach are that it monitors performance
from the service-user perspective and identifies what is valued by service users(21).
14
2.5.3 Third-party assessments

Third-party assessments are often voluntary and usually combine internal selfassessments with external audits and include International Organisation for
Standardisation (ISO) certification, peer review and accreditation.
Certification against ISO standards involves monitoring compliance with quality systems
rather than hospital performance alone and usually involves measuring aspects of the
organisation, such as laboratory systems.
Peer review is a form of professional self-assessment, usually done for the purpose of
gaining recognition as a training facility. It involves professionals visiting from an
external organisation to peer review other professionals from their own discipline.
Accreditation involves measuring hospital performance through self-assessment,
external review by a multi-disciplinary team and benchmarking with selected KPIs.
Accreditation is usually done for the purpose of organisational development rather than
regulation(21).
2.5.4 Key Performance Indicators
KPIs are specific and measurable elements of health and social care that can be used to
assess quality of care(23). KPIs are measures of performance, based on standards
determined through evidence-based academic literature or through the consensus of
experts when evidence is unavailable.
According to the Joint Commission on Accreditation of Healthcare Organisations
(JCAHO) in the United States, KPIs are not intended to be direct measures of quality but
instead act as alerts to warn us of opportunities for improvement in the process and
outcome of service-user care(24).
15
3 Key performance indicators for healthcare quality assessment

The recognition that there are variations in the quality of healthcare delivered,
combined with concerns about the costs of poor quality healthcare has driven the need
for the measurement and evaluation of healthcare(25). Together with quality
improvement, measurement contributes to learning, regulation and accountability and
assists healthcare staff in their quest to provide optimal care(26).
The ability to monitor healthcare quality is essential in order to effectively measure
performance which can be done with the assistance of KPIs(11). KPIs facilitate the
capture of healthcare trends as a quantitative measure of quality. They make an
inference about the quality of care provided and indicate areas that require further
investigation(27).
3.1 Types of indicators
KPIs can be characterised according to whether they are generic or specific and by both
the type and function of care for which the measurement is intended (see Figure 2).
3.1.1 Generic or specific KPIs
KPIs can be targeted to measure performance that is relevant to all service-users or
they can measure aspects of a service that are relevant to a specific service user
population:
generic KPIs measure aspects of performance relevant to the majority of service
users and do not target a specific service user population. For example, the number
of service users awaiting admission from the emergency department for more than
six hours
specific KPIs are related to a specific service user population and measure particular
aspects of care related to those service users. For example, the percentage of
children that have been referred for speech and language therapy that wait more
than three months from referral to assessment.
3.1.2 Type of care
KPIs can be classified according to the type of care for which the measurement process
was developed. For example, preventive, acute or chronic care. Preventive care refers to
the maintenance of health and prevention of illness such as in immunisation
programmes. Acute care usually refers to care given for a new onset illness or for a
sudden deterioration in chronic conditions and may involve short term medical care or
surgery. Chronic care usually refers to the long term care of chronic diseases or
conditions such as maintaining acceptable blood glucose levels and prevention of
complications in diabetes through medication and lifestyle.
16
3.1.3 Function of care

KPIs can be classified according to the function of care, which can be screening,
diagnosis, treatment and follow-up.
Figure 2 outlines the many pathways that can be considered when choosing a KPI and
demonstrates that the final indicator can be a combination of different classifications of
indicators.
Figure 2: Types of KPIs
To illustrate, Figure 3 outlines two indicators. The first KPI measures the percentage of
women between the ages of 25 to 60 that have a cervical screening test result within
the last five years. It is a process KPI, it is specific to a particular service user
population, it is preventive and is done for the purpose of screening.
The second KPI measures the number of service users that return to the emergency
department for an unscheduled visit within seven days with the same condition. It is an
outcome KPI, it is generic as it is applicable to all service users, the type of care is acute
and the function of care is intervention/treatment.
17
Figure 3: Examples of types of KPIs
3.2 Benefits
Since the measurement of performance itself contributes to improvement, it is
necessary to monitor performance in order to improve the quality and safety of
healthcare delivery.
3.2.1 Benchmarking
KPIs facilitate the improvement of performance through benchmarking, which makes it
possible for organisations to document the quality of care they provide against that
18
provided by similar organisations. KPIs also facilitate benchmarking within an

organisation to highlight improvements in quality and safety over time. These
benchmark processes help to identify where there are opportunities for improvement or
where improvements have already occurred as a result of changes in the way care is
delivered.
3.2.2 Accountability
Through the further process of performance reporting, KPIs promote accountability to
all stakeholders. They promote accountability to service users by facilitating
comparisons with other organisations and to stated objectives or targets of an
organisation. They promote accountability to central government for the efficient use of
resources with other comparable organisations.
3.2.3 Service user choice
KPIs support service-users in their choice of providers by making performance results
available to them to facilitate comparisons. It is important for healthcare providers to
recognise that, although KPIs have the potential to identify variations in service quality,
they cannot provide information on why this variation exists(28). KPIs are intended to be
flags or signals to alert us about the level of performance and that further investigation
may be warranted.
3.2.4 Public reporting
A recent study in the United Kingdom indicates that the death rate for patients
undergoing coronary artery bypass graft (CABG) surgery declined following the
introduction of performance reporting with no evidence to suggest that surgeons were
avoiding high-risk patients to improve performance results(29). It had been suggested
that performance reporting can lead to healthcare providers avoidance of providing
treatment to service users with complicated high-risk conditions(30). This latest United
Kingdom study supports the view that public reporting of performance does not
adversely effect the chances of high-risk patients undergoing elective surgical
procedures.
3.2.5 Identify areas for further investigation
In the United Kingdom in 2007, the Healthcare Commission became aware of high
mortality rates in the Mid-Staffordshire Trust in comparison with other Trusts(31). On
further investigation, the Commission were able to determine the reasons behind the
high level of mortality, which included under-staffing, poor equipment in the emergency
department (ED), lack of training for staff, and poor patient care.
19
The investigation occurred because a key performance indicator measuring mortality

had flagged that this was an issue and further supports the positive use of KPIs to flag
or alert that performance is not at the desired standard.
3.3 Considerations
KPIs represent a standard method of performance monitoring, but there are a number
of risks associated with their use which warrant consideration when KPIs are being
decided upon, and in advance of implementation.
In order to maximise the positive impact of performance measurement, the set of KPIs
used must provide a comprehensive view of the service without placing an excessive
burden on organisations to collect data. A suite of KPIs that reflects the service-user
experience or care pathway from primary and community care, including social services,
through to secondary care and subsequent follow-up is one method to counteract the
temptation for healthcare providers to focus on one particular aspect of care to the
detriment of others. The use of a single KPI, or even the use of a limited set of KPIs,
may not provide sufficient information for measuring performance and may instead
encourage organisations to focus on the activity being measured to the detriment of the
service as a whole - leading to a what gets measured gets done situation(28).
3.3.1 Data quality
The KPI needs to be interpreted on the basis of the quality of the data and the
definitions that constitute the KPI. If the definitions are not explicitly stated or there are
no checks to verify the quality of the data, then organisations may not be accurately
recording activity, making benchmarking impossible. This can be overcome by ensuring
that there are explicit definitions for each KPI and built in data quality checks to verify
that the required data is accurate.
3.3.2 Service-user profile
Casemix is an internationally recognised system of measuring clinical activity
incorporating the age, gender and health status of the population served by an
organisation and allows for a fair comparison between organisations.
Not all organisations have an equivalent service-user profile and therefore, Casemix may
need to be incorporated into the performance indicator to account for variations that
may be demonstrated by presenting raw data. Variations in the service-user profile such
as age, gender, co-morbidity and severity of disease can account for variations in the
results of the measurement process.
20
Also, healthcare outcomes are usually the result of a combination of factors and so it is
important that the KPI used appropriately measures outcomes that are attributable to
the performance of the healthcare system in which they are employed(32).
3.3.3 Data availability
The decision to select or develop a KPI based solely on available data is another factor
which must be considered. Basing KPIs on what the organisation considers an intrinsic
component of a quality service will lead to measurements that enhance quality within
the organisation. In contrast, basing KPIs on available data, while more expedient, may
lead to measurements that do not contribute or have a negative impact on quality
improvement. It is however important to identify what information is available with the
aim of identifying significant gaps.
3.3.4 Local application of KPIs
National targets may allow services to be benchmarked against international
comparators, but they provide little information as to why there are variations in
results(33). As a result, national KPIs need to be supported by local operational KPIs to
provide information at a local level to inform practice.
Performance data, captured at the point of care delivery, can be used locally to involve
and inform clinicians in performance improvement. In order to be effective and not
overburden an organisations available resources, healthcare performance data needs to
be relevant to the healthcare provider and must not divert resources from the primary
purpose of providing frontline healthcare. In the United Kingdom the Healthcare
Commission developed the Better Metrics project(34) in response to the recognition
that clinicians were not always aware of targets being used in performance
measurement. This project aims to develop metrics that are relevant to clinicians dayto-day practice and to assist local services in developing their own metrics.
21
22
Part 2:
Development of Key Performance Indicators and
Minimum Data Sets
23
4 Development of KPIs
A number of factors, outlined below, should to be considered when developing and
evaluating KPIs (Figure 4)(33-35).
These factors are not presented as a series of steps and even though some may follow
a logical order, others can happen at any stage in the process or throughout the whole
process. These factors have been identified through a synthesis and analysis of
literature following an extensive review and should be considered when developing
KPIs.
4.1 Define the audience and use for measurement
It is important to define the goals of the measurement, reasons for measurement and
the intended audience in order to identify and develop a suitable KPI.
It is essential to note that whether the goal of the measurement is for benchmarking,
either internally for quality improvement purposes or externally against standards or
other organisations, will influence the KPI selection process. For example, if the KPI is
being developed for the purpose of benchmarking performance internationally, then a
KPI must be selected that is widely used internationally and has a clear definition.
There are many quality domains such as safety, effectiveness, efficiency, person
centredness and equity. Before embarking on the performance measurement process,
it is necessary to identify the domains for which the measurement is intended, which
may in turn be dependent on the audience. In order to fully evaluate quality it will be
necessary to identify a balanced suite of KPIs.
The intended audience can influence the unit of analysis or the way in which the result
is presented. The audience refers to the person or group for whom the KPI will aid
decision-making and can be the service-user, the clinician, the public, the facility or the
healthcare system. For example a patient waiting for surgery will be more interested in
the average waiting time for that surgery, rather than the number of people on the
waiting list.
4.2 Consult with stakeholders and advisory group
There should be consultation with all stakeholders throughout the data development
process. Consultation facilitates the identification of the needs of stakeholders while
simultaneously contributes to the acceptance of the selected KPIs. Consultation also
facilitates agreement about data elements and assists in familiarisation with the data
and standards(35).
24
Figure 4: KPI development process
25
Consultation with decision-makers can assist in identifying their information needs and
subsequent use for that information. Consultation with service providers can also assist
in identifying their information needs, and elicit what data they can provide. Discussions
with data capture and analysis staff can assist in determining skills base and training
requirements. Service user engagement can assist in identifying their information needs
and if the proposed data collection process raises any privacy and confidentiality
concerns(36).
Where appropriate, consultation should include ongoing engagement and eventual
endorsement by national or regional committees that have responsibility for health
information and standards to ensure compliance.
Methods of consultation can vary from once-off meetings to regularly scheduled
meetings with the advisory group and web forums. In keeping with best practice,
consultation should be tailored to appropriately meet the needs of the situation - the
chosen method should be based on the most efficient method of communicating with
the intended audience to disseminate the desired information and obtain the required
feedback. Consultation facilitates guidance from all stakeholders and in particular from
the expert panel.
The advisory group membership should include the relevant health professionals and
stakeholders for the area being measured. An appropriately constituted advisory group
will increase the likelihood that the chosen KPIs are fit-for-purpose and will be adopted.
Group members should be independent, should not have a conflict of interest and have
the primary objective of developing KPIs that provide a fair and accurate reflection of
the area being measured. Processes are required to ensure advisory group members
have the ability to be objective, have good teamwork and communication skills and be
willing to commit sufficient time for background reading and to attend meetings(37).
The service user is the most important stakeholder in healthcare and their involvement
is essential to help incorporate the consideration of those issues that are important to
service users into the decision-making process for the delivery of healthcare. Sufficient
support and processes should be put in place to facilitate the active participation of
service users in the advisory group. Service users have a broad perception of healthcare
quality that can include the availability of information, interpersonal relationships and
the environment whereas healthcare professionals are more likely to focus on treatment
outcomes(38). In addition, the inclusion of service users will encourage confidence in,
and support for, healthcare delivery decisions when they are made(39). Service-user
representation does not need to be in the form of a formally qualified member of the
public but should be an individual who has experience and knowledge of issues that are
important to service users(40).
26
Clinician membership should be multidisciplinary with members recognised as experts in

their respective professions. This will enhance confidence in the validity of KPIs and will
increase the likelihood of acceptance by professionals in the area being evaluated.
The advisory group should also include epidemiologists (or healthcare quality experts
with experience in epidemiology) to ensure that the data collection and analysis
methodology is reliable and valid. If it is not feasible for these professionals to be
included then the team should, at a minimum, have access to this expertise in an
advisory capacity. Consideration should also be given for the inclusion of data collectors
and/or database managers as their experience in data collection will allow them
contribute to the selection process.
Finally, the team should have access to administrators responsible for resource
distribution for the topic area within the health and social care system for which KPIs
are being developed. For national projects, the team should include membership from
different geographical regions however, the team should be kept relatively small to keep
it focussed. For example, in the Danish National Indicator Project, groups consisted of
eight to fifteen members representing healthcare professionals, relevant to the care of
each condition being measured, such as physicians, nurses, physiotherapists, dieticians,
etc. The team included representation from clinical and scientific aspects of the
condition and also included a project manager, project coordinator, an epidemiologist
and a person with responsibility for literature searches(41).
Finally, a protocol should be developed to provide an opportunity for written comment
from interested parties prior to the conclusion of the data development process.
4.3 Choose the area to measure
Choosing the area to be measured should be based on the importance of the problem,
service-user safety, potential for improvement and controllability by health or social care
system/professionals.
A healthcare problem is important if it is associated with significant morbidity and
mortality, has high service-user volumes and is costly to treat(42) for example, diabetes.
Morbidity and mortality can be determined by epidemiological data, including mortality
rates and disease prevalence. The importance of a health or social care problem can
also be determined by resource utilisation associated with a particular condition.
Service-user safety should always be paramount in the delivery of healthcare and is
recognised as a core domain of healthcare quality(43). Even though care is delivered by
individuals, KPIs that identify patterns and trends can demonstrate the need for
improvement in systems(44). Service-user safety KPIs can be generic, measuring
standardised mortality rates and adverse events or they can be more specific,
measuring healthcare associated infection, preventable surgical complications and
27
medication safety(43). Other service-user safety KPIs monitor adverse events such as
falls and bedsores.
As it is not possible to exhaustively monitor every aspect of healthcare delivery, priority
should be given to conditions for which there is evidence to support potential for
improvement. Areas that have demonstrated variability in the quality of care or where
there is a clear gap between actual and potential levels of healthcare should be
considered(45).
The process or outcome measure being assessed should be susceptible to influence by
the healthcare system in relation to quality improvement(46). In other words, the
healthcare system should have the ability to address any problems identified through
measurement and likewise the measure should reflect policy/practice changes that
contribute to quality improvement.
Together with the reasons for collecting data, such as improvements in the safety and
quality of services, issues such as the efficient use of resources as a consequence of
improvements resulting from the measurement process should be considered and are
drivers towards the introduction of such a system.
4.4 Achieve a balance in measurement
The diversity of stakeholders in health and social care requires that there is a need for
measures across multiple domains to satisfy their different information needs(47). A
number of approaches have been developed to assist in identifying a balanced set of
KPIs including:
28
the balanced scorecard which was originally developed by Kaplan and Norton(48)
and suggests four perspectives of a performance indicator set to provide a
comprehensive view of the performance of an organisation:
- service user perspective measures how an organisation meets the assessed
needs and expectations of the service user
- internal management perspective measures the key business processes that
have been identified as necessary for a high quality and effective service
- continuous improvement perspective measures the ability of the
organisations systems and people to learn and improve
- financial perspective measures the efficient use of resources to achieve the
organisations objectives.
The Three Es framework(28) uses the three domains of economy, efficiency and
effectiveness:
- economy which measures the acquisition of human and material resources of
the appropriate quality and quantity at the lowest cost
- efficiency which measures the capacity to provide effective healthcare using
minimum resources
- effectiveness which measures the degree to which the organisation attains
established goals.
performance frameworks identify domains of healthcare performance that can be

used as a basis for the development of performance indicator sets, such as the HCQI
project(18). In the United Kingdom, the performance assessment framework
measures performance in six main areas:
-
health improvement
fair access
effective delivery of appropriate care
efficiency
service-user/carer experience
health outcomes.
the process of achieving a balanced set of KPIs can be assisted by incorporating the
structure, process and outcome classification into the methodology for assessing the
healthcare system. These classifications are interdependent and structure can have
an impact on processes which in turn can have an impact on outcomes.
4.5 Determine selection criteria

KPIs should be chosen based on the judgement and consensus of experts and potential
users(49). Table 1 outlines a list of characteristics and related questions which can be
used to assist in the identification of KPIs. Adapted from criteria developed by the World
Health Organization (WHO)(36) and incorporating suggestions from stakeholders in the
Irish health service, the questions outlined in Table 1 can also be used for testing KPIs
once they have been developed.
29
Table 1: Selection Criteria
Validity
Does the KPI measure what it is supposed to measure?
A valid KPI measures what it is supposed to measure and captures an important aspect of
quality that can be influenced by the healthcare facility or system. Ideally KPIs selected
should have links to processes and outcomes through scientific evidence. Measures that
have been selected using scientific evidence possess high content validity and measures
selected through consensus and guidelines will have high face validity.
Content validity refers to whether the KPI captures important aspects of the quality of care
provided. Face validity can be determined by the KPI making sense logically and clinically or
from previous usage.
Reliability
Does the KPI provide a consistent measure?
The KPI should provide a consistent measure in the same population and settings
irrespective of who performs the measurement.
Reliability is similar to reproducibility to the extent that if the measure is repeated you
should get the same result. Any variations in the result of the KPI should reflect actual
changes in the process or outcome. Reliability can be influenced by training, the KPI
definition and the precision of the data collection methods(6).
Inter-rater reliability compares differences between evaluators performing the same
measurement. Internal consistency examines the relationship between sub-indicators of the
same overall measurement, and, if reliable, there should be correlation of the results. Testretest reliability compares the difference between results when the same evaluator
performs the measurement at different times.
Explicit evidence
base
Is the KPI supported by scientific evidence or the consensus of

experts?
KPIs should be based on scientific evidence, the consensus of expert opinions among
health professionals or on clinical guidelines.
The preferred method of choosing KPIs is through evaluating scientific evidence in support
of each KPI and rating the strength of that evidence. One example of a rating system is to
give the highest rating to evidence (A evidence) from meta-analysis of randomised
controlled trials and give a lesser rating (B evidence) to evidence for controlled studies
without randomisation and a further lower rating (C evidence) to data from
epidemiological studies(46).
In healthcare, there may only be limited scientific evidence to support a KPI and it becomes
30
necessary to avail of expert opinion(27). There are a number of methods by which a KPI can
be developed through facilitating group consensus from a panel of experts, such as the
Delphi technique, the RAND appropriateness method and from clinical guidelines. Appendix
2 gives a brief description of each method and Appendix 3 provides an example of a Delphi
assessment instrument. The expert panel can exist independently of the advisory group and
are used as a point of reference for the KPI development process.
Acceptability
Are the KPIs acceptable?
The data collected should be acceptable to those being assessed and to those carrying out
the assessment.
Feasibility
Is it possible to collect the required data and is it worth the

resources?
There should be a feasibility analysis carried out to determine what data are currently
collected and the resources required to collect any additional required data.
The feasibility analysis should determine what data sources are currently available and if
they are relevant to the needs of the current project. This will include determining if there
are existing KPIs or benchmarking processes based on these data sources.
The reporting burden of collecting the data contained in the KPI should not outweigh the
value of the information obtained. Preferably, data should be integrated into servicedelivery, and, where additional data are required that are not currently part of service
delivery, there should be cost benefit analysis to determine if it is cost-effective to collect.
The feasibility analysis should also include what means are used to collect data and the
limitations of the systems used for collection. It should also outline the reporting
arrangements, including reporting arrangements for existing data collection and frequency
of data collection and analyses.
Sensitivity
Are small changes reflected in the results?
Changes in the component of care being measured should be captured by the

measurement process and reflected in the results. The performance indicator should be
capable of detecting changes in the quality of care and these changes must be reflected in
the resulting values.
31
Specificity
Does the KPI actually capture changes that occur in the service for
which the measure is intended?
Only changes in the area being measured are reflected in the measurement results
Relevance
What useful decisions can be made from the KPI?
The results of the measurement should be of use in planning and the subsequent delivery
of healthcare and contribute to performance improvement
Balance
Do we have a set of KPIs that measure different aspects of the

service?
The final suite of indicators should measure different aspects of the service in order to
provide a comprehensive picture of performance, including user perspective(28).
Tested
Have national and international KPIs been considered?
There should be due consideration given to indicators that have been tried and tested in
the national and international arena rather than developing new indicators for the same
purpose.
Safe
Will an undue focus on the KPI lead to potential adverse effects on

other aspects of quality and safety?
The indicator should not lead to an undue focus on the aspect of care being measured that
may in turn lead to a compromise in the quality and safety of other aspects of the service.
Avoid duplication
Has consideration been given to other projects or initiatives?
Prior to developing the indicator due consideration should be given to other projects or
initiatives to ensure that there will not be a duplication of data collection.
Timeliness
Is the information available within an acceptable period of time to

inform decision-makers?
The data should be available within a time period that enables decision-makers utilise the
data to inform their decision-making process. If the data is required for operational
purposes, then it will be required within a shorter timeframe than data used for long term
strategic purposes.
32
4.6 Define the indicator

A clear definition of the indicator ensures that it is appropriately interpreted by those
with responsibility for collecting the data. Appendix 4 provides an example of the type of
detail that should be included when defining a KPI. The definition should not be too
complex to ensure that only the desired data is collected(28).
Including the rationale for the measurement will provide context and highlight the
importance of the subject being measured.
4.6.1 Identify the target population
The calculation and presentation of results of the measurement requires that the target
population are clearly identified. The target population is called the denominator and
includes all service users or events that qualify for inclusion in the measurement
process. The subset of the target population that meet the criteria as defined in the
indicator is called the numerator.
For example, if we were measuring the percentage of service users that receive
thrombolytic therapy within 60 minutes of presenting to the emergency department
following a myocardial infarction, the denominator includes all service users that receive
thrombolytic therapy following presentation to the emergency department with acute
myocardial infarction and the numerator includes all of the service users within that
group that received thrombolytic therapy within 60 minutes.
More specific information regarding the target population can be given under the
headings of inclusion criteria and exclusion criteria. Inclusion criteria outline specific
parameters of the population for inclusion in the numerator and/or denominator that
may not have been included in the KPI definitions.
Exclusion criteria describe the specific criteria for excluding cases from both the
numerator and denominator. For example, a metric measuring the rate of caesarean
sections to determine if it is an overused option would exclude abnormal presentations,
multiple gestations, foetal deaths, etc. from the denominator as these are recognised
reasons for caesarean delivery and will not contribute to determining if the procedure is
overused.
We can also use tracer conditions to identify the target population, particularly when
searching electronic records and in the above example the tracer condition is caesarean
section. Using a tracer condition broadly identifies the target service-user population,
but a more detailed definition of the target population will be specified by the KPI
definition. The definition of the tracer condition should also include synonyms and
International Classification of Diseases (ICD) and SNOMED (Systematised Nomenclature
of Medicine-Clinical Terms) codes where applicable (see Appendix 5. ICD is an
international diagnostic classification system used to classify diseases and other
33
healthcare problems. SNOMED is a standard clinical language used by information

systems to record healthcare encounters.
4.6.2 Define the target to be achieved
There should be a target set for the KPI to inform progress towards an acceptable level
of performance and also to challenge the organisation or service to improve. According
the Sutherland and Leatherman(50) there are four distinct purposes for setting targets:
to motivate towards a common goal

as a management tool to:
-
operationalise policy
achieve agreement and promote discussion regarding priorities and expectations
set benchmarks and monitor progress
as a means for performance contracting
to communicate to stakeholders regarding priorities and expectations

to hold decision-makers accountable.
Targets can be(28):
all-the-time targets, which aim to provide a level of service all the time
percentage achievement targets, which aim to achieve a specified level of
performance against a standard
qualitative targets, which are descriptive of what standard of service to expect
time-bound targets, which are one-off for a specific service
national, regional or service specific targets, which are determined for a specific
demographic or service area.
Targets should be realistic but also challenge service delivery towards improvement.
They should be SMART, that is: specific, measurable, achievable, relevant and timebound. For example, service users presenting with myocardial infarction should receive
thrombolytic therapy within 60 minutes of calling for professional help, where that is the
treatment of choice. However, not all service users with myocardial infarction should
receive thrombolysis, some service users undergo alternative treatment such as primary
angioplasty. Therefore the target should be based on an agreed acceptable level of
performance that can be achieved incrementally over a specified timeframe. It will be
necessary to have baseline data in order to identify a target that is both achievable and
challenging.
34
4.6.3 Threshold for action

Determining a threshold for action assists in deciding when it is appropriate or
necessary to institute changes in response to the measurement. The threshold should
be negotiated with the service provider and will depend on the resources and level of
service available.
4.6.4 Action
Unless actions are taken based on results, the measurement process will become an end
in itself and will not contribute to quality improvement. There should be an agreement
reached with stakeholders for actions in response to performance indicator results.
There may be a series of incremental actions depending on the variation of the result
from the target.
35
5 Develop the Minimum Data Set (MDS)

A data set is a set of data that is collected for a specific purpose(35) and a minimum
data set (MDS) is the core data identified as the minimum required for that purpose.
Once KPIs have been developed, it is necessary to determine what data needs to be
collected for each KPI being used to measure performance. This should be achieved by
creating a minimum data set and be based on what data is feasible to collect.
The minimum data set should be developed based solely on the essential data required
to operationalise the KPI. As data collection can involve the use of additional resources
it is essential that only the minimum amount of data, required to enable effective
decision making, is collected(36). The MDS should be incorporated into a data dictionary
to ensure the data is clearly defined and values are agreed. A data dictionary contains a
list of data element definitions and attributes which supports the consistent collection of
data for different purposes and by different people/organisations.
5.1 Define the level of health information
Ideally, for optimal data quality and in order to minimise any burden on resources, the
required data should be collected as part of routine service delivery. Whether or not the
available data meets the requirements of the measurement process should have been
determined during feasibility testing. If requirements are not met there will be a need to
collect additional data.
Data is routinely collected during the delivery of healthcare in order to manage care.
This data is then processed at different levels within the healthcare system according to
the needs of the system and the purpose of the information as follows:
episode-level: which is necessary to facilitate the management of care for each

individual service contact. Episode level data records details of a service users
journey through the health service and includes data such as socio-demographic
details, referral details, and clinical details. Episode level data is based on the
concept of an episode of care which commences at the first contact with the service
and is a means of describing and recording relevant information in relation to the
care provided to an individual service user during a defined period of time. A unique
identifier for individuals is necessary in order to report episode-level information
case-level: which is necessary to facilitate the management of care for each

individual service user. Case-level data is an aggregate of all the episodes an
individual service user has during a reporting period and is derived from episodelevel data
36
facility-level: which is necessary to facilitate the management of the service

facility. Facility-level data includes data relating to the facility such as number of
beds, staffing, expenditure and also includes episode-level and case-level data
system-level: which is necessary for policy and planning purposes on a systemwide or national basis. System-level data is an aggregate of all data elements in a
particular region and is derived from episode, case and facility-level information.
Frequently, the KPI will require data to be processed from different levels, using a
combination of data during analysis, to achieve a measurement. For example, episodelevel information will need to be combined with facility-level information to determine
the ratio of emergency physicians to the number of attendees at an emergency
department. In this example, episode-level information will be collected for each
service-user, while facility-level information needs only to be collected on an annual
basis.
5.2 Define the frequency of collection
Some data may need to be collected on a daily basis while other data can be collected
annually. The urgency of decisions to be made based on the KPI or the level of
monitoring required, will determine the frequency of data collection.
5.3 Document the data collection process
It is necessary to write detailed data collection specifications to ensure that data are
collected and measured consistently and to reduce the risk of bias. There should be a
data development process which results in data standards that contribute to a
consistent approach to data collection and use. Data standards are agreements on the
representation, format, and definition of common data. These data standards will then
assist in the process of ensuring data collection is of high quality and enable consistent
and comparable reporting of data and information(51).
Data can be collected manually, electronically or a combination of both. Methods of data
collection need to be explored with the advisory group to determine the feasibility of the
KPI and answer the following questions:
can existing data sources be used? During the feasibility analysis existing data
sources will have been identified and where possible these should be utilised.
However, if an existing data source does not meet the needs of the project, then it
should not be used
can existing data sources be enhanced? If the existing data source provides data
closely aligned with the required data but not completely fulfilling the requirements,
it may be possible to enhance the existing data source. Before enhancing an existing
37
data source it is necessary to consult with others using the data source to ensure the
modification does not impact on other uses of the data
is a new method of data collection needed? If a new data source is required it should
be determined that the reporting burden does not exceed the benefits gained from
collecting the data.
5.4 Identify data sources

As part of documenting the data collection process data sources should be identified.
The most efficient way to collect data is to incorporate the collection process into
routine service-user care, which involves standardising documentation to ensure the
required information is already being recorded for operational purposes.
Data sources/methods include:
administrative databases, which are readily available and therefore will involve
minimal expenditure for data collection, however the information may not be specific
enough and may not be reliable
medical record data, which are also readily available and contain more detail than
administrative data, including diagnosis, treatment and outcome
prospective data collection, which involves collecting data specifically for quality
measurement purposes - it is more specific and can define exactly what data are
required. It is, however, not readily available and expensive to collect
survey data, which involves collecting data regarding knowledge, attitudes and
behaviours and is not otherwise available. It is not readily available and is expensive
to collect.
5.5 Identify data for development

Once a decision has been made on a KPI that fulfills the performance measurement
aims and the MDS has been identified, each data element within the MDS should then
be described in detail. High quality data collection processes in which the data set is well
defined and standardised (see Table 2) ensures that the same data are not collected,
counted or reported differently for different purposes(35). This results in a reduction in
the burden and use of resources for data collection and facilitates the principle of
create once, use often.
Data should be clearly defined and standardised for comparability purposes and should
not be reliant on or limited by the capability of one particular system, organisation or
data collection tool(35).
38
Data set name

Data element name
Synonyms
Metadata item type
Technical name
Registration status
Definition
Data element concept
Value domain
Field length - Maximum
Field length - Minimum
Instructions
Reference source
Waiting times for Emergency Department

Time service user presents
Presentation time
Data element
Health Service event Presentation time
Is this a National Data Standard item
The time at which the service-user presents for the delivery
of a service
Health service event presentation time
hhmm
4
4
The time of service-user presentation at the Emergency
Department is the earliest occasion of being registered
clerically or triaged
ISO 8601:2000: data elements and interchange formats
Information interchange Representation of dates and times
Table 2: Example of data element attributes
5.6 Assess compliance with Information Governance

Healthcare information is sensitive and therefore provision must be made to ensure
security and confidentiality of data held on service users. Information governance is the
process whereby organisations and individuals ensure that personal information is
handled legally, securely, efficiently and effectively, in order to deliver the best possible
care. The data set should comply with data protection legislation and guidelines and
should have respect for service users privacy and confidentiality.
5.7 Plan data quality checks

There should be clear definitions for each data element in the MDS to ensure data
collectors have a good understanding of what, how and when data needs to be
collected. There should also be routine data quality checks to minimise the occurrence
of reporting and input errors. Quality checks can be introduced at all stages of the
measurement process, such as data collection, processing, analysis, use and
dissemination.
39
6 Data reporting to stakeholders

There should be a plan to outline how and when the results of the measurement
process are released to stakeholders and the public. The results should be presented to
allow the intended audience to easily interpret and use the information generated by the
measure(42). The frequency of publication of results should ensure that information is
made available in a timely manner and continues to be relevant to the information
needs of the stakeholders and service users.
Priority should be given to supporting the interpretation of results by multiple audiences
rather than an individual audience. For example clinicians will have a better
understanding of information presented with clinical detail, whereas service users may
prefer information at a more summary level. The purpose of data reporting is to inform
all relevant audiences so that improvements can be made based on the available
information.
Dashboards are one example of a method for presenting information to inform decisionmaking. Performance results are presented graphically through a series of charts,
gauges or tables and facilitate comparison of actual performance against desired results.
Dashboards can sometimes be colour coded to indicate performance against a goal or
target. Colours can be used to represent performance, for example:
green, to indicate good performance

amber, to indicate average performance with room for improvement
red, to indicate unsatisfactory performance requiring attention.
Information can also be presented using composite measures which present the results
of performance measurement using a single score representing an aggregation of a
number of underlying KPIs(52). Composite measures can provide a rounded picture of
the performance of an organisation or system rather than trying to identify a trend from
a range of individual KPIs(53). Each of the individual KPIs within the composite measure
must satisfy the requirements of a good KPI, otherwise the composite measure will not
represent an accurate picture of performance.
In certain instances weights are assigned to individual KPIs within a composite measure
to reflect their priority or importance, so that individual KPIs within the subset
contribute to a higher proportion of the result than the remaining KPIs.
For example, a composite measure that comprises seven indicators may assign a weight
of 0.25 to 2 of the KPIs and 0.1 to each of the other 5. This weighting is then reflected
in the overall result. There are, however, risks associated with aggregating the KPIs into
a composite measure. It is possible to lose important information, such as serious
failings in a particular part of the organisation, or to fail to identify specific areas where
40
significant improvement is required. The weighting system can also influence the result,
particularly when used for benchmarking between service providers. Service providers
that excel in the higher weighted KPIs will perform better than those that excel in the
lower weighted KPIs, so the weighting methodology needs to be robust.
6.1 Determine frequency of processing and analysis
The frequency of processing and analysing the data collected should be determined to
ensure the efficient use of resources and also meet the needs of the information user. It
may not always be necessary to process and analyse data at the same frequency as
data collection. It may be practical to collect data on a daily basis, but for analysis and
comparison purposes it may be appropriate that this data is processed and analysed on
a weekly, monthly or even annual basis.
6.2 Define method of analysis
A detailed protocol should be developed for presenting the result of the KPI. This should
address issues such as missing data, risk adjustment, and also what is an acceptable
level of performance or target to be achieved. In some cases the result can be
presented as the proportion of the total population that have experienced the particular
aspect of the service being measured. Other results can be based on the proportion that
has achieved a particular standard or threshold.
6.2.1 Define type of measure
The chosen method for analysing and presenting the results should be determined and
this is based on the topic/service being measured. The following is an example of
various ways of presenting the results of the measurement process(46):
rate-based KPIs: use information about events that are expected to happen
frequently. The measurements can be represented as proportions or ratios, detailed
as follows:
- proportion KPIs: to allow comparisons between organisations or trends over a
specified time they require both a numerator and a denominator. The KPI must
identify the population at risk of the event and the period of time within which
the event might take place. They are usually expressed as a percentage and the
numerator is contained in the denominator. An example of proportion KPIs is the
proportion of cardiovascular related deaths that are male
- ratio KPIs: the numerator is not contained in the denominator e.g. ratio of male
to female cardiovascular related deaths.
41
count KPIs: measure the number of events without a denominator. An example of a

count KPI is the number of newly detected cases of tuberculosis in a given year
sentinel KPIs: identify individual events that are inherently undesirable and usually
warrant detailed analysis to determine why the event occurred. Sentinel events
depict extremely poor performance. An example of a sentinel KPI is the number of
deaths resulting from medical error.
6.3 Determine level of aggregation

Aggregation refers to the combination of results to provide a broader picture of
performance over a geographic region or time period. Aggregation over space refers to
the geographical region by which data will be reported, which could be nationally, within
a specific health-delivery region or within a hospital or organisation. Aggregation over
time refers to the time period for which the information will be reported, which could be
daily, weekly, monthly or annually.
These factors may be determined by the level within the system to which data are
reported. For example, a hospital might request that data be aggregated to reflect
performance within a specific service within the hospital on a monthly basis, whereas a
health system might want the same data aggregated over a geographical region on an
annual basis. The same data informs decision-making a both levels, but both needs are
met by different levels of aggregation. What is important is that the same data is
utilised both locally and nationally to inform practice and quality improvement initiatives.
6.4 Develop risk-adjustment strategy
There should always be consideration given to determine if a risk-adjustment strategy is
necessary. A risk adjustment strategy reduces the possibility of external factors
influencing the measure and ensures that the measure is a true reflection of the process
being measured.
Certain characteristics related to the service-user or disease may influence the outcome,
including age profile of the service-user population, co-morbidities, socio-economic
features and service-user compliance. These prognostic factors should be identified and
factored in to the measurement specifications through casemix adjustment models by
epidemiologists to facilitate comparability. This may involve collecting additional data to
assist in the analysis. Alternatively, restricting the measurement to a specific serviceuser population will ensure that service-user characteristics do not have an undue
influence on the comparison process.
42
7 Pilot test KPIs

Even though a considerable amount of time and effort will have been spent designing
the specifications, it is necessary to test the KPI as there may be a need for refinement.
This can generally be done through a small pilot and can assist in identifying issues such
as gaps in data collection processes.
Prior to commencing the pilot test there should be clear plan for the pilot. Issues
covered in the plan should include the criteria for selecting the pilot site(s), proposed
length of pilot test, training and education of participants and information to be
obtained from the pilot.
The information to be obtained from the pilot can be posed as a number of questions,
such as(54):
are there validity and reliability issues in relation to data collection?
is the information obtained from the KPI of use in decision-making?
can the KPIs contribute to improved service and quality of care?
have there been any issues identified through quality checks and are data recorded
consistently?
what additional measures that were not in place for the pilot, need to be instituted
for the KPIs to be rolled-out successfully?
are there any modifications necessary to the KPI specifications?
The pilot test can also be used to validate the KPIs against the selection criteria used for
developing the KPIs (Table 1).
Once the pilot test has been completed to the satisfaction of the advisory group, it will
be necessary to develop a plan for the roll-out of the KPI project to the identified sites.
7.1 Determine review frequency
There should be a plan to review the KPI at regular intervals with a view to refinement
in response to stakeholder demands or improved data availability. Health services are
continually evolving and it is important that KPIs respond to these changes. There
should be a date set for reviewing the KPI to ensure that it is still relevant and up to
date.
The review may highlight the need to modify the KPI or aspects of the KPI in response
to stakeholder demands, improved data availability and changes in practice. Changes
may involve modifying the target, threshold or definition based on new evidence or
alterations in the health system. However, for the purpose of comparability and
monitoring long-term trends, KPIs should not be amended too frequently. To support
trending over time and comparability there should be strong business rules on which the
decision to amend or discontinue existing KPIs is based.
43
8 Conclusion
Access to and use of good quality information is a key component of performance
measurement and improvement for high quality, safe and reliable healthcare.
Performance improvement involves monitoring the current level of performance and
instituting changes where performance is not at the desired level. KPIs support
organisations improve the safety and quality of care by providing information about the
current level of performance and identifying where there are opportunities for
improvement.
This document has been developed as a resource to support stakeholders in the process
of developing KPIs and associated MDSs used for monitoring the quality and safety of
health and social care. The guidance identifies important factors to be considered when
developing and identifying KPIs and has been identified through an extensive synthesis
and analysis of the literature.
KPIs that have been identified and developed based on the factors identified in this
document are more likely to lead to measurements that can be confidently relied upon
by decision makers. Data collection to support the KPI measurement process is more
efficient if it is incorporated into routine care. It is important that each KPI and the
associated MDS is clearly defined, so that the result of the measurement reflects actual
changes in the quality and safety of care.
Having completed this guidance, the Authority will continue to develop and publish
additional documents to support improvements in the quality and safety of healthcare.
44
Reference List
(1) Smith PC, Mossialos E, Papanicolas I. Performance measurement for health
system improvement: experiences, challenges and prospects. World Health
Organisation; 2008.
(2) Kohn LT, Corrigan JM, Donaldson MS. To Err is Human: Building a Safer Health
System. Washington: National Academies Press; 1999.
(3) Berwick DM, James B, Coye MC. Connections Between Quality Measurement and
Improvement. Medical Care. 2003; 43(1, Supplement): p.i30-i38.
(4) Department of Health (United Kingdom). High Quality Care for All: NHS Next
Stage Review Final Report. London: Department of Health; 2010.
(5) Raleigh VS, Foot C. Getting the Measure of Quality: Opportunities and challenges.
The Kings Fund; 2010.
(6) Rubin HR, Pronovost P, Diette GB. From a process of care to a measure: the
development and testing of a quality indicator. International Journal for Quality in
Health Care. 2001; 13(6): pp.489-96. Available online from:
http://intqhc.oxfordjournals.org/cgi/reprint/13/6/489.
(7) Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of processbased measures of health acre quality. International Journal for Quality in Health
Care. 2001; 13(6): pp.469-74.
(8) Donabedian A. An Introduction to Quality Assurance in Health Care. Oxford
University Press; 2003.
(9) Donabedian A. The Quality of Care: How Can It Be Assessed? Journal of the
American Medical Association. 1988; 260(12): pp.1743-8. Available online from:
http://post.queensu.ca/~hh11/assets/applets/The_Quality_of_Care__How_Can_it
_Be_Assessed_-_Donabedian.pdf.
(10) Mainz J. Defining and classifying clinical indicators for quality improvement.
International Journal for Quality in Health Care. 2003; 15(6): pp.523-30.
(11) Arah OA, Klazinga N, Delnoij DMJA, Ten Asbroek AHA, Custers T. Conceptual
frameworks for health systems performance: a quest for effectiveness, quality
and improvement. International Journal for Quality in Health Care. 2003; 15
pp.377-98. Available online from:
45
(12) Silimperi D, Miller Franco L, Veldhuyzen Van Zanten T, Macaulay C. A framework

for institutionalising quality assurance. International Journal for Quality in Health
Care. 2002; 14, Supplement 1 pp.67-73.
(13) vretveit J. Formulating a health quality improvement strategy for a developing
country. International Journal of Health Care Quality Assurance. 2004; 17(7):
pp.368-76.
(14) Institute of Medicine. Health, Health Care, and Quality of Care. Kathleen N.Lohr,
(Ed) In: Medicare: A Strategy for Quality Assurance. 1st Edition. 1990. p.21.
Available online from:
http://books.nap.edu/openbook.php?record_id=1547&page=21. Accessed on: 7
January 2009.
(15) McGlynn EA. Six Challenges in Measuring the Quality of Health Care. Health
Affairs. 1997; 16(3):
(16) Mant J. Process versus outcome indicators in the assessment of quality of health
care. International Journal for Quality in Health Care. 2001; 13(6): pp.475-80.
(17) Miller T, Leatherman S. The National Quality Forum: A 'Me-Too' or a
Breakthrough in Quality Measurement and Reporting? Health Affairs. 1999;
18(6): pp.233-7.
(18) Kelley E, Hurst J. Health Working Papers. Health Care Quality Indicators Project
Conceptual Framework Paper. Organisation for Economic Co-operation and
Development; Report No.: Health Working Papers No. 23. 2006.
(19) Arah OA, Westert GP, Hurst J, Klazinga NS. A conceptual framework for the
OECD Health Care Quality Indicators Project. International Journal for Quality in
Health Care. 2006; pp.5-13. Available online from:
http://intqhc.oxfordjournals.org/cgi/reprint/18/suppl_1/5.
(20) World Health Organisation. The determinants of health [Online]. Available from:
http://www.who.int/hia/evidence/doh/en/. Accessed on: 2 February 2010.
(21) Shaw C. How can hospital performance be measured and monitored? World
Health Organisation; 2003.
(22) Selznick P. Noll R, (Ed.). Focusing organisational research on regulation.
Berkeley: University of California Press; 1985. pp.363-8.
(23) Marshall M, Campbell Stephen. Introduction to quality indicators in general
practice. Marshall M, Campbell Stephen, Hacker J, Roland M, (Eds) In: Quality
indicators for general practice: A practical guide for health professionals and
managers. The Royal Society of Medicine Press Ltd; 2002.
46
(24) Turpin RS, Darcy LA, McMahill C, Meyne K, Morton D, Rodriguez J, et al. A Model
to Assess the Usefulness of Performance Indicators. International Journal for
Quality in Health Care. 1996; 8(4): pp.321-9. Available online from:
(25) Kelley E, Arispe I, Holmes J. Beyond the initial indicators: lessons from the OECD
Health Care Quality Indicators Project and the US National Healthcare Quality
Report. International Journal for Quality in Health Care. 2006; pp.45-51.
(26) Pringle M, Wilson T, Grol R. Measuring "goodness" in individuals and healthcare
systems. British Medical Journal. 2002; 325 pp.704-7.
(27) Campbell SM, Braspenning J, Hutchinson A, Marshall M. Research methods used
in developing and applying quality indicators in primary care. Quality and Safety
in Health Care. 2002; 11 pp.358-64.
(28) Audit Commission. On Target: the practice of performance indicators. Audit
Commission for Local Authorities and the National Health Service in England and
Wales; 2000. Available online from: http://www.auditcommission.gov.uk/Products/NATIONAL-REPORT/266D51B7-0C33-4b4b-98327484511275E6/archive_mptarget.pdf.
(29) Bridgewater B, Grayson AD, Brooks N, Grotte G, Fabri BM, Au J, et al. Has the
publication of cardiac surgery outcome data been associated with changes in
practice in northwest England: an analysis of 25,730 patients undergoing CABG
surgery under 30 surgeons over eight years. Heart. 2007; 93 pp.744-8.
(30) Dranove D, Kessler D, McClellan M, Satterthwaite M. Is more information better?
The effects of 'report cards' on health care providers. Journal of Political
Economy. 2003; 111(3):
(31) Commission for Healthcare Audit and Inspection. Investigation into Mid
Staffordshire NHS Foundation Trust. Healthcare Commission; 2009. Available
online from:
http://www.cqc.org.uk/_db/_documents/Investigation_into_Mid_Staffordshire_N
HS_Foundation_Trust.pdf.
(32) Helfert M, Henry P, Leist S, Zellnor G. Koliman KS, (Ed.). Healthcare performance
indicators - Preview of frameworks and an approach for healthcare processdevelopment. 2005. pp. 371-8.
(33) McLoughlin V, Leatherman S, Fletcher M, Owen J. Improving performances using

indicators. Recent experiences in the United States, United Kingdom and
Australia. International Journal for Quality in Health Care. 2009; 13(6): pp.45562.
47
(34) Healthcare Commission. The Better Metrics project. Healthcare Commission;

Report No.: Version 8. 2007. Available online from:
http://www.cqc.org.uk/_db/_documents/Better_Metrics_full_report.pdf.
(35) AIHW. A guide to data development. Australian Institute of Health and Welfare
(AIHW); Report No.: AIHW Cat. no. HWI 94. 2007.
(36) WHO. Mental Health Information Systems (Mental Health Policy and Service
Guidance Package). World Health Organisation; 2005.
(37) Scottish Intercollegiate Guideline Network. SIGN 50: A guideline developer's
handbook. 2008.
(38) Comptroller and Auditor General. Special Report: Health Service Executive Emergency Departments. Government of Ireland; 2009.
(39) Bruni RA, Martin DK. Public engagement in setting priorities in health care.
Canadian Medical Association Journal. 2008; 179(1): pp.15-8.
(40) National Health and Medical Research Council. A guide to the development,
implementation and evaluation of clinical practice guidelines. Australia: National
Health and Medical Research Council; 1999.
(41) Mainz J. The National Indicator Project [Online]. Available from:
http://www.hpm.org/survey/dk/a12/3 . Accessed on: 4 February 2008.
(42) McGlynn E, Asch S. Developing a Clinical Performance Measure. American Journal
of Preventive Medicine. 1998;(14): pp.14-21.
(43) Kristensen S, Mainz J, Bartels P. Safety Improvement for Patients in Europe.
Establishing a Set of Patient Safety Indicators: Safety Improvement for Patients

in Europe. European Society for Quality in Health Care; Report No.: WP4. 2007.
Available online from:

http://www.esqh.net/www/projects/simpatie/files/folder_1183979816/Patient_saf
ety_indicator_development.pdf. Accessed on: 13 January 2009.
(44) Rivard P, Rosen A, Carroll J. Enhancing Patient Safety through Organisational
Learning: Are Patient Safety Indicators a Step in the Right Direction? Health
Research and Educational Trust. 2006; 41(4 Part 2): pp.1633-53.
(45) Millar J, Mattke S. Selecting Indicators for Patient Safety at the Health Systems
Level in OECD Countries. Organisation for Economic Co-operation and
Development; Report No.: OECD Health Technical Papers No.18. 2004.
(46) Mainz J. Developing evidence-based clinical indicators: a state of the art methods
primer. International Journal for Quality in Health Care. 2003; 15(Supplement 1):
48
p.i5-i11. Available online from:

http://intqhc.oxfordjournals.org/cgi/reprint/15/suppl_1/i5.
(47) Ballantine J, Brignall S, Modell S. Performance measurement and management in
public health services: a comparison of UK and Swedish practice. Management
Accounting Research. 1998; 9 pp.71-4.
(48) Kaplan RS, Norton DP. The Balanced Scorecard - Measures that Drive
Performance. Harvard Business Review. 1992; 70(1): pp.71-9.
(49) Bodart C, Shresta L. Identifying information needs and indicators. In: WHO,
Design and implementation of health information systems. Geneva. World Health
Organisation; 2000. pp.49-72.
(50) Sutherland K, Leatherman S. Quest for Quality and Improved Performance.
Regulation and Quality Improvement: A review of the evidence. London: The
Health Foundation; 2006.
(51) AIHW. National Health Data Dictionary: Version 14. Canberra, Australia:
Australian Institute of Health and Welfare; 2008.
(52) Jacobs R, Goddard M, Smith PC. Public Services: Are composite measures a
robust reflection of performance in the public sector. Center for Health
Economics, York: CHE Research Paper 16; 2006.
(53) Smith PC. Developing composite indicators for assessing health system efficiency.
In: Measuring up: Improving the performance of health systems in OECD
countries. Paris. OECD; 2002.
(54) Kazandjian V, Reid T. Healthcare Performance Measurement. American Society
for Quality; 1999. Available online from:
http://books.google.com/books?id=GyPmaat8lcUC&pg=PA155&lpg=PA155&dq=p
ilot+test+performance+indicators&source=bl&ots=ADwhvHk_qa&sig=3ETbXpqWlB3VwGupzpkCC79k6s&hl=en&ei=i1bCSYThOS1jAelo9msCw&sa=X&oi=book_result&resnum=3&ct=result#PPR12,M1.
Accessed on: 19 March 2009.
49
Glossary of terms
BALANCED SCORECARD:
A framework developed by Robert Kaplan and David Norton that suggests four
perspectives of performance measurement to provide a comprehensive view of an
organisation. These are service user perspective, internal management perspective,
continuous improvement perspective and financial perspective.
BENCHMARK:
A point of reference or standard by which something can be measured
BENCHMARKING:
The process of comparing the cost, cycle time, productivity, or quality of a specific
process or method to another that is widely considered to be an industry standard or
best practice.
CASEMIX:
Casemix is an internationally recognised system of measuring clinical activity
incorporating the age, gender and health status of the population served by an
organisation with a view to objective determination of hospital reimbursement.
DATA:
Data are numbers, symbols, words, images, graphics that have yet to be organised or
analysed
DATA DICTIONARY:
A descriptive list of names (also called representations or displays), definitions, and
attributes of data elements to be collected in an information system or database.
DATA ELEMENT:
A unit of data for which the definition, identification, representation, and permissible
values are specified by means of a set of attributes.
DELPHI TECHNIQUE:
A method for obtaining group consensus involving the use of a series of mailed
questionnaires and controlled feedback to respondents which continues until consensus
is reached.
DENOMINATOR:
The specifications that describe the sampling, inclusion and exclusion criteria that
determine the eligibility of data for a measure.
50
DOMAINS OF QUALITY:
Are those definable, preferably measurable and actionable, attributes of the system that
are related to its functioning to maintain, restore or improve health
HEALTH INFORMATION:
Health Information is defined as information, recorded in any form or medium, which is
created or communicated by an organisation or individual relating to the past, present
or future, physical or mental health or social care of an individual or cohort. It also
includes information relating to the management of the health and social care system
METADATA:
Data that defines and describes other data
MINIMUM DATA SET:
The minimum set of data elements that are required to be collected for a specific
purpose
NHS TRUST:
A National Health Service Trust provides services on behalf of the National Health
Service (NHS) in England and Wales. There are different types of Trusts, each
responsible for specific services such as Primary Care Trusts, Acute Trusts, Ambulance
Trusts, Care Trusts and Mental Health Trusts
NUMERATOR:
The specifications that define the subset of data items in the denominator that meet
the indicator criteria.
KEY PERFORMANCE INDICATORS:
Performance Indicators are specific and measurable elements of practice that can be
used to assess quality of care. Indicators are quantitative measures of structures,
processes or outcomes that may be correlated with the quality of care delivered by the
healthcare system.
51
PROCESS INDICATORS:
Performance indicators that monitor the activities carried out in the
assessment/diagnosis and treatment of service users.
OUTCOME INDICATORS:
Performance indicators that monitor the desired states resulting from care processes,
which may include reduction in morbidity and mortality, and improvement in the quality
of life.
RELIABILITY:
Reliability is the consistency of your measurement, or the degree to which an
instrument measures the same way each time it is used under the same condition with
the same subjects.
STRUCTURE INDICATORS:
Performance indicators that monitor the attributes of the health system that contribute
to its ability to meet the healthcare needs of the population.
VALIDITY:
Validity of indicators refers to whether performance indicators are measuring what they
are supposed to measure.
52
Appendix 1: HCQI Framework
Adapted from Arah et al
Health
How healthy are the citizens of Ireland?
Health
Conditions
Human Function
and Quality of Life
Life Expectancy and

Well-being
Mortality
Non-healthcare determinants of health

What are the non-healthcare factors that influence health, and
occasionally, how and when care is accessed
Health
Personal or
Socio-economic Physical
Behaviours Host
Conditions and
Environment
and
Resources
Environment
Lifestyles
E
q
u
i
t
y
Healthcare System Performance

How does the health system perform? What is the level of quality
of care across the range of service user needs? What does this
performance cost?
Domains of Care
Quality
Healthcare
Needs
Staying
healthy
Getting better
Safety
Effectiveness
Personcentred
Efficient
Equitable
Living with
illness/
disability
Coping with
end of life
Efficiency
Health System design, policy and context
Other determinants of
performance
(e.g. country capacity)
Health System Delivery Features
53
Appendix 2: Examples of consensus techniques
The Delphi technique is a facilitated structured process whereby a panel of experts

complete questionnaires (see Appendix 3 for example) remotely and, through feedback
and scoring over a number of rounds where some KPIs are discarded, a consensus is
achieved on a final set of KPIs. The panel need not ever meet face-to-face and each
individuals feedback is provided anonymously to the panel, which eliminates the
possibility of undue influence by dominant personalities within the panel.
The RAND appropriateness method combines scientific evidence with expert
opinion by facilitating experts to rate, discuss and re-rate KPIs. Unlike the Delphi
technique the expert panel meet face-to-face to discuss possible KPIs and are given a
copy of the scientific literature in support of the KPIs so that they can ground their
opinion on evidence-based literature(27).
KPIs can also be developed based on clinical guidelines. An acceptable method of
developing KPIs using guidelines is the iterated consensus technique whereby KPIs are
selected based on the perceived impact of the guideline on the outcome of care(27).
54
Appendix 3: Delphi study example of brief assessment instrument
Scoring Matrix
Domain
Validity
Reliability
Definition
Is the indicator satisfactory in
terms of:
Face validity
Content validity
Is the indicator satisfactory in
terms of reliability?
Score
1 3 Low degree of relevance
4 6 Medium degree of relevance
7 9 High degree of relevance
1
4
7
Acceptability Is the indicator acceptable?
1
4
7
Feasibility
How is the:
1
4
Availability of data
Burden of data collection 7
Scoring Sheet
Title:
Scores
Validity
Reliability
3
6
9
3
6
9
3
6
9
Low degree of relevance

Medium degree of relevance
High degree of relevance
Acceptability Feasibility
Additional
Comments
55
Appendix 4: Example of a clinical KPI

1.
KPI Title
2.
Description
3.
Rationale
4.
Target
5.
KPI collection
frequency
; Daily
Weekly
Monthly
Quarterly
Bi-annually
Annually
Other give details: ______________________________
6.
KPI reporting
frequency
7.
KPI calculation
Daily
Weekly
Monthly
; Quarterly
Bi-annually
Annually
Other give details: ________________________________
Numerator divided by denominator expressed as a percentage
Numerator: Total number of patients with a diagnosis of AMI
requiring reperfusion who receive thrombolytic therapy within 60
minutes of presentation to the Emergency Department
Denominator: Total number of patients with a diagnosis of AMI
requiring reperfusion who receive thrombolytic therapy following
presentation to the Emergency Department
8.
Reporting
aggregation
9.
Data Source(s)
Time to Thrombolysis
Percentage of patients with Acute Myocardial Infarction (AMI)
requiring thrombolysis who receive thrombolytic therapy within 60
minutes of presentation to the Emergency Department
Cardiovascular disease is the leading cause of death in Ireland and
research indicates that mortality is directly proportional to the time
delay from onset of symptoms to the commencement of definitive
therapy. The Cardiovascular Health Strategy in Ireland recommends
that eligible patients receive thrombolysis within 90 minutes of
seeking professional help. In the United Kingdom the Coronary
Heart Disease National Service Framework sets out that patients
suffering from Myocardial Infarction should receive thrombolysis
within 60 minutes of calling for professional help.
65% of eligible patients presenting with AMI

or
> 50% with a 20% improvement from 2009
National
; Regional
LHO Area ; Hospital
County
Institution
Age
Gender
Socio Economic Class
Other give details:
Administrative data
Medical Record
56
10.
Tracer conditions
11.
Minimum Data Set
Acute Myocardial Infarction (AMI)

Synonyms: Myocardial Infarction (MI)
Cardiac Infarction
Heart Attack
SNOMED CT: ConceptID 57054005
ICD-10-AM codes: 121.0. 121.1, 121.2, 121.3, 121.4, 121.9
Local service-user
identifier
UHI:
The number used to identify a service user

uniquely by a care provider
Unique Health Identifier (not yet applicable)
First name:
Surname:
Date of birth:
Gender:
Date patient presents:
Time patient presents:
AMI Diagnosis:
Main first name of service user

Surname on birth cert or passport
Date on birth cert or passport
At time of diagnosis
The date of arrival
The time of arrival
This is a working diagnosis at the time of
admission
Thrombolysis, PTCA, referred elsewhere for
PTCA, reperfusion not attempted
The name of the drug administered as
thrombolytic therapy
The time thrombolysis commenced
Reperfusion type:
Thrombolytic drug:
Time thrombolysis
started:
Thrombolysis treatment
location:
Thrombolysis delay
reason:
Reason thrombolysis
not given:
12.
13.
14.
57
International
comparison
Web link to data
(where available)
Additional
Information
Where was thrombolysis commenced e.g.

pre-hospital, ED, CCU, ward, etc..
Identifies a justified reason for delay in
commencing thrombolysis e.g. uncontrolled
hypertension, delay in obtaining consent,
initial ECG not diagnostic, etc..
Identifies the reason why thrombolysis was
not given e.g. too late, patient refused,
uncontrolled hypertension, elective decision,
etc..
Care Quality Commission, UK
58
Published by the Health Information and Quality Authority

For further information please contact:
Georges Court
Georges Lane
Dublin 7
Phone: +353 (0)1 814 7400
URL: www.hiqa.ie
Health Information and Quality Authority 201

HI KPI Guidelines

Uploaded by

Copyright:

Available Formats

HI KPI Guidelines

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

HI KPI Guidelines

Uploaded by

Copyright:

Available Formats

Guidance on developing Key Performance Indicators and Minimum Data Sets to Monitor Healthcare Quality

Health Information and Quality Authority

Safer Better Care

About the Health Information and Quality Authority

Overview of the Health Information function

The role of KPIs

Methods of monitoring and improving quality

3 Key performance indicators for healthcare quality assessment

Define the audience and use for measurement

Consult with stakeholders and advisory group

Choose the area to measure

Achieve a balance in measurement

Determine selection criteria

Define the indicator

5 Develop the Minimum Data Set (MDS)

Define the level of health information

Define the frequency of collection

Document the data collection process

Identify data sources

Identify data for development

Assess compliance with Information Governance

Plan data quality checks

6 Data reporting to stakeholders

Determine frequency of processing and analysis

Define method of analysis

Determine level of aggregation

Develop risk-adjustment strategy

7 Pilot test KPIs

Determine review frequency

Appendix 2: Examples of consensus techniques

Appendix 3: Delphi study example of brief assessment instrument

Appendix 4: Example of a clinical KPI

Figure 2: Types of KPIs

Figure 3: Examples of types of KPIs

Figure 4: KPI development process

Table 2: Example of data element attributes

meaningful comparison. This can be achieved through the development of a minimum

Figure 1: Quality Assurance Triangle(12)

2.2.1 Defining quality

informed decisions and facilitates system-wide quality improvement by informing

2.4 Conceptual frameworks

health to capture the health status of the population

health system design and context to capture contextual information regarding

Healthcare systems should develop a structure for performance measurement within

2.5.1 Regulatory inspection

2.5.3 Third-party assessments

3 Key performance indicators for healthcare quality assessment

3.1.3 Function of care

Figure 2: Types of KPIs

Figure 3: Examples of types of KPIs

provided by similar organisations. KPIs also facilitate benchmarking within an

The investigation occurred because a key performance indicator measuring mortality

Figure 4: KPI development process

Clinician membership should be multidisciplinary with members recognised as experts in

performance frameworks identify domains of healthcare performance that can be

4.5 Determine selection criteria

Table 1: Selection Criteria

Does the KPI measure what it is supposed to measure?

Does the KPI provide a consistent measure?

Is the KPI supported by scientific evidence or the consensus of