LAPPEENRANTA-LAHTI UNIVERSITY OF TECHNOLOGY

LUT School of Business and Management

Business Analytics
Master’s Thesis

KEY PERFORMANCE INDICATORS AND

THEIR VISUALIZATION IN AN ELECTRIC
VEHICLE CHARGING COMPANY

Valeriya Bulaeva
May 2019
Helsinki, Finland

Examiner: Professor, D.Sc. (Econ. & BA), Mikael Collan

Instructor: Postdoctoral researcher, D.Sc. (Tech.) Jan Stoklasa
ABSTRACT
Author: Valeriya Bulaeva
Title: Key performance indicators and their visualization in an electric vehicle charging
company
Year: 2019 Place: Helsinki
Master’s thesis, Lappeenranta-Lahti University of Technology, LUT School of Business
and Management, Business Analytics
76 pages, 13 figures, 7 tables and 3 appendices
Examiner: Professor, D.Sc. (Econ. & BA), Mikael Collan
Instructor: Postdoctoral researcher, D.Sc. (Tech.) Jan Stoklasa
Keywords: Key performance indicators, reliability assessment, electric vehicle charging,
visualization, dashboard
In different industries companies implement KPIs in order to get an overview of a current
situation, to maintain and improve the quality of service. Furthermore, KPIs are necessary
for tracking and analyzing the most important processes in companies. The purpose of this
study is to obtain and visualize a set of KPIs that will be used to monitor EV charging
points reliability in the case company that provides EV charging services globally. Many
KPIs development and ranking approaches that are applied in various industries can be
found in literature. This thesis offers the 7- Step KPIs identification approach that was
specifically developed for reliability assessment in the case company.

In the first part of the study the review of EV charging infrastructure, technologies and
locations is presented in order to understand the industry in which the case company
operates. Then the theories about performance measurement, KPIs and methods of their
identification are discussed. In the second part of this thesis the process of KPIs
identification approach based on data analysis in the case company is introduced. The
failure rate calculation model, proposal for company’s reporting system improvement and
created dashboard of visualized KPIs are presented at the end.

According to the results of the research, nine KPIs were identified for tracking reliability
of EV charging points. KPIs were validated and then visualized in a form of a dashboard.
However, it was not possible to visualize all the defined KPIs and it was concluded that
software that is currently used the company is not the best for KPIs visualization.
Recommendations related to failure rate calculation, visualization tool, and reporting
system are presented in the end. Some of the proposed improvements are already taken in
to consideration and will be implemented by the case company in the following months.
Table of Contents

1. Introduction ........................................................................................................... 9

1.1 The case company .................................................................................................... 9

1.2 Background of the study......................................................................................... 10

1.3 Purpose of the study and research questions .......................................................... 11

1.4 Research methodology and research process.......................................................... 12

1.5 Structure of the Thesis ............................................................................................ 14

2. Charging of Electric Vehicles ................................................................................. 16

2.1 EV charging infrastructure today ............................................................................ 16

2.2 Development of EV charging infrastructure in Finland ............................................ 17

2.3 EV charging technologies and locations .................................................................. 18

3. Performance measurements and key performance indicators ............................... 21

3.1 Characteristics of KPIs ............................................................................................ 23

3.2 Categorization of KPIs ............................................................................................ 25

3.3 Methods and techniques for KPIs creation .............................................................. 28

4. Identification of EV charging points KPIs in the case company .............................. 31

4.1 A 7-step KPIs identification approach for CI reliability assessment in the case
company ...................................................................................................................... 32

4.1.1 Step 1. The notion of reliability in the case company ........................................... 33

4.1.2 Step 2. Reliability goals setting and result indicators recognition......................... 34

4.1.3 Step 3. EV charging problems identification ......................................................... 35

4.1.4 Step 4. Assigning variables from the database to the defined reliability goals ..... 37

4.1.5 Step 5. KPIs identification using S.M.A.R.T. method ............................................. 39

4.1.6 Data evaluation for KPIs calculation. ................................................................... 45

4.1.7 Validation of identified KPIs. ............................................................................... 47

4.2 Failure rate calculation ........................................................................................... 49

4.3 Admin Panel improvement ..................................................................................... 51

4.3.1 Current state of maintenance section in Admin Panel .......................................... 52

4.3.2 Maintenance section improvement proposal ....................................................... 52

5. Visualization of KPIs in the case company ............................................................. 54

5.1 Dashboard visualization ......................................................................................... 54

5.2 Kibana as a tool for visualization ............................................................................ 54

5.3 KPIs Dashboard in the case company ...................................................................... 55

6. Conclusions and recommendations ....................................................................... 60

6.1 Conclusions of the study ......................................................................................... 60

6.2 Recommendations for further development ........................................................... 64

7. Summary .............................................................................................................. 66

References ................................................................................................................... 68

Appendices

Appendix 1. Failure rate calculation model created in MC Excel

Appendix 2. The uptime/downtime percentage ratio of a total network
Appendix 3. Reliability report created in MC Excel
LIST OF FIGURES

Figure 1 Structure of the thesis. ........................................................................................... 15

Figure 2 Deployment scenarios for the stock of electric cars to 2030. (IEA Global EV
Outlook 2017) ...................................................................................................................... 16
Figure 3 The number of EV charging points in Finland in 2013-2018. (EAFO 2018) ....... 18
Figure 4 Three types of performance measurement presented by Parmenter (2007). ......... 23
Figure 5 A 7-step KPIs identification approach for reliability assessment in the case
company. .............................................................................................................................. 33
Figure 6 The formula for failure rate calculation (Jones, 2006). ......................................... 49
Figure 7 The formula for failure rate calculation in the case company. .............................. 50
Figure 8 The example of a dashboard created in Kibana using global flights data.
(Retrieved from: https://www.elastic.co/guide/en/kibana/current/dashboard.html
[Accessed: 05.01.2019]) ...................................................................................................... 55
Figure 9 kWh charged heat map. ......................................................................................... 57
Figure 10 Usage frequency per station in the last 24 hours. ................................................ 58
Figure 11 Usage frequency data table.................................................................................. 58
Figure 12 The option that customer uses to start charging in the last 24 hours................... 59
Figure 13 The outline of research questions and answers. .................................................. 63
LIST OF TABLES

Table 1 Classification of EV charging methods. ................................................................. 20

Table 2 Classification of KPI done by different researchers. (Anand and Grover, 2015)... 27
Table 3 The methods introduced by different authors. ........................................................ 28
Table 4 Goals, result indicators and variables assigned from the database. ........................ 39
Table 5 KPIs, data for their calculation, KPIs interpretation, and data problems. .............. 45
Table 6 Checklist for KPIs validation.................................................................................. 48
Table 7 The types of visualizations created in Kibana. ....................................................... 57
LIST OF ABBREVIATIONS

AC - Alternating Current
AHP - Analytical Hierarchy Process
App - Application
BI - Business Intelligence
BSC - The Balanced Scorecard
CI – Charging Infrastructure
CPO - Charging Point Owners
DC - Direct Current
DLM - Dynamic Load Management
EAFO - European Alternative Fuels Observatory
EC - Electronic Cash
EV - Electric Vehicle
EVSE - Electric Vehicle Supply Equipment
ICT - Information and Communication Technology
ID - Identifier
IEA - International Energy Agency
KPI - Key Performance Indicators
kW - kilowatt
kWh - kilowatt hour
LAN - Local Area Network
OSH - Occupational Safety and Health
PLC - Powerline Communication
RFID - Radio-Frequency Identification
S.M.A.R.T. - Specific, Measurable, Assignable, Realistic, Time related
SMS - Short Message Service
SQL - Structured Query Language
TOPSIS - Technique for Order of Preference by Similarity to Ideal Solution
WLAN - Wireless Local Area Network
 9

1. Introduction

According to Parmenter (2007), key performance indicators (KPIs) represent a set of

measures focusing on those aspects of organizational performance that are the most critical
for the current and future success of the organization. If defined correctly, KPIs can
positively influence the performance of different operations such as production, finance,
customer service, marketing, and others. That is why, monitoring, managing and analyzing
KPIs is an important issue for many companies in different industries. (Anand and Grover
2015, Durkacova et al., 2012, Stricker et al. 2017)

This thesis focuses on creation of charging points KPIs and their visualization in an electric
vehicle (EV) charging company. In the recent years EV charging infrastructure (CI) has been
continuously developing worldwide and the case company is in forefront of this
development. (IEA Global EV Outlook 2018) Due to the fast-growing amount of case
company’s charging points, the company needs to obtain the appropriate performance
indicators for monitoring of EV charging points that can be used for further improvement of
their usability and reliability. After the suitable set of KPIs is defined, it has to be visualized
and introduced to the company’s employees and customers. As a result, the benefits of key
performance indicators implementation will be such as improved reliability of EV charging
points and increased customer satisfaction.

1.1 The case company

The case company is an electric vehicle charging service platform that connects electric
vehicle owners, manufacturers, charging point owners (CPO) and energy utilities. The
company offers a wide range of functionalities such as dynamic load management (DLM),
optimized charging based on prices at the local energy market, and grid frequency-controlled
charging. The case company’s mission is to solve the climate change through bringing
mobility to the everyday life and energy systems.

The case company was established in 2013 and its headquarters is located in Helsinki,
Finland. The company’s network currently connects more that 8000 charging points in over
 10

20 countries and it is a leading platform in France, Switzerland, Germany, Finland, Sweden,

and Iceland. There are about 70 employees in Finland, Sweden, Germany and France.

1.2 Background of the study

EV charging industry is relatively new, however, it is rapidly developing. The number of

electric cars worldwide exceeded 3 million in 2017. (IEA Global EV Outlook 2018) And it
is estimated that by 2030 there will be more than 100 million electric vehicles in the world.
(IEA Global EV Outlook 2017) Deployment of EVs is influenced by many factors such as
the depletion of fossil fuels and the desire to reduce the pollutant emissions. EVs adoption
can be significantly stimulated by evaluating and improving EV charging services.

Currently the case company does not have any defined performance indicators to measure
the reliability of charging stations. Thus, in some cases it is complicated to determine and as
a result to eliminate the problems that negatively influence the charging services. One of the
most important goals that the case company is planning to achieve by implementing KPIs is
a higher level of reliability that will lead to better reporting system in the company.
Moreover, the appropriate way of KPIs visualizations has to be found in order to efficiently
communicate the existing KPIs to company’s employees and customers.

This thesis was requested by Services and Operations team that includes project managers,
service developers, and technical support specialists. The ongoing project of the team is
related to service development, reporting systems and support operations improvement. In
the previous months the company hired some professionals who are working on improving
security, reliability and quality of services. Moreover, the company is preparing for ISO
certification that will take place in spring 2019. This thesis is a part of a bigger project
intended to create appropriate metrics for measuring the overall performance of the
company. This study describes the part of the project that focuses on developing a set of
performance indicators for reliability measurement of charging points that can be visualized
and communicated across the company. KPIs will be a starting point for EV charging
experience monitoring and improvement.
 11

1.3 Purpose of the study and research questions

The purpose of the thesis is to obtain a clearly defined and documented set of KPIs that can
be quantified and further used to monitor EV charging points reliability. After defining the
purpose of the thesis, the main research question was composed:

1. What set of KPIs has to be used at the case company in order to measure
reliability of charging points?

The sub-questions of the study are the following:

1.1 How to define a suitable set of KPIs for the case company?

1.2 What is the appropriate way to visualize the selected KPIs?

1.3 What improvements can be proposed based on identified KPIs?

To answer the main research question, first the case company and its EV charging network
are examined. As requested by company’s management this thesis is particularly focused on
creation of operational KPIs for company’s EV charging points not taking into consideration
any financial performance indicators. This study evaluates the technical performance of
charging points that affects the reliability of EV charging service.

The first sub question is answered by creating the approach for KPIs identification in the
case company. Furthermore, the available methods of KPIs identification are studied and
researches by different authors are examined. The steps for KPIs selection are defined based
of case company’s specifications and limitations.

The aim of the second sub question is to define the most appropriate content, design and
layout of KPIs visualizations. The visualization tool has already been chosen by the case
company’s management and has to be learnt by the author of the thesis. The conclusions
related to visualization results and suitability of the selected tool will be presented in the end
of the study.
 12

The third sub question is answered by proposing the improvements that can be implemented
in the case company in the future. The proposal is done based on the determined KPIs and
related to case company’s reporting system, database and visualization tool.

1.4 Research methodology and research process

According to different researches, the mixing of quantitative and qualitative approaches will
result in the most accurate and complete description under investigation. (Tashakkori,et al
2003) For this study a combination of both approaches is used, starting with qualitative and
continuing with quantitative methods.

The first part of the project is completed using qualitative research in a form of a case study.
According to McMillan (1996) studies deriving from qualitative perspective focus on
meaning and understanding and take place in naturally occurring situations. Yin (1989)
states that even though case studies often use quantitative measures, most of the time they
take qualitative perspective and focus on exploring and describing the phenomenon. He
defines a case study as “an empirical inquiry that investigates a contemporary phenomenon
in depth and within its real-life context, especially when the boundaries between
phenomenon and context are not clearly evident. [...]” There are two types of a case study:
“a single case” and a “multiple case”. (Yin, 2003, Baxter & Jack, 2008; Stake, 1995)
Considering the fact that this thesis focuses on the analysis of one organization, the singe
case study approach is followed.

Data collection was completed with respect to the defined research problem using various
research methods. The theoretical research was done by conducting a literature review in
order to familiarize readers with previous studies related to performance measurements and
EV charging industry.

Another material collection method was active participant observation that aims to maximize
discovery and description (McCal1 and Simmons, 1969). It is a method where a researcher
takes an active part in the activities of the community. The author of this thesis worked
fulltime as a Reliability Analyst in the case company, attended different company’s meetings
and workshops. As this study is a part of a bigger project that involves team work, the author
 13

also participated in project planning and customer meetings. In participant observation

method, a researcher's discipline-based interests and commitments define which events he
or she examines are valuable and relevant to the research inquiry. (Emerson et al. 2001).
That is why participant observation is supposed to be a subjective method that means that
two different researchers can observe the same subject differently and end up with
contrasting conclusions. It can be also considered as a positive thing because the method
gives an opportunity to look at the same subject from various perspectives.

Interviews is the method that was actively used in the first months of this project. The
interviews were semi-structured, meaning that the questions were selected beforehand, but
there was an opportunity for an open discussion, depending on the role and experience of an
employee. This interview type is “particularly good at enabling the researcher to learn, first
hand, about people’s perspectives on the subject chosen as the project focus” (Davies, 2007,
p.29). All in all, 14 employees of case company from Technical, Services and Operations,
Sales, and Marketing teams were interviewed. The questions were about their
responsibilities in the company, their opinion about the level of reliability of EV charging
points and ways of its improvement.

The second part of this research is quantitative, and it is executed in a form of data analysis.
According to Leedy (1993), quantitative research approach is used to answer questions on
relationships within measurable variables with an intention to explain, predict and control a
phenomena (Leedy 1993). The major part of this project includes work with company’s
database and data visualization software. As a result of this research, the method for KPIs
identification in the EV charging company was developed by the author of this thesis.
Furthermore, the calculation models of KPIs an failure rate are done using quantitative data.

The total duration of the project was 6 months, starting with a kick off meeting with the case
company’s supervisor in October 2018. The scope of the study was defined and the
interviews with the company’s employees from different department were scheduled. After
that the project plan was created and submitted to the company’s and university’s
supervisors. The study started with reviewing available literature related to electric mobility,
EV charging networks, and existing performance indicators in the industry. For better
understanding of the company and EV charging industry the interviews with the company’s
employees were conducted during the first two months of a project.
 14

Then analysis of case company’s database started and the draft of a KPIs determination
approach was created. The approach consisted of 7 steps that were conducted as the second
part of the project. The outcome of every step was checked and confirmed by management
of the case company. After that the determined KPIs were visualized using the case
company’s software. The conclusion of the study and recommendation for further work are
presented in the end of the study. The project was completed in April and all the results were
presented to company’s management and university supervisors.

1.5 Structure of the Thesis

This thesis is composed of seven chapters that are demonstrated in Figure 1. Chapter one is
an introduction and the next two chapters present the theoretical background of this study.
Chapter two is dedicated to EV charging. In the beginning of the chapter the information
about EV charging infrastructure and its development is introduced. Then EV charging
process, technologies and locations are explained. Chapter three discusses the definition of
performance measurement and key performance indicators. Furthermore, characteristics and
ways of KPIs categorization are presented. In the end of this chapter various methods and
frameworks for KPIs creation are summarized. Chapter four presents the approach that was
created for KPIs identification. First, the seven steps that form the approach are introduced
and explained. Afterwards, the model for failure rate calculation is presented and proposal
for company’s reporting system improvement is discussed. Chapter five focuses on
visualization of KPIs. The visualization tool used in the company is described and the
selected types of KPIs visualization are discussed. Chapter six is dedicated to conclusions
and recommendations that present the outcome of this study. The last chapter is the summary
of this thesis.
 15

Introduction

EV charging Performance
infrastructure and measurement and
technologies KPIs

KPIs identification
and validation
approach for
reliability
assessment in the
case company

Visualization of
identified KPIs

Conclusions and
recommendations

Summary

Figure 1 Structure of the thesis.

 16

2. Charging of Electric Vehicles

2.1 EV charging infrastructure today

A sufficient infrastructure is required for EV charging points based on electric energy needed
for recharging. (Trip et al. 2012) GTM Research predicts that the amount of charging points
worldwide will grow up and range from 40 to 70 million by 2030. (Figure 2) The reason is
the increase in sales of electric vehicles, GTM research sates that 11 percent of sold vehicles
will be electric. In Europe the amount of residential charging points is estimated to raise up
to 9 million, and the amount of public charging points will be around 1.6 million by 2030.
(GTM Research 2018)

Figure 2 Deployment scenarios for the stock of electric cars to 2030. (IEA Global EV Outlook 2017)

The report about current and planned public charging infrastructure conducted by the
Platform of Electromobility concludes that nowadays there is no lack of infrastructure and
there are enough charging points available for the number of EV vehicles on the road.
Moreover, the report estimates that if the goals for EV charging infrastructure roll-out are
reached there will be an appropriate quantity of chargers until 2020 to charge the growing
amount of sold EVs.

EVs are changing the automotive sector all around the world. For instance, such countries
as the Netherlands, UK and Norway are in forefront of EV charging infrastructure
development that stimulates the growth of EV charging companies. Thus, the competition
 17

in EV charging industry is tough, that pushes companies to continuously work on providing

the most convenient and reliable service for their customers. The demonstration of reliable
charging services is an essential prerequisite for the successful EVs charging companies
operation. The question that arises is how to monitor and evaluate the performance of EV
charging networks. (J. Helmus, 2016)

CleanTechnica is a cleantech-focused website that presented a report “Electric Car Drivers:

Desires, Demands, & Who They Are” (2017). Over 2000 electric car owners were surveyed
in 28 countries located in North America and Europe. The responses were divided into three
EV groups: Tesla drivers, pure electric but non-Tesla drivers, and plug-in hybrids drivers.

One of the questions was about the level of reliability of EV charging stations that drivers
use. According to the report, from 10 to 30 percent of respondents stated that they experience
reliability problems while using EV charging points. However, most of EV drivers evaluated
the stations as somewhat or very reliable. 93 percent of Tesla drivers in North America and
83 percent of Tesla drivers in Europe consider the charging points to be reliable. A bit less
positive were pure EV non-Tesla drivers, the results showed 82 percent reliability rate in
North America and 79 percent in Europe. 84 percent of plug-in hybrids drivers in North
America and only 70 percent plug-in hybrids drivers in Europe are satisfied with the
reliability of charging points. The other questions were related to accessibility of charging
stations and their convenience for EV drivers. The results presented in the report show the
overall satisfaction of EV drivers but state that there is a need for EV charging points
improvement.

2.2 Development of EV charging infrastructure in Finland

In 2017 it was announced that in 2017-2019 4,8 million euros will be allocated to
development of EV charging infrastructure. The goal is to catalyze 15 million euros in
investments and triple the amount of public charging stations. This initiative aims to raise
the number of fast chargers in Finland, thus 35 percent of subsidy rate is allocated to fast
chargers and 30 percent – to normal chargers. (Ministry of Economic affairs and
Employment of Finland 2017)
 18

According to statistics provided by European Alternative Fuels Observatory (EAFO 2018),

in 2018 the total number of charging points in Finland was 973. (Figure 3) To see the growth
of EV charging infrastructure in the recent years these statistics can be compared to statistics
of year 2013. EAFO reported that in 2013 the total number of charging points in Finland
was 267. (EAFO 2013)

Figure 3 The number of EV charging points in Finland in 2013-2018. (EAFO 2018)

2.3 EV charging technologies and locations

Two components of an EV charging infrastructure can be defined: the electric charging

facilities and the information and communication technology (ICT). Charging facility, also
referred as charging equipment or EV supply equipment (EVSE) is an electric facility for
charging EVs that can have several socket outlets or several connectors. ICT is used by the
operators to use, operate, and control the charging infrastructure. In simple terms, EV
charging can be done by connecting the EV to a charging point and transferring the electric
power to the vehicle. Charging point is an arrangement of charging one EV at a time, and
charging station is a group of charging facilities. (Wirges, 2016)

Manufacturers of charging facilities develop and implement different technologies of user

interaction, identification, payment and communication to back-end systems. To start
charging, EV drivers have to identify themselves using for instance radio-frequency
 19

identification (RFID) card, website, mobile application (App) or SMS method. After
verification is done, the charging can start by plugging the charging stations plug into the
vehicle’s power socket. Charging duration depends on the following factors: grid
connection, EV type, charging station, socket, and charging cable. The payment for the EV
charging service can be done via direct payment in cash, Electronic Cash (EC), credit card,
cell phone bill, electricity bill, or via debit if the user is a registered customer at the charging
station operator. Connection to back-end system can be maintained by different means such
as connection to a local area network (LAN) or wireless local area network (WLAN).
Another option is communication over mobile phone networks using SIM cards or powerline
communication (PLC). (Wirges, 2016)

There are different available classifications of EV charging technologies available in

literature. (Rautiainen, 2015, Jackson, 2016, Rahman et al, 2016) According to Jackson,
2016 EV charging methods are divided into 3 types:

§ Slow charging or Level 1.

This type of charging provides power up to 3 kW that is suitable for overnight charging as
it takes from 6 to 8 hours to fully charge an EV vehicle.

§ Fast charging or Level 2.

This type of charging provides power up to 22 kW and it takes about 3 hours to charge an
EV.

§ Rapid charging or Level 3.

Rapid chargers are divided into two types: alternating current (AC) and direct current (DC).
AC chargers supply power at the rate of 43kW and DC units reach 50 kW. These chargers
fill about 80% of the EVs batteries in maximum an hour, the time depends on the battery.

The international standard IEC 61851-1 provides another classification of EV charging

types. It specifies four charging modes that define the communication protocol used between
the vehicle and the charging device. (IEC, 2003) Modes 1 and 2 are both AC methods and
they are used for slow charging. But “Mode 1 charging is intended mostly for light vehicle
charging and mode 2 for temporary or very restricted charging of cars”. (Rautiainen, 2015,
page 12)
 20

Mode 3 is also an AC charging method that has the same safety protocol as Mode 2 but
supports a higher power level. It also provides different control and safety functions. Mode
4 is a fast DC charging method that differs from other modes in a way that the charger is not
a part of a car, but a part of a charging station. (Rautiainen, 2015, page 13) Mode 1 can be
compared to slow charging Level 1, Modes 2 and 3 are referred to Level 2 fast charging, and
Mode 4 is related to rapid charging Level 3. (Table 1)

There are three main locations where EV drivers can charge their cars. The first way to
charge an EV is public charging. Public charging points are available to everybody and allow
to charge an EV in less than an hour. Another way to charge an EV is to use a semi-public
charger that is open to a certain community of people. For instance, companies can install
EV charging points for their employees and visitors. One more way to recharge an EV is
private charging. This type of charging is the most restricted one, because it is only open to
one user, EV or household. (Filho, Kotter, 2015, Kley et al., 2011) In this categorization of
EV charging locations accessibility is evaluated as a differentiation factor, and not the
ownership of the ground on which the charging station is placed. (Wirges, 2016)

Charging Charging Current Power (kWh) Location

method mode
Level 1 Mode 1 AC Up to3 kWh Private/domestic

Slow charging
Level 2 Modes 2 and3 AC Up to22 kWh Semi-public

Fast charging
Level 3 Mode4 AC/DC Up to 50 kWh Public

Rapid charging

Table 1 Classification of EV charging methods.

 21

3. Performance measurements and key performance

indicators

It is important for organizations to define the relevant indicators, their influence on the
formulated goals and how they rely on the activities performed (Popova et al., 2012). Kaplan
(1990) emphasized the importance of measurement with the following sentence ‘‘No
measures, no improvement’’. In other words, if something needs to be improved, it has to
be measured first. Gunasekaran & Kobu (2007) have presented six common purposes why
measuring organizational performance is beneficial for any company. The purposes are:

§ to identify success,
§ to identify if the customer needs are met,
§ to help organizations to define its processes and understand what information they
are missing,
§ to identify the bottlenecks, problems, waste, etc. and what improvements have to be
done,
§ to ensure that decisions are based on facts, not on intuition, emotion, etc.,
§ to show that planned improvements are happening in reality.

The purposes listed above state how beneficial it is for organization to measure its
performance. However, the process of performance measurement is long and
complicated. Decision maker has to understand the main objective of performance
measurement and focus on evaluating what really matters for the company. And the most
important phase of measurement process is not to make the list of all the existing
performance indicators, but to identify the performance indicators that will create value
for the company and affect the main processes. These performance indicators are KPIs.
(Franceschini et al. 2007, 7-10) Enns et al., (2005) define key performance indicators as
“measurements that reflect the health of an organization, and the health of its business
development system. They connect the firm’s goals and strategies to its activities and
outcomes, keeping management informed of overall health: past, current, future.” (Enns
et al., 2005). Franceschini et al. (2007) define performance indicators using the following
requirements:
 22

§ Representativeness,
§ Easy and simple to interpret the results,
§ Possible to indicate time-trends,
§ Adaptive to changes within or outside the company,
§ Easy collecting and processing of data,
§ Update can be done easily and quickly.

Dean R. Spitzer (2007) stated in his book that “93% of organizational leaders believe that
measurement is important in influencing business outcomes, but only 51% are satisfied with
their current systems, and only 15% are very satisfied.” Eckerson (2009) conducted a survey
and interviewed different performance management practitioners and solution providers. His
goal was to define the status of these professional’s initiative to develop KPIs. The results
were based on the answers of 678 respondents and showed that more than one-third of them
either partially (31 percent) or fully deployed (9 percent) an initiative to define KPIs. More
than a half of of interviewees answered that the status of their KPIs development process is
planned (24 percent) or under way (28 percent). And only about 8 percent of respondents
stated that currently they do not have any KPIs projects.

Despite the fact that companies are developing and using KPIs, the results of performance
measurement is not always positive. Parmenter (2007) states that many decision makers
define the KPIs incorrectly and very few organizations really monitor their true KPIs”. The
researcher suggests that it happens because the definition of KPIs is not understood correctly.
He defined three types of performance measures (Figure 4):

§ Key result indicators (KRIs) show the results of the pervious actions.
§ Performance indicators (PIs) define what has to be done.
§ Key performance indicators (KPIs) state what has to be done to increase
performance significantly.

 23

Figure 4 Three types of performance measurement presented by Parmenter (2007).

Different opinions about the required number of KPIs can be found in literature. Kaplan and
Norton (1996) suggest that the appropriate number of KPIs is not more than 20. According
to Franceschini (2012), this number has to be between 3 and 15. Parmenter (2007) proposed
the 10/80/10 rule meaning that companies should use about 10 KRIs, 80 PIs, and 10 KPIs.
Such factors as the organization’s size and the industry in which the company operates can
influence the number of required KPIs. Furthermore, KPIs have to be frequently measured
in order to be efficient. Depending on their importance, KPIs can be tracked hourly, daily or
weekly. The shorter the measurement interval the better for organization. (Parmenter 2007)

3.1 Characteristics of KPIs

The characteristics of KPIs were introduced by different authors in their publications. (Marr,
2015, Bauer, 2004, Parmenter, 2015, Eckerson 2005) In order to provide the overview of
existing characteristics the list of KPIs attributes presented by Wayne W. Eckerson (2005)
was chosen. The author provides a detailed description of 12 characteristics of effective KPIs
that are listed and described below.

1. Strategic
The KPIs always have to be focused on the desired outcome.
2. Simple
KPIs have to be created in a way to be easily understood by users. They should not be based
on complex indexes that can complicate their usability in the company.
3. Owned
 24

All the indicators have to be assigned to a particular user or group of users who is
accountable for the particular KPI and its outcome.
4. Actionable
KPIs have to actionable to allow users to take actions in order to continuously improve the
performance and positively influence the outcome.
5. Timely
KPIs have to be constantly updated in order to immediately improve the performance if
needed.
6. Referenceable
The users have to able to understand the origins of the created KPI in order to use and trust
their outcome.
7. Accurate
KPIs have to be measured and communicated with the appropriate level of accuracy.
8. Correlated
KPIs can be used for achieving the desired results.
9. Game-proof
KPIs have to be checked and analyzed frequently in order to confirm their effectiveness.
10. Aligned
KPIs have to be connected to organization’s strategy and goals.
11. Standardized
Standardized KPIs have to be communicated throughout the organization and everyone has
to agree on them.
12. Relevant
KPIs tend to lose their effectiveness, thus they have to be revised and improved over time.

Parmenter (2009) in his book Key Performance Indicators: Developing, Implementing, and
Using Winning KPIs presented a different list of effective KPIs. According to the author,
the seven characteristics of KPIs are:

1. Nonfinancial
KPIs are not financial measures and they are not expressed in currency.
2. Frequently measured
Weekly and daily measures are preferred for KPIs tracking.
 25

3. Acted on by the senior managers

Senior managers have a constant focus on existing KPIs.
4. Understood by the employees
Employees have to be able to understand what actions to take based on KPIs.
5. Assigned to the individual or team
An employee or a team have to take responsibility for KPI tracking.
6. Have significant impact
KPIs have to be bring value to the company.
7. Have positive impact
KPIs have to affect other performance measurements positively.

3.2 Categorization of KPIs

There are different ways to distinguish KPIs types depending on point of view of a decision
maker and the purpose of the research. One of the most common categorizations is done by
industry. In different industries companies actively use various KPIs in order to understand
how to gain value to the company, define problems and reach customer satisfaction. Some
of the performance indicators vary depending on business area, and some of them are the
same for different type of industries.

Eckerson (2009) states that there are two most fundamental categories of KPIs: outcomes
and drivers. Outcomes are lagging indicators that measure the results of the past activity.
They can be financial in nature and include such KPIs as return on equity, revenue, margins,
etc. They can be easily measured, but their improvement takes a lot of effort. Driver KPIs
are also defined as leading or value drivers. They track the activities that have the important
impact on the outcome KPIs. This type of KPIs measure current and future states such as
number of scheduled meetings for this and next week. It is difficult to measure driver KPIs,
but it is easy to influence them.

Parmenter (2010) and Kaplan et al. (1996), argued the reliability of categorization KPIs into
lagging and leading. They stated that some KPIs can be related to both categories. Parmenter
proved it with the example of “late-planes-in-the-air KPI” that can be a leading indicator
because it is related to taking actions to land the plane on time. But at the same time, it can
 26

be a lagging KPI because the plane is already late, meaning that there will be such
consequences as dissatisfaction of passengers or extra consumption of fuel needed to speed
up the plane.

Anand and Grover (2015, 141-142) in their work presented various categorizations of KPI
done by different authors. Table 2 presents the researchers and KPIs types that they defined.
The oldest classification that is included in the table was conducted in 1995 and the newest
was done in 2006.
 27

Table 2 Classification of KPI done by different researchers. (Anand and Grover, 2015)
 28

3.3 Methods and techniques for KPIs creation

Most of the authors come to the conclusion that KPIs have to be linked to strategy and
objectives of the company, however, the universal approach for KPIs identification does not
exist. (Parmenter 2007, Neely et al,1997) Various KPIs identification models and
approaches were created in order to simplify the process of performance indicators
identification. The process of KPIs selection is complicated and its success depends not only
on selected model, but on the experience and initiative of the decision maker. (Franceschini,
F., et al. 2007) The table below presents the work of the authors who developed different
methods for KPIs identification.

Author Introduced method(s)

Podgorski (2015) The Analytical Hierarchy Process (AHP) method for
the selection of leading KPIs for measuring OSH MS
operational performance.

Parmenter (2010) A 12-Step Model for developing and using KPIs.

Kaplan & Norton (1996) Balanced Scorecard (BSC) to track KPIs.
Doran (1981) S.M.A.R.T. method to write management's goals and
objectives.
Shahin & Mahbod (2015) The Analytical Hierarchy Process (AHP) technique
and S.M.A.R.T criteria for KPI prioritization.
Kadarsah (2007) Framework of measuring key performance indicators
using Analytic Hierarchy Process (AHP) technique.
Sun (2010) A performance evaluation model by integrating fuzzy
AHP and fuzzy technique for order of preference by
similarity to ideal solution (TOPSIS) methods.

Table 3 The methods introduced by different authors.

Multiple researchers used Analytical Hierarchy Process (AHP) method for KPIs
identification or ranking. In many papers, this method was used in combination with other
approaches and frameworks. Podgorski (2015) worked on measuring operational
 29

performance of occupational safety and health management system (OSH) and introduced
the AHP method for the selection of leading KPIs. Kadarsah (2007) created a framework for
KPIs measuring of higher educational institution. He categorized KPIs into academic,
research and supporting KPI and then weighted them using Analytic Hierarchy Process,
trend analysis and comparative data.

Parmenter (2007) in his book described a 12-Step Model for developing and using KPIs.
The model is based on the following four foundation stones for KPI development:

§ Collaboration with employee, unions, suppliers and customers,

§ Allocation of power to the front line,
§ Combination of measurement, reporting and performance improvement,
§ Connection of performance measurement to strategy.

The Balanced Scorecard (BSC) introduced by Kaplan & Norton (1996) is a common way of
KPIs selection. BSC is used for measuring performance from four perspectives: financial,
customers, internal business processes, and learning and growth. By using these
perspectives, decision-makers can translate company’s strategy and objectives into
performance measures that provide a framework for strategic measurement and management
system.

S.M.A.R.T. method is also widely used in performance measurement systems. First it was
introduced by Doran (1981) who created a managerial method for goal setting based on
S.M.A.R.T. criteria. (specific, measurable, assignable, realistic, time related).

Shahin & Mahbod (2015) combined both AHP and S.M.A.R.T. methods in their work and
established guidelines for KPIs ranking in organizations. They created an approach for
prioritization of KPIs in terms of S.M.A.R.T criteria goal setting. AHP method was applied
as the basis for S.M.A.R.T criteria comparisons for KPIs.

Fuzzy methodology is also used for performance measurement. Sun (2010) applied fuzzy
TOPSIS method together with Fuzzy AHP. The research presents the combination of these
methods in order to help industrial practitioners to evaluate performance measurements in a
fuzzy environment.
 30

Depending on the operational industry and goals of a researcher the suitable KPIs
identification method can be chosen. The methods presented above can be divided into
manual (BSC, S.M.A.R.T) and selection (AHP, fuzzy TOPSIS) groups. Such methods as
S.M.A.R.T and AHP are typical for KPIs identification and used in multiple works presented
in the Table 2. Fuzzy methods are not so widely used, but also getting more common for
measuring the performance. Furthermore, machine learning and data mining techniques are
used for making predictions and recommendations on KPIs.
 31

4. Identification of EV charging points KPIs in the case

company

A lot of information about EV charging infrastructure can be found in literature.

Infrastructure development and improvement are the topics widely presented by different
authors. (San Roman 2011, Rahman 2016, Nie 2013) However, there is also a lack of
information on KPIs used in EV charging companies and only a few research works related
to this topic were found. Different performance indicators applied in various industries can
be found in literature, but they cannot be an appropriate base for KPIs creation for such
complex industry as electric vehicle charging.

The research “Key Performance Indicators of Charging Infrastructure” conducted by J.

Helmus (2016) presents KPIs that can be used to monitor and improve the public EV
charging infrastructure in the Netherlands. The author concludes that the amount of literature
related to this topic is limited and develops his own approach for KPIs identification. As the
large number of stakeholders such as municipalities, EV drivers, CPOs, etc. are interested
in CI development, he takes a stakeholder perspective in his research. In the context of this
thesis work that focuses on one case company, the approach presented by J. Helmus cannot
be followed.

Part of the methods listed in Chapter 3.3 are intended for KPIs selection in the companies
that already have the list of performance indicators. Another part is suitable for ranking or
improvement of performance measurement in organizations. However, there are not
methods presented in the literature that can help to define KPIs for EV charging company.
This study is focused specifically on developing KPIs for charging points reliability
monitoring and not for the whole organization. Thus, it was necessary to create a new
approach for KPIs identification based on the example of the case company.

As EV charging companies are growing and developing, the need for performance
monitoring of charging points occurs. The approach presented in this study can be a guidance
for decision makers who are in the beginning of KPIs identification for EV charging
 32

companies. The next chapters present the developed approach that consists of seven steps of
KPIs identification in an EV charging company.

4.1 A 7-step KPIs identification approach for CI reliability assessment

in the case company

As mentioned before, currently the case company does not have any established KPIs for
monitoring reliability of their EV charging points. Thus, performance indicators had to be
created from scratch. When starting the project, the available resources included company’s
database and documentation. Hence, a lot of information needed for the project completion
was obtained during the interviews and meetings with the company’s employees.

The methods created by different researchers for KPIs determination were studied first in
order to choose the suitable one for the case company. As concluded before, none of the
reviewed methods can be selected as a framework for KPIs identification in the case
company. However, S.M.A.R.T method will be used in the fifth step of the approach.

The approach for KPIs identification in the case company includes the following steps
(Figure 6):

1. The definition of reliability of charging points.

2. Reliability goals setting and result indicators recognition.
3. EV charging points problems identification.
4. Assigning the variables from the database for the defined reliability goals.
5. KPIs identification based on S.M.A.R.T. method.
6. Data evaluation for KPIs calculation.
7. Validation of identified KPIs.

As KPIs were required for monitoring of reliability of EV charging points, it was crucial to
define what is a reliable charging station. The definition included seven important
characteristics that define a reliable charging station in the case company. Then the goals for
reliability of EV charging points were established based on the notion of reliability. The
result indicators were also defined and linked to reliability goals. After the desired level of
 33

reliability was described, the current state of charging points was evaluated. As a result, the
detailed list of existing EV charging problems was composed. The next step included
identifying of variables from the database and assigning them to the reliability goals. After
that the selected variables were tested against SMART method and a set of KPIs was
identified. The company’s database was analyzed in order to define whether it is possible to
calculate the defined KPIs. The quality and availability of required data was also evaluated
and discussed. As a final step of the method the KPIs were validated by testing them against
such properties as quantifiability, sensitivity, congruency, reliability, efficiency and
improvement-orientation. Every step of the research process was checked and approved by
support team and managers of the case company.

1. The notion of reliability of charging points

2. Reliability goals setting and result indicators recognition

3. EV charging points problems identification

4. Assigning variables from the database to the defined reliability goals

5. KPIs identification based on S.M.A.R.T. method

6. Data evaluation for KPIs calculation

7. Validation of identified KPIs.

Figure 5 A 7-step KPIs identification approach for reliability assessment in the case company.

4.1.1 Step 1. The notion of reliability in the case company

The first step to create a set of KPIs is to define the notion of reliability in the case company.
In other words, it has to be specified what is a reliable charging station from the point of
 34

view of the case company. According to Kececioglu (2002), reliability is “the conditional
probability, at a given confidence level, that the equipment will perform its intended
functions satisfactorily or without failure [...]” For the case company it was decided that the
term of a reliable charging station includes seven characteristic that are listed below.
Reliable charging station:

1. is easily accessed,
2. allows fast and easy start of charging event using RFID, App or SMS,
3. allows smooth charging process without any problems or interruptions,
4. allows fast and easy ending of charging event using RFID, App or SMS,
5. provides all the data related to charging event to back end and to customers,
6. is constantly maintaining connection to back end, e.g. shows that it is online,
7. provides up to date and useful information about its status.

4.1.2 Step 2. Reliability goals setting and result indicators

recognition

The characteristics of reliable charging station described in Chapter 4.1 form a desired state
for company’s chargers. They are also the base for reliability goals in the case company. If
all of them are met, the charging point is considered to be functioning correctly. Reliability
goals and result indicators are shown in the first and second columns of Table 3.

The following EV charging reliability goals were defined for the case company:

§ Accessibility of a charging point.

§ Fast and easy start of charging event.
§ Smooth charging process without any problems or interactions.
§ Fast and easy ending of charging event.
§ Data related to charging event is provided to back end and to customers.
§ Connection to back end is maintained.
§ Up to date information is provided to back end and to customers.
 35

Result indicators show the benefits for the case company and for the customers that can be
obtained if the reliability goals are met. Based on the reliability goals the following result
indicators were identified and presented in the second column of the table 4:

§ Customer is always able to access the station.

§ Customer is able to start charging using RFID, App or SMS method without a need
to call support.
§ The electric vehicle is charged without any problems or interruptions.
§ Customer is able to stop charging using RFID, App or SMS method without a need
to call support.
§ Support team and customers get up to date and trustworthy data such as station id,
customer id, start and stop time, kWh charged, price of charging event. Payment is
done correctly, and the data related to charging event and payment is visible in the
reporting system - Admin Panel.
§ Chargers stay online permanently.
§ Chargers send up to date and useful data about its status.

4.1.3 Step 3. EV charging problems identification

The next step is to evaluate the current state of charging points at the company. The case
company’s employees from Support or Technical teams are aware of the problems related
to their charging network. Based on available data and interviews with company’s
employees the list of all the problems related to charging points is created. The data required
for this study was retrieved from SQL database. From several tables data about charging
points, charging events, created bills, stations disconnects and alerts was analyzed.
Employees of the case company perform tests of chargers and testing data is also stored in
the same data tables as data from normal charges.

This step is important in order to see where charging stations do not comply with the reliable
station definition. Precisely defined problems will give an overview of charging points
reliability and will lead to better solutions to these problems. All in all, 14 problems are
defined and listed below:
 36

1. The charging station goes offline.

2. Charging event does not start or stop.
3. The system does not identify customers, e.g. roaming customers.
4. Charging stops unexpectedly, e.g. car or cable problem.
5. Reservation on the App does not work well, e.g. the App shows that the charger is
reserved for you, but it is not, so another EV drivers can use this charger.
6. There are chargers that do not function properly, but they are visible to the customers.
The App shows the availability of a charging station, even though it is taken, broken,
offline, reserved, or not activated yet.
7. RFID or App does not work well, e.g. does not start or stop charging.
8. Error messages from the charging points do not specify what is the problem. It is
difficult to define if there are any problems with actual charging station and where is
the problem.
9. Data that comes from back end or stations is not always trustworthy, e.g. charger
does not provide data about the amount of used energy.
10. If customers experience problems while using the chargers, they use hot line call
service. In some cases, hot line waiting time can be about 30 minutes.
11. The car is connected to the charging point, but there is no energy coming.
12. Charger does not recognize the car, e.g. car-to-charger communication problem.
13. Sim card problems, e.g. weak signal in a parking garage, so sim card loses the
connection completely and needs to be rebooted on the site.
14. Payment problems based on lack of data or incorrect data. There are situations when
energy is not coming from charger, but the customer pays for the time his or her car
was connected to the charger

The mentioned problems negatively influence the usability of charging points and have to
be eliminated in order to reach the desired level of reliability. However, there are some
charging problems that the case company cannot solve due to some constraints. The
following limitations to reaching the reliability goals are identified:

§ Parking next to EV chargers due to some reasons might be impossible.

§ Bad quality of connection cables purchased by customers might influence
charging negatively.
 37

§ Due to weather conditions such as snow falls access to chargers might be

blocked.

4.1.4 Step 4. Assigning variables from the database to the defined

reliability goals

For each defined goal from 4 to 8 variables were defined using the company’s database.
Reliability goals can be assessed and judged using the following variables that are also
presented in the third column of Table 4:

1. Usage frequency, low utilized stations.

2. Number of hotline calls, number of complaints.
3. kWh charged.
4. Number of frequent users per charging station.
5. Number of online stations.
6. Average occupancy of charging point, occupancy rate.
7. The option that customer uses to start charging: RFID, App or SMS.
8. Number of errors in Admin System.
9. The option that customer uses to end charging.
10. Number of long charges.
11. Database update frequency, heart beat every 1-15 minutes.
12. Number of payments.
13. Charge time and connection time ratio.

Goals Result Indicators Variables

Accessibility of a Customer is always § Usage frequency (low utilized stations)
charging point able to access the § Number of hotline calls/complaints
station § kWh charged
§ Number of frequent users/charging
station
§ Number of online stations
 38

§ Avg. occupancy of charging point-

occupancy rate

Fast and easy start Customer is able to § Number of online stations

of charging event start charging using § The option that customer uses to start
RFID/App without charging (RFID/App/SMS)
a need to call § kWh charged
support § Number of errors in Admin System
§ Number of frequent users/charging
station
§ Usage frequency (low utilized stations)
§ Number of hotline calls/complaints

Smooth charging The electric vehicle § Number of online stations

process without is charged without § Usage frequency (low utilized stations)
any problems or any problems or § Number of hotline calls/complaints
interactions interruptions § kWh charged
§ Number of frequent users/charging
station
§ Number of errors in Admin System

Fast and easy Customer is able to § Number of errors in Admin System

ending of stop charging using § The option that customer uses to end
charging event RFID/App without charging
a need to call § Number of frequent users/charging
support station
§ Usage frequency, low utilized stations
§ Number of long charges
§ Number of hotline calls/complaints

Data related to Support team and § Database update frequency- heart beat
charging event is customers get up to 1-15 min
provided to back date and § Amount of errors in admin system
 39

end and to trustworthy data. § Number of hotline calls/complaints

customers Payment is done. § Number of online stations

This data is in § Number of payments

Admin Panel

Connection to Chargers stay § Number of errors in Admin System

back end is online permanently § Number of hotline calls/complaints
maintained § Usage frequency, low utilized stations
§ Charge time/connection time ratio (e.g.
car is connected but no energy coming)
§ kWh charged
§ Database update frequency- heart beat
1-15 min
§ Number of payments
§ Number of online stations

Up to date info is Chargers send data § Number of hotline calls/complaints

provided about their status Database update frequency - heart beat
1-15 min
§ Number of errors in Admin System
§ Number of online stations

Table 4 Goals, result indicators and variables assigned from the database.

4.1.5 Step 5. KPIs identification using S.M.A.R.T. method

The problems that were identified in Step 3 can be divided into the following main groups
that have to be covered by KPIs:

§ Station errors.
§ Frequency of charging and number of frequent users per charging point.
§ Monitoring and reports that station sends.
§ Energy transfer.
 40

§ Type of initiation or completion of charging events.

Variables that were repeated multiple times and assigned to the groups mentioned above
were selected and then tested against S.M.A.R.T. criteria. Doran (1981) introduced the
S.M.A.R.T. goal setting method that was later adapted to KPIs identification process. The
S.M.A.R.T. acronym refers to the following concepts: specific (S), measurable (M),
assignable (A), realistic (R), time related (T). And the goal of applying this method is to
check if KPIs are referred to each of these concepts.

§ Specific (S) – It has to be clearly defined and communicated what the KPI measures.
The definition has to be precise and it has to be understood and accepted across the
organization.
§ Measurable (M) – It has to be possible to measure the KPI. To do that there should
be a possibility to access the necessary data and to interpret it in a right way.
§ Assignable (A) – A person or a group of people have to be defined as responsible for
a KPI. They have to continuously monitor, improve and report the results of this KPI.
§ Realistic (R) – It has to be possible to achieve the defined KPI. On the other hand,
KPIs should be motivating and set at right level.
§ Time related (T) – The timeline for measuring and reviewing a KPI has to be set in
advance. KPIs are meaningful only if the time frame in which it is realized is
determined.

As a result, the variables that covered the problems of EV charging points and complied with
S.M.A.R.T criteria were identified as KPIs in the case company. The set of 9 KPIs is
presented below:

KPI 1: Usage frequency of chargers.

KPI 2: Number of hotline calls/complaints.
KPI 3: kWh charged.
KPI 4: Number of frequent users per charging station(s).
KPI 5: Total number of offline stations.
KPI 6: The option that customer uses to start charging (RFID/App/SMS).
KPI 7: Number of errors in admin system.
 41

KPI 8: Number of long charges.

KPI 9: Number of payments/ the amount of bills created correctly.

Table 5 consists of four columns and represents the defined key performance indicators,
available data required for calculation, KPI interpretation and problems related to data.
Below the description and interpretation of each KPIs is presented. The content of the table
is specific to the case company, however the structure of the table can be generalized and
applied to other companies.

The first selected KPI is usage frequency of EV charging points. This indicator provides an
overview of utilized stations and helps to spot the low utilized ones. High usage frequency
of a charger is an indicator of a well performing station, on the other hand low usage
frequency identifies possible problems. It is important to note that low usage frequency
might also mean the remote location of the charger. Hence the usage frequency results have
to be compared to the perilous period of time, e.g. two weeks, in order to spot the difference
in usage rate.

Number of hotline calls received from an outsourced service is an another identified KPI.
The increased number of complains can point out the drop in the reliability of charging
points, it is also important for measuring the customer satisfaction. The reports related to the
number of hotline calls are sent to the case company every week and analyzed by the Support
team.

Amount of kWh charged is another KPI that is useful for monitoring the performance of
station. The drop in the charged energy is a potential identification of a problem. The same
as with user frequency if a charging point is not used often and there are not sudden drops
in usage it can also mean the remote location of the charger.

The number of frequent users per charging station is a KPI that identifies well performing
stations by counting the number of frequent customers of public charging stations. If EV
owners prefer to use the same charger(s) frequently it means the high reliability of these
charger(s). However, for measuring this KPI home chargers have to be filtered out.
 42

The number of offline stations is another KPI relevant to EV charging industry. It provides
information about the number of offline chargers that directly influences usability and
reliability of the whole charging network.

The next KPI is the option that customer uses to start charging: RFID, App or SMS. It shows
the most popular thus the most reliable method of charge initiation. It also identifies what
method is the worst from the point of view of customers and has to be improved or
discontinued.

The number of errors in admin system is a KPI that identifies the most problematic stations
based on the amount of error messages coming from charging points. There are various
errors received from chargers that can be classified according to type and level of severity.

The number of long charges is the KPI that counts the charges that used more than 100 kWh.
It is evaluated as an abnormal event, that can mean that the charging process did not stop
due to some reason. Long charges can identify bus charging, on other hand it can pinpoint
the unreliable stations.

The number of payments or the number of bills created correctly shows that charging event
was successful, and the payment can be processed. The sudden drop in created bills can
indicate the chargers that do not function properly or the ones that were set to free charging
by mistake.

KPI Data KPI interpretation Problems with

data
KPI 1. Table: High usage frequency-well § Watt hour data

Usage frequency Charge performing station. is not always

of chargers Low frequency- problem or correct,

Columns:
remote location. contains “0”
Station ID and numbers
Compare to the last period of
Start_time with ”- “
time e.g. 3 weeks to see the
Stop_time changes in usage frequency
 43

Watt_hour § Not clear why

start time and
stop time are
the same

KPI 2. Number Outsourced The less complains the better § Reports are
of hotline reports sufficient, sent
calls/complaints to Virta every
week

KPI 3. kWh Table: High amount of kWh § Watt hour data

charged Charge-energy charged-well performing is not always

usage station. correct,

Low amount of kWh contains ‘0’
Columns:
charged - problem or remote and numbers
Station_id location. with’-‘
Watt_hour Compare to the last e.g. 3
weeks to see the changes in
kWh charged

KPI 4. Table: This KPI can identify well

Number of Charge-energy performing stations or home

frequent users usage chargers

per charging Column:

station(s)
Station_id
Customer_id

KPI 5. Table: KPI shows % of time the § Difficult or

Total number of Station status charger is online impossible to
offline stations notification, understand
 44

Stations what the error

disconnects is
Columns:

Offline

KPI 6. Table: KPI shows the most reliable § Difficult or

The option that Charge start charge initiation method impossible to
customer uses to Columns: understand
start charging what the error
Method
(RFID/App/Sms) is

KPI 7. Table: An increase in error reports

Number of errors Station status indicates problems
in admin system notification,

Station alert
Columns:

Error code

KPI 8. Table: KPI shows the most reliable § Difficult or

The option that Charge start charge initiation method impossible to
customer uses to Columns: understand
start charging what the error
Method
(RFID/App/Sms) is

KPI 9. Table: If there is a drop in a

Number of Bill number of created bills,

payments/ there might be a possible

Columns:
problem
Created
 45

number of bills Status

created correctly

Table 5 KPIs, data for their calculation, KPIs interpretation, and data problems.

Such variables as average occupancy of charging points, charge time and connection time
ratio, and database update frequency are not included in the set of identified KPIs. These
variables can be indicators of the charging points problems, but they do not measure the
critical processes and cannot create a value for the company.

Average occupancy of charging points was not chosen as a KPI because it is similar to
previously described usage frequency KPI. Exactly the same data (table: charge; columns:
station ID, start_time, stop_time) is required for calculation of both usage frequency and
average occupancy of charging points.

Charge time and connection time ratio can show when a car is connected to a charger, but
the energy is not being transferred. This variable is too weak to become a KPI, because it
does not measure a process critical for the company.

Database update frequency is related to charger’s status data update frequency. It can be set
differently for each charging point and varies from one 1 to 15 minutes. It means that
charging points send signal that they are online every 1- 15 minutes. This signal is also
defined as “heart beat” of a charger. And if a charger has not sent a signal 3 times in a row,
it is considered to be offline. Database update frequency was not chosen as a KPIs because
it is directly related to another KPI – the number of offline stations. Thus, there is no need
to monitor how often the database is updated, it is enough to track the number of online
stations.

4.1.6 Data evaluation for KPIs calculation.

After KPIs identification the database of the case company was analyzed in order to evaluate
the quality and availability of data that can be used for further KPIs calculation. The second
and the fourth columns of Table 5 include information about data. After describing the
 46

selected KPIs, it is necessary to discuss the data quality and availability at the case company.
Wang & Strong, 1996 described data quality as “fitness for use” and emphasized that quality
of data is judged by particular data consumers.

The second column presents the list of the tables and columns that include the data used for
KPIs calculation. All the KPIs can be calculated using the available data from the database
except the number of hotline calls. It can be concluded that in general the company’s
database stores a lot of valuable data that can be used for analysis and KPIs identification.

The remarks related to data quality are listed in the last column of Table 5. First of them is
about the kWh data stored in the database. It is not always correct and can contain zeros or
negative numbers. In some cases, the chargers do not send the kWh data or send incorrect
data. In other cases, the incorrect numbers are displayed due to the changes made in the
database manually. All that complicates the usage frequency calculation and causes
inaccurate results.

Another data problem is related to start time and stop time of a charging event that can be
exactly the same. It can have two meanings such as the unsuccessful charging attempt or the
incorrect data sent by stations.

Furthermore, stations send error notifications, but it is problematic to understand their

meaning. For instance, some charging points send such notifications as “other error” or
“OCPP error” that are not clear. As a result, it is complicated and time consuming to
investigate what is the actual problem.

Test data is stored in the same data tables as data from normal chargers and has to be filtered
for analysis. But sometimes the data from test stations is displayed in company’s Admin Panel
visualizations that can distort the presented results. For instance, test data is included in
today’s overview of charging network that gives incorrect information.
 47

4.1.7 Validation of identified KPIs.

Several researchers (Scholten 2017, Ella Roubtsova 2013 and Vaughan Michell 2013) in
their work applied the KPIs validation framework presented by Kitchenham (1996), Kueng
(2000), Winchell (1996). These researchers performed validation of KPIs by testing them
against the following properties: quantifiability, sensitivity, congruency, reliability,
efficiency and improvement-orientation.

§ Quantifiability of KPIs means that they have to be presented in a quantifiable form.

In the case of this study, quantifiability means that it is possible to get the data for
calculation of selected KPIs.
§ Sensitivity means that KPIs have to be sensitive to possible changes. Even small
changes in the values of the database influence the value of the performance
measured through the indicator.
§ Congruency or linearity means that changes in performance have to be in line with
the indicator’s value via a clear mathematical relationship. For instance, if a value
changes negatively, KPI changes negatively too.
§ Reliability means that KPIs are semantically reliable, free from subjective factors
and measurement errors. Reliable KPIs specifically measure what they claim to
measure.
§ Improvement-orientation KPIs are referred to possibility to improve the system or
process in the future. Conformance is not the goal of improvement-oriented KPIs.

The KPIs determined for the case company were tested using the properties described above.
Table 6 shows the obtained validation results. If cell is colored in green color, it means that
a KPI complies with a property that is shown on the left side of the table. Gray color indicates
the opposite result. (Table 6)
 48

KPI KPI KPI KPI KPI KPI KPI KPI KPI

1 2 3 4 5 6 7 8 9
Quantifiability

Sensitivity

Congruency

Reliability

Improvement-
orientation

Table 6 Checklist for KPIs validation.

The list of identified KPIs is the following:

KPI 1: Usage frequency of chargers.

KPI 2: Number of hotline calls/complaints.
KPI 3: kWh charged.
KPI 4: Number of frequent users per charging station(s).
KPI 5: Total number of offline stations.
KPI 6: The option that customer uses to start charging (RFID/App/SMS).
KPI 7: Number of errors in Admin System.
KPI 8: Number of long charges.
KPI 9: Number of payments/ the amount of bills created correctly.

All of the identified KPIs comply with quantifiability and sensitivity properties. It means
that the data for their calculation is available and they are sensitive to changes in the values
of the company’s database.

KPI 2 is not in line with the indicator’s value via a clear mathematical relationship, thus it is
not congruent at the moment. KPIs 4,7 an 8 do not fully satisfy the reliability property,
 49

because of the possible errors in the case company’s database. These KPIs can also point out
the regular situations such as distant location of a charger. KPIs 4 and 8 are not improvement-
oriented, but they are still important for measuring the reliability of charging points.

If there is a possible problem to comply with one of the properties, the validation is not
considered to be passed. Gray color was used even if some KPIs point a regular situation
that cannot be filtered from a charging problem. However, KPIs that currently do not comply
with some of the properties can be improved in the future and pass the validation.

4.2 Failure rate calculation

One of the selected KPIs is the number of errors in admin system that requires the failure
rate calculation. Failure rate can be described “as the anticipated number of times that an
item fails in a specified period of time. It is a calculated value that provides a measure of
reliability for a product.” (Greeff, Ghoshal, 2004) The Greek letter lambda ( 𝜆 ) represents
the failure rate that is calculated by taking the number of failures that occurred in a specific
length of time and dividing it by the length of time. (Jones, 2006) The formula of failure rate
calculation is presented below:

𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑓𝑎𝑖𝑙𝑢𝑟𝑒𝑠
𝜆=
𝑡𝑜𝑡𝑎𝑙 𝑚𝑒𝑎𝑠𝑢𝑟𝑒𝑑 𝑢𝑠𝑎𝑔𝑒

Figure 6 The formula for failure rate calculation (Jones, 2006).

In the case company failure rate is the measure of reliability of charging points and it can be
calculated as the percentage of time the station is faulted in a period of time. The
management of the company requested the failure rate calculation model to be done in
Microsoft Excel. The created model is presented in Appendix 1.

For the failure rate calculation, the “station status notification” table was used, and the
following data was required:

§ Station identification number.

 50

For each station a unique identification number is assigned, and it can contain from
four to six numbers.
§ Status of a charger.
There are three types of status notifications in the company’s database: available,
faulted, and occupied.
§ Timestamp.
The timestamp indicated the date and time when the charger became available,
faulted or occupied.

The failure rate calculation for the case company is based on the “faulted” charging events
or error that the station sends. According to database, chargers send data about its status and
a timestamp, that defines when the error happened. But the data table does not contain data
related to time when the error was eliminated. Thus, it is not possible to calculate the period
of time the station was faulted.

However, it is possible to estimate for how long the station was faulted using time intervals.
It was decided to create 15 minutes time slots for one month (31 days) and calculate the
number of 15 minutes slots when the fault was reported. It means that if the fault was
registered at 0.03, it has to be assigned to the time period of 00.00 - 00.15.

For the case company the failure rate calculation formula is the following:

𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑓𝑎𝑢𝑙𝑡𝑒𝑑 15 𝑚𝑖𝑛𝑢𝑡𝑒𝑠 𝑠𝑙𝑜𝑡𝑠

λ4 =
𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 15 𝑚𝑖𝑛𝑢𝑡𝑒𝑠 𝑠𝑙𝑜𝑡𝑠 𝑖𝑛 𝑎 𝑚𝑜𝑛𝑡ℎ

Figure 7 The formula for failure rate calculation in the case company.

The charger number 3039 was used as an example for failure rate calculation in a month of
December 2018. To calculate the number of 15 minutes slots when faults were reported the
columns that represents the time slots were created in Excel. All in all, the charger reported
the faulted status 1436 times and all of these faults were assigned to created time slots. Then
the sum of the time slots when faults were reported was calculated and the result was 106.
 51

As mentioned before in the case company failure rate is the percentage of time the station is
faulted in a period of time. In this calculation the period of time is one month. Thus, to obtain
the failure rate in December 2018 the number of 15 minutes slots when fault was reported
was divided into the number of 15 minutes slots in December 2018. It has been already
calculated that the sum of the time slots when faults were reported is 106 and the number of
15 minutes slots in December is 2976. Thus, when dividing 106 by 2976 the result is 3,46
percent.

Each charging point has its own failure rate. And by calculating the average number of the
individual failure rates of charging points, the average failure rate of the whole charging
network can be calculated.

As the calculation described above is not precise enough, it was decided to add the second
timestamp to the “station status notification” table in the database. The developers will work
on that in the following months. The timestamp will indicate the time of error elimination
that will help to calculate the duration of time the station was “faulted”.

4.3 Admin Panel improvement

Admin Panel is a system that provides various tools for managing and monitoring the
charging points. Case company’s employees and customers can access different reports,
statistics, and inform about problems using Admin Panel.

As stated earlier, Kibana was chosen as a tool for dashboard creation of KPIs for charging
points. But currently only a limited number of case company’s employees are using this tool
and the clients do not have access to it at all. Furthermore, it has not been decided yet if
Kibana will be used in the company in the future. Hence, together with the company’s
manager it was decided to make a proposal for changes in Admin Panel that is the commonly
used tool by both employees and customers.
 52

4.3.1 Current state of maintenance section in Admin Panel

Currently the maintenance section in Admin Panel presents the review of charging network
performance and consists of three parts. The first part presents the today’s overview, such
as:

§ Latest error message,

§ Latest station went offline,
§ Latest zero kWh event,
§ Latest charge started, and
§ Latest messaging error.

The second part of the maintenance section includes statistics presented in the following
charts:

§ Total error messages from stations.

§ Count of offline stations.
§ Total zero kWh events from stations.

The last part of the maintenance section includes the stations uptime information that is
shown in a form of a data table that contains uptime percentage, duration and number of
disconnects.

4.3.2 Maintenance section improvement proposal

The improved maintenance section will include four parts instead of three. The first part will
not be changed and as the current version of maintenance section it will present the todays
overview. The second part will also contain the charts, but their type will be different. The
following visualizations will be shown:

§ The failure rate of a total network in a period of time.

§ The uptime/downtime percentage ratio of a total network. (Appendix 2)
 53

The third pat of maintenance section will display the network KPIs scorecard. It is a form of
reliability report that presents reliability results on each station and includes (Appendix 3):
§ the station’s identification number
§ the number of charging sessions,
§ the number of errors,
§ percentage of time the station was offline,
§ kWh charged, and
§ profit generated by stations.

The last part stations that shows the station uptime information will remain the same. The
filter that will allow choosing the station model has to be added to the Stations Uptime part
of the Maintenance section.
 54

5. Visualization of KPIs in the case company

5.1 Dashboard visualization

Visual management is used by many organizations in order to improve performance through

connecting and linking vision, values, goals and culture of the company with other
management systems, processes, and stakeholders. (Liff and Posey, 2004).

Dashboard is a common way to visualize key performance indicators in the form of graphs,
gauges, or charts. Dashboards can be built in a different way, but all of them have the same
goal – to provide the relevant information to the right people quickly. Usually dashboards
are created by KPIs users to visualize what has to be done to improve their business.
(Rasmussen et al., 2009) Bose (2006) gives the similar definition of a dashboard, he states
that dashboard is a software application that provides a single screen layout of the most
important data that enables managers to take the necessary decisions faster. Malik (2005)
states that dashboards are essential for growing companies as the amount of data is rapidly
growing and it requires information flow and decision-making process improvement. As per
Pauwels et al. (2009) dashboards have three main objectives such as monitoring, planning
and communicating the obtained results.

Nowadays a big variety of KPIs dashboard tools such as Tableau, Excel, Qlik Sense,
QlikView are available. Kibana is a software that is currently used for data visualization and
monitoring at the case company. As requested by the case company, KPIs have to be
visualized in a dashboard. In the next sub chapter, the KPIs visualization tool that is used in
the company will be discussed.

5.2 Kibana as a tool for visualization

Kibana (Elastic.co., 2018) is an open source data visualization tool for Elasticsearch that can
be easily accessed with a browser. The benefits of Kibana include interactive charts and
reports such as histograms, line graphs, pie charts, heat maps. Moreover, it provides
geospatial capabilities that allow to present geographical locations in maps. Another feature
is pre-build aggregations and filters, that makes data analysis and visualization easier and
 55

faster. With Kibana it is also possible to create various dashboards and share them with
others. Moreover, the tool provides such options as unsupervised machine learning and
advanced time series analysis and allows to customize the data using Canvas feature.

It is possible to customize the dashboard according to the needs of the company and
customers. Furthermore, the speed of producing different types of visualizations in Kibana
is relatively fast and efficient. Figure 9 represents the example of a dashboard created in
Kibana.

Figure 8 The example of a dashboard created in Kibana using global flights data. (Retrieved from:
https://www.elastic.co/guide/en/kibana/current/dashboard.html [Accessed: 05.01.2019])

5.3 KPIs Dashboard in the case company

After the KPIs are identified and validated, the process of visualization can start. The case
company requested KPIs to be presented in a form of dashboard using the company’s
software. This process and visualization results are presented in this chapter. As described
in the Chapter 4.1.5, the following KPIs were defined for the monitoring EV charging points
reliability:
 56

§ KPI 1: Usage frequency of chargers.

§ KPI 2: Number of hotline calls/complaints.
§ KPI 3: kWh charged.
§ KPI 4: Number of frequent users per charging station(s).
§ KPI 5: Total number of offline stations.
§ KPI 6: The option that customer uses to start charging (RFID/App/SMS).
§ KPI 7: Number of errors in Admin System.
§ KPI 8: Number of long charges.
§ KPI 9: Number of payments or the number of bills created correctly.

However, at the moment the following KPIs cannot be visualized in Kibana:

§ Number of hotline calls or complaints

As mentioned before hotline service is outsourced and data received from hotline service
company is not stored in Elasticsearch. It is received by Support team once a week and stored
in Excel files in the company’s Dropbox.

§ Number of payments or the number of bills created correctly

The data related to created and paid bills is not currently available in Elasticsearch and as a
result it cannot be visualized in Kibana.

The following KPIs were visualized using time series visual builder: kWh charged, usage
frequency, the number of offline stations, the option that customer uses to start charging
(RFID/App/Sms), the number of errors in Admin System, and the number of long charges.
Furthermore, for some of them additional visualizations such as data table, pie chart,
coordinate map, and data table were created.

The table below includes KPIs that were visualized in Kibana and the types of their
visualization.

KPI The type of visualization in Kibana

kWh charged 1. Time series visual builder,
 57

2. Coordinate map

Usage frequency 1. Time Series Visual Builder,

2. Data table

Total number of offline stations 1. Time series visual builder,

2. Data metric

The option that customer uses to start charging 1. Time series visual builder
(RFID/App/Sms)

Number of errors in Admin System 1. Time series visual builder,

2. Pie chart

Number of long charges 1. Time series visual builder

Table 7 The types of visualizations created in Kibana.

KPIs dashboard combines different types of charts and structured in the same order as
presented in the table. The examples of created Kibana visualizations are presented below.

Figure 9 kWh charged heat map.

 58

The map displayed in Figure 9 shows in red color the geographical location of places where
the highest amounts of kWh are charged.

Figure 10 Usage frequency per station in the last 24 hours.

Figure 10 presents the usage frequency of charging points per station in the last 24 hours. The
station names on the right side of the visualization are blurred.

Figure 11 Usage frequency data table.

 59

The data table shown in Figure 11 was created to visualize usage frequency KPI. It shows
station ID, name and the number of charging events that happened in the last 24 hours.

Figure 12 The option that customer uses to start charging in the last 24 hours.

Figure 12 presents such methods of charge initiation as RFID, admin, App, onetime, and
SMS. Admin option means that case company’s support team started a charging event.
Onetime payment option is used by the customers that are not registered as case company’s
customers and payed online using their bank card.
 60

6. Conclusions and recommendations

6.1 Conclusions of the study

The case company is continuously working on improving EV charging experience for their
customers. And the topic of charging points reliability is an important issue that takes a lot
of attention from the management team of the company. This study’s aim was to create a
value for the case company by identifying and visualizing EV charging points KPIs for
reliability monitoring.

In terms of the thesis practical contribution to the case company:

1. KPIs for EV charging points reliability tracking were identified,

2. EV charging problems were recognized,
3. database analysis was performed and as a result data errors were discovered, and data
that has to be obtained was defined,
4. KPIs dashboard for EV charging points reliability tracking was built,
5. recommendation for further improvement were stated.

One main research question and four sub questions were defined and introduced in the
beginning of the thesis. In order to make conclusions of the study, the answers to these
questions are provided in this chapter. Figure 14 shows the outline of research questions and
answers.

1. What set of KPIs has to be used at the case company in order to measure reliability
of charging points?

For the case company a set of nine key performance indicators was created. They cover such
areas as station errors, frequency of charging and number of frequent users, monitoring and
reports sent by stations, energy transfer, and the type of initiation or completion of charging
events. The KPIs are the following: the usage frequency of chargers , the number of hotline
calls/complaints, the amount kWh charged, the number of frequent users per charging
 61

station(s), the number of offline stations, the option that customer uses to start charging
(RFID/App/SMS), the number of errors in admin system-failure rate, the number of long
charges, the number of payments or bills created correctly. All together these KPIs give an
overview of reliability of EV charging points and help to spot the negative changes. After
the KPIs are implemented and tracked for a period of time, they should be evaluated again.
If needed, they can be improved, eliminated, or replaced with more suitable KPIs.

1.1 How to define a suitable set of KPIs for the case company?

To answer this question the case company’s charging network was analyzed first and then
different methods for KPIs identification were evaluated. Based on that it was decided to
create a unique approach that will help to identify KPIs for measuring reliability of charging
points. The approach is created in a way that it can be applied in other EV charging
companies if needed. It included seven steps and focused on identifying KPIs specifically
for tracking reliability of charging points. It was supported with literature review of EV
charging, performance measurement and various methods for KPIs creation. The approach
that was described in detail in chapter 4 combined the idea of current and future state
evaluation, data analysis and KPIs validation.

The KPIs identification process started with defining reliability in the case company and as
a result the term included seven characteristics. Then the goals for reliable charging stations
were defined that formed a desired level of reliability in the future. After that the list of EV
charging points was composed that gave an overview of the current situation. Database
analysis was performed in order to assign the variables from the database for the defined
reliability goals. Data errors were also defined during this step. A set of nine KPIs was
selected using S.M.A.R.T. method and then data was defined for their calculation. The final
step included KPIs validation by testing them against the following properties:
quantifiability, sensitivity, congruency, reliability, efficiency and improvement-orientation.

1.2 What is the appropriate way to visualize the selected KPIs?

Dashboard is a popular way of KPIs visualization that was also used in the case company.
KPIs dashboard for the case company was composed using Kibana that offers various
 62

visualization opportunities for instance data tables, line graphs, pie charts, and heat maps.
As shown in chapter 5.3, for the KPIs dashboard such types of visualization as time series
visual builder, data table, coordinate map, and data metric were chosen. Most of identified
KPIs were displayed using time series visual builder and then supported by other types of
visualizations depending on the nature of a KPI. The reason to select time series visual
builder was that the KPIs have to be measured on daily basis and their results have to be
compared to previous periods of time. Not all of the KPIs were visualized in a dashboard
because not all of the data is available in Elasticsearch at the moment. Furthermore, Kibana
is not the best tool for KPIs tracking that is more discussed in sub chapter 6.2 that discusses
recommendations of this study.

1.3 What improvements can be proposed based on identified KPIs?

This thesis offers two improvement proposals related to failure rate calculation and
maintenance section in Admin Panel. The failure rate calculation was done using 15 minutes
intervals that provides a percentage of time the EV charging network was “faulted”. The
more precise calculation is not possible at the moment due to nonexistence of required data.
This calculation can be improved by adding the second timestamp that indicates the time of
error elimination. If we know the duration of time the station was “faulted”, the failure rate
calculation can be done easily. A detail description of a proposed failure rate calculation can
be found in chapter 4.2.

Maintenance section in Admin Panel displays an important information regarding charging

network that is used by the case company’s employees and customers. It has to be improved
in order to provide more insights on reliability and also display different type of
visualizations. First, KPIs scorecard has to be added to the maintenance section. And then
the current visualizations have to be substituted with the ones that provide information about
failure rate and uptime percentage of a total network.

The company has already started a project that aims to make the changes that were proposed
in this thesis. The developers are currently working on making Admin Panel improvements
that were described more thoroughly in Chapter 4.3. Furthermore, the necessary data will be
added to database that is required for calculating failure rate of EV charging network.
 63

1. What set of KPIs has to be used at the case company in

order to measure reliability of charging points?

KPI 1: Usage frequency of chargers

KPI 2: Number of hotline calls/complaints
KPI 3: kWh charged
KPI 4: Number of frequent users per charging station(s)
KPI 5: Total number of offline stations
KPI 6: The option that customer uses to start charging (RFID/App/SMS)
KPI 7: Number of errors in Admin System
KPI 8: Number of long charges
KPI 9: Number of payments/ the amount of bills created correctly

1.1 How to define a suitable set of KPIs for the case company?

The approach for KPIs identification includes the following steps:

1.The definition of reliability of charging points
2.Reliability goals setting and result indicators recognition
3.EV charging points problems identification
4.Assigning the variables from the database for the defined reliability goals
5.KPIs identification based on S.M.A.R.T. method
6.Data evaluation for KPIs calculation
7.Validation of identified KPIs

1.2 What is the appropriate way to visualize the selected KPIs?

• KPIs visualization in a form of a dashboard

• Visualization types are: time series visual builder, data table, coordinate
map, and data metric

1.3 What improvements can be proposed based on identified

KPIs?

• Changes in maintenance section in Admin Panel

• Failure rate calculation model implementation

Figure 13 The outline of research questions and answers.

 64

6.2 Recommendations for further development

The improvement proposals related to Admin Panel and Failure rate calculation were already
introduced in Chapters 4.2 and 4.3. In this chapter recommendations concerning other topics
are described.

Kibana is a tool that has been tested in the company for a period of six months. It allowed to
create most of the defined KPIs, but it can be concluded that it is not the best software for
KPIs dashboard creation. The reason is that it is not possible to make a visually appealing
time series that will present the comparison of values obtained. For example, it is
complicated to create a chart that will visualize and present the clear difference in kWh
charged this and the last month. It is already investigated by the company’s management
with what tool to substitute Kibana for internal monitoring of charging network and
dashboard visualization.

Furthermore, the dashboard is intended to be used not only internally but also by case
company’s customers. It is crucial to communicate the KPIs dashboard in an efficient way
so it will be understood and used by different user groups. For this purpose, the KPIs have
to be integrated into Admin Panel that is actively used by customers. KPIs can be presented
in a form of reliability report and visualizations that will provide the most valuable
information related to charging points usability and reliability. The style of reports and charts
has to be simple, visually appealing and it also has to be easily found in one of the sections
in Admin Panel.

Defined and visualized KPIs for EV charging network reliability cannot bring any real value
to the company if they are not shared and communicated to the responsible stakeholders.
KPIs dashboard immediately shows the occurring problems that simplifies the process of
charging network monitoring. It also allows to take faster decisions and promptly react to
changes.

As for the possible changes in Admin panel, Business Intelligence (BI) section also requires
some improvements. The section consists of a big variety of reports in a form of
visualizations such as active customers and charging events per country. The presented
 65

information is definitely important, but the style of the charts is not easily understandable
and visually attractive. The change of charts type and the order of their location on the page
will make the BI section more useful and user friendly.

One of the most important KPIs that was defined is the number of errors sent by charging
points. The case company monitors various errors that are differentiated by type. The error
types are set by charger’s manufacturers and may vary. The problem is that it is not always
understandable what the error actually means and how severe it is. The research related to
the error classification has to be done that will help to better understand the error
notifications and be able to react to them in a short period of time. Moreover, it will be useful
for further selection of the most reliable manufacturers of hardware.

Error notifications is not the only topic that needs attention. It also has to be defined on the
company level what is abnormal event and what charging events belong to this category. It
is also critical to identify such terms as long charge, downtime, failure rate, and some others.
These terms have to be documented and explained to company’s employees, partners and
also customers.

It was stated in Chapter 6.1 that the proposed failure rate calculation can be modified by
adding the second timestamp that indicates the time of error elimination. If it is done, the
precise failure rate can be calculated easily. The ticket that describes this required change
has already been created and explained to developers.
 66

7. Summary

The case company is an EV charging services provider that connects more than 8000
charging stations globally. The company is focused on increasing performance of their
charging network and improving the quality of their services. This Master’s thesis was
commissioned by Services and Operations team in order to identify a set of KPIs that can be
visualized and further used specifically for monitoring EV charging points reliability. First,
the theoretical part of the study was conducted and then followed by the practical part of the
study. The literature review of EV charging and performance measurement was introduced
and followed with available frameworks for KPIs determination. Such methods as practical
observation and interviews with company’s employees were used to learn about the
company and the industry of its operation. Then the data analysis part started and the created
approach for KPIs identification was introduced and described. The big part of the project
required working with company’s database and with the tool used for charging points
monitoring.

The final results of the research include the KPIs that can be used for monitoring of EV
charging points reliability. In the beginning of the project the available recourses were
limited to company’s database and documentation. Furthermore, there is a lack of
information concerning the methods for creating EV charging points reliability KPIs. Thus,
the method had to be created from scratch based on EV charging company specifications.

The approach proposed in this thesis consists of 7 steps that aim to determine KPIs for
monitoring of EV charging points reliability. These steps can be also used as a guideline for
other EV charging companies that plan to track the reliability of their stations by applying
KPIs. The created approach combined the evaluation of current and future state of charging
points, data analysis, S.M.A.R.T. method and KPIs validation.

This study is focused on providing a practical contribution to the case company by defining
EV charging problems and goals, executing data analysis and discovering data errors,
creating KPIs for EV charging points reliability tracking, validating and visualizing them.
The presented approach can be also applied in other EV charging companies. KPIs
dashboard does not present all the identified KPIs due to the reason that not all the data is
 67

available for visualization. The recommendations for further improvement of dashboard,

Admin Panel, and other issues related to charging points reliability were proposed in the end
of the study. The improvement proposals that are described in Chapter 4.3 will be executed
in the following months.
 68

References
Anand, N., and Grover, N., 2015. Measuring retail supply chain performance: Theoretical
model using key performance indicators (KPIs). Benchmarking: An International Journal,
22(1), pp.135-166.

Bauer, K., 2004. KPIs-The metrics that drive performance management. Information
Management, 14(9), p. 63.

Baxter, P. and Jack, S., 2008. Qualitative case study methodology: Study design and
implementation for novice researchers. The qualitative report, 13(4), pp.544-559.

Bose, R., 2006. Understanding management data systems for enterprise performance
management. Industrial Management & Data Systems, 106(1), pp.43-59.

Bunsen, T., Cazzola, P., Gorner, M., Paoli, L., Scheffer, S., Schuitmaker, R., Tattini, J. and
Teter, J., 2018. Global EV Outlook 2018: Towards cross-modal electrification.

Cazzola, P., Gorner, M., Schuitmaker, R. and Maroney, E., 2017. Global EV outlook 2017:
Two million and counting.

Clean Technica., 2016. Electric Car Drivers: Desires, Demands, & Who They Are. [online]
[Accessed on 20.10.2018] Available at: https://c1cleantechnicacom-wpengine.netdna-
ssl.com/files/2017/05/Electric-Car-Drivers-Report-Surveys-CleanTechnica-Free-
Report.pdf

Davies, M. R., 2007. Doing a successful research project: Using qualitative or quantitative
methods. Basingstoke, UK: Palgrave Macmillian.

Deason, K., 2010. Governmental Programs on E-Mobility. In IPHE Workshop Report, Ulm.

Doran, G. T., 1981. There’s a S.M.A.R.T. Way to Write Management’s Goals and
Objectives. Management Review, vol.70, pp. 35-36.
 69

Durkacova, M., Lavin, J. and Karjust, K., 2012. KPI optimization for product development
process. In Annals of DAAAM for 2012 & Proceedings of the 23rd International DAAAM
Symposium (Vol. 23, No. 1, pp. 1079-1084).

Eckerson, W.W., 2005. Performance Dashboards: Measuring, Monitoring and Managing

Your business, second edition, pp. 209-212.

Eckerson, W.W., 2009. Performance Management Stratagies: How to Create and Deploy
Effective Metrics. The Data Warehouse Institute, pp. 3, 18-23.

Elastic.co., 2018. Kibana: Explore, Visualize, Discover Data | Elastic. [online] [Accessed on
7.11.2018] Available at: https://www.elastic.co/products/kibana.

Emerson, R.M., Fretz, R.I. and Shaw, L.L., 2001. Participant observation and fieldnotes.
Handbook of ethnography, pp.352-368.

European Alternative Fuels Observatory. Europe., 2018. [online] [Accessed on 8.11.2018]

Available at: http://www.eafo.eu/europe.

Franceschini, F., Galetto, M. and Maisano, D., 2007. Management by measurement:

Designing key indicators and performance measurement systems. Springer Science &
Business Media.

Greeff, G. and Ghoshal, R., 2004. Practical E-manufacturing and supply chain management.
Elsevier.

Greentechmedia.com., 2018. Up to 40 Million EV Charging Points Forecast Worldwide by

2030. [online] [Accessed on 25.10.2018]. Available at:
https://www.greentechmedia.com/articles/read/electric-vehicle-charging-points-40-
million-gtm.

Helmus, J. and Van den Hoed, R., 2016. Key performance indicators of charging
infrastructure. World Electric Vehicle Journal, 8(4), pp.733-741.
 70

Jackson, C., 2016. Solar car parks—A guide for owners and developers. BRE National Solar
Centre [online] [Accessed on 19.11.2018] Available at: https://www.r-e-a.net/upload/rea-
bre_solar-carpark-guide-v2_bre114153_lowres.pdf.

Jones, J. V., 2006. Integrated logistics support handbook. New York: McGraw-Hill.

Kadarsah, S., 2007. Key performance indicators measurement model based on analytic
hierarchy process and trend-comparative dimension in higher education institution.
In International symposium on the analytic hierarchy process (ISAHP), Vi na Del Mar,
Chile (Vol. 3).

Kaplan, R.S. and David, P. Norton.,1996. The balanced scorecard: translating strategy into
action.

Kececioglu, D., 2002. Reliability engineering handbook (Vol. 1). DEStech Publications,
Inc, pp. 61-62.

Kitchenham, B. A.,1996. Software metrics: measurement for software process

improvement: Blackwell Publishers, Inc.

Kueng, P., 2000. Process performance measurement system: a tool to support process-based
organizations. Total Quality Management, 11(1), pp.67-85.

Leal Filho, W. and Kotter, R. eds., 2015. E-Mobility in Europe: Trends and good practice.
Springer.

Leedy, P., 1993. Practical Research: Planning and Design, vol. 5. Macmillan Publishing
Company, New York.

Liff, S. and Posey, P.A., 2004. Seeing is believing: how the new art of visual management
can boost performance throughout your organization. AMACOM/American Management
Association.
 71

Malik, S., 2005. Enterprise dashboards: design and best practices for IT. John Wiley & Sons.

Marr, B., 2015. Key performance indicators for dummies. England: John Wiley & Sons.

McCall, G. J., & Simmons, J. L., 1969. Issues in participant observation: A text and reader.
Reading, MA : Addison-Wesley.

Ministry of Economic Affairs and Employment of Finland., 2017. TEM tukee yhtätoista
energiateknologian kärkihanketta vuonna 2017 [online] [Accessed on 8.11.2018] Available
at: https://tem.fi/artikkeli/-/asset_publisher/tem-tukee-yhtatoista-energiateknologian-
karkihanketta-vuonna-2017.

Neely, A., Richards, H., Mills, J., Platts, K. and Bourne, M., 1997. Designing performance
measures: a structured approach. International journal of operations & Production
management, 17(11), pp.1131-1152.

Nie, Y.M. and Ghamami, M., 2013. A corridor-centric approach to planning electric vehicle
charging infrastructure. Transportation Research Part B: Methodological, 57, pp.172-190.

Ning, Z., Jian-hai, W. and Jia-xin, W., 2011, August. Design Study on Human-Computer
Interface in Kpi-System of Enterprises. In International Conference on Applied Informatics
and Communication (pp. 189-195). Springer, Berlin, Heidelberg.

Parmenter, D., 2007. Key Performance Indicators. Developing, Implementing and Using
Winning KPIs. Hoboken, New Jersey: John Wiley & Sons.

Parmenter, D., 2015. Key performance indicators: developing, implementing, and using
winning KPIs. John Wiley & Sons.

Pauwels, K., Ambler, T., Clark, B.H., LaPointe, P., Reibstein, D., Skiera, B., Wierenga, B.
and Wiesel, T., 2009. Dashboards as a service: why, what, how, and what research is needed?
Journal of service research, 12(2), pp.175-189.
 72

Podgorski, D., 2015. Measuring operational performance of OSH management system–A

demonstration of AHP-based selection of leading key performance indicators. Safety
science, 73, pp.146-166.

Rahman, I., Vasant, P. M., Singh, B. S. M., Abdullah-Al-Wadud, M., & Adnan, N., 2016.
Review of recent trends in optimization techniques for plug-in hybrid, and electric vehicle
charging infrastructures. Renewable and Sustainable Energy Reviews, 58, 1039-1047.

Rasmussen, N.H., Bansal, M. and Chen, C.Y., 2009. Business dashboards: a visual catalog
for design and deployment. John Wiley & Sons.

Rasmussen, N.H., Bansal, M. and Chen, C.Y., 2009. Business dashboards: a visual catalog
for design and deployment. John Wiley & Sons.

Rautiainen, A., 2015. Aspects of electric vehicles and demand response in electricity grids.
Tampere University of Technology. Publication; Vol. 1327.

Roubtsova, E. and Michell, V., 2013, July. Modelling and validation of KPIs. In Proceedings
of the Third International Symposium on Business Modelling and Software Design, BMSD
2013, 8-10 July 2013.

San Román, T. G., Momber, I., Abbad, M. R., & Miralles, A. S., 2011. Regulatory
framework and business models for charging plug-in electric vehicles: Infrastructure, agents,
and commercial relationships. Energy policy, 39(10), 6360-6375. Chicago.

Scholten, C.F., 2017. Specification, Modeling and Validation of KPIs and their required
properties (Master's thesis, Open Universiteit Nederland).

Shahin, A. and Mahbod, M.A., 2007. Prioritization of key performance indicators: An

integration of analytical hierarchy process and goal setting. International Journal of
Productivity and Performance Management, 56(3), pp.226-240.
 73

Spitzer, D. R., 2007. Transforming performance measurement: Rethinking the way we

measure and drive organizational success. New York: Amacom.

Stake, R.E., 1995. The art of case study research. Sage.

Stricker, N., Echsler Minguillon, F. and Lanza, G., 2017. Selecting key performance
indicators for production with a linear programming approach. International Journal of
Production Research, 55(19), pp.5537-5549.

Sun, C.C., 2010. A performance evaluation model by integrating fuzzy AHP and fuzzy
TOPSIS methods. Expert systems with applications, 37(12), pp.7745-7754.

Tashakkori, Abbas, and Charles Teddlie., 2003. Handbook of mixed methods in social &
behavioral research. Thousand Oaks, CA: Sage.

Trip, J.J., Lima, J. and Bakker, S., 2012. Electric mobility policies in the North Sea Region
countries. Delft University of Technology.

Winchell, W., 1996. Inspection and measurement in manufacturing: keys to process

planning and improvement. Society of Manufacturing Engineers.

Wirges, J., 2016. Planning the charging infrastructure for electric vehicles in cities and
regions. KIT Scientific Publishing.

Yin, R. K., 1989. Case study research: Design and methods. Applied Social Research Series,
vol. 5. London: Sage.

Yin, R. K., 2009. Case Study Research: Design and Methods (4th ed.). Thousand Oaks, CA:
Sage.

Zhu, D., Patella, D.P., Steinmetz, R. and Peamsilpakulchorn, P., 2016. The Bhutan Electric
Vehicle Initiative: Scenarios, Implications, and Economic Impact. The World Bank.
 74

Appendix 1. Failure rate calculation model created in MC Excel

 75

Appendix 2. The uptime/downtime percentage ratio of a total network for months

from 1 to 12
 76

Appendix 3. Reliability report created in MC Excel