Applications of Business Analytics in Healthcare: Sciencedirect

Download as pdf or txt
Download as pdf or txt
You are on page 1of 12

Business Horizons (2014) 57, 571—582

Available online at www.sciencedirect.com

ScienceDirect
www.elsevier.com/locate/bushor

Applications of business analytics in


healthcare
Michael J. Ward a,^, Keith A. Marsolo b,^, Craig M. Froehle c,d,e,*

a
Department of Emergency Medicine, Vanderbilt University, 703 Oxford House, 1313 21st Avenue,
Nashville, TN 37232-4700, U.S.A.
b
Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center, 3333 Burnet Avenue,
Cincinnati, OH 45229-3026, U.S.A.
c
Carl H. Lindner College of Business, University of Cincinnati, 2925 Campus Green Drive, Cincinnati,
OH 45221-0130, U.S.A.
d
College of Medicine, University of Cincinnati, 3230 Eden Avenue, Cincinnati, OH 45267, U.S.A.
e
Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital Medical Center,
3333 Burnet Avenue, Cincinnati, OH 45229-3026, U.S.A.

KEYWORDS Abstract The American healthcare system is at a crossroads, and analytics, as an


Healthcare; organizational skill, figures to play a pivotal role in its future. As more healthcare
Analytics; systems capture information electronically and begin to collect more novel forms of
Information technology data, such as human DNA, how will we leverage these resources and use them to
improve human health at a manageable cost? In this article, we argue that analytics
will play a fundamental role in the transformation of the American healthcare system.
However, there are numerous challenges to the application and use of analytics: the
lack of data standards, barriers to the collection of high-quality data, and a shortage
of qualified personnel to conduct such analyses. There are also multiple managerial
issues, such as how to get end users of electronic data to employ it consistently to
improve healthcare delivery and how to manage the public reporting and sharing of
data. In this article, we explore applications of analytics in healthcare, barriers and
facilitators to its widespread adoption, and ways in which analytics can help us
achieve the goals of the modern healthcare system: high-quality, responsive, afford-
able, and efficient care.
# 2014 Kelley School of Business, Indiana University. Published by Elsevier Inc. All
rights reserved.

1. Healthcare: An industry in need of


* Corresponding author
analytics
E-mail addresses: [email protected]
(M.J. Ward), [email protected] (K.A. Marsolo),
The American healthcare system has long suffered
[email protected] (C.M. Froehle) from constrained resources, increasing demand,
^
See disclosure at end of article. and questionable value, yet the future looks more

0007-6813/$ — see front matter # 2014 Kelley School of Business, Indiana University. Published by Elsevier Inc. All rights reserved.
http://dx.doi.org/10.1016/j.bushor.2014.06.003
572 M.J. Ward et al.

promising due to increasingly sophisticated and advanced analytics. As the ACA is implemented,
widespread uses of data and analytics. Past perfor- incentives should be more in line with patient health
mance of the healthcare system provides insight and well-being while achieving value for limited
regarding why change was, and still is, necessary. healthcare resources.
The Centers for Medicare and Medicaid (CMS) esti- Although healthcare has taken longer than other
mate that healthcare represents a staggering 17.9% industries to incorporate the use of analytics, such
of U.S. gross domestic product (GDP) and that the adoption is radically transforming healthcare deliv-
United States spent $2.7 trillion, or $8,680 per ery for the better. In this article, we discuss how
person, on healthcare in 2011 (CMS, 2013a). Accord- healthcare is fundamentally changing in response to
ing to the Organization for Economic Co-operation the application of analytics. We also discuss how
and Development (OECD), which ranks the perfor- data are collected, organized, and analyzed as well
mance of international healthcare systems, the as the challenges facing the widespread adoption of
United States ranked 27th in life expectancy at birth analytics in healthcare. In addition, we discuss
in 2009 despite having the highest proportion of GDP managerial issues and ways in which analytics can
spent on healthcare (OECD, 2011). produce a meaningful output for organizations and
This raises questions regarding the value of the individuals alike. Finally, we conclude with specific
U.S. healthcare system. There are multiple reasons examples illustrating the application of analytics to
for this value deficit. First, the third-party-payer healthcare delivery. We use examples from the
system decouples the payer from the individual re- visualization of data in quality improvement, genet-
ceiving services, mitigating some checks and balan- ics, comparative effectiveness, chronic disease da-
ces on costs. Second, there is a lack of aligned tabases, disaster planning, and asset tracking to
incentives in the existing fee-for-service system, demonstrate how the application of analytics to
which promotes consumption of resources and over- healthcare is improving the way healthcare is deliv-
use rather than overall patient health and well- ered and to demonstrate the unique analytical
being. Third, there are unique barriers to competi- issues this application raises.
tion that prevent innovation, which are not present in
other industries. Fourth, for-profit insurers, fraud,
and waste divert a portion of healthcare funds away 2. The analytics process in healthcare
from paying for care. Finally, despite information
technology’s (IT) role in rapidly advancing the pro- 2.1. Data generation
ductivity of many other industries (e.g., Rawley &
Simcoe, 2012), IT adoption in healthcare has sorely There is a tendency for hospitals and healthcare
lagged behind. systems to operate and manage a wide range of
The Affordable Care Act (ACA) and the Health clinical and operational information systems. While
Information Technology for Economic and Clinical the interoperability requirements of ‘meaningful
Health (HITECH) Act–—a component of the American use’ (MU) (Blumenthal & Tavenner, 2010) are causing
Recovery and Reinvestment Act (ARRA) of 2009–— institutions to consolidate their clinical information
have initiated tremendous change in healthcare. systems into enterprise-wide EHRs (Marsolo &
Fueled by the carrot-and-stick approach of the Spooner, 2013), most institutions still rely on a host
HITECH Act, hospital adoption of at least basic of platforms. While not an exhaustive list, examples
electronic health records (EHRs) has nearly doubled of such platforms are described next and are sum-
from 2008 to 2012, with 44% of U.S. hospitals using marized in Table 1:
at least a basic EHR (DesRoches et al., 2013). With-
out an EHR, much healthcare data are contained in  EHRs–—EHRs have become one of the largest sour-
paper format. Widespread EHR adoption sets the ces of digital information on the health and well-
stage for electronic data collection and subsequent being of patients. Spurred in part by the ARRA and
analysis. The next phase entails transforming these MU, the rate of EHR adoption has grown dramati-
data into actionable information packets that can cally (DesRoches et al., 2013; HealthIT.gov, 2013).
be used to improve healthcare delivery. EHRs are used to capture family, social, surgical,
Now that the necessary data pieces are being put and medical history; allergies and immunizations;
into place, analytics can and must play a pivotal role laboratory results; clinical findings; clinical or-
in the transformation of American healthcare into ders; and other condition-specific information.
an efficient, value-driven system. With investments Depending on the configuration of the EHR, this
in healthcare IT implementation and a shift in focus information may either exist in discrete fields or
from quantity of treatment to overall healthcare be captured as part of free-text notes (Marsolo &
value, the stage is set for the application of Spooner, 2013).
Applications of business analytics in healthcare 573

Table 1. Example data sources within a healthcare delivery system


Data Source Data Generated
EHRs Clinical documentation, patient history, results reporting, and patient
orders.
LIMS Laboratory results. Typically interfaced with EHRs.
Diagnostic or monitoring Range from images (e.g., magnetic resonance imaging) to numbers
instruments (e.g., vital signs) to text report (result interpretation). May or may not
be interfaced with EHRs.
Insurance claims/billing Information on what was done to the patient during a visit, the cost of
those services and the expected payment. The level of service is often
determined from data in EHRs.
Pharmacy Information on the fulfillment of medication orders. Not typically part
of EHRs.
Human resources and Lists of employees and their roles in the institution and the location and
supply chain utilization of medical supplies. Not typically interfaced with EHRs.
Real-time locating systems Positions and interactions of assets and people.

 Laboratory information management systems patient actually took the medication as pre-
(LIMSs)–—An LIMS is used for processing laboratory scribed. Pharmacy refills are being supplanted
samples and for storing the interim and final by electronic pill bottles as a better way of deter-
results for a particular test. These systems typi- mining medication adherence (Aardex Group,
cally contain sample metadata (e.g., collection 2012).
date/time, container type, preservative) that are
useful for quality-assurance purposes.  Human resources and supply chain–—Many health-
care systems now use typical enterprise-level IT
 Instruments used for diagnostic or monitoring systems (e.g., PeopleSoft, SAP) to manage their
purposes–—These instruments range from magnet- human resources and supply chains. These are
ic resonance imaging or computed tomography typically not connected to the other systems
scanners to echo- and electrocardiograms and vital mentioned previously in this section.
sign monitors. The level of integration with these
instruments varies based on importance and the  Real-time locating systems–—Increasingly, hospi-
sophistication of the underlying system. Some in- tals and large healthcare organizations are inves-
struments will simply generate a text report that is ting in systems that provide the real-time location
transmitted to the EHR. Others may produce im- of assets (e.g., intravenous pumps) and/or people
ages or other raw data that can be used for analyti- (e.g., staff and patients) in order to better man-
cal purposes or, like radiology picture archiving and age operations (Froehle & Magazine, 2013). These
communication systems, for improving the man- systems locate the asset or person through some
agement of these imaging databases. combination of wireless technologies, such as
radio-frequency identification (RFID), Wi-Fi, ul-
 Insurance claims/billing–—These systems are used trasound, infrared, and global positioning sys-
to generate bills for services during each clinic or tems. Combined with management front ends,
hospital visit and keep track of what was paid by these technologies can reduce asset loss and theft
patients, insurance providers, and other payers. and improve the situational awareness of staff
who direct workflow.
 Pharmacy–—Until now, pharmacy information sys-
tems had rather limited uses regarding inventory 2.2. Data extraction
management. However, pharmacy information
systems are becoming increasingly sophisticated Most clinical information systems were not designed
to address clinical problems such as medication with analytics in mind and, as such, do not necessarily
non-adherence, a major reason for lack of im- make it easy to extract data. Systems typically sup-
provement in patient outcomes (Martin, Williams, port data transmission using Health Level Seven (HL7)
Haskard, & Dimatteo, 2005). In an outpatient messages (www.hl7.org), but only a fraction of the
setting, it is possible to determine whether an total information in the system may be accessible
order was placed for a particular medication, but using such an interface (Garrido et al., 2014). Sys-
it is more challenging to determine whether the tems may also provide a back-end reporting database
574 M.J. Ward et al.

that can be used for research and analytics, but there attempt to put patients more in control of their
may be a lag in how often that data is refreshed. health and their health data. Health systems will
Access to real-time data can be problematic. Typi- also be required to demonstrate the ability to ex-
cally, the only potential avenues available are (1) change patient records with other health systems in
HL7, which limits the data that can be accessed and their region, moving closer to the vision of patients
the types of questions that can be asked, and (2) web having a single record that contains all their health
services, which provide a richer interface, but the data. This means that it will become much easier to
details of which may be considered the vendor’s perform population-level analytics on ‘standard’
intellectual property and, therefore, may only be data elements (e.g., allergies, medication orders,
made available to its customers. surgical history, vital signs, diagnoses) that can be
While there is no shortage of data standards in exchanged via these mechanisms.
the healthcare industry, there is a distinct lack of
uptake of those standards by the health IT commu- 2.3. Analysis
nity. Within a given clinical information system,
vendors are free to define their own data structures The application of analytics in healthcare requires
and often do. The same element may be stored and the transformation of data into usable information
coded in myriad ways by vendors and sometimes that can be relayed back to end users. The adoption
even within different systems from the same vendor. of EHRs and other electronic data mechanisms makes
In one classic example, it was reported that there the application of analytical tools more tractable by
were more than 40 different ways of capturing blood providing the basic electronic data upon which to act.
pressure within a single EHR (Koppel, n.d.). All of This coincides with the rise of the ‘data scientist,’ a
them were valid within the clinical context in which term sometimes applied to those who use analytics
the measurements were taken. The only standards and can serve as a one-stop shop for the manage-
that do exist are those tied to payment: the Inter- ment, analysis, and interpretation of electronic data.
national Classification of Diseases (ICD) codes. In healthcare, this is particularly important for trans-
These codes are used as billing diagnoses (CMS, lating electronic bits into meaningful data.
2014) because they are required by CMS in order Data scientists often need to draw from a dizzy-
to receive payment from Medicare and Medicaid. ingly broad spectrum of analytical methodologies.
Changes in Medicare and Medicaid prompted the Well-established techniques, such as biostatistics
healthcare industry to move from ICD-9 to ICD-10 and epidemiologic analysis, Monte Carlo and dis-
in October 2014 (CMS, 2013b). crete-event simulation, and causal modeling, are
Despite the challenges detailed heretofore, ef- being joined with methods previously uncommon in
forts are underway to facilitate data sharing and healthcare. These newer methods include data min-
exchange through data-format standardization. The ing, Bayesian statistics, optimization modeling, so-
primary drivers behind this are the MU regulations. cial network analysis, and agent-based simulation,
They call for clinical findings to be coded to SNOMED just to name a few.
CT (International Health Technology Standards Devel- Analysis depends upon the context in which it is
opment Organisation, n.d.), laboratory results to be being performed. Clinical care and performance
coded to Logical Observation Identifiers Names and improvement can require very different data per-
Codes (LOINC) (LOINC, n.d.), and medication orders spectives and use the data in unique ways. Clinical
to be coded to RxNorm (National Library of Medicine, analytics involves improving patient care. This type
n.d.). Within the research community, such mappings of data is very different than process-oriented data
are standard when performing inter-institutional and may include genetic data as well as clinical
analyses; making them standard throughout the in- records, which are often narrative and may be more
dustry will significantly reduce the burden of sharing difficult to analyze on a large scale. Performance
data in a coherent fashion. data, on the other hand, may be subject to the
In addition to calling for the use of standard aforementioned issues; namely, availability and
terminologies, another key element of MU is to quality. Considering that EHRs were not designed
increase the interoperability and exchange of EHR with system performance in mind, figuring out how
data (Blumenthal & Tavenner, 2010). In Phase 2, to capture these data with high quality at a low cost
there are explicit measures to enable patients is a daunting yet fundamentally important task.
to view, download, and transmit their results.
Modeled after the Blue Button initiative (http:// 2.4. Visualization and reporting
www.healthit.gov/bluebutton), which was started
by several federal agencies to allow patients to view Traditionally, healthcare has used business data far
and download their personal health data, this is an less regularly and comprehensively than most other
Applications of business analytics in healthcare 575

industries. It has underinvested in advanced mana- use of sophisticated visualization techniques. A


gerial technologies like reporting systems and data growing amount of research has demonstrated that
visualization. This underinvestment may be partly users make better decisions, or at least are more
due to some healthcare providers viewing invest- confident in the decisions they make, when data are
ments in managerial and operational information provided to them in graphs or tables that are easy to
systems as less important than investments in clini- interpret and understand (e.g., Tait, Voepel-Lewis,
cal information systems. & Zikmund-Fisher, 2010). Advanced visualization
Whereas many organizations outside healthcare techniques can provide more consistent, clean,
have developed or purchased real-time reporting and unambiguous charts, which can improve the
systems that push targeted updates to specific end speed and reliability of users’ decision making. Com-
users, healthcare has typically relied on the central- bining these advanced visualization methods with
ized production of static, undifferentiated report real-time dashboards can put unprecedented power
documents that provide the same view of historical in the hands of end users to better understand how
performance to all recipients. Contemporary key metrics are changing and what should be done to
reporting systems often incorporate features like address problems.
interactive dashboards that provide customized,
up-to-the-minute (or at least frequently updated)
graphical displays of critical performance metrics, 3. Problems, challenges, barriers, and
historical trends, and reference benchmarks or goals. issues in healthcare analytics
These dashboards are designed to help end users
focus on those data that are most informative about 3.1. Managerial issues
how their systems are performing. In healthcare,
decision-support dashboards are increasingly com- A key objective for any analytics system in health-
mon on the clinical side, especially in EHR environ- care is to produce a valuable output for those taking
ments, but far less so when it comes to supporting care of patients, doing research, or making other
managerial or operational decisions (Figure 1). decisions about how the organization functions. For
One of the distinguishing features of many con- example, one of the core motivations for the recent
temporary dashboards and analytical systems is the massive investment in EHR infrastructure is the
Figure 1. A sample dashboard of emergency department performance measures*

*Used with permission from Emergency Medicine Business Intelligence (www.embiondemand.com)


576 M.J. Ward et al.

assumption that these systems will provide data to will be necessary to determine the cost of the
enable clinicians and researchers to develop better benefits achieved from such technology.
interventions, protocols, drugs, and policies that
lead to improved patient outcomes. 3.2. Data quality
In healthcare, the foremost concern for manage-
ment is the people that comprise it–—namely, the While it is a widely held belief that clinical informa-
key stakeholders–—be they patients, physicians, tion systems, particularly EHRs, can serve as a rich
nurses, or other medical staff; referring providers; source of data for analytical and research purposes,
or representatives from the local community. Em- not all data are created equal. Some data elements
powering these individuals and increasing the quali- are captured in a more consistent fashion and on a
ty and transparency of decision making are key goals greater percentage of the patient population. Most
for any business analytics initiative. Therefore, be- EHRs are designed to allow the same piece of infor-
cause of the pervasive influence these systems have, mation to be captured in many different ways. A
the organization needs to establish business analyt- diagnosis, for instance, could be listed on the pa-
ics as an organizational and cultural objective, a tient’s problem list or in his or her medical history,
component of its long-term strategy. billing records, reason for visit, clinical narrative,
Such a culture would result in some fundamental etc. Institutions can implement best practices for
improvements to the organization. First, decision where certain information is supposed to be docu-
making based primarily on data and information mented on the patient’s chart and use quality im-
would become the expectation and the norm. This provement (QI) reports to ensure compliance.
is essential to complete the transformation to evi- In most cases, however, when conducting
dence-based medicine. Additionally, because data population-level analytics, one needs to remember
are shared and updated frequently, routine deci- to look in all possible locations where data may
sions can be more easily automated or augmented exist. Otherwise, the user is left with the population
with decision-support systems. Tools like computer- which has data in the locations being searched. The
ized physician order entry systems that verify and latter approach introduces its own set of biases, but
validate medication orders in real time are but one in some cases, it may be sufficient (e.g., ensuring
example of the power and promise of analytics. that all patients with documented diabetes are
However, to realize these benefits, clinicians, identified so they can have their hemoglobin A1c
support staff, and leadership all need to under- levels checked versus trying to identify every pa-
stand and appreciate the importance of business tient in the hospital who might possibly be sus-
analytics as tools and as a fundamental process pected of having diabetes). As a result, there is a
within the organization. Otherwise, the organiza- growing awareness about the role of data quality in
tion will continue to underinvest and staff will be EHR-based analytics and a need to characterize the
skeptical of the value of recording data as a matter data’s ‘fitness for use’ before utilizing it for any
of course. ancillary purposes (Weiskopf & Weng, 2013).
Another key managerial challenge lies in finding Another major challenge facing the use of ana-
and retaining personnel capable of performing these lytics includes the availability and the cost of ac-
often complex analytical and data-management quiring electronic data. A project to automate
tasks. An issue that has limited the expansion of metric reporting at the integrated health system
healthcare analytics is lack of qualified individuals Kaiser Permanente found that the necessary elec-
with the appropriate background and skills in com- tronic data were frequently not available for public
puting and mathematics to perform these analyses, reporting of system metrics (Garrido et al., 2014).
combined with increasing demand for such individ- This resulted in nearly $7 million in administrative
uals. McKinsey and Company (2011) estimated that costs to obtain and report these data. After imple-
by 2018, the United States could face a shortage of menting automated data reporting, Kaiser estimat-
140,000 to 190,000 people with the appropriate ed that the company reduced abstraction time by
analytical skills. more than 50% and saved approximately $1 million
Additional managerial issues involve the mea- in administrative costs. However, just because data
surement of outcomes from the use of electronic are collected does not mean they are readily able to
data and their subsequent application to human be mined. Some data may be in a clinical narrative
health. Do they improve health? Do they save that is more difficult to mine, requiring natural lan-
money? What metrics should be used to quantify guage processing algorithms to determine whether a
their benefit or cost? Currently, there are no clear- particular action (e.g., smoking cessation) was per-
cut answers (Mandl & Kohane, 2012; Menachemi & formed. While discrete elements, such as check box-
Collum, 2011). To answer these questions, analytics es, could be added to the user interface, this
Applications of business analytics in healthcare 577

approach threatens to prolong the time a provider approach taken by an increasing number of orga-
uses electronic tools (e.g., Poissant, Pereira, nizations is to have patients take on a larger amount
Tamblyn, & Kawasumi, 2005). Even when discrete of the data-entry burden. By providing kiosks or
fields are available, many electronic data are often tablets to allow patients to fill out forms in the
incomplete (Staroselsky et al., 2006). waiting room or allowing them to enter the data
Data standardization also raises another chal- at home through a patient portal, physicians need
lenge: the accuracy of electronic data. EHRs may only review the responses instead of keying them in
not improve–—and may even worsen–—data quality themselves. (Whether these patient-reported data
(Tse & You, 2011). For example, in one emergency are as complete or as of high quality as data provid-
department, EHR implementation increased the ed by clinicians is an open question.)
number of systematic errors during implementation From an analytical perspective, this approach is
compared to the legacy system (Ward, Froehle, limited by the quality of the data supplied by pa-
Hart, & Lindsell, 2013). Compromised data quality tients and is subject to recall bias. As one physician
poses risks for interpretation as well as any actions stated: ‘‘For example, when I ask a patient if they
resulting from such data. have any medical problems, I have had multiple
patients respond ‘no,’ only to later see that they
3.3. Data collection have human immunodeficiency virus (HIV) in their
chart.’’ Another challenge posed by patient-entered
Data quality and the data-collection process data is that these responses are typically segregated
are inextricably linked. Once data quality is com- in the EHR’s reporting database from those entered
promised, it can be tremendously expensive to by clinicians. Even with a large percentage of pa-
overcome; therefore, it is critical to focus on tients entering data, clinicians will still need the
high-quality data collection (Redman, 2013). In ability to enter the same data elements through
healthcare, the generation of high-quality, useful their EHR interfaces. Therefore, when using these
data does not necessarily happen as a byproduct of data for analytical purposes, one must remember to
the system. In the vast majority of cases, to produce merge both the patient-entered data tables with
high-quality data, someone needs to collect it. the clinician-entered data tables in order to get a
Therefore, workflows must be designed in a way that comprehensive dataset.
ensures the important data elements will be cap-
tured during a visit and that these tasks minimally 3.4. Competitive concerns and public
disrupt workflow, particularly expensive resources reporting
such as nurses and physicians. Even if this interaction
is as trivial as a key press, information-processing One of the most significant ongoing debates about
theory tells us that burden can greatly undermine the analytics in healthcare involves the public reporting
consistency and quality of the data being collected of results. The trend has been toward more trans-
(Payne, Bettman, & Johnson, 1993). parency. CMS, for example, now provides a public
Instead of collecting as much data as possible, report card of hospital quality measures (Hospital
institutions should actually take the opposite ap- Compare, n.d.), and there are numerous private
proach, ensuring that they collect the minimal set of initiatives to spur public reporting on quality and
data elements required. It is far better to have a cost (James, 2012). However, there are competitive
smaller set of high-quality elements with a high concerns about sharing quality and outcome data, so
completion percentage than a large set with spotty many institutions are reluctant to share data that
coverage. There are ways that organizations can are not required by federal or state regulations.
encourage employees to collect specific data ele- An innovative approach to sharing health data has
ments, including publicizing the capture rates of been through the establishment of collaborative
individual employees within a clinic (anonymous or multi-center quality-improvement networks. These
identified), tying a portion of salary to data-entry networks, such as Solutions for Patient Safety (Ohio
compliance, and providing a tangible benefit from Children’s Hospitals, 2014) and ImproveCareNow
data collection (e.g., the data captured can be used (Crandall et al., 2011), set goals such as eliminating
to automate a downstream process, saving time and patient harm and improving the care and outcomes
effort). of children with inflammatory bowel disease. An-
There is only a certain amount of data that can be other example of inter-institutional data sharing,
collected in any single visit. After a certain instant, the Emergency Department Benchmarking Alliance
data entry will increase visit length to the point that (http://www.edbenchmarking.org/), allows mem-
it affects patient flow, potentially impacting patient ber emergency departments to review and compare
satisfaction and revenues. As a result, another blinded operational data of similar facilities. The
578 M.J. Ward et al.

organization defines a set of outcome measures and govern what data can be used for research, who may
standardizes data collection. Center-level out- have access to that data, and the type of patient
comes are shared within the collaborative, and consent required before access is granted. This
participants learn from those centers with the best poses challenges to analytical staff and their IT
outcomes. In addition, metrics can be shared with systems. In many cases, an organization may want
the public while keeping the healthcare facility to have a common set of business rules that can be
anonymous. Because many of the centers partici- applied regardless of whether the data are used for
pating in the network are in competition with one clinical care, internal performance improvement,
another, the networks are largely built on trust and a or research (as in the public reporting example
sense of duty to public good. By ensuring that no mentioned previously). If the logic for the business
center’s results are used against it in a disparaging rules are encoded into the analytical system, the
way, the Emergency Department Benchmarking Al- organization will either need to determine how to
liance is able to improve outcomes for the popula- assign role-level access to that data to control who
tion as a whole. can see the data for research purposes or implement
a completely stand-alone research infrastructure,
3.5. Data privacy and governance which poses its own set of costs and challenges.

With increased access to and use of electronic data,


privacy concerns are escalating. Specifically, results 4. Specific examples
that involve genetic data require special consider-
ation. The federal government has had a longstand- As we have discussed, analytics has begun to
ing policy of making public, as much as possible, the improve and inform healthcare in many varied
genetic findings obtained with public research fund- ways. Some analytics in particular have experi-
ing (National Institutes of Health, 2007). That has enced dramatic adoption or seem to present a
led to the establishment of public databases like the potentially revolutionary approach to medical de-
database of Genotypes and Phenotypes (Mailman cision making and healthcare management. These
et al., 2007; Zhang et al., 2008) and international examples and others can be loosely grouped into
initiatives like the 1000 Genomes Project (Genomes two categories–—discovery/efficacy and care de-
Project, 2010). By nature, genetic information can- livery–—as shown in Figure 2. We now discuss
not be de-identified, but it was widely believed that several of these examples.
the size and complexity of the data would at least
confer some degree of anonymity. Recent studies 4.1. Dashboards and control charts
have shown that such expectations are not valid
(Gymrek, McGuire, Golan, Halperin, & Erlich, 2013), Dashboards and control charts are tools from QI
and due to the concern of using genetic information science that are used to monitor outcomes and look
to discriminate for employment or insurance pur- for process variations. Control charts consist of
poses–—despite it being illegal (Pulley et al., 2012)–— time-series plots with upper- and lower-control
there may be a move to decrease transparency when limits. Based on certain criteria, data points outside
it comes to sharing genetic results. the control limits may be deemed a special cause,
Healthcare faces another managerial challenge which would indicate an improvement or deteriora-
that is not present in other industries: the use tion in a particular process. Applied to healthcare,
of data for research is governed by different rules control charts are used to monitor outcomes, such as
than that for non-research purposes. Federal regu- surgical site infections, remission rate, or opera-
lations like the Health Insurance Portability and tional metrics (e.g., patient wait times, room
Accountability Act (HIPAA) and the Common Rule utilization).

Figure 2. Some analytical areas and their contributions to healthcare


Discovery/Efficacy Care Delivery

• Chronic disease databases • RFID • Paent flow


• Comparave effecveness • Locaon-based • Dashboards and control charts
• Genecs analycs • Disaster planning
Applications of business analytics in healthcare 579

4.2. Genetics been a theoretical exercise, and to implement its


findings.
The integration of genetic data into routine health-
care practices provides tantalizing possibilities for 4.4. Chronic disease databases
the field of analytics. Private companies have com-
menced straight-to-consumer testing on risk factors The vision of many within medicine is to move from
for certain genetic conditions. Some healthcare our current health system to that of a learning
organizations have even begun to prospectively health system (LHS), for which every clinic visit
genotype patients at risk for conditions for which provides an opportunity to generate new evidence
treatment involves medication that is metabolized and knowledge (Skiba, 2011; Slutsky, 2007). The
differently depending on genotype (Pulley et al., Institute of Medicine envisions an LHS in which
2012). The results are stored in the patient’s EHR healthcare, the continuous improvement of health-
and can be used to drive decision support around care, technological innovation, and research are all
medication dosage should the patient develop the purposefully integrated. As one example of this
condition in question. Federally funded initiatives vision, the ImproveCareNow network is developing
like the Electronic Medical Records and Genomics a registry that is linked to EHRs, allowing data to be
Network seek to derive phenotypes from EHR data collected at the point of care and used for clinical
and use genomic analyses to determine if there is an support, QI, and research (Marsolo, 2013). This
underlying genetic basis that can be used to identify registry has allowed the network to (1) create tools
treatment (McCarty et al., 2011). While some con- and processes to capture data directly from EHRs,
troversy exists among the informatics and health IT improve the quality of information obtained during
community regarding how much genetic information the patient encounter, and support the use of QI
should be stored in EHRs and how much control methods; (2) use the registry to automate pre-visit
patients should have over those data (Masys et al., planning and population-management activities and
2012), we are approaching the era of the $1,000 develop personalized software applications aimed
genome (Mardis, 2006), meaning we will soon be at increasing patient participation (i.e., activation)
awash in genomic data. Given the incredible size in chronic care; (3) use data from the registry to
and dimensionality of these datasets, the field of conduct novel CE research by generating estimates
analytics will need to borrow techniques from large of the comparative treatment effects of biologic
analytics companies like Google and Facebook and agents; and (4) develop governance and data-
from fields like astronomy and physics to architect sharing processes. Effective data capture and tech-
their solutions (Schatz & Langmead, 2013). nology use are foundational in making the LHS model
work, but patients and clinicians must be sufficient-
4.3. Cost of treatment and guiding ly motivated and engaged to participate. As a result,
investments the network is providing data, tools, and support
aimed at meeting the important needs of each of the
Borrowing from the field of comparative effective- stakeholders who contribute to the LHS model and
ness (CE) research, analytics can also be applied in to using technology to reduce the time and effort
healthcare to compare the cost and effectiveness of required to participate.
interventions, treatments, public health policies, or
medical devices. For instance, organizations could 4.5. Disaster planning
use cost and outcome data to examine the perfor-
mance of replacement hips and stop utilizing those As another example, analytics could play a pivotal
that perform worse (i.e., fail earlier) or cost more role in disaster response. Disasters can take many
and do not provide improved patient outcomes. forms. Simply put, a disaster is when demand for
Similarly, organizations could stop prescribing med- resources far outstrips available supply. Disasters
ications that cost more and do not yield a thera- can result from forces of nature (e.g., hurricanes,
peutic benefit that justifies the cost. While CE snow storms), infectious diseases, terrorist attacks,
research has become politically polarized in the airplane crashes, or even building collapses. Partic-
United States in recent years, insurance companies ularly in disasters, specific types of resources may
routinely use the practice to decide what they will be needed more than others. The key is matching
reimburse, and in a time of decreasing federal appropriate resources with demand in order to mit-
budgets, healthcare organizations, which largely igate harm and potential lives lost. For example,
depend on those federal dollars, will be forced to following a building collapse, dialysis machines are a
do the same. This could force healthcare organiza- specific resource in high demand since many pa-
tions to reconsider CE research, which has largely tients will go into kidney failure, and the need for
580 M.J. Ward et al.

this equipment can quickly outstrip a hospital’s from the ICU often lead to rework in the form of
resources. Similarly, during influenza outbreaks, readmissions to the ICU and subsequent clinical de-
respiratory failure can require the use of ventilators terioration (KC & Terwiesch, 2012). A study of car-
to assist with breathing until a patient recovers. diothoracic surgery found that workload is an
However, having real-time data on the availability important predictor of patient safety and service
of such equipment and resources within a location times and that peak performance cannot be sus-
(e.g., hospital, city, state, region) could enable tained despite healthcare’s unwritten motto of ‘work
resources to be organized more efficiently, resulting harder’ (KC & Terwiesch, 2009). Healthcare is in-
in improved outcomes and negating deleterious creasingly time dependent, and analytics plays a
outcomes from delayed treatment. As another pivotal role in ensuring that patient flow is smooth
example, analytics is used to create Google Flu and that patients with time-critical illnesses are
Trends, which estimates the prevalence of influenza cared for as soon as possible.
based on search term activity (Google, 2011). Social
media is also increasingly used to identify influenza- 4.7. RFID
like illnesses to detect potential outbreaks before
formal diagnoses are even made (Corley, Cook, In addition to asset management, RFID is increas-
Mikler, & Singh, 2010). The use of analytics tools ingly being integrated into healthcare to provide
allows patterns indicative of future outbreaks to be real-time management, identification, and tracking
detected earlier, before situations get out of hand, of patients and staff (Ingrassia et al., 2012; Yu &
costs spiral, and lives are lost. Ganz, 2011). This highly granular location data can
be invaluable when modeling patient flow and care-
4.6. Patient flow delivery processes. More novel uses are also being
explored. For instance, RFID can be used to monitor
For decades, patient flow in healthcare facilities has compliance with hospital policies, such as hand
depended on experienced managers with no training washing (Perna, 2013). Surgical patients can be
in operations to predict arrival surges and resource evaluated for potential retained surgical equip-
needs (e.g., patient beds). Physician and nurse staff- ment–—a major source of morbidity and medicolegal
ing is frequently performed based on provider prefer- risk–—through RFID-embedded equipment (Rogers,
ence rather than system requirements. As healthcare Jones, & Oleynikov, 2007). More than just patient
utilization continues to increase and the number of flow and tracking, RFID holds potential for disease
entities and coordination needed among providers, management as well. Patients with chronic diseases
patients, and their families becomes increasingly like diabetes and hypertension can be monitored for
complex, expertise is no longer enough. Tools like achieving specific numeric targets in their blood
closed-form mathematical modeling, discrete-event sugar and blood pressure, respectively, with mini-
simulation, and empirical/statistical analyses have mal effort as smart devices embedded with RFID
been employed to improve patient flow, reduce communicate to their providers (Moore, 2009). Sim-
waiting, and potentially improve patient outcomes ilarly, elderly patients can be monitored to prevent
(Froehle & Magazine, 2013; White, Froehle, & falls (Ranasinghe et al., 2012). In this scenario, prior
Klassen, 2011). Initially, these projects began as to patients needing expensive healthcare resources
department-specific improvement initiatives but like an emergency department visit or hospital ad-
have grown to encompass more and more of the mission, care coordination could potentially inter-
complex systems they seek to represent. For exam- vene and identify the reason for out-of-control
ple, several clinical disciplines have begun using values and return the patient to his or her normal
approaches to predict resource utilization and down- state of health. These applications have unique
stream consequences. Emergency departments have analytics issues, such as ensuring data accuracy
used predictive models to forecast and plan for and handling the large quantities of data that were
excessive patient waiting (i.e., crowding), to staff not previously available. Privacy issues can also be a
and build an observation unit for patients requiring concern for both patients and staff, as individuals
abbreviated admissions, and to plan for downstream are tracked (Hawrylak, Schimke, Hale, & Papa,
inpatient bed use (Hoot et al., 2008; Lovejoy & 2012).
Desmond, 2011; Peck, Benneyan, Nightingale, &
Gaehde, 2012). Internal medicine has used such tools
to improve patient flow in outpatient ambulatory 5. Conclusion
clinics (Chand, Moskowitz, Norris, Shade, & Willis,
2009). Similarly, intensive care units (ICUs) have used Analytics is increasingly weaving itself into the fab-
empirical analyses to identify that early discharges ric of healthcare and will fundamentally shape the
Applications of business analytics in healthcare 581

future of medicine and care delivery. The number of Corley, C. D., Cook, D. J., Mikler, A. R., & Singh, K. P. (2010). Using
applications seems to grow unabated. With oppor- web and social media for influenza surveillance. Advances in
Experimental Medicine and Biology, 680, 559—564.
tunities like improving healthcare efficiency Crandall, W., Kappelman, M. D., Colletti, R. B., Leibowitz, I.,
while enhancing the quality of care as well as the Grunow, J. E., Ali, S., & . . . Margolis, P. (2011). ImproveCar-
ability to mine genetic data, reduce costs, effectively eNow: The development of a pediatric inflammatory bowel
respond to disasters, and achieve numerous other disease improvement network. Inflammatory Bowel Diseases,
goals, the application of analytics is broad and far- 17(1), 450—457.
DesRoches, C. M., Charles, D., Furukawa, M. F., Joshi, M. S.,
reaching. Such a vision will not be easily achieved, Kralovec, F. M., Mostashari, F., & . . . Jha, A. K. (2013).
however, without surmounting numerous challenges Adoption of electronic health records grows rapidly, but fewer
along the way. These include overcoming privacy than half of US hospitals had at least a basic system in 2012.
concerns, collecting high-quality data and making Health Affairs, 32(8), 1478—1485.
Froehle, C. M., & Magazine, M. J. (2013). Improving scheduling
it available, developing data standards, communi-
and flow in complex outpatient clinics. In B. Denton (Ed.),
cating data to end users, and having enough qualified Handbook of healthcare operations management: Methods
personnel to deliver on such promises. While formi- and applications (pp. 229—250). New York: Springer.
dable, these issues are not insurmountable. Indeed, Garrido, T., Kumar, S., Lekas, J., Lindberg, M., Kadiyala, D.,
success is necessary to ensure that the application Whippy, A., & . . .Weissberg, J. (2014). e-measures: Insight
and understanding of analytics in healthcare will into the challenges and opportunities of automating publicly
reported quality measures. Journal of the American Medical
continue to grow and play an increasing role in Informatics Association, 21(1), 181—184.
healthcare systems of the future. Genomes Project. (2010). A map of human genome variation from
population-scale sequencing. Nature, 467(7319), 1061—1073.
Google. (2011). Flu trends. Retrieved July 18, 2013, from http://
www.google.org/flutrends/us/#US
DISCLOSURE Gymrek, M., McGuire, A. L., Golan, D., Halperin, E., & Erlich, Y.
(2013). Identifying personal genomes by surname inference.
The project described was supported by Award Science, 339(6117), 321—324.
Number K12 HL109019 from the National Heart, Hawrylak, P. J., Schimke, N., Hale, J., & Papa, M. (2012). Security
risks associated with radio frequency identification in medical
Lung, and Blood Institute and by R01HS20024 from environments. Journal of Medical Systems, 36(6), 3491—3505.
the Agency for Healthcare Research and Quality. HealthIT.gov. (2013). Health IT dashboard. Retrieved April 11,
The content is solely the responsibility of the 2013, from http://dashboard.healthit.gov/
authors and does not necessarily represent the Hoot, N. R., LeBlanc, L. J., Jones, I., Levin, S. R., Zhou, C., Gadd,
official views of the National Heart, Lung, and C. S., & Aronsky, D. (2008). Forecasting emergency depart-
ment crowding: A discrete event simulation. Annals of Emer-
Blood Institute, the National Institutes of Health gency Medicine, 52(2), 116—125.
or the Agency for Healthcare Research and Quality. Hospital Compare. (n.d.). About Hospital Compare data. Re-
trieved July 14, 2013, from http://www.medicare.gov/
hospitalcompare/Data/About.html
Ingrassia, P. L., Carenzo, L., Barra, F. L., Colombo, D., Ragazzoni,
L., Tengattini, M., & . . .Della Corte, F. (2012). Data collection
References in a live mass casualty incident simulation: Automated RFID
technology versus manually recorded system. European Jour-
Aardex Group. (2012). Data collection. Retrieved June 20, 2012, nal of Emergency Medicine, 19(1), 35—39.
from http://www.mwvaardex.com/Products/DataCollection/ International Health Technology Standards Development Organi-
index.htm sation. (n.d.). SNOMED CT. Retrieved April 11, 2013, from
Blumenthal, D., & Tavenner, M. (2010). The ‘‘meaningful use’’ http://www.ihtsdo.org/snomed-ct/
regulation for electronic health records. New England Journal James, J. (2012, March 8). Health policy brief: Public reporting on
of Medicine, 363(6), 501—504. quality and costs. Health Affairs. Retrieved April 11, 2013, from
Chand, S., Moskowitz, H., Norris, J. B., Shade, S., & Willis, D. R. http://healthaffairs.org/healthpolicybriefs/brief.php?brief_
(2009). Improving patient flow at an outpatient clinic: Study of id=65
sources of variability and improvement factors. Health Care KC, D.-S., & Terwiesch, C. (2009). Impact of workload on service
Management Science, 12(3), 325—340. time and patient safety: An econometric analysis of hospital
CMS. (2013a). National health expenditure data 2011. Retrieved operations. Management Science, 55(9), 1486—1498.
July 23, 2013, from http://www.cms.gov/Research-Statistics- KC, D.-S., & Terwiesch, C. (2012). An econometric analysis of
Data-and-Systems/Statistics-Trends-and-Reports/National patient flows in the cardiac intensive care unit. Manufacturing
HealthExpendData/NHE-Fact-Sheet.html and Service Operations Management, 14(1), 50—65.
CMS. (2013b). International classification of diseases, 9th revision, Koppel, R. (n.d.). Usability: Present and future. Retrieved July 14,
clinical modification (ICD-9-CM) ICD-10. Retrieved April 11, 2013, from www.uthouston.edu/nccd/pre-amia-symposium/
2013, from http://www.cms.gov/Medicare/Coding/ICD10/ koppel.pdf
index.html?%20redirect=/icd10 LOINC. (n.d.). Logical observation identifiers names and codes
CMS. (2014). International classification of diseases, 9th revision, [website]. Retrieved April 11, 2013, from http://loinc.org/
clinical modification (ICD-9-CM). Retrieved April 11, 2013, Lovejoy, W. S., & Desmond, J. S. (2011). Little’s law flow analysis
from http://www.cms.gov/Medicare/Coding/ICD9Provider of observation unit impact and sizing. Academic Emergency
DiagnosticCodes/index.html Medicine, 18(2), 183—189.
582 M.J. Ward et al.

Mailman, M. D., Feolo, M., Jin, Y., Kimura, M., Tryka, K., Bagout- physicians and nurses: A systematic review. Journal of the
dinov, R., & . . . Sherry, S. T. (2007). The NCBI dbGaP database American Medical Informatics Association, 12(5), 505—516.
of genotypes and phenotypes. Nature Genetics, 39(10), Pulley, J. M., Denny, J. C., Peterson, J. F., Bernard, G. R.,
1181—1186. Vnencak-Jones, C. L., Ramirez, A. H., & . . .Roden, D. M.
Mandl, K. D., & Kohane, I. S. (2012). Escaping the HER trap–—The (2012). Operational implementation of prospective genotyp-
future of health IT. New England Journal of Medicine, 366(10), ing for personalized medicine: The design of the Vanderbilt
2240—2242. PREDICT project. Clinical Pharmacology and Therapeutics,
Mardis, E. R. (2006). Anticipating the 1,000 dollar genome. 92(1), 87—95.
Genome Biology, 7(7), 112. Ranasinghe, D. C., Shinmoto Torres, R. L., Sample, A. P., Smith, J.
Marsolo, K. (2013). In search of a data-in-once, electronic health R., Hill, K., & Visvanathan, R. (2012). Towards falls preven-
record-linked, multicenter registry–— How far we have come tion: A wearable wireless and battery-less sensing and auto-
and how far we still have to go. eGEMs, 1(1), 1—4. matic identification tag for real time monitoring of human
Marsolo, K., & Spooner, S. A. (2013). Clinical genomics in the movements. Conference Proceedings of the IEEE Engineering
world of the electronic health record. Genetics in Medicine: in Medicine and Biology Society, 6402—6405.
Official Journal of the American College of Medical Genetics, Rawley, E., & Simcoe, T. S. (2012). Information technology,
15(10), 786—791. productivity, and asset ownership: Evidence from taxicab
Martin, L. R., Williams, S. L., Haskard, K. B., & Dimatteo, M. R. fleets. Organization Science, 24(3), 831—845.
(2005). The challenge of patient adherence. Therapeutics and Redman, T. C. (2013, December). Data’s credibility problem.
Clinical Risk Management, 1(3), 189—199. Harvard Business Review. Retrieved from http://hbr.org/
Masys, D. R., Jarvik, G. P., Abernethy, N. F., Anderson, N. R., 2013/12/datas-credibility-problem/ar/1
Papanicolaou, G. J., Paltoo, D. N., & . . .Levy, H. P. (2012). Rogers, A., Jones, E., & Oleynikov, D. (2007). Radio frequency
Technical desiderata for the integration of genomic data into identification (RFID) applied to surgical sponges. Surgical
electronic health records. Journal of Biomedical Informatics, Endoscopy, 21(7), 1235—1237.
45(3), 419—422. Schatz, M. C., & Langmead, B. (2013, June 27). The DNA data
McCarty, C. A., Chisholm, R. L., Chute, C. G., Iftikhar, J. K., deluge. IEEE Spectrum. Retrieved from http://spectrum.
Jarvik, G. P., Larson, E. B., & . . .Wolf, W. A. (2011). The ieee.org/biomedical/devices/the-dna-data-deluge
eMERGE network: A consortium of biorepositories linked to Skiba, D. J. (2011). Informatics and the learning healthcare
electronic medical records data for conducting genomic stud- system. Nursing Education Perspectives, 32(5), 334—336.
ies. BMC Medical Genomics, 4(13), 1—11. Slutsky, J. R. (2007). Moving closer to a rapid-learning health care
McKinsey and Company. (2011, May). Big data: The next frontier system. Health Affairs, 26(2), 122—124.
for innovation, competition, and productivity. Retrieved July Staroselsky, M., Volk, L. A., Tsurikova, R., Pizziferri, L., Lippin-
17, 2013, from http://www.mckinsey.com/insights/business_ cott, M., Wald, J., & Bates, D. W. (2006). Improving electronic
technology/big_data_the_next_frontier_for_innovation health record (EHR) accuracy and increasing compliance with
Menachemi, N., & Collum, T. H. (2011). Benefits and drawbacks of health maintenance clinical guidelines through patient access
electronic health record systems. Risk Management and and input. International Journal of Medical Informatics,
Healthcare Policy, 4, 47—55. 75(10/11), 693—700.
Moore, B. (2009). The potential use of radio frequency identifi- Tait, A. R., Voepel-Lewis, T., & Zikmund-Fisher, B. J. (2010). The
cation devices for active monitoring of blood glucose levels. effect of format on parents’ understanding of the risks and
Journal of Diabetes Science and Technology, 3(1), 180—183. benefits of clinical research: A comparison between text,
National Institutes of Health. (2007). NIH data sharing informa- tables, and graphics. Journal of Health Communication: In-
tion. Retrieved July 14, 2013, from http://grants.nih.gov/ ternational Perspectives, 15(5), 487—501.
grants/policy/data_sharing/ Tse, J., & You, W. (2011). How accurate is the electronic health
National Library of Medicine. (n.d.). RxNorm. Retrieved April 11, record? A pilot study evaluating information accuracy in a
2013, from http://www.nlm.nih.gov/research/umls/rxnorm/ primary care setting. Studies in Health Technology and Infor-
OECD. (2011). Health at a glance 2011: OECD indicators. Re- matics, 168, 158—164.
trieved July 17, 2013, from http://www.oecd.org/els/health- Ward, M. J., Froehle, C. M., Hart, K. W., & Lindsell, C. J. (2013).
systems/49105858.pdf Operational data integrity during electronic health record
Ohio Children’s Hospitals. (2014). Solutions for patient safety. implementation in the ED. American Journal of Emergency
Retrieved July 14, 2013, from http://solutionsforpatient Medicine, 31(7), 1029—1033.
safety.org/ Weiskopf, N. G., & Weng, C. (2013). Methods and dimensions of
Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The electronic health record data quality assessment: Enabling
adaptive decision maker. Cambridge, UK: Cambridge Univer- reuse for clinical research. Journal of the American Medical
sity Press. Informatics Association, 20(1), 144—151.
Peck, J. S., Benneyan, J. C., Nightingale, D. J., & Gaehde, S. A. White, D. L., Froehle, C. M., & Klassen, K. J. (2011). The effect of
(2012). Predicting emergency department inpatient admis- integrated scheduling and capacity planning on clinical efficien-
sions to improve same-day patient flow. Academic Emergency cy. Production and Operations Management, 20(3), 442—455.
Medicine, 19(9), E1045—E1055. Yu, X., & Ganz, A. (2011). Scalable patients tracking framework
Perna, G. (2013, January 28). Leaders at one Texas-based hospital for mass casualty incidents. Conference Proceedings of the
are not taking hand hygiene for granted. Healthcare Infor- IEEE Engineering in Medicine and Biology Society, 860—863.
matics. Retrieved from http://www.healthcare-informatics. Zhang, H., Morrison, M. A., Dewan, A., Adams, S., Andreoli, M.,
com/article/2013-healthcare-informatics-innovator-awards- Huynh, N., & . . .DeAngelis, M. M. (2008). The NEI/NCBI dbGAP
third-place-winner-shannon-medical-center database: Genotypes and haplotypes that may specifically
Poissant, L., Pereira, J., Tamblyn, R., & Kawasumi, Y. (2005). The predispose to risk of neovascular age-related macular degen-
impact of electronic health records on time efficiency of eration. BMC Medical Genomics, 9(1), 1—12.

You might also like