Kawamoto-2019-Association of An Electronic Hea

Original Investigation | Health Informatics
Association of an Electronic Health Record Add-on App for Neonatal

Bilirubin Management With Physician Efficiency and Care Quality
Kensaku Kawamoto, MD, PhD, MHS; Polina Kukhareva, PhD, MPH; Julie H. Shakib, DO, MS, MPH; Heidi Kramer, PhD; Salvador Rodriguez, PhD; Phillip B. Warner, MS;
David Shields, BA; Charlene Weir, PhD, RN; Guilherme Del Fiol, MD, PhD; Teresa Taft, PhD; Carole H. Stipelman, MD, MPH
Abstract Key Points

Question Is an electronic health record
IMPORTANCE The usefulness of electronic health record (EHR) systems could be significantly
add-on app for neonatal bilirubin
enhanced by innovative, third-party EHR add-on apps.
management associated with time
savings for clinicians and improved
OBJECTIVE To evaluate whether an EHR add-on app for neonatal bilirubin management can save
quality of care?
clinicians time and improve patient care.
Findings In this quality improvement
DESIGN, SETTING, AND PARTICIPANTS This quality improvement study was conducted at the study, an electronic health record
University of Utah Health Well Baby nursery and outpatient clinics and consisted of 4 substudies: (1) add-on app for neonatal bilirubin
time savings were estimated in an experimental task-timing study comparing the time required for management saved clinicians a mean of
physicians to manage newborns’ bilirubin levels with and without the add-on app, (2) app use was 66 seconds for bilirubin management
estimated from app logs, (3) health care use measures and guideline compliance were compared tasks compared with a commonly used
retrospectively before and after the intervention, and (4) clinician-perceived usability was measured tool. In a retrospective pre-post analysis,
through System Usability Scale surveys. The study took place between April 1, 2016, and September the odds of clinically appropriate
3, 2019. Data analyses were conducted from October 30, 2018, to September 23, 2019. phototherapy orders during
hospitalization increased significantly
INTERVENTIONS At baseline, clinicians used a manual approach to ensure compliance with an by 84%.
evidence-based clinical guideline for neonatal bilirubin management. To facilitate guideline
Meaning These findings suggest that
compliance, an EHR add-on app that automatically retrieves, organizes, and visualizes relevant
well-designed electronic health record
patient data was developed. The app provides patient-specific assessments and recommendations,
add-on apps may be associated with
including the risk of rebound hyperbilirubinemia following phototherapy based on a predictive
time savings for physicians and
model. The add-on app was integrated with the University of Utah Health EHR on April 12, 2017.
improvements in patient care.
MAIN OUTCOMES AND MEASURES Clinician time savings, app use, health care use measures,
guideline-compliant phototherapy ordering, and perceived usability as measured by the System Author affiliations and article information are
listed at the end of this article.
Usability Scale survey. The survey is composed of 10 statements with responses ranging from 1
(strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100,
with ratings described as worst imaginable (mean System Usability Scale score, 12.5), awful (20.3),
poor (35.7), okay (50.9), good (71.4), excellent (85.5), and best imaginable (90.9).
RESULTS In 2018, the application was used 20 516 times by clinicians for 91.84% of eligible
newborns. Use of the app saved 66 seconds for bilirubin management tasks compared with a
commonly used tool (95% CI, 53-79 seconds; P < .001). Following the intervention, health care use
rates remained stable, while orders for clinically appropriate phototherapy during hospitalization
increased for newborns with bilirubin levels above the guideline-recommended threshold (odds
ratio, 1.84; 95% CI, 1.16-2.90; P = .009). Surveys indicated excellent usability (System Usability Scale
score, 83.90; 95% CI, 81.49-86.31).
(continued)
Open Access. This is an open access article distributed under the terms of the CC-BY License.
JAMA Network Open. 2019;2(11):e1915343. doi:10.1001/jamanetworkopen.2019.15343 (Reprinted) November 15, 2019 1/11
Downloaded From: https://jamanetwork.com/ on 07/17/2020

JAMA Network Open | Health Informatics Association of Health Record App for Neonatal Bilirubin Management With Physician Performance
Abstract (continued)
CONCLUSIONS AND RELEVANCE Well-designed EHR add-on apps may save clinicians time and
improve patient care. If time-saving apps, such as the bilirubin app, were implemented widely across
institutions and care domains, the potential association with improved patient care and clinician
efficiency could be significant. The University of Utah Health bilirubin app is being prepared for
release into EHR app stores as free-to-use software.
JAMA Network Open. 2019;2(11):e1915343. doi:10.1001/jamanetworkopen.2019.15343
Introduction
Despite billions of dollars invested, electronic health records (EHRs) often fall short in supporting
efficient, high-quality patient care.1 Ambulatory care physicians can spend 2 hours on EHR and desk
work for every hour spent in direct clinical face time,2 and inpatient physicians can spend 5 hours on
such tasks for every hour spent in direct patient care.3 In a statewide survey of Rhode Island
physicians conducted in 2017, close to two-thirds of physicians reported that use of EHRs added to
their daily frustration.4
Through emerging technology standards, EHRs now allow the integration of add-on apps,
whereby third-party app developers can deliver innovative solutions that enhance the utility of EHRs
for targeted health care tasks and domains.5,6 Just as smartphones deliver a better user experience
through a variety of add-on apps, the hope is that EHRs can deliver improved user experiences
through such apps. Unlike EHR vendors, who may be overwhelmed with competing priorities, such
as maintaining legacy functionality and providing support to thousands of customers for a wide range
of clinical and administrative tasks, a third-party app developer can focus entirely on optimizing a
comparatively small range of decision tasks. Apps also use a technical framework designed to
facilitate substituting one app for another, whereas switching EHRs is a disruptive and costly process.
Thus, app developers may be inherently incentivized to optimize patient care and the user
experience in their domain of focus, whereas EHR vendors are unlikely to gain or lose a customer
based on their level of support for any specific decision task. Also, unlike EHR vendors, who may have
years-long cycles for feature prioritization and development, third-party app developers can adopt
a much more rapid and iterative development approach, with the ability to develop and release
enhancements much more quickly based on user feedback.
The technology enabling EHR add-on apps is known as Substitutable Medical Applications and
Reusable Technologies on Fast Healthcare Interoperability Resources (SMART on FHIR; pronounced
smart on fire).7 Apps enabled by SMART on FHIR use the Health Level Seven International (HL7)
SMART standard8 to enable single sign-on and integration with the EHR user interface, as well as the
HL7 FHIR data interface standard9 to pull in relevant patient data automatically. Although the
evidence of the influence of SMART on FHIR apps on clinical outcomes is limited, investigators at
Boston Children's Hospital showed in 2017 that the introduction of a SMART on FHIR app for
pediatric blood pressure visualization was associated with an increase in the recognition of abnormal
blood pressure (7.1% vs 4.9%; P < .001).10
The present study reports on an evaluation of an EHR add-on app that was implemented in 2017
at an academic medical center to support the American Academy of Pediatrics (AAP) guideline on
the management of neonatal hyperbilirubinemia.11 To manage newborns’ care according to this
guideline, the clinician must retrieve disparate data scattered across the medical record, synthesize
the data for risk classification, and apply guideline algorithms to identify patient-specific care needs,
such as the administration of phototherapy when bilirubin levels exceed risk-based treatment
thresholds.11 The app was designed to support these tasks by retrieving relevant data, providing a
visual data summary, and delivering guideline-based recommendations on next steps. We also
designed the app to support known success factors for clinical decision support (CDS) systems,
including providing CDS at the time and location of decision-making, providing recommendations

rather than just assessments, integrating with the EHR, and minimizing the need for additional
clinician data entry.12 The goal of this study was to evaluate our hypothesis that this EHR add-on app
would save clinicians time and improve care quality.
Methods
Study Design
This was a mixed-methods quality improvement study that included 4 separate substudies: (1) an
experimental task-timing study to estimate time savings, (2) an observational study of app use, (3) a
retrospective pre-post intervention study evaluating patient outcomes, and (4) a usability survey
study. App implementation and the subsequent quantitative evaluations of app use and patient
outcomes were exempted as quality improvement by the University of Utah Institutional Review
Board (IRB). The timing study and survey were approved by the IRB, and participants consented
verbally after reading an IRB-approved informed consent cover letter. This study report follows the
Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) reporting guidelines.13
The study was conducted at University of Utah Health (UUH), an academic health care system
using the Epic EHR (EPIC Systems Corp). Inpatient care for well newborns is provided in a newborn
nursery as mother-baby couplet care. Follow-up visits are conducted in 27 UUH clinics. Universal
bilirubin screening was implemented in the nursery on March 31, 2016. Prior to the UUH app
intervention, a stand-alone web-based tool known as BiliTool (BiliTool Inc) was used for bilirubin
management.14
Newborns born at 35 weeks’ gestation or longer and admitted to the nursery were included in
the analysis. Follow-up visits at UUH clinics within 14 days from discharge were included.
Before proceeding with the design and development of the Bili App, and in accordance with
UUH’s standard operating procedures for new app development, the project team explored whether
native EHR approaches could be used to meet the user needs. In particular, conventional EHR
mechanisms for CDS, such as alerts, reminders, and order sets, were considered. However, it was
determined that these conventional CDS mechanisms would not be able to support the desired
functionality, including providing a graphical summary of relevant patient data, supporting the full
complexity of the underlying clinical decision logic, and adjusting recommendations based on user
input. Thus, the decision was made to proceed with app development.
The app leverages the SMART on FHIR framework. Starting from a basic app developed by
Intermountain Healthcare to graph bilirubin levels against a risk nomogram,15,16 the University of
Utah ReImagine EHR team iteratively refined the app based on physician feedback. The ReImagine
EHR team includes clinical informaticists with expertise in areas including software development,
software architecture, standards-based interoperability, cognitive psychology, and biostatistics. Both
the SMART and a core set of FHIR data standards, known as the US Core FHIR profiles,17 were
natively supported by the Epic EHR, facilitating integration of the app. However, to achieve the full
set of functionality requested by clinicians, custom FHIR interfaces had to be developed to pull
additional data elements, including the mother’s laboratory data and outpatient phototherapy
orders. Such custom FHIR interfaces can be shared across health care systems and may become
unnecessary as EHR vendors increase the scope of data covered by their native FHIR data interfaces.
The resulting app (Figure) supports neonatal bilirubin management according to the AAP
guideline11 and estimates the risk of postphototherapy rebound hyperbilirubinemia.18 The app was
deployed institution-wide on April 12, 2017. Beyond a brief email message to EHR users on app
availability, app awareness was spread by word-of-mouth.
Resident physicians screen for and manage hyperbilirubinemia in the nursery under the
guidance of attending physicians. Prior to the introduction of the UUH app, clinicians used the
BiliTool website, which is available as a link in the EHR. Clinicians would open the website and
manually enter the time of birth as well as the last total bilirubin level and associated specimen
collection time. The website takes these inputs and provides recommendations for all 3 potential risk

levels as defined by the AAP guideline (lower, medium, and higher risk).11 BiliTool does not calculate
the patient-specific risk level; as such, clinicians independently assessed the patient’s risk status
using data in the EHR, with the patient’s risk level determined primarily by the patient’s gestational
age and direct Coombs test results.11
With the UUH app, clinicians launch the app through the EHR’s sidebar (Figure, A, item 1). The
app then loads as any other EHR screen does. There is no need for additional login and all the
information is pulled automatically from the EHR. If the patient does not have direct Coombs test
results, the app prompts the clinician whether other neurotoxicity risk factors are present before
Figure. App Screenshots for Bilirubin Analysis
A Patient with hyperbilirubinemia
B Patient with projected hyperbilirubinemia
Screenshots shown for a patient with

hyperbilirubinemia (A) and projected
hyperbilirubinemia (B). 1, link to open the app in the
EHR sidebar; 2, bilirubin levels over time (blue line); 3,
patient-specific phototherapy threshold according to
American Academy of Pediatrics guideline (green line);
4, inpatient phototherapy (yellow area); 5, patient-
specific risk factors; 6, mother’s laboratory
information; 7, other neurotoxicity risk factors; 8,
guideline-based care recommendations (in blue box);
and 9, predicted risk of rebound hyperbilirubinemia
following phototherapy (in green box). AAP indicates
American Academy of Pediatrics; InPt, inpatient; Neg,
negative; OutPt, outpatient; Pos, positive; TSB, total
serum bilirubin. Used with permission from Epic
Systems Corp.

providing a recommendation (Figure, A, item 7). Clinicians then review the patient data, including the
patient’s bilirubin levels over time (Figure, A, item 2); the patient’s risk-specific threshold where
phototherapy is indicated (Figure, A, item 3); history of phototherapy administration (Figure, A, item
4); other relevant data, including the mother’s blood type and indirect Coombs test results (Figure,
A, item 6); patient-specific recommendations on next steps (Figure, A, item 8); and, if applicable, the
patient’s estimated risk of rebound hyperbilirubinemia following phototherapy (Figure, A, item 9).
Clinicians can also evaluate the influence of potential changes on patient risk factors, such as if the
patient’s gestational age at birth was 37 weeks 6 days rather than 38 weeks 0 days (Figure, A, item 5).
Substudies
In a time-savings evaluation, 12 pediatric and family medicine resident physicians on service in the
nursery (including 7 interns, 3-second-year residents, and 2 third-year residents) were recruited to
complete bilirubin management tasks for their current patients (n = 42 patients). Each resident
physician completed these bilirubin management tasks for 2 to 5 patients, with every patient
randomly assigned to either Bili App or BiliTool use. Participants were asked to complete their work
as they usually would for making a decision regarding the patients' care needs with regard to
hyperbilirubinemia screening and management. Typically, this process involved reviewing the
bilirubin levels, identifying risk factors, and making a decision about appropriate next steps, including
possible additional bilirubin testing, phototherapy use, and discharge. We recorded the EHR screen
as the residents completed these bilirubin management tasks. Participants were asked to act as if the
observer was not present. Specifically, participants were asked to not “think aloud” or describe what
they were doing while performing these tasks.
Task time was calculated from screen recordings from the moment the patient’s medical record
was opened in the EHR to the moment when residents indicated that they were finished. Time
savings were estimated using generalized linear regression with generalized estimating equations to
account for correlation within timings from the same residents. We included subset analyses for
users with different experience levels.
Based on EHR logs for eligible newborns born in 2018, app use was measured as the number of
uses and the proportion of newborns with admissions (n = 3826) and follow-up visits (n = 1932) for
whom the Bili App was used. To calculate annual time savings for UUH, we multiplied the number of
uses in 2018 by the mean time savings from the task-timing study.
The preintervention period was April 1, 2016, through March 31, 2017 (n = 3714), and the
postintervention period was May 1, 2017, through April 30, 2019 (n = 7520), excluding 5 newborns
with missing gestational age or sex. There were no other interventions related to bilirubin
management introduced in the nursery during the study timeframe.
Reported patient characteristics include sex, gestational age less than 38 weeks, direct Coombs
test results, and performance of a serum bilirubin test. Patient characteristics before and after the
intervention were compared using χ2 tests. Health care use measures were length of stay, intensive
care unit admissions, urgent care visits, and readmissions. Risk-specific thresholds for phototherapy
(Figure, A, item 3) were determined at 12 to 144 hours of life based on whether the gestational age
was less than 38 weeks and whether a positive direct Coombs test result was documented.11 The
potential existence of other risk factors was not assessed for these purposes. Phototherapy use was
evaluated for 3 mutually exclusive patient subsets based on whether patients (1) had a bilirubin level
that was above the phototherapy threshold (hyperbilirubinemia, n = 591), (2) had a bilirubin level
that was projected from the rate of rise to be above the threshold within 24 hours (projected
hyperbilirubinemia, n = 557) (Figure, B, provides an example), or (3) met neither condition
(nonhyperbilirubinemia, n = 9980). For patients with hyperbilirubinemia, we evaluated
phototherapy ordering rates within 4 hours of the first documented serum bilirubin level result above
the AAP phototherapy threshold during hospitalization. For patients with projected
hyperbilirubinemia or nonhyperbilirubinemia, we evaluated phototherapy ordering rates for any time
during hospitalization.

Generalized linear models were used for the evaluation of all measures. Gamma regression was
used for length of stay and logistic regression was used for all other measures. Covariates included
in the model were gestational age less than 38 weeks, positive direct Coombs test results, and, for
the phototherapy-ordering measures for patients with projected hyperbilirubinemia or
nonhyperbilirubinemia, the shortest distance between a bilirubin level and the phototherapy
threshold. Estimated marginal means and percentages were calculated at mean covariate values.
Odds ratios and fold increases were estimated by exponentiating the regression coefficients.
Percentage changes in odds were calculated as the odds ratio −1, then multiplied by 100.
In February 2019, an invitation to participate in a System Usability Scale (SUS) survey was
emailed to 208 clinician users who had used the Bili App in January 2019; of these, 109 users
(52.40%) participated.19 The SUS survey is composed of 10 statements with responses ranging from
1 (strongly disagree) to 5 (strongly agree). The survey results in a single score ranging from 0 to 100.
Bangor et al20 described the adjective ratings associated with SUS scores: worst imaginable (mean
SUS score, 12.5), awful (20.3), poor (35.7), okay (50.9), good (71.4), excellent (85.5), and best
imaginable (90.9).
Statistical Analysis
Data for analyses were retrieved from the enterprise data warehouse. All statistical analyses were
performed using R, version 3.5.1 (R Foundation) and 2-tailed tests. P values <.05 were considered
significant. The study took place between April 1, 2016, and September 3, 2019. Data analyses were
conducted from October 30, 2018, to September 23, 2019.
Results
The Bili App required a mean of 35 seconds (95% CI, 27-42 seconds) to complete the bilirubin
management tasks compared with 100 seconds (95% CI, 89-112 seconds) with a commonly used tool
(66-second time savings, 95% CI, 53-79 seconds; P < .001) (Table 1). Interns saved 81 seconds, while
second and third-year residents saved 56 seconds.
In 2018, the app was used 20 516 times (Table 2), including 17 812 times for 91.84% of births in
the inpatient setting and 2704 times in outpatient settings for 39.18% of patients with follow-up
Table 1. Randomized Task-Timing Study: Time Required to Complete Bilirubin Management Tasks
No. Time, Estimated Marginal Mean (95% CI), s

Physician Group Patients Residents BiliTool Time UUH App Time Time Savings P Valuea
All resident physicians 42 12 100 (89-112) 35 (27-42) 66 (53-79) <.001
Interns 21 7 117 (99-134) 35 (24-47) 81 (64-98) <.001
Second- and third-year residents 21 5 88 (82-94) 32 (29-35) 56 (49-63) <.001
a
P values are based on linear regression using generalized estimating equations to account for within-physician correlation.
Table 2. App Use Study: Number of Uses and Estimated Time Savings per Year in 2018a
Clinician Role No. of Clinicians Uses per Clinician, Mean (95% CI) Sum of Uses per Role Estimated UUH Time Savings (95% CI)
All clinicians 527 38.93 (32.78-45.08) 20 516 374.36 (301.19-447.59)
Physician
Resident 107 120.42 (100.18-140.66) 12 885 235.12 (189.16-281.11)
Fellow and attending 77 51.77 (34.35-69.19) 3986 72.73 (58.52-86.96)
Registered nurse 162 13.25 (9.86-16.64) 2146 39.16 (31.50-46.82)
Medical student 136 8.34 (7.56-9.12) 1134 20.69 (16.65-24.74)
Other 45 8.11 (3.49-12.73) 365 6.66 (5.36-7.96)
Abbreviation: UUH, University of Utah Health.

a
Time savings are estimated based on an assumption that users saved 66 (95% CI,
53-79) seconds per use.

visits. The app was accessed by 77 fellows and attending physicians, 107 residents, 162 registered
nurses, 136 medical students, and 45 other clinicians, such as nurse practitioners and physician
assistants (Table 2). Given that the app was used 20 516 times overall, we estimated 374.36 (95% CI,
301.19-447.59) hours of time savings annually for all clinicians at UUH, including 235.12 (95% CI,
189.16-281.11) hours of time savings for residents.
Patient characteristics and outcomes pre-post intervention are summarized in Table 3. There
were no significant changes in patient characteristics and health care use measures. For patients with
hyperbilirubinemia, the adjusted percentage of newborns for whom phototherapy was ordered
within 4 hours of the first documented inpatient serum bilirubin test result above the AAP treatment
threshold increased significantly from 74.27% to 84.12% (odds ratio [OR], 1.84; 95% CI, 1.16-2.90;
P = .009). For patients with projected hyperbilirubinemia, the adjusted percentage of newborns for
whom phototherapy was ordered increased significantly from 27.34% to 53.41% (OR, 3.05; 95% CI,
2.01-4.62; P < .001). For patients without hyperbilirubinemia, the adjusted percentage of newborns
for whom phototherapy was ordered remained stable (OR, 0.95; 95% CI, 0.68-1.33; P = .78).
The mean SUS score indicated excellent usability (SUS, 83.90; 95% CI, 81.49-86.31) (Table 4).20
Attending physicians rated the system at 91.05 (95% CI, 86.31-95.79), resident physicians at 86.56
(95% CI, 82.14-90.98), registered nurses at 81.63 (95% CI, 81.63 77.85-85.41), and medical students
at 76.25 (95% CI, 65.25-87.25). The benchmark mean SUS score across industries is 68.21 The app
Table 3. Retrospective Pre-Post Intervention Study: Patient Characteristics and Outcomes
Variable Preintervention Postintervention OR/Fold Increase P Value

Patient Characteristics, No. (%)a
Eligible newborns 3714 7520 NA
Female 1875 (50.48) 3737 (49.69) NA .43
Gestational age <38 wk 662 (17.82) 1443 (19.19) NA .08
Newborns with positive direct Coombs test result 303 (8.16) 550 (7.31) NA .11
Newborns with serum bilirubin test result 3678 (99.03) 7450 (99.07) NA .84
Patient Outcomes, Estimated Marginal Mean (95% CI)b
Health care use measuresc
Length of stay after birth, d 3.12 (3.00-3.25) 3.08 (2.97-3.19) 0.99 (0.95-1.02) .43
Intensive care unit admission, % 8.51 (7.25-9.97) 9.36 (8.22-10.63) 1.11 (0.94-1.31) .21
Urgent care visit, % 2.58 (1.95-3.4) 2.97 (2.36-3.74) 1.16 (0.90-1.49) .25
Readmission, % 10.71 (9.34-12.26) 10.69 (9.51-11.99) 1.00 (0.87-1.15) .97
Phototherapy ordering, %
Patients with hyperbilirubinemia (n = 591)c 74.27 (66.51-80.75) 84.12 (80.11-87.45) 1.84 (1.16-2.90) .009
Patients with projected hyperbilirubinemia (n = 557)d 27.34 (20.36-35.65) 53.41 (46.44-60.25) 3.05 (2.01-4.62) <.001
Patients without hyperbilirubinemia (n = 9980)d 0.25 (0.16-0.39) 0.24 (0.15-0.37) 0.95 (0.68-1.33) .78
d
Abbreviations: NA, not applicable; OR, odds ratio. Values adjusted for the gestational age less than 38 weeks, positive direct Coombs test
a 2
P values based on χ tests. results, and distance of the bilirubin level from the phototherapy threshold.
b
P values based on multivariate regression.
c
Values adjusted for gestational age less than 38 weeks and positive direct Coombs
test results.
Table 4. Usability Survey Study: SUS Scoresa
Clinician Role No. of Clinicians SUS (95% CI)

Abbreviation: SUS, System Usability Scale.
All clinicians 109 83.90 (81.49-86.31)
a
The SUS survey is composed of 10 statements with
Physician
responses ranging from 1 (strongly disagree) to 5
Resident 24 86.56 (82.14-90.98) (strongly agree). The survey results in a single score
Attending 19 91.05 (86.31-95.79) ranging from 0 to 100. Bangor et al20 described the
Registered nurse 49 81.63 (77.85-85.41) adjective ratings associated with SUS scores: worst
Medical student 8 76.25 (65.25-87.25) imaginable (mean SUS score, 12.5), awful (20.3),
poor (35.7), okay (50.9), good (71.4), excellent
Other 9 80.83 (69.58-92.08)
(85.5), and best imaginable (90.9).

was recognized with multiple awards from the Department of Health and Human Services’ Provider
User-Experience Challenge for EHR-integrated apps.22
Discussion
Electronic health records are evolving into platforms in which third-party apps add value to their
users through integrated tools targeted at specific decisions and tasks. This study may provide
empirical support for this vision, in which an add-on app for neonatal bilirubin management was
widely used, was associated with clinicians time savings and improved guideline-compliant care, and
had high perceived usability. To our knowledge, this is one of the first studies to provide real-world,
long-term data suggesting that EHR add-on apps using the emerging SMART on FHIR standard can
save clinicians time, improve care, and provide a positive user experience.
The app focuses on a narrow domain and saves only a minute per use, and yet the potential for
time savings if deployed on a national scale is significant. If extrapolated to the approximately 3.4
million nonpreterm births annually in the United States,23 universal use of the app could potentially
save more than 300 000 hours of clinician time every year. Given the limited extent to which EHRs
currently support task-optimized data retrieval and synthesis, there are likely millions of hours of
clinician time each year that could be saved through the widespread deployment of similar EHR
add-on apps developed through a user-centered design process.
Beyond the time savings, another finding was the improvement in appropriate phototherapy
ordering. Studies have shown that patients often do not receive recommended care.24,25 Moreover,
these challenges in care quality may be associated with the time constraints that clinicians face;
Yarnall et al26 found, for example, that just satisfying US Preventive Services Task Force A and B
recommendations would require 7.4 hours per working day of physician time. By making it easier to
provide appropriate care, tools such as the app could engender a positive cycle of clinician time
savings and improved patient care.
Through the University of Utah ReImagine EHR initiative, we now have experience with
implementing a number of SMART on FHIR apps in addition to the bilirubin app. Based on this
experience, we believe that several specific features of the bilirubin app have helped to make this
implementation successful. These system features include those identified in a meta-analysis of the
literature as being important to CDS success: provision of CDS at the time and location of decision-
making, provision of recommendations rather than just assessments, integration with the EHR, and
the minimal need for additional clinician data entry.12 We believe our findings are also congruent with
systematic reviews of information displays for critical care, which found that comprehensive
information displays that integrate information from multiple sources (eg, laboratory test results,
procedures) and present trend data graphically had the strongest association with positive clinician
performance and patient outcomes, possibly owing to improved pattern recognition and situational
awareness.27,28 By automating low-level cognitive tasks, such as retrieving, organizing, sorting, and
graphically representing data, integrated information displays let clinicians spare valuable cognitive
resources for high-level and complex cognitive tasks. For example, the significantly higher odds of
phototherapy orders among patients with projected hyperbilirubinemia suggest that the app may
help clinicians to more efficiently identify patients for whom phototherapy may be indicated (Figure,
B). Our findings are consistent with cognitive research on working memory, which has found that
cognitive load can be reduced—and user performance improved—through displays that match users’
mental models29 and automate subtasks, such as information searching, that distract and reduce
working memory.30
In addition, we believe key factors in the app’s widespread adoption are the reduction in
cognitive effort and associated time savings, which can provide a relief from the burden otherwise
imposed by EHRs.2,3 Because the app applies to practically every newborn, its use became almost
universal in the newborn nursery. As such, the app was able to automatically provide CDS as a part of

routine clinician workflow, which is a factor that has repeatedly been found to be critical to
CDS success.31,32
We are actively working to disseminate the app as a free tool through EHR app stores and hope
to report on the adoption and outcome of the app’s use at other institutions. We are also developing
numerous additional EHR add-on apps through the University of Utah ReImagine EHR initiative to
optimize patient care and the clinicians’ EHR experience. Further research is also needed on how to
best support clinical decision-making through EHR add-on apps, such as through the application of
cognitive science principles and CDS best practices.
Limitations
This study has limitations. Although the app uses a standards-based approach that should be
deployable across health care systems and EHR products, this was a single-center study, and further
studies are needed to demonstrate generalizability. The pre-post intervention design could be
confounded by secular trends; however, we used multivariate, generalized linear models to adjust for
covariates. Also, for the health care use measures (length of stay, intensive care unit admission,
urgent care visit, and readmission), we did not limit our analyses to events related to
hyperbilirubinemia (eg, intensive care unit admissions due to hyperbilirubinemia). Conversely, the
task-timing study was limited to inpatient residents, such that overall time savings may differ from
our estimates if other types of clinicians save more or less time from use of the app. Another
limitation is that local users were actively involved in the development of the intervention—a factor
known to correlate with CDS success12 and thus potentially overestimate the outcome that can be
expected at other institutions. However, most users of the app are residents, who rotate in and out of
the nursery service on a regular basis and had no involvement in the development of the app. In
addition, while EHR add-on apps are a promising solution to support clinicians in complex cognitive
tasks, they are unlikely to replace more transactional EHR functionality that is tightly coupled with
the EHR, such as order entry. Where possible, native EHR functionality should be improved to
preserve a cohesive user experience. To this end, as EHR add-on apps grow in their prevalence, it will
become increasingly important for SMART on FHIR apps and the underlying EHRs to adopt common
user interface conventions.
Conclusions
The findings of this study suggest that well-designed EHR add-on-apps can save clinicians time,
improve care, and enhance the EHR user experience by supporting complex decision tasks. Further
research is needed to evaluate the generalizability of these findings in other health care domains and
settings. If well-designed EHR add-on apps were widely implemented across health care
organizations, the potential for improved patient care and clinician efficiency could be significant.
ARTICLE INFORMATION
Accepted for Publication: September 24, 2019.
Published: November 15, 2019. doi:10.1001/jamanetworkopen.2019.15343
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Kawamoto
K et al. JAMA Network Open.
Corresponding Author: Kensaku Kawamoto, MD, PhD, MHS, Department of Biomedical Informatics, University of
Utah, 421 Wakara Way, Ste 108, Salt Lake City, UT 84108 ([email protected]).
Author Affiliations: Department of Biomedical Informatics, University of Utah, Salt Lake City (Kawamoto,
Kukhareva, Kramer, Rodriguez, Warner, Shields, Weir, Del Fiol, Taft); Department of Pediatrics, University of Utah,
Salt Lake City (Shakib, Stipelman).

Author Contributions: Dr Kawamoto had full access to all of the data in the study and takes responsibility for the
integrity of the data and the accuracy of the data analysis.
Concept and design: All authors.
Acquisition, analysis, or interpretation of data: Kukhareva, Shakib, Kramer, Shields, Weir, Del Fiol, Taft, Stipelman.
Drafting of the manuscript: Kawamoto, Kukhareva, Shields, Weir.
Critical revision of the manuscript for important intellectual content: Kawamoto, Kukhareva, Shakib, Kramer,
Rodriguez, Warner, Weir, Del Fiol, Taft, Stipelman.
Statistical analysis: Kukhareva, Weir.
Obtained funding: Kawamoto.
Administrative, technical, or material support: Rodriguez, Warner, Shields, Weir.
Supervision: Kawamoto, Weir, Del Fiol, Stipelman.
Conflict of Interest Disclosures: Dr Kawamoto reported receiving awards and associated cash prizes for the Bili
App from the US Department of Health and Human Services (HHS) Provider User-Experience Challenge during the
conduct of the study; grants and personal fees from Hitachi, personal fees from McKesson InterQual, Premier,
Klesis Healthcare, Research Triangle Institute, the Mayo Clinic, Vanderbilt University, University of Washington,
University of California at San Francisco, and the US Office of the National Coordinator for Health IT (via Enterprise
Science and Computing, JBS International, A+ Government Solutions, Hausam Consulting, and Security Risk
Solutions) outside the submitted work. All authors reported receiving awards and associated cash prizes for the Bili
App from the HHS Provider User-Experience Challenge during the conduct of the study. Mr Warner reported
receiving personal fees from Phast-Services during the conduct of the study. No other disclosures were reported.
Funding/Support: This work was funded by the University of Utah.
Role of the Funder/Sponsor: The University of Utah had no role in the design and conduct of the study; collection,
management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and
decision to submit the manuscript for publication.
Additional Information: The app was recognized with multiple awards and associated cash prizes from the
Department of Health and Human Services’ Provider User-Experience Challenge for electronic health records-
integrated apps.
REFERENCES
1. Schulte F, Fry E. Death by a thousand clicks: where electronic health records went wrong. Kaiser Health News.
https://khn.org/news/death-by-a-thousand-clicks/. Published March 18, 2019. Accessed June 21, 2019.
2. Sinsky C, Colligan L, Li L, et al. Allocation of physician time in ambulatory practice: a time and motion study in 4
specialties. Ann Intern Med. 2016;165(11):753-760. doi:10.7326/M16-0961
3. Chaiyachati KH, Shea JA, Asch DA, et al. Assessment of inpatient time allocation among first-year internal
medicine residents using time-motion observations. JAMA Intern Med. 2019;179(6):760-767. doi:10.1001/
jamainternmed.2019.0095
4. Gardner RL, Cooper E, Haskell J, et al. Physician stress and burnout: the impact of health information
technology. J Am Med Inform Assoc. 2019;26(2):106-114. doi:10.1093/jamia/ocy145
5. Mandl KD, Mandel JC, Kohane IS. Driving innovation in health systems through an apps-based information
economy. Cell Syst. 2015;1(1):8-13. doi:10.1016/j.cels.2015.05.001
6. Mandl KD, Kohane IS. No small change for the health information economy. N Engl J Med. 2009;360(13):
1278-1281. doi:10.1056/NEJMp0900411
7. Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps
platform for electronic health records. J Am Med Inform Assoc. 2016;23(5):899-908. doi:10.1093/jamia/ocv189
8. Health Level Seven International (HL7). SMART App Launch Framework. http://www.hl7.org/fhir/smart-
app-launch/. Accessed September 23, 2019.
9. Health Level Seven International (HL7). Fast Healthcare Interoperability Resources (FHIR) standard. https://www.
hl7.org/fhir/. Accessed September 23, 2019.
10. Twichell SA, Rea CJ, Melvin P, et al. The effect of an electronic health record–based tool on abnormal pediatric
blood pressure recognition. Congenit Heart Dis. 2017;12(4):484-490. doi:10.1111/chd.12469
11. American Academy of Pediatrics Subcommittee on Hyperbilirubinemia. Management of hyperbilirubinemia in
the newborn infant 35 or more weeks of gestation. Pediatrics. 2004;114(1):297-316. doi:10.1542/peds.114.1.297
12. Lobach D, Sanders GD, Bright TJ, et al. Enabling health care decisionmaking through clinical decision support
and knowledge management. Evid Rep Technol Assess (Full Rep). 2012;(203):1-784.

13. Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality
Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual
Saf. 2016;25:986-992. doi:10.1136/bmjqs-2015-004411
14. Longhurst C, Turner S, Burgos AE. Development of a Web-based decision support tool to increase use of
neonatal hyperbilirubinemia guidelines. Jt Comm J Qual Patient Saf. 2009;35(5):256-262. doi:10.1016/S1553-7250
(09)35035-7
15. Thornton SN, Thompson BS, Millar JA, Eggert LD, Wilcox AB. Neonatal bilirubin management as an
implementation example of interdisciplinary continuum of care tools. AMIA. Annual Symposium Proceedings.
2007;2007:726-730.
16. SMART App Gallery. Intermountain Healthcare bilirubin chart. https://apps.smarthealthit.org/app/bilirubin-chart.
Accessed July 10, 2019.
17. US Core FHIR Profiles. (Release 1.0.1). Health Level Seven International (HL7). http://hl7.org/fhir/us/core/1.0.1/.
Accessed July 10, 2019.
18. Chang PW, Kuzniewicz MW, McCulloch CE, Newman TB. A clinical prediction rule for rebound
hyperbilirubinemia following inpatient phototherapy. Pediatrics. 2017;139(3):e20162896. doi:10.1542/peds.
2016-2896
19. Brooke J. SUS—a quick and dirty usability scale. In: Usability Evaluation in Industry. Vol 189. 1996:4-7. https://cui.
unige.ch/isi/icle-wiki/_media/ipm:test-suschapt.pdf. Accessed October 15, 2019.
20. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale.
J Usability Stud. 2009;4(3):114-123. http://uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS_Bangor_
May2009.pdf. Accessed October 15, 2019.
21. Sauro J. SUStisfied? Little-known system usability scale facts. User Exp Mag. 2011;10(3). https://uxpamagazine.
org/sustified/. Accessed September 23, 2019.
22. Office of the National Coordinator for Health Information Technology. Provider user-experience challenge. https://
www.healthit.gov/techlab/ipg/node/4/submission/2131. Published 2017. Accessed July 10, 2019.
23. Centers for Disease Control and Prevention. National Vital Statistics System—Birth Data. https://www.cdc.gov/
nchs/nvss/births.htm. Accessed June 21, 2019.
24. Levine DM, Linder JA, Landon BE. The quality of outpatient care delivered to adults in the United States, 2002
to 2013. JAMA Intern Med. 2016;176(12):1778-1790. doi:10.1001/jamainternmed.2016.6217
25. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl
J Med. 2003;348(26):2635-2645. doi:10.1056/NEJMsa022615
26. Yarnall KSH, Pollak KI, Østbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention?
Am J Public Health. 2003;93(4):635-641. doi:10.2105/AJPH.93.4.635
27. Wright MC, Borbolla D, Waller RG, et al. Critical care information display approaches and design frameworks:
a systematic review and meta-analysis. J Biomed Inform X. 2019;3:3. doi:10.1016/j.yjbinx.2019.100041
28. Waller RG, Wright MC, Segall N, et al. Novel displays of patient information in critical care settings: a systematic
review. J Am Med Inform Assoc. 2019;26(5):479-489. doi:10.1093/jamia/ocy193
29. Ericsson KA. Superior working memory in experts. In: The Cambridge Handbook of Expertise and Expert
Performance. Cambridge, UK: Cambridge University Press; 2018:696-713. doi:10.1017/9781316480748.036
30. Camos V. Domain-specific versus domain-general maintenance in working memory: reconciliation within the
time-based resource sharing model. Prog Brain Res. 2017;232:135-171. doi:10.1016/bs.plm.2017.03.005
31. Van de Velde S, Heselmans A, Delvaux N, et al. A systematic review of trials evaluating success factors of
interventions with computerised clinical decision support. Implement Sci. 2018;13(1):114. doi:10.1186/s13012-018-
0790-1
32. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support
systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi:10.
1136/bmj.38398.500764.8F

Kawamoto-2019-Association of An Electronic Hea

Uploaded by

Copyright:

Available Formats

Kawamoto-2019-Association of An Electronic Hea

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Kawamoto-2019-Association of An Electronic Hea

Uploaded by

Copyright:

Available Formats

Original Investigation | Health Informatics

Association of an Electronic Health Record Add-on App for Neonatal

Abstract Key Points

Downloaded From: https://jamanetwork.com/ on 07/17/2020

JAMA Network Open. 2019;2(11):e1915343. doi:10.1001/jamanetworkopen.2019.15343

Downloaded From: https://jamanetwork.com/ on 07/17/2020

Downloaded From: https://jamanetwork.com/ on 07/17/2020

Figure. App Screenshots for Bilirubin Analysis

A Patient with hyperbilirubinemia

B Patient with projected hyperbilirubinemia

Screenshots shown for a patient with

Downloaded From: https://jamanetwork.com/ on 07/17/2020

Downloaded From: https://jamanetwork.com/ on 07/17/2020

No. Time, Estimated Marginal Mean (95% CI), s

Abbreviation: UUH, University of Utah Health.

Downloaded From: https://jamanetwork.com/ on 07/17/2020

Table 3. Retrospective Pre-Post Intervention Study: Patient Characteristics and Outcomes

Variable Preintervention Postintervention OR/Fold Increase P Value

Table 4. Usability Survey Study: SUS Scoresa

Clinician Role No. of Clinicians SUS (95% CI)

Downloaded From: https://jamanetwork.com/ on 07/17/2020

Downloaded From: https://jamanetwork.com/ on 07/17/2020

Downloaded From: https://jamanetwork.com/ on 07/17/2020

Downloaded From: https://jamanetwork.com/ on 07/17/2020

Downloaded From: https://jamanetwork.com/ on 07/17/2020

You might also like