Quality Improvement Studies in Pediatric Critical Care Medicine
Quality Improvement Studies in Pediatric Critical Care Medicine
Quality Improvement Studies in Pediatric Critical Care Medicine
O
ver the past 15 years, healthcare leaders and practitioners have begun
Downloaded from https://journals.lww.com/pccmjournal by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8KKGKV0Ymy+78= on 07/11/2021
Well-executed QI advances patient outcomes (13, 14), while QI science DOI: 10.1097/PCC.0000000000002744
inappropriately or half-heartedly applied slows im- sedation (either under-or over-sedation), leading to a
provement rate and magnitude, ultimately harming different aim and strategy. Taking time to ponder the
patients. When improvement work and subsequent re- problem (open to adjusting understanding in early
porting are weak – not addressing the local problem project stages) leads to a more effective project. For
in context, not describing problem drivers and Plan- example, thoroughly describing the problem in the
Do-Study-Act (PDSA) testing cycles, and failing to manuscript helps readers compare their local situation
incorporate QI science statistical analysis (e.g., run or to that described by the authors, thereby increasing
control charts) – local and extrapolated learning from generalizability.
projects are hampered (lower return on investment). Rationale: Just as traditional research requires a hy-
Poorly executed QI and poorly written QI reports di- pothesis before experimentation, QI projects (experi-
minish the critical care community’s opportunity to ments to improve a system) require a rationale. A
learn from others’ good work, emulate and implement rationale is an opportunity to develop theories re-
that good work, and ultimately improve outcomes. garding current system functioning. The QI team
Here we describe the necessary components of a and the published manuscript should address specific
QI project and the subsequent manuscript describing questions: 1) is the problem solvable and why?; 2) what
that project. Because there are many publications and is causing current system underperformance?; and 3)
textbooks on this topic (15–17), we emphasize essen- why does the team believe specific leverage points and
tial issues or frequently omitted SQUIRE guideline interventions are important? This exercise challenges
elements (7, 8) in many published QI manuscripts. A the QI team to understand the problem entirely, and
well-performed improvement project and well-written when described in the manuscript, the reader is more
manuscript address each of these topics. Notably, a informed regarding how this project could be locally
well-done peer review of the submitted paper insists emulated.
these elements are included before publication. We Specific Aim: Improvement projects require an aim
suggest reviewing SQUIRE elements before and dur- statement, guiding the team on the improvement
ing the project, not just when writing, which helps im- journey (15). Current and goal performance should
prove project execution from the start. be described, outlining for the QI team “where we are
now” and “where we are going.” When absent, the pro-
ject is prone to lose focus and more likely to find the
ELEMENTS REQUIRED FOR QI WORK work at a dead-end instead of reaching the intended
IN PCCM goal. A well-conceived aim statement clearly defines
Several elements make a QI report good and useful. the project target (patients and/or system), makes it
Table 1 provides a summary, with excerpts from exem- easier to identify data collection needs, and most often
plary publications (18–23), to support each descrip- prevents scope creep. Finally, a date by when achieve-
tion below. ment is expected motivates a team to keep making
Problem Description: Before performing QI, the team progress. A QI manuscript without an aim statement
must identify the problem, which requires more than in the introduction or methods is a marker the docu-
superficial questioning and introspection. Einstein ment may not describe actual improvement work.
said, “If I had an hour to solve a problem, I’d spend Context: All QI work happens in a complex socio-
55 minutes thinking about the problem and 5 min- technical system. Without considering and reporting
utes thinking about the solution.” Using the 5-Why’s on institutional system specifics, others cannot repli-
approach is applicable here, repeatedly asking “why is cate the work or adapt learnings to their environment.
that a problem?” leading the team to core issues requir- Just as medication effectiveness studies thoroughly de-
ing change before starting. For example, a team may scribe patient characteristics and basic science papers
believe their problem is insufficient sedation in post- report cell culture conditions, describing the institu-
operative patients, leading them to pursue specific tional system and microsystem is essential, allowing
interventions. However, with in-depth problem con- the reader to better understand (contextualize) imple-
sideration before starting the project, they realize the mented interventions and results (24). One resource to
problem isn’t insufficient sedation but inappropriate aid teams and authors in broadening their thinking at
TABLE 1.
Exemplary Writing from the Pediatric Critical Care Medicine Quality Improvement Literature
Element Good examples in the text
Problem “Sedation for critically ill pediatric cardiac surgery patients reduces distress, eases adaption to
Description the ventilator, reduces oxygen consumption and carbon dioxide production, and facilitates
(18) stable hemodynamics. In this context, patients benefit from the sedative side effects of opioids
typically used to manage pain in patients recovering from cardiac surgery. However, dispropor-
tionate sedation leads to longer ICU and hospital stays, significant morbidities and mortality,
iatrogenic withdrawal syndrome, delirium, and long-term neurologic and behavioral difficulties.
Protocolized sedation for mechanically ventilated adult patients improves outcomes including
decreased morbidity, length of stay, and ventilation, however pediatric data are limited.”
Rationale “Our institution displayed practice variations in the timing of orders, class of ASMs (anti-seizure
(19) medications), and timing of neurology consults. The process of dispensing ASMs from the phar-
macy was often delayed due to a request for verbal confirmation from an ordering physician.
The treatment model for patients with CSE (convulsive status epilepticus) was a collaborative
approach between pediatrics and neurology departments. We describe the development of an
intervention to reduce variability in patient care while promoting adherence to evidence-based
practices which were subsequently published as a treatment guideline by the American Epi-
lepsy Society (AES).”
Specific Aim “The multidisciplinary team identified a specific, measurable, attainable, relevant, time-bound aim
(20) to decrease average IV acetaminophen usage by 30% from 0.55 doses per patient day (DPPD)
to 0.38 DPPD by December 2017.”
Context “This QI initiative took place within the Cardiac Center at The Children’s Hospital of Philadelphia.
(21) All cardiac patients with arterial catheters placed between January 2015 and July 2017 were
included. Umbilical arterial catheters and catheters placed at outside institutions were excluded.
The Cardiac Center performs approximately 750–1,000 cardiac surgical interventions annu-
ally in addition to several thousand cardiac catheterizations and electrophysiology studies. On
average, 250 patients per month require indwelling peripheral arterial access for procedural and
CICU monitoring needs.”
Measures “Primary outcome measures included SWI (sternal wound infection) rates per 100 sternotomies
(22) for patients undergoing either PC (primary closure) or DSC (delayed sternal closure) before
and after implementation of the SWPB (sternal wound prevention bundle). Process measures
included compliance with SWPB elements.”
Interpretation “In our study, the delirium prevalence declined over the course of the 22-month project with the ex-
(23) ception of 4 months: March 2014, November 2014, December 2014, and May 2015. The high-
est monthly prevalence (28%) in March 2014 coincided with the rollout of the updated Cornell
Assessment of Pediatric Delirium revised (CAP-D[R]) instrument. The CAP-D(R) was published
a few months after project implementation. The CAP-D(R) included an additional question to bet-
ter capture a fluctuating course of delirium and included developmental anchor points to define
expected and observable behaviors for newborns to children 2 years old. Staff acknowledged
that this guidance based on age aided in their ability to screen young infants which may explain
the initial increased prevalence peak until staff became confident using the updated tool.”
multiple levels about contextual factors that can impact and denominators for rates or percentages, is essential.
the QI project is the Model for Understanding Success in Specifying when the data collection begins and ends is
Quality (MUSIQ) (25). vital. QI projects require multiple measures. Minimally
Measures: For all metrics, precisely describing what one process measure (the uptake of an intervention
is and is not being counted, including the numerators such as a new protocol) and one outcome measure
(the intended result) must be analyzed and reported. and after interventions, and conclude that improved
Without quantifying both measures, one cannot link outcomes derive from the intervention(s). With pooled
interventions with outcomes. Often a project will test data, random noise can lead to false-positive results
multiple interventions. Consequently, numerous pro- over short time periods, and longer time periods may
cess measures may be required, including a few related obscure a shift in system performance not temporally
outcome measures (e.g., proximal outcomes such as related to the intervention. For example, statistically,
sternal wound infection rate and more distal outcomes significant pre-/post-intervention differences are fre-
such as survival to discharge). quently shown as a table (Table 2) or a bar graph (Fig.
Analysis: A comprehensive QI data analysis review is 1A). With this analysis, the QI team and manuscript
not within our scope; therefore, we focus on one com- readers might falsely conclude the results derive from
mon and widespread QI analytical error. Authors often the intervention(s) because the different results in the
report pooled data from two time periods, usually before two time periods are statistically different. However,
Figure 1. Four ways of displaying data after an intervention. A. Pre-/Post Bar graph: does not show when unplanned extubation (UE)
rate improved relative to the intervention. B. Time series of pre-intervention data: correctly shows the 18-week pre-intervention period. C.
Time series with improper shift: the graph has the data broken into two periods, forced at the time of the intervention, although the data
(using statistical process control rules) does not indicate system performance changed then. D. Time series with the proper shift: the
graph shows the appropriate shift (using Nelson Rules, (27)), when eight or more consecutive points were first below the old baseline.
The shift started three weeks before the intervention.
TABLE 2. data analysis are available elsewhere (16, 17, 30), in-
Unplanned Extubation Percentages After cluding data sampling appropriate for different control
a New Protocol chart types and achieving appropriate power (29).
Interpretation: Describing project outcomes requires
UE No UE Row Totals
discussion beyond just restating “intervention x led to
Old Protocol 134 (47%) 151 (53%) 285 improvement y.” The interpretation should outline why
New Protocol 36 (16%) 185 (84%) 221
interventions did or did not work, why the change mag-
nitude was as large or small as observed, and what the de-
Column Totals 170 336 506 gree of improvement reveals about the interplay between
UE = unplanned extubation. interventions and the context/system. The rationale
Data from each period covers 18 weeks. Chi-square p-value (described earlier) and interpretation are paired sections
< 0.00001, indicating the percentage of patients with one or enhancing understanding of the complex sociotechnical
more UEs was significantly lower (16%) in the period after a new systems, thus improving study generalization. Without a
protocol was introduced compared to when the old protocol was well-developed interpretation, the project spread could be
used (47%). This data does not prove the intervention (new pro-
limited to primarily those with nearly identical problems
tocol) caused or was associated with the improvement.
and contexts. Further interpretation of QI work must
place the results in the context of others who have done
analyzing and reporting QI data using an interrupted
similar work. Repeating one’s conclusions is insufficient.
time-series / statistical process control chart (Shewhart
chart) approach (26, 27) (Fig. 1B), displaying normal
variation, and system performance over time, are NEXT STEPS TO A NEW PARADIGM
preferred. A frequent error in displaying time-series
FOR PCCM QI REPORTS
control charts is annotating a centerline shift inappro- We can and must improve by raising the quality of
priately. The centerline shift is driven by the data, not QI project execution, manuscript writing, and peer
by intervention timing. Specifically, the chart in Figure review.
1C is divided into two time periods, arbitrarily split QI Execution: The primary barriers to excellent QI
when the intervention occurred, replicating the error execution include: 1) inadequate institutional dedi-
in Table 2 and Fig. 1A. Table 2, Fig 1A, and Fig 1C in- cated QI resources; 2) insufficient allocated faculty non-
form the team and reader that system performance clinical time dedicated to QI endeavors; 3) inadequate
was different after the intervention (e.g., new pro- training and experience for PICU practitioners. The
tocol), but not that system performance was different rigors of clinical care in the PICU may inhibit practi-
because of the intervention. Figure 1D is the proper tioners seeking QI science training. Although requiring
analysis, wherein a shift in centerline mean is indicated important institutional investment, faculty QI science
when the data first show improvement [i.e., 8 points in training can yield a significant return for patients and
a row below the previous centerline (mean) (Nelson healthcare systems (31–34). Intra-institutional QI
rules)] (27). Here the system had two different stable training options for practicing faculty are becoming
performance periods, first at 51%, then at 11%, and more abundant [e.g., Nationwide Children’s, Cincinnati
the shift occurred three weeks before the intervention. Children’s, Seattle Children’s, Children’s National, and
Therefore, claiming the improvement arose from the others (personal communications)]. Notably, institu-
intervention would be incorrect, as a reader might de- tions without such programs should consider devel-
duce from Table 2, Figs. 1A, and 1C. Consequently, the oping internal courses or send PICU staff to external
QI team and those reading the manuscript regarding courses. While online QI courses (e.g., Institutes for
this work should consider other improvement causes. Healthcare Improvement) are good introductory infor-
A key advantage of using control charts as part of mation, we believe that QI is learned by doing projects,
the analysis is they are based on rigorous statistics and not reading about them. Another barrier in many ac-
account for sample size and the number of events (28, ademic institutions is the low value placed on having
29). More detailed and complete explanations of inter- faculty perform and publish QI, often signaled through
rupted time series analysis and control chart-driven promotion and tenure (P&T) processes (35). Suppose
P&T committees or Department Chairs indicate a ca- the bar to describe “what should be” and “how we get
reer focusing on QI is less viable than more traditional there”, our patients everywhere will benefit.
paths for academic promotion. In that case, faculty
will vote with their feet and de-emphasize QI work to 1 Division of Neonatology, Department of Pediatrics,
prioritize the many competing interests for their time Nationwide Children’s Hospital, Columbus, OH
(36). Fortunately, we observe QI career paths are be- 2 John F. Wolfe Endowed Chair in Medical Leadership and
coming more popular as more PICU professionals seek Pediatric Quality and Safety; Division of Critical Care
advanced QI training, and Department Chairs are in- Medicine, Department of Pediatrics, Nationwide Children’s
creasingly supporting such activity. Hospital, Columbus, OH
QI Writing: The SQUIRE guidelines were released 13 The authors have disclosed that they do not have any potential
conflicts of interest.
years ago (7, 8). It is time that all QI publications adhere
to the guidelines and incorporate the majority of the For information regarding this article, E-mail: rbrilli@nationwide-
childrens.org
components. While writing a QI manuscript is not fun-
damentally different than other research manuscripts,
participating in QI writing training may be helpful (37).
Readers should note that PCCM’s new 2,500 to 3,000 REFERENCES
word-length article with a structured format to all sec- 1. Jones E, Furnival J, Carter W: Identifying and resolving the
tions (Introduction, Methods, Results, Discussion) requires frustrations of reviewing the improvement literature: The
adherence to the SQUIRE template for QI reports (38). experiences of two improvement researchers. BMJ Open Qual
QI Manuscript Peer Review: As previously mentioned, 2019; 8:e000701
the quality of many published QI manuscripts is sub-opti- 2. Batalden PB, Davidoff F: What is “quality improvement” and
how can it transform healthcare? Qual Saf Health Care 2007;
mal. Peer-reviewers are instrumental in driving improve-
16:2–3
ment in manuscript quality. Reviewers should thoroughly
3. Kourtis SA, Burns JP: Quality improvement in pediatric inten-
critique papers and look for: 1) significant work that is sive care: A systematic review of the literature. Pediatr Investig
generalizable to other contexts; 2) novel work, even when 2019; 3:110–116
placed in the context of previous work; 3) a thorough de- 4. Kourtis SA, Burns JP: Frameworks for quality improvement
scription of a well-designed project which incorporates in pediatric intensive care: A concise review. Pediatr Investig
QI science tools and methods; 4) SQUIRE guideline for- 2019; 3:117–121
matting. As always, the editorial process must follow best 5. Inata Y, Nakagami-Yamaguchi E, Ogawa Y, et al: Quality
practices (39). As reviewers and editors elevate the pub- Assessment of the Literature on Quality Improvement in
PICUs: A Systematic Review. Pediatr Crit Care Med 2021 Feb
lication bar, future authors will have better literature on
26. [online ahead of print]
which to model their work and writing, creating a cycle
6. Hempel S, Shekelle PG, Liu JL, et al: Development of the
of improvement. Most importantly, as QI work quality Quality Improvement Minimum Quality Criteria Set (QI-MQCS):
(executed and published) improves, PICUs will begin to A tool for critical appraisal of quality improvement intervention
achieve better outcomes. The path of achieving those out- publications. BMJ Qual Saf 2015; 24:796–804
comes will be easier to discern. 7. Ogrinc G, Mooney SE, Estrada C, et al: The SQUIRE (Standards
for QUality Improvement Reporting Excellence) guidelines for
quality improvement reporting: Explanation and elaboration.
CONCLUDING COMMENTS Qual Saf Health Care 2008;17 (Suppl 1):i13–32
QI work and publishing have accelerated over the last 8. Ogrinc G, Davies L, Goodman D, et al: SQUIRE 2.0 (Standards
decade, albeit in our opinion, not fast enough. Now is for QUality Improvement Reporting Excellence): revised pub-
lication guidelines from a detailed consensus process. BMJ
the time to consider ourselves at the end of the begin-
Qual Saf 2016; 25:986–992
ning. Importantly, improvement science is becoming
9. Sacks OA, Lambour AJ, Wilcox AR, et al: Quality assessment
mainstream. However, going forward, we should be of the literature on surgical quality improvement. Surgery
dissatisfied with improvement projects performed 2019; 166:764–768
without rigorous QI science methodology and QI 10. Heneghan JA, Pollack MM: Morbidity: Changing the outcome
manuscripts omitting essential information. A new paradigm for pediatric critical care. Pediatr Clin North Am
paradigm of critical care QI is upon us. As we raise 2017; 64:1147–1165