COVID-19: Minimizing Critical Facility Risk: Authors Uptime Institute Intelligence Team
COVID-19: Minimizing Critical Facility Risk: Authors Uptime Institute Intelligence Team
COVID-19: Minimizing Critical Facility Risk: Authors Uptime Institute Intelligence Team
RESILIENCY
UI Intelligence report 37
COVID-19: Minimizing
critical facility risk
Authors
Uptime Institute Intelligence team
UII-37 v1.2 published March 16, 2020 last updated March 27, 2020
COVID-19: Uptime Institute Advisory, March 16, 2020
Executive summary 3
Responses 3
Prepare the business 4
Protect site and staff 5
Safeguard operations 10
Consider factors that raise risk 11
Conclusions 16
Appendix: Resources 17
EXECUTIVE SUMMARY
This advisory report has been produced by Uptime Institute, with the assistance of its clients and
members, to help operators of critical infrastructure facilities prepare for, and respond to, the
impact of the novel coronavirus that causes COVID-19. The steps discussed in this report will also
help operators develop strategies and procedures for future epidemics.
Responses
COVID-19 is a pandemic. In situations such as this, mission-
critical facilities face particular challenges, due to both the risk of
unavailability of key staff through illness or quarantine and other long-
term impacts that might affect the ability of the operator to maintain
continuous availability. Fortunately, preparedness is in the industry’s
DNA; thanks to their focus on performance, efficiency and reliability
— tested through prior experience with power blackouts, wildfire,
adverse weather and other potentially disruptive events — most data
centers owner/operators have contingency plans in place that can be
adapted to the challenges of the current pandemic.
As the virus spreads, more organizations are moving from updating
their response plans to implementing them. Each organization is
different, and responses vary based on site environment, the number
of COVID-19 cases in the area and government-mandated restrictions.
One thing all companies hold in common, though, is their priorities:
the health and safety of their staff, partners and customers; business
continuity; and compliance with the guidelines and regulations issued
by public health and government agencies.
This report reviews the current status of the data center industry’s
response to the pandemic and details recommendations and possible
next steps. These recommendations are based on feedback and
comments from Uptime Institute Network members and clients, the
Uptime Institute Intelligence team and Uptime Institute consultants in
five continents. The Appendix includes a list of relevant resources.
To further support owners and operators, Uptime Institute will
introduce a regular bulletin with updates on COVID-19. Details will be
posted on the Uptime Institute website, Uptime’s Inside Track portal
and LinkedIn after March 20, 2020.
In March 2020, Uptime Institute began a series of roundtable
discussions for its global network of data center operators and
partners. Participants reported their organizations have taken or are
considering taking many of the steps discussed below.
Prepare the A first and essential step is to be prepared. This includes the following
components:
business • Develop a specific pandemic preparedness plan. If a pandemic-
specific plan is not in place, use another emergency plan that may
have been prepared for civic emergences, etc. The plan should
incorporate a tiered response, clearly identifying the actions to
be taken at each level and the circumstances that would trigger
implementation of the next level. Most organizations have a three
to five-level contingency plan, ranging from taking reasonable
precautions through lights-out operation and, in worst cases, a
complete site shutdown with transfer of critical applications and
operations to backup sites. The plan should consider situations
in which staff may be unable to access or leave the site on short
notice.
• Confer with insurance companies and legal advisors on relevant
items, such as cleaning requirements, service level agreements
(SLAs), notifications, etc.
• Consider IT service (client) impact. Responses to COVID-19 may
affect internet traffic, workloads and availability requirements for
some clients. Operators should confer with clients, internal and
external, to discuss any impact, especially if upgrades or migrations
are planned or new capacity is being added and delays to those
projects may impact business unit operations or projects.
• Maintain communication with staff, customers and partners. This
is a dynamic situation, so frequent — daily or even twice-daily —
briefings may be appropriate as the conditions change and may
affect business operations.
• Share news updates and links to public resources to keep
staff informed of the current status of the pandemic and best
practices for maintaining a safe and healthy work environment
(see Appendix).
• Provide clear guidance to staff on company policies (and
regulatory policies) related to symptoms (personally or in
family members), cases of possible exposure, self-quarantine
parameters and duration, and implications for sick leave/paid
time off limits, insurance coverages, etc.
• Keep employees updated on a regular basis of current response
level and its effect on daily activities.
• If activities such as operations and maintenance are outsourced,
collaborate with partners to set and align policies.
• Anticipate supply chain disruptions. In addition to resources core
to business functionality, procure an appropriate supply level of
products that reduce the spread of infectious agents: disinfectant
wipes, hand sanitizer, masks, gloves, noncontact thermometers,
Protect site and While many of the steps that need to be taken involve external partners,
protection of the immediate site and staff are the first concern.
staff Site
Increase sanitization
For a virus pandemic, sanitization is, of course, critical. Critical facilities
present challenges, because of access/security, the need for specialized
procedures and the need to protect equipment. The following steps will
improve protection:
• Intensify housekeeping measures — conduct multiple rounds
of cleaning daily, especially of heavy-contact surfaces (e.g.,
door handles, light switches, elevator buttons, handrails, faucet
handles). If possible, have a cleaner continually cycle through the
facility disinfecting high-touch surfaces during hours of operation.
(This includes workstations, offices and personal and shared
technology.)
• Place hand sanitizer and disinfectant wipes (with disposal units)
throughout the facility, as well as signs to remind staff and visitors
to use them frequently.
• Place signs in bathrooms reminding staff to wash hands often,
using proper techniques.
• Post signs through the facility reminding staff to carry tissues and
sneeze and cough into those tissues, then dispose of the tissues in
a waste receptacle.
• Note that person traps could present a repository for the virus
in that they are small contained spaces, they are not usually
ventilated, and they have surfaces that could allow the virus to live
for hours, if not days. Consider limiting the use of person traps and/
or sanitizing after each use.
• Provide cleaning supplies and require staff to disinfect all work
areas at the beginning and end of each shift.
Limit access
Access to critical facilities, almost by definition, is strictly controlled
already — this will prove helpful in reducing infection risks. Consider the
following:
• Security checkpoints at the data center entry gates should inspect
entry passes, take temperature measurement by noncontact
methods (if possible/available) and disinfect (use sanitizers). Entry
to the site is allowed only if visitor is qualified.
© COPYRIGHT 2020 UPTIME I NS TITUTE . A L L R I GHTS R E S E RV E D 6
COVID-19: Uptime Institute Advisory, March 16, 2020
Staff
Working practices, legislation and attitudes to working conditions and/
or safety can vary significantly from country to country. Similarly, rules
regarding remote working, remote access to data, and on-site attendance
can vary widely by country and industry. The following suggested
practices should be considered in association with HR and security
management:
• Test all virtual private network (VPN) connections to ensure reliable
access, then consider instructing all staff noncritical to data center
operations to work from home.
• Ensure VPN access to building management systems (BMS) for
remote data center monitoring.
• Provide city/region-specific instruction on which VPN server
to log into (particularly important since most of company’s
workforce will temporarily be telecommuting).
• Ensure access to standard operation procedures (SOPs) and
emergency operation procedures (EOPs) to allow for remote co-
piloting if needed.
• Ensure SOPs/EOPs are accurate and could be followed by a
resource not normally working at the facility.
• Depending on circumstances, consider postponing/cancelling
all in-person meetings — use email, phone and audio/video
conferencing.
• Remind staff (using signs, daily briefings) of their responsibility for
sanitization — provide protection equipment, cleaning materials
and reminders to wash hands thoroughly and often.
• Anticipate the challenges of operating with reduced staff. Develop
a staffing threat matrix for various scenarios of employee
absenteeism (e.g., under 25 percent, 25-50 percent, 50-75 percent,
75-99 percent, 100 percent). For each scenario, summarize the
following:
© COPYRIGHT 2020 UPTIME I NS TITUTE . A L L R I GHTS R E S E RV E D 7
COVID-19: Uptime Institute Advisory, March 16, 2020
Limit travel
Travel limitations are being applied by companies and governments
during the COVID-19 outbreak. Government rules in affected countries
should always be followed. Rules will be relaxed as the pandemic
subsides, so different policies should be applied at different times. The
following should be considered:
• Ban/reduce all unnecessary travel. Organizations should be clear
about what constitutes travel (for example, short local journeys
versus longer/international travel) and develop appropriate
guidance.
• Prohibit or reduce travel between sites. Where travel between
sites is necessary, take steps to ensure cross-contamination is
minimized — one site may be backing up another.
• Plan for essential maintenance visits. Governments or companies
may relax rules, or provide exemptions, for the maintenance of
essential equipment. Most of the “lockdowns” currently in place
make exception for people going to work, however other authorities
having jurisdiction may apply stricter controls on travel within their
areas of control. Operators must plan for how to manage this in
advance and obtain the necessary permissions where required.
Permissions may depend on the applications/services being run in
the data center.
Manage shifts
Ideally, the principles of redundancy that underpin data center design and
operation should apply to the staff too. In many sites, of course, such
principles are already applied. During the virus outbreak, the following
should be considered:
• Create teams of mission-critical staff, ensuring each team has
a mix of skills/experience sufficient to effectively manage the
facility (if this is appropriate and if the site is adequately staffed).
Segregate teams between sites, especially by not allowing
personnel who work in a primary site to visit that site’s backup
location or have any contact with the backup site’s staff. If possible,
organize site tasks so that teams work in separate areas of the
facility, never coming into contact with each other or the others’
workspaces. Ensure that team members always work the same
shift, so there is no cross-shift contact.
• Allow no cross-contact of teams, even outside the work
environment.
• Allow no cross-shift interaction. Incoming shift workers should
maintain at least a 6-foot (roughly 2-meter) distance from the
outgoing shift workers. This includes elevators.
• Shared workspaces should be wiped down with disinfectant
wipes by the incoming shift staff.
• Depending on the appropriate medical or management advice,
workers should use masks during shift turnover.
• Depending on the appropriate medical or management advice,
training pairs (e.g., senior engineer and trainee) must wear masks
at all times.
• Shift leaders should report regularly (via email) to managers
on staff compliance with mitigation efforts (cleaning, social
distancing, etc.) and notify of any concerns (e.g., worker issues,
shortage of disinfecting supplies, etc.).
• Consider implementing a contact tracing system. Register the
health information and location of personnel, supplier personnel
and other related personnel every day to monitor for possible
exposure to the virus or any symptoms (including those of the
common cold).
Consider factors The most predictable and routine tasks, conducted by expert in-house
or contracted staff very familiar with the environment, have the lowest
that raise risk risks. Operators should attempt to eliminate other factors, processes and
behaviors that introduce uncertainties. The management of third parties
needs active attention. In these cases, consider the following:
surfaces that could allow the virus to live for hours, if not days.
Clients should consider limiting the use of person traps and/or
sanitizing after each use.
• Consider limiting the accessibility to shared spaces, such as client
lounges, etc. Ensure there are sanitization supplies (and disposal
units) in all shared areas, including next to vending machines.
Mixed-use facilities
Some small data centers, sometimes designated as server rooms,
are sited in mixed-use buildings, such as headquarters, factories or
administrative centers. In this situation, while the principles described
in this document largely apply, policies and rules will likely be set by
noncritical facilities management.
• Requirements (maintenance, access) for critical staff, and for
critical facility exceptions to the general building rules, should be
clearly identified to establish exception policies where appropriate.
• Operators should test all VPN connections to ensure reliable
access, then instruct all staff noncritical to data center operations
to access the systems remotely where possible.
Corporate response
• Stay current. Consult available information sources for updates
and guidance (see Appendix).
• Share lessons. Because many organizations have data centers
in multiple regions, responses may vary by location or facility
characteristics. Share lessons learned in more affected regions
with those less/not yet affected to strengthen their response.
• Secure documentation. Management may need to obtain
permissions/official documents that permit key employees to travel
to work (especially if cross-border commutes are common in the
area).
• Clarify escalation process. Ensure that business units — especially
mission-critical units — are fully briefed on response levels and the
specific events that would trigger escalation.
• Ensure business/technical alignment. Encourage business units
to be in frequent communication with data center operations and
IT operations about policy changes that may impact data center/
IT operations. For example, directing employees to telecommute
Conclusions
COVID-19 has been active for many weeks (at the time of this report),
but its impact on many organizations has been limited until recently.
As a result, many businesses are (and will remain) in a reactive mode.
Eventually this will be replaced by review and iterative improvement of
policies and procedures; in the severity of the pandemic, it is likely that
many of the policies will become permanently incorporated into critical
facility management. This may increase overall costs.
The graying of the workforce in some geographies means that despite
best efforts, the data center industry may be more vulnerable than other
industries to COVID-19. This presents a challenge, given the existing
and well-documented staffing shortages the industry faces. Current
events reinforce the need for increased efforts on the part of the
industry, educational institutions and trade organizations to strengthen
recruitment and training programs.
© COPYRIGHT 2020 UPTIME I NS TITUTE . A L L R I GHTS R E S E RV E D 16
COVID-19: Uptime Institute Advisory, March 16, 2020
Appendix: Resources
• National and international news, public health and governmental
agencies — for example,
• US Centers for Disease Control and Prevention
• US Occupational Health and Safety Administration
• World Health Organization
• Johns Hopkins University
• Local resources (news sites, state and regional agencies)
• Coronavirus tech handbook (a crowd-sourced reference for the
tech industry)
• Uptime Institute Inside Track COVID-19 bulletins
• Uptime Institute’s COVID-19 Intelligence Collection (available to
members of Uptime’s Inside Track community/Uptime Network
members), which is updated frequently and includes sample
business continuity plans, site-visitor questionnaires and more
• Uptime live support or on-demand emergency management
resources
© COPYRIGHT 2020 UPTIME I NS TITUTE . A L L R I GHTS R E S E RV E D 17
COVID-19: Uptime Institute Advisory, March 16, 2020
ACKNOWLEDGEMENTS
The Uptime Institute Intelligence team thanks Chris Brown, Fred Dickerman, Scott Killian, Scott
Good, Ryan Orr, Matt Stansberry, Alex Tay and global members of the Uptime Institute Network
and Uptime’s Inside Track community for their contributions to this report.
Uptime Institute – The Global Data Center Authority®, a division of The 451 Group, has office locations in the US, Mexico, Costa Rica, Brazil,
UK, Spain, UAE, Russia, Taiwan, Singapore and Malaysia. Visit uptimeinstitute.com for more information.