by Paul Kirvan, CISA, CISSP, FBCI, CBCP
by Paul Kirvan, CISA, CISSP, FBCI, CBCP
by Paul Kirvan, CISA, CISSP, FBCI, CBCP
Revision History
DATE
NAME
DESCRIPTION
Table of Contents
Information Technology Statement of Intent ...................................................................... 5 Policy Statement ................................................................................................................. 5 Objectives ........................................................................................................................... 5 Key Personnel Contact Info ................................................................................................ 6 Notification Calling Tree ................................................................................................ 7 External Contacts ................................................................................................................ 8 External Contacts Calling Tree ..................................................................................... 10 1 Plan Overview ........................................................................................................... 11 1.1 Plan Updating.................................................................................................... 11 1.2 Plan Documentation Storage............................................................................. 11 1.3 Backup Strategy ................................................................................................ 11 1.4 Risk Management ............................................................................................. 11 2 Emergency Response ................................................................................................ 12 2.1 Alert, escalation and plan invocation ................................................................ 12 2.1.1 Plan Triggering Events ................................................................................. 12 2.1.2 Assembly Points ............................................................................................ 12 2.1.3 Activation of Emergency Response Team .................................................... 12 2.2 Disaster Recovery Team ................................................................................... 13 2.3 Emergency Alert, Escalation and DRP Activation ........................................... 13 2.3.1 Emergency Alert ........................................................................................... 13 2.3.2 DR Procedures for Management ................................................................... 14 2.3.3 Contact with Employees ............................................................................... 14 2.3.4 Backup Staff.................................................................................................. 14 2.3.5 Recorded Messages / Updates ...................................................................... 14 2.3.7 Alternate Recovery Facilities / Hot Site ....................................................... 14 2.3.8 Personnel and Family Notification ............................................................... 14 3 Media ........................................................................................................................ 14 3.1 Media Contact ................................................................................................... 14 3.2 Media Strategies................................................................................................ 15 3.3 Media Team ...................................................................................................... 15 3.4 Rules for Dealing with Media ........................................................................... 15 4 Insurance ................................................................................................................... 15 5 Financial and Legal Issues ........................................................................................ 16 5.1 Financial Assessment ........................................................................................ 16 5.2 Financial Requirements .................................................................................... 16 5.3 Legal Actions .................................................................................................... 16 6 DRP Exercising......................................................................................................... 16 Appendix A Technology Disaster Recovery Plan Templates........................................ 17 Disaster Recovery Plan for <System One> .................................................................. 17 Disaster Recovery Plan for <System Two> .................................................................. 19 Disaster Recovery Plan for Local Area Network (LAN).............................................. 21 Disaster Recovery Plan for Wide Area Network (WAN) ............................................. 23 Disaster Recovery Plan for Remote Connectivity ........................................................ 25
Disaster Recovery Plan for Voice Communications .................................................... 27 Appendix B Suggested Forms ....................................................................................... 29 Damage Assessment Form ............................................................................................ 29 Management of DR Activities Form............................................................................. 29 Disaster Recovery Event Recording Form ................................................................... 29 Disaster Recovery Activity Report Form ..................................................................... 30 Mobilizing the Disaster Recovery Team Form ............................................................. 31 Mobilizing the Business Recovery Team Form............................................................ 31 Monitoring Business Recovery Task Progress Form.................................................... 32 Preparing the Business Recovery Report Form ............................................................ 32 Communications Form.................................................................................................. 32 Returning Recovered Business Operations to Business Unit Leadership .................... 33 Business Process/Function Recovery Completion Form .............................................. 33
Policy Statement
Corporate management has approved the following policy statement: The company shall develop a comprehensive IT disaster recovery plan. A formal risk assessment shall be undertaken to determine the requirements for the disaster recovery plan. The disaster recovery plan should cover all essential and critical infrastructure elements, systems and networks, in accordance with key business activities. The disaster recovery plan should be periodically tested in a simulated environment to ensure that it can be implemented in emergency situations and that the management and staff understand how it is to be executed. All staff must be made aware of the disaster recovery plan and their own respective roles. The disaster recovery plan is to be kept up to date to take into account changing circumstances.
Objectives
The principal objective of the disaster recovery program is to develop, test and document a wellstructured and easily understood plan which will help the company recover as quickly and effectively as possible from an unforeseen disaster or emergency which interrupts information systems and business operations. Additional objectives include the following: The need to ensure that all employees fully understand their duties in implementing such a plan The need to ensure that operational policies are adhered to within all planned activities The need to ensure that proposed contingency arrangements are cost-effective The need to consider implications on other company sites Disaster recovery capabilities as applicable to key customers, vendors and others
Work Alternate Mobile Home Email Address Alternate Email Work Alternate Mobile Home Email Address Alternate Email Work Alternate Mobile Home Email Address Alternate Email Work Alternate Mobile Home Email Address Alternate Email Work Alternate Mobile Home Email Address Alternate Email Work Alternate Mobile Home Email Address Alternate Email
External Contacts
Name, Title Contact Option Contact Number
Landlord / Property Manager Account Number None Work Mobile Home Email Address Power Company Account Number
Insurance Name
Name, Title
Contact Option
Contact Number
Account Number
1 1.1
It is necessary for the DRP updating process to be properly structured and controlled. Whenever changes are made to the plan they are to be fully tested and appropriate amendments should be made to the training materials. This will involve the use of formalized change control procedures under the control of the IT Director.
1.2
Copies of this Plan, CD, and hard copies will be stored in secure locations to be defined by the company. Each member of senior management will be issued a CD and hard copy of this plan to be filed at home. Each member of the Disaster Recovery Team and the Business Recovery Team will be issued a CD and hard copy of this plan. A master protected copy will be stored on specific resources established for this purpose.
1.3
Backup Strategy
Key business processes and the agreed backup strategy for each are listed below. The strategy chosen is for a fully mirrored recovery site at the companys offices in _____. This strategy entails the maintenance of a fully mirrored duplicate site which will enable instantaneous switching between the live site (headquarters) and the backup site.
KEY BUSINESS PROCESS BACKUP STRATEGY
IT Operations Tech Support - Hardware Tech Support - Software Facilities Management Email Purchasing Disaster Recovery Finance Contracts Admin Warehouse & Inventory Product Sales Maintenance Sales Human Resources Testing Fully Mirrored Recovery site Workshop Fully Mirrored Recovery site Call Center Web Site
Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Off-site data storage facility Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site Fully mirrored recovery site
1.4
Risk Management
There are many potential disruptive threats which can occur at any time and affect the normal business process. We have considered a wide range of potential threats and the results of our deliberations are included in this section. Each potential environmental disaster or emergency
situation has been examined. The focus here is on the level of business disruption which could arise from each type of disaster. Potential disasters have been assessed as follows:
Potential Disaster Probability Rating Impact Rating Brief Description Of Potential Consequences & Remedial Actions
Flood Fire
3 3
4 4
All critical equipment is located on 1st Floor FM200 suppression system installed in main computer centers. Fire and smoke detectors on all floors.
Tornado Electrical storms Act of terrorism Act of sabotage Electrical power failure
5 5 5 5 3
Redundant UPS array together with auto standby generator that is tested weekly & remotely monitored 24/7. UPSs also remotely monitored. Two diversely routed T1 trunks into building. WAN redundancy, voice network resilience
2
2.1
Emergency Response
Alert, escalation and plan invocation
Respond immediately to a potential disaster and call emergency services; Assess the extent of the disaster and its impact on the business, data center, etc.; Decide which elements of the DR Plan should be activated; Establish and manage disaster recovery team to maintain vital services and return to normal operation; Ensure employees are notified and allocate responsibilities and activities as required.
2.2
The team will be contacted and assembled by the ERT. The team's responsibilities include: Establish facilities for an emergency level of service within 2.0 business hours; Restore key services within 4.0 business hours of the incident; Recover to business as usual within 8.0 to 24.0 hours after the incident; Coordinate activities with disaster recovery team, first responders, etc. Report to the emergency response team.
2.3
This policy and procedure has been established to ensure that in the event of a disaster or crisis, personnel will have a clear understanding of who should be contacted. Procedures have been addressed to ensure that communications can be quickly established while activating disaster recovery. The DR plan will rely principally on key members of management and staff who will provide the technical and management skills necessary to achieve a smooth technology and business recovery. Suppliers of critical goods and services will continue to support recovery of business operations as the company returns to normal operating mode.
One of the tasks during the early stages of the emergency is to notify the Disaster Recovery Team (DRT) that an emergency has occurred. The notification will request DRT members to assemble at the site of the problem and will involve sufficient information to have this request effectively communicated. The Business Recovery Team (BRT) will consist of senior representatives from the main business departments. The BRT Leader will be a senior member of the company's management team, and will be responsible for taking overall charge of the process and ensuring that the company returns to normal working operations as early as possible.
3 3.1
Assigned staff will coordinate with the media, working according to guidelines that have been previously approved and issued for dealing with post-disaster communications.
3.2
Media Strategies
1. Avoiding adverse publicity 2. Take advantage of opportunities for useful publicity 3. Have answers to the following basic questions: What happened? How did it happen? What are you going to do about it?
3.3
Media Team
3.4
Only the media team is permitted direct contact with the media; anyone else contacted should refer callers or in-person media representatives to the media team.
Insurance
As part of the companys disaster recovery and business continuity strategies a number of insurance policies have been put in place. These include errors and omissions, directors & officers liability, general liability, and business interruption insurance. If insurance-related assistance is required following an emergency out of normal business hours, please contact: ____________________________________________
Policy Name Coverage Type Coverage Period Amount Of Coverage Person Responsible For Coverage Next Renewal Date
5 5.1
The emergency response team shall prepare an initial assessment of the impact of the incident on the financial affairs of the company. The assessment should include: Loss of financial documents Loss of revenue Theft of check books, credit cards, etc. Loss of cash
5.2
Financial Requirements
The immediate financial needs of the company must be addressed. These can include: Cash flow position Temporary borrowing capability Upcoming payments for taxes, payroll taxes, Social Security, etc. Availability of company credit cards to pay for supplies and services required post-disaster
5.3
Legal Actions
The company legal department and ERT will jointly review the aftermath of the incident and decide whether there may be legal actions resulting from the event; in particular, the possibility of claims by or against the company for regulatory violations, etc.
DRP Exercising
Disaster recovery plan exercises are an essential part of the plan development process. In a DRP exercise no one passes or fails; everyone who participates learns from exercises what needs to be improved, and how the improvements can be implemented. Plan exercising ensures that emergency teams are familiar with their assignments and, more importantly, are confident in their capabilities. Successful DR plans launch into action smoothly and effectively when they are needed. This will only happen if everyone with a role to play in the plan has rehearsed the role one or more times. The plan should also be validated by simulating the circumstances within which it has to work and seeing what happens.
Appendix A Technology Disaster Recovery Plan Templates Disaster Recovery Plan for <System One>
SYSTEM OVERVIEW PRODUCTION SERVER
HOT SITE SERVER APPLICATIONS (Use bold for Hot Site) ASSOCIATED SERVERS KEY CONTACTS Hardware Vendor System Owners Database Owner Application Owners Software Vendors Offsite Storage BACKUP STRATEGY FOR SYSTEM ONE Daily Monthly Quarterly SYSTEM ONE DISASTER RECOVERY PROCEDURE
Location: Server Model: Operating System: CPUs: Memory: Total Disk: System Handle: System Serial #: DNS Entry: IP Address: Other: Provide details
Provide details Provide details Provide details Provide details Provide details Provide details
Provide details Scenario 1 Total Loss of Data Provide details Scenario 2 Total Loss of HW
ADDENDUM CONTACTS
File Systems <date> File System as of <date> Minimal file systems to be created and restored from backup: <List> Filesystem <Provide details> kbytes Used Avail %used Mounted on
Other critical files to modify Necessary directories to create Critical files to restore Secondary files to restore Other files to restore
<Provide details> <Provide details> <Provide details> <Provide details> <Provide details>
HOT SITE SERVER APPLICATIONS (Use bold for Hot Site) ASSOCIATED SERVERS KEY CONTACTS Hardware Vendor System Owners Database Owner Application Owners Software Vendors Offsite Storage BACKUP STRATEGY for SYSTEM TWO Daily Monthly Quarterly SYSTEM TWO DISASTER RECOVERY PROCEDURE
Location: Server Model: Operating System: CPUs: Memory: Total Disk: System Handle: System Serial #: DNS Entry: IP Address: Other: Provide details
Provide details Provide details Provide details Provide details Provide details Provide details
Provide details Scenario 1 Total Loss of Data Provide details Scenario 2 Total Loss of HW
ADDENDUM CONTACTS
File Systems <date> File System as of <date> Minimal file systems to be created and restored from backup: <List> Filesystem <Provide details> kbytes Used Avail %used Mounted on
Other critical files to modify Necessary directories to create Critical files to restore Secondary files to restore Other files to restore
<Provide details> <Provide details> <Provide details> <Provide details> <Provide details>
HOT SITE SERVER APPLICATIONS (Use bold for Hot Site) ASSOCIATED SERVERS KEY CONTACTS Hardware Vendor System Owners Database Owner Application Owners Software Vendors Offsite Storage BACKUP STRATEGY for SYSTEM TWO Daily Monthly Quarterly SYSTEM TWO DISASTER RECOVERY PROCEDURE
Location: Server Model: Operating System: CPUs: Memory: Total Disk: System Handle: System Serial #: DNS Entry: IP Address: Other: Provide details
Provide details Provide details Provide details Provide details Provide details Provide details
Provide details Scenario 1 Total Loss of Data Provide details Scenario 2 Total Loss of HW
ADDENDUM CONTACTS
File Systems <date> File System as of <date> Minimal file systems to be created and restored from backup: <List> Filesystem <Provide details> kbytes Used Avail %used Mounted on
Other critical files to modify Necessary directories to create Critical files to restore Secondary files to restore Other files to restore
<Provide details> <Provide details> <Provide details> <Provide details> <Provide details>
HOT SITE EQUIPMENT SPECIAL APPLICATIONS ASSOCIATED DEVICES KEY CONTACTS Hardware Vendor System Owners Database Owner Application Owners Software Vendors Offsite Storage Network Services BACKUP STRATEGY for SYSTEM TWO Daily Monthly Quarterly SYSTEM TWO DISASTER RECOVERY PROCEDURE
Location: Device Type: Model No.: Technical Specifications: Network Interfaces: Power Requirements; System Serial #: DNS Entry: IP Address: Other: Provide details
Provide details Provide details Provide details Provide details Provide details Provide details Provide details
Provide details Scenario 1 Total Loss of Network Provide details Scenario 2 Total Loss of HW
ADDENDUM CONTACTS
Support Systems <date> Support system Critical network assets Critical interfaces Critical files to restore Critical network services to restore Other services <Provide details> <Provide details> <Provide details> <Provide details> <Provide details> <Provide details>
HOT SITE EQUIPMENT SPECIAL APPLICATIONS ASSOCIATED DEVICES KEY CONTACTS Hardware Vendor System Owners Database Owner Application Owners Software Vendors Offsite Storage Network Services BACKUP STRATEGY for SYSTEM TWO Daily Monthly Quarterly SYSTEM TWO DISASTER RECOVERY PROCEDURE
Location: Device Type: Model No.: Technical Specifications: Network Interfaces: Power Requirements; System Serial #: DNS Entry: IP Address: Other: Provide details
Provide details Provide details Provide details Provide details Provide details Provide details Provide details
Provide details Scenario 1 Total Loss of Network Provide details Scenario 2 Total Loss of HW
ADDENDUM CONTACTS
Support Systems <date> Support system Critical network assets Critical interfaces Critical files to restore Critical network services to restore Other services <Provide details> <Provide details> <Provide details> <Provide details> <Provide details> <Provide details>
HOT SITE EQUIPMENT SPECIAL APPLICATIONS ASSOCIATED DEVICES KEY CONTACTS Hardware Vendor System Owners Database Owner Application Owners Software Vendors Offsite Storage Network Services BACKUP STRATEGY for SYSTEM TWO Daily Monthly Quarterly SYSTEM TWO DISASTER RECOVERY PROCEDURE
Location: Device Type: Model No.: Technical Specifications: Network Interfaces: Power Requirements; System Serial #: DNS Entry: IP Address: Other: Provide details
Provide details Provide details Provide details Provide details Provide details Provide details Provide details
Provide details Scenario 1 Total Loss of Switch Provide details Scenario 2 Total Loss of Network
ADDENDUM CONTACTS
Support Systems <date> Support system Critical network assets Critical interfaces Critical files to restore Critical network services to restore Other services <Provide details> <Provide details> <Provide details> <Provide details> <Provide details> <Provide details>
_____________
Commencement Date/Time
Completion Date/Time
Resources Involved
In Charge
__________________
All key events that occur during the disaster recovery phase must be recorded. An event log shall be maintained by the disaster recovery team leader. This event log should be started at the commencement of the emergency and a copy of the log passed on to the business recovery team once the initial dangers have been controlled. The following event log should be completed by the disaster recovery team leader to record all key events during disaster recovery, until such time as responsibility is handed over to the business recovery team.
Disaster Recovery Team's Work Completed: <Date> Event Log Passed to Business Recovery Team: <Date>
_________________
The report will include: A description of the emergency or incident Those people notified of the emergency (including dates) Action taken by members of the DRT Outcomes arising from actions taken An assessment of the impact to normal business operations
Assessment of the effectiveness of the BCP and lessons learned Lessons learned __________
Description of Emergency: Date Occurred: Date Work of Disaster Recovery Team Completed:
Name of Team Member Contact Details Contacted On (Time / Date) By Whom Response Start Date Required
___________
Following an emergency requiring activation of the disaster recovery team, the business recovery team should be notified of the situation and placed on standby. The format shown below will be used for recording the activation of the business recovery team once the work of the disaster recovery team has been completed.
Description of Emergency: Date Occurred: Date Work of Business Recovery Team Completed:
Name of Team Member Contact Details Contacted On (Time / Date) By Whom Response Start Date Required
____________
Note: A priority sequence must be identified although, where possible, activities will be carried out simultaneously.
Recovery Tasks (Order of Priority) Person(s) Responsible Completion Date Estimated Actual Milestones Identified Other Relevant Information
1. 2. 3. 4. 5. 6. 7.
___________
The contents of the report shall include: A description of the incident People notified of the emergency (including dates) Action taken by the business recovery team Outcomes arising from actions taken An assessment of the impact to normal business operations Problems identified Suggestions for enhancing the disaster recovery and/or business continuity plan Lessons learned
Communications Form
It is very important during the disaster recovery and business recovery activities that all affected persons and organizations are kept properly informed. The information given to all parties must be accurate and timely.
In particular, any estimate of the timing to return to normal working operations should be announced with care. It is also very important that only authorized personnel deal with media queries.
Persons Selected To Coordinate Communications to Affected Persons / Organizations Name Position Contact Details
____________
____________
I confirm that the work of the business recovery team has been completed in accordance with the disaster recovery plan for the above process, and that normal business operations have been effectively restored. Business Recovery Team Leader Name: ________________________________________ Signature: ________________________________________________________________ Date: __________________________ (Any relevant comments by the BRT leader in connection with the return of this business process should be made here.)
I confirm that above business process is now acceptable for normal working conditions. Name: ___________________________________________________________________ Title: ____________________________________________________________________ Signature: ________________________________________________________________ Date: __________________________