5 phases of disaster recovery

Disaster Recovery Plan

Class – Please review the attached lessons learned from Hurricane Sandy.
1. Review the key phases of disaster recovery as outlined in chapter 10 of your textbook
2. Briefly discuss the disaster recovery phases in chapter 10 and their relation to the lessons learned from Hurricane Sandy article. I need to see information systems related planning.
3. Discuss how you would alter sample plan based on the lessons learned from Hurricane Sandy Article. (Do not give me a copy of the altered plan, simply explain in the paper what you would alter)

• Include at least Six (6) reputable sources.

Chapter 10 Disaster Recovery: Operation and

Maintenance

Objectives

• Describe the key challenges an organization faces when engaged in DR operations

• Discuss what actions organizations should take to prepare for the activation of the DR plan

• List the critical elements that comprise the response phase of the DR plan

• Explain what occurs in the recovery phase of the DR plan

• Describe how an organization uses the resumption phase of the DR plan

Principles of Incident Response and Disaster Recovery, 2nd Edition 2

Objectives (cont’d.)

• Discuss how an organization resumes normal operations using the restoration phase of the DR plan

Principles of Incident Response and Disaster Recovery, 2nd Edition 3

Introduction

• When disaster occurs organizations need – Meticulous preparation and ongoing diligence – Quick and decisive reaction to restore operations – To prepare to promptly reestablish operations at a

new permanent location • Each area of the world has its own challenges and

risks of disaster – Natural or man-made

• DR plans and procedures are similar to those undertaken for IR and BC actions

Principles of Incident Response and Disaster Recovery, 2nd Edition 4

Facing Key Challenges

• Widespread disasters frequently affect: – Departments and various organization levels – Communities encompassing the organization – Vendors and suppliers

• Outside help may be unavailable for days or weeks – Emergency services, public services, grocers and

other suppliers, utility services, private services, telecommunications services, and air and surface transportation

• Worst-case scenario – Seemingly routine event quickly spins out of control

Principles of Incident Response and Disaster Recovery, 2nd Edition 5

Facing Key Challenges (cont’d.)

• Most disaster-related losses cause: – Inability to react properly to the disaster – A need to improvise, adapt, and overcome obstacles

• Most disasters last hours or a few days • DR plan phases

– Preparation – Response – Recovery – Resumption – Restoration

Principles of Incident Response and Disaster Recovery, 2nd Edition 6

Preparation: Training the DR Team and the Users

• No prevention phase in DR planning – Reason: majority of disasters cannot be prevented

• Can minimize disaster probability by planning • Preparation

– Being ready for possible contingencies that can escalate to become disasters

• Develop BIA and DR plans • Organize and staff various DR teams • Train various stakeholders and practice the plan

Principles of Incident Response and Disaster Recovery, 2nd Edition 7

Plan Distribution

• Must distribute plan to those who need it most • Ensure that all personnel:

– Have access to the plan – Have fully read the plan – Understand the plan

• IR, DR, or BC plan storage – Physical copy easy to misplace – Online storage locations

• Electronic disruptions could prevent access

Principles of Incident Response and Disaster Recovery, 2nd Edition 8

Plan Distribution (cont’d.)

• Store password-protected plans where employees can access them – At the office – Away from the office – Online (anytime, anywhere)

• Password-protecting all electronic files • Store physical copies in secure locations

Principles of Incident Response and Disaster Recovery, 2nd Edition 9

Plan Triggers and Notification

• Preparation phase: continuous • Other phases: activated by triggers

– Management notification – Employee notification – Emergency management notification – Local emergency services – Media outlets

Principles of Incident Response and Disaster Recovery, 2nd Edition 10

Disaster Recovery Planning as Preparation

• Cornerstone of preparation – Developing an effective DR plan

• DR plan primary goals – Eliminate or reduce

• Potential for injuries, loss of human life, damage to facilities, loss of assets and records

– Immediately invoke DR plan emergency provisions • Stabilize disaster effects • Allow appropriate assessment; begin recovery efforts

– Implement procedures contained in the DR plan

Principles of Incident Response and Disaster Recovery, 2nd Edition 11

Disaster Recovery Planning as Preparation (cont’d.)

• CP team engages in scenario development and impact analysis – Categorizes threat level each potential disaster poses

• Generating DR scenario – Start with most important asset: people

• Must test DR plan regularly – Ensure DR team can lead recovery effort quickly and

efficiently

Principles of Incident Response and Disaster Recovery, 2nd Edition 12

Disaster Recovery Planning as Preparation (cont’d.)

• Key features of the DR plan – Clear delegation of roles and responsibilities – Execution of the alert roster and notification of key

personnel – Use of employee check-in systems – Clear establishment and communication of business

resumption priorities – Complete and timely documentation of the disaster – Preparations for alternative implementations – DR team members should know their disaster duties

• Key personnel may include external groups Principles of Incident Response and Disaster Recovery, 2nd Edition 13

Disaster Recovery Planning as Preparation (cont’d.)

• Key features of the DR plan (cont’d.) – During a disaster response

• Verify status of employees, contractors, consultants using manual or automatic procedures

• First priority: preservation of human life • Carefully record disaster from the onset

– Mitigation of impact • Action steps to minimize damage associated with the

disaster on operations

Principles of Incident Response and Disaster Recovery, 2nd Edition 14

Disaster Recovery Planning as Preparation (cont’d.)

• Additional preparations – Two types of emergency information employees need

• Personal emergency information • Snapshot of the DR plan

– Emergency information often encapsulated into a wallet-sized, laminated card

– Crisis management • Focused steps dealing primarily with the safety and

state of the people involved in the disaster • DR team works closely with crisis management team

Principles of Incident Response and Disaster Recovery, 2nd Edition 15

DR Training and Awareness

• Training involves different approaches • Training should focus on roles individual expected to

execute during an actual disaster • Disaster preparation limited to awareness training

– Part of annual or semiannual security education, training, and awareness (SETA) program

– Make employees aware of general procedures for responding to disasters

Principles of Incident Response and Disaster Recovery, 2nd Edition 16

General Training for All Teams

• Best crisis preparation – Ensure employees trained and comfortable in

completing normal tasks • Training and rehearsals purpose

– Identify individuals with rusty technical skills – Provide opportunity to brush up on responsibilities

• Vertical and horizontal job rotation – Allows preparation for normal personnel shortages or

outages • Practice degraded mode operations

Principles of Incident Response and Disaster Recovery, 2nd Edition 17

Disaster Management Team Training

• Command and control group – Responsible for all planning and coordination

activities • Training, rehearsal, and testing

– Predominantly communicative in nature • Must quickly and effectively communicate resources

needed for subordinate teams to function • Must communicate directives from higher teams and

peer teams

Principles of Incident Response and Disaster Recovery, 2nd Edition 18

Communications Team Training

• Information-dissemination group – Responsible for interacting and communicating with

the external environment • Training, rehearsal, and testing

– Prepares information notices, news releases, and internal memorandums and directives

– Sends communications to all groups and teams – Informs people of their tasks and responsibilities

• Should be involved in routine rehearsal and testing

Principles of Incident Response and Disaster Recovery, 2nd Edition 19

Computer Recovery (Hardware) Team Training

• Hardware recovery and reconstitution team • Ideally practices and trains during normal operation • Training requirements

– Advanced training to rebuild systems by scavenging parts

– Knowledge in how to deal with systems damaged by water, heat, and dust

• Team should work closely with other technology teams

Principles of Incident Response and Disaster Recovery, 2nd Edition 20

Principles of Incident Response and Disaster Recovery, 2nd Edition 21

Systems Recovery Team Training

• Responsible for recovering and reestablishing operating systems (OSs)

• May rehearse DR duties during normal operations • Train to quickly recover system’s operating system • Responsibilities may be combined with other IT

teams

Principles of Incident Response and Disaster Recovery, 2nd Edition 22

Network Recovery Team Training

• Responsible for reestablishing – Connectivity between systems and to the Internet – Voice communication networks

• Focus of training – Establishing ad hoc networks quickly but securely – Wireless technology

• Team requirements – Stash of wireless networking components stored

outside the organization • Difficult internet connectivity may need vendor

interaction Principles of Incident Response and Disaster Recovery, 2nd Edition 23

Storage Recovery Team Training

• Responsible for information recovery and reestablishment of operations – In storage area networks or network attached storage

• Training needs – Rebuilding damaged systems – Recovering data from off-site locations

Principles of Incident Response and Disaster Recovery, 2nd Edition 24

Principles of Incident Response and Disaster Recovery, 2nd Edition 25

Applications Recovery Team Training

• Responsible for recovering and reestablishing critical business applications operations

• Requirements – Skills performed during normal operations – Coordination and training in operating under adverse

circumstances • Team will have user representation • Team effectiveness

– Heavily influenced by ability to create an effective liaison with application business units

Principles of Incident Response and Disaster Recovery, 2nd Edition 26

Data Management Team Training

• Responsible for data restoration and recovery • Focus of training

– Quick and accurate restoration of data from backup – Should include data recovery from damaged systems

• May need vendor help to extract data

Principles of Incident Response and Disaster Recovery, 2nd Edition 27

Principles of Incident Response and Disaster Recovery, 2nd Edition 28

Vendor Contact Team Training

• Responsible for working with suppliers and vendors – Need to replace damaged or destroyed equipment or

services determined by other teams • Training best obtained through normal work in

equipment procurement • Focus of training

– Methods of obtaining resources quickly as possible – Familiarity with preferred vendors

• Vendor relationships: crucial during a disaster

Principles of Incident Response and Disaster Recovery, 2nd Edition 29

Damage Assessment and Salvage Team Training

• Provides assessment for: – Initial damage to equipment and systems on-site – Physically recovering equipment transported to

location where other teams evaluate it • Requires basic background in hardware repair • May need to outsource the function

Principles of Incident Response and Disaster Recovery, 2nd Edition 30

Business Interface Team Training

• Works with remainder of the organization – Assists in recovery of nontechnology functions

• Training – Combines technical and nontechnical functions – Involves interfacing with various business groups to

determine routine needs • Help desk representatives well suited for this team

Principles of Incident Response and Disaster Recovery, 2nd Edition 31

Logistics Team Training

• Provides needed supplies, space, materials, food, services, or facilities needed at the primary site

• Require basic training in local purchasing • Primary function

– Serve as health, welfare, and morale support for the other teams doing their jobs

Principles of Incident Response and Disaster Recovery, 2nd Edition 32

DR Plan Testing and Rehearsal

• Testing DR plan elements – Can overlap with plan training and rehearsal

• Rehearsal – Occurs when organization practices steps performed

during a disaster • Testing involves assessment (internal or external) • Before performing in a large-scale exercise

– Provide classroom-style, structured training • Plan rehearsal

– Start small and escalate to larger-scale exercises Principles of Incident Response and Disaster Recovery, 2nd Edition 33

DR Plan Testing and Rehearsal (cont’d.)

• Rehearsal and testing strategies – Desk check – Structured walk-through – Simulation – Parallel testing – Full-interruption – War gaming

Principles of Incident Response and Disaster Recovery, 2nd Edition 34

Rehearsal and Testing of the Alert Roster

• Alert roster – Used in IR and BC planning, and crisis management

• Alert roster document – Contains contact information on individuals notified in

the event of an actual incident or disaster – Must be tested frequently because it is subject to

continual change • Two activation methods: sequential and hierarchical

Principles of Incident Response and Disaster Recovery, 2nd Edition 35

Rehearsal and Testing of the Alert Roster (cont’d.)

• Alert message – Scripted disaster description – Consists of just enough information so that each

responder knows what portion of the DR plan to implement

– Does not impede notification process • Auxiliary phone alert and reporting system

– Information system with a telephony interface – Used to automate the alert process

• “I’m okay” automated emergency response line – Employees call a predetermined number

Principles of Incident Response and Disaster Recovery, 2nd Edition 36

Disaster Response Phase

• Response phase – Associated with implementing initial reaction to a

disaster – Focus

• Controlling or stabilizing the situation, if possible • Response phase designed to:

– Protect human life and well-being (physical safety) – Attempt to limit and contain the damage to the

organization’s facilities and equipment – Manage communications with employees and other

stakeholders Principles of Incident Response and Disaster Recovery, 2nd Edition 37

Recovery Phase

• Recovery of the most time-critical business functions • Get back up and running as quickly as possible

– Even if operations limited to some degree • Less critical operations wait until resumption phase • Primary goals of the recovery phase

– Recover critical business functions – Coordinate recovery efforts – Acquire resources to replace damaged or destroyed

materials and equipment – Evaluate the need to implement the BC plan

Principles of Incident Response and Disaster Recovery, 2nd Edition 38

Resumption Phase

• Focuses on non-critical functions • BIA: guiding document for creating list of primary

and secondary functions • Goals of the resumption phase

– Initiate implementation of secondary functions – Finalize implementation of primary functions – Identify additional needed resources – Continue planning for restoration

• Complex interaction exist between DR plan and BC plan

Principles of Incident Response and Disaster Recovery, 2nd Edition 39

Restoration Phase

• Formally begins once: – All damage assessments accomplished – Rebuilding of primary site has commenced

• Restoration phase primary goals – Repair damage or select or build replacement facility – Replace primary site damaged or destroyed contents – Coordinate relocation from temporary offices to

primary site or to new replacement facility – Restore normal operations at the primary site – Stand down the DR teams and conduct the after-

action review Principles of Incident Response and Disaster Recovery, 2nd Edition 40

Repair or Replacement

• Two possibilities in the restoration phase – Reestablish operations at the primary site – Establish operations at a new permanent site

• Reestablish operations at the primary site – Organization can rebuild facilities at the primary site

• Continue partial operations while repairs made • Best to temporarily relocate the administrative function

Principles of Incident Response and Disaster Recovery, 2nd Edition 41

Repair or Replacement (cont’d.)

• Move to a new permanent site – Occurs if primary site becomes uninhabitable – Bulldoze and rebuild

• Good if organization owns the land • May be months before the organization can relocate

– Select a new location • Necessary when organization cannot relocate for an

extended stay at temporary locations • Selection of new permanent site: complex decision • Staff may not be available to relocate families

Principles of Incident Response and Disaster Recovery, 2nd Edition 42

Restoration of the Primary Site

• Occurs once physical facilities rebuilt – Must replace office furniture, desktop computers,

photocopying equipment, filing systems, office supplies

– Determine what insurance will and will not cover – Examine service contracts

• Determine if damage or destruction to leased equipment is covered by the provider

Principles of Incident Response and Disaster Recovery, 2nd Edition 43

Relocation from Temporary Offices

• Movement back to the primary site – Signals beginning of the end of disaster operations – Must be carefully coordinated: not simple

• Must relocate people and administrative paperwork • Must restore data functions and associated computing

equipment • Data management practices

– More crucial before and after moves – May require movement coordinator

Principles of Incident Response and Disaster Recovery, 2nd Edition 44

Resumption at the Primary Site

• Recover day-to-day operations to stabilize organization and keep it running efficiently – Management of employee benefit packages – Employee training and awareness programs – Organizational planning retreats and meetings – Routine progress meetings and reports – Long-term planning activities – Research and development activities

• Business now reconstituted and functioning as it did before the disaster

Principles of Incident Response and Disaster Recovery, 2nd Edition 45

Standing Down and the After-Action Review

• Standing down – Deactivation of DR teams

• Releasing individuals back to their normal duties – After-action review (AAR)

• Last activity before declaring disaster officially over • Management obtains input and feedback from teams • Information combined with official disaster log • Official log: legal and planning record and training tool

– Last step: creation and archiving of the official report • Legal document for insurance, parent organization • Once archived, disaster over

Principles of Incident Response and Disaster Recovery, 2nd Edition 46

Summary

• Matter of time until a disaster strikes – Meticulous preparation and ongoing diligence needed

to properly respond • DR plan implementation involves five phases

– Preparation, response, recovery, resumption, restoration

• DR and business resumption planning goals – Eliminate or reduce potential injuries or loss of human

life, facility damage, loss of assets and records – Stabilize the effects of the disaster – Implement DR and BR procedures

Principles of Incident Response and Disaster Recovery, 2nd Edition 47

Summary (cont’d.)

• Recovery phase – Quick critical business function recovery

• Resumption – Focuses on remaining unrestored functions

• Restoration phase primary goals – Repair damage or select or build replacement facility – Replace primary site damaged or destroyed contents – Coordinate relocation from temporary offices to

primary site or to new replacement facility – Restore normal operations at the primary site – Stand down the DR teams; conduct the AAR

Principles of Incident Response and Disaster Recovery, 2nd Edition 48

Principles of Incident Response and Disaster Recovery, 2nd Edition
Objectives
Objectives (cont’d.)
Introduction
Facing Key Challenges
Facing Key Challenges (cont’d.)
Preparation: Training the DR Team and the Users
Plan Distribution
Plan Distribution (cont’d.)
Plan Triggers and Notification
Disaster Recovery Planning as Preparation
Disaster Recovery Planning as Preparation (cont’d.)
Disaster Recovery Planning as Preparation (cont’d.)
Disaster Recovery Planning as Preparation (cont’d.)
Disaster Recovery Planning as Preparation (cont’d.)
DR Training and Awareness
General Training for All Teams
Disaster Management Team Training
Communications Team Training
Computer Recovery (Hardware) Team Training
Slide Number 21
Systems Recovery Team Training
Network Recovery Team Training
Storage Recovery Team Training
Slide Number 25
Applications Recovery Team Training
Data Management Team Training
Slide Number 28
Vendor Contact Team Training
Damage Assessment and Salvage Team Training
Business Interface Team Training
Logistics Team Training
DR Plan Testing and Rehearsal
DR Plan Testing and Rehearsal (cont’d.)
Rehearsal and Testing of the Alert Roster
Rehearsal and Testing of the Alert Roster (cont’d.)
Disaster Response Phase
Recovery Phase
Resumption Phase
Restoration Phase
Repair or Replacement
Repair or Replacement (cont’d.)
Restoration of the Primary Site
Relocation from Temporary Offices
Resumption at the Primary Site
Standing Down and the After-Action Review
Summary
Summary (cont’d.)