Incident Response Lifecycle

Structured process management for network incidents from detection through
post-incident review. This skill covers the organizational coordination
layer: severity classification, escalation, role assignment, stakeholder
communication, recovery coordination, and root cause analysis. It does not
cover technical evidence collection, device forensics, or containment
execution — use the incident-response-network skill for network-level
evidence gathering and forensic analysis.

The procedure follows the operational lifecycle shape: detect and classify
the incident, triage and escalate to the right people, coordinate the
investigation across teams, manage communications to all audiences, drive
resolution and recovery, then conduct a blameless post-incident review.

See references/communication-templates.md for notification templates by
audience and severity level. See references/rca-framework.md for
the 5-whys methodology, fishbone diagram guidance, and post-mortem
document structure.

When to Use

• - Service-affecting incident declared — a P1 or P2 event requires

formal incident management with role assignment and communications

• - Escalation decision needed — determining who to notify at what

severity level and when to engage vendor support or management

• - Multi-team coordination required — investigation spans network,

security, application, and infrastructure teams needing a single command structure

• - Customer or regulatory notification required — incident has

external communication obligations (SLA breach, data exposure, regulatory reporting)

• - Post-incident review facilitation — scheduling, structuring, and

running blameless post-mortems with 5-whys root cause analysis

• - Incident metrics reporting — collecting MTTD, MTTI, MTTR, and

recurrence data for continuous improvement

Prerequisites

• - Incident management authority — the person initiating this process

must have authorization to declare incidents and assign roles within the organization

• - Contact directory — current on-call rosters, escalation contacts

for management, vendor TAC numbers, and regulatory notification contacts must be accessible

• - Communication channels — bridge call infrastructure (conference

line or collaboration tool), status page access, and email distribution lists for each stakeholder group must be established

• - Incident tracking system — a ticketing system to record the

incident, track actions, and maintain the timeline of events

• - Defined severity criteria — organizational agreement on what

constitutes P1 through P4 severity (see Threshold Tables below for a reference framework)

Procedure

Follow these six steps in sequence. Steps 3 and 4 run in parallel once
roles are assigned — investigation coordination and communication
management proceed simultaneously. Each step references templates from
references/communication-templates.md and methodology from
references/rca-framework.md where applicable.

Step 1: Detection and Classification

Classify the incident by severity, type, and scope to determine the
appropriate response level.

Severity assignment — apply the P1–P4 taxonomy from the Threshold
Tables section. Base severity on the highest-impact criterion met.
When multiple criteria apply at different levels, the highest governs.

Incident type classification — categorize as outage (service
unavailable), degradation (reduced capacity), security (unauthorized
access or data exposure), or data loss (corruption or deletion).

Scope determination — assess whether the incident affects a single
device, a network segment, an entire site, or multiple sites. Scope
drives staffing, communication breadth, and recovery complexity.

Initial impact assessment — estimate affected user count, impacted
services and their business criticality, data at risk, and revenue
impact per hour. Record estimates in the incident ticket.

Step 2: Triage and Escalation

Assign roles, notify stakeholders, and set response timeline
expectations based on the severity classification from Step 1.

Role assignment — every P1 or P2 incident needs four named roles:
Incident Commander (IC) — owns the incident end-to-end and makes
escalation decisions; Technical Lead — coordinates diagnostics and
synthesizes findings; Communications Lead — drafts stakeholder
notifications and manages the status page; Scribe — maintains the
real-time timeline and records bridge call decisions. For P3, IC and
Technical Lead may be combined. P4 uses normal operations workflows.

Escalation matrix execution — notify by severity:
P1 — all four roles plus engineering management, VP/director on-call,
vendor TAC if vendor equipment is involved, executive notification
within 30 minutes. P2 — all four roles plus engineering management
within 1 hour. P3 — Technical Lead plus team lead within 4 hours.
P4 — assigned engineer via normal ticket queue.

Response timeline expectations:
P1 — bridge in 15 minutes, first update in 30 minutes, then every
30 minutes. P2 — bridge in 30 minutes, first update in 1 hour, then
every 2 hours. P3 — initial assessment in 4 hours, daily updates.
P4 — acknowledgment within 1 business day.

Vendor engagement criteria — engage vendor TAC when the incident
involves hardware failure, software defects requiring patches, or when
internal triage has not identified root cause within the severity time
window.

Step 3: Investigation Coordination

Coordinate the technical investigation across teams and evidence
sources. For network-level evidence collection (device state, routing
tables, interface data, log retrieval), reference the
incident-response-network skill — this step focuses on organizing the
investigation, not executing forensic commands.

Evidence collection tasking — assign team members to collect
evidence from relevant domains: network devices (via
incident-response-network procedures), application logs, infrastructure
metrics, and security tooling alerts. Each assignee reports findings
to the Technical Lead.

Parallel investigation streams — for complex incidents, run
multiple investigation threads simultaneously. Common parallel
tracks: (1) symptom analysis — what is failing and for whom,
(2) change correlation — what changed recently (deployments, config
modifications, maintenance), (3) external factors — upstream provider
issues, DDoS, DNS resolution failures.

Hypothesis tracking — maintain a running list of hypotheses with
current status (investigating, confirmed, ruled out). Each hypothesis
should have an owner and a validation method. Update the list on every
bridge call.

Timeline of events (ToE) — the Scribe maintains a running
chronological log of when events occurred, when they were detected,
what actions were taken, and what was discovered. The ToE becomes the
foundation for the post-incident review in Step 6.

Subject matter expert engagement — when investigation stalls or
enters an unfamiliar domain, escalate to specialists. Define clear
handoff: what has been tried, what data is available, and what
specific question needs answering.

Step 4: Communication Management

Manage stakeholder communications throughout the incident. Use the
templates in references/communication-templates.md for consistent
messaging across audiences.

Stakeholder notification by audience — executive summary (business
impact, estimated resolution, customer exposure — no technical detail),
technical detail (root cause hypothesis, diagnostics, remediation plan
— delivered on bridge call), customer-facing (service impact, workaround
if available, estimated resolution — via status page), regulatory
(formal notification per compliance framework when required). Use
templates from references/communication-templates.md.

Status update cadence — follow severity-based cadence from Step 2.
Each update includes: current status, progress since last update, next
planned action, and revised time-to-resolution estimate.

Bridge call management — the IC runs calls with a fixed agenda:
(1) technical status from Tech Lead, (2) communication status from
Comms Lead, (3) hypothesis updates, (4) decisions needed, (5) action
items with owners and deadlines. Keep calls focused — park side
discussions as action items.

External notification requirements — track regulatory reporting
deadlines, law enforcement notification when criminal activity is
suspected, customer SLA breach notification per contractual terms, and
vendor escalation for ongoing support.

Step 5: Resolution and Recovery

Drive service restoration through validated recovery steps with
monitoring to confirm the fix holds.

Recovery validation criteria — before declaring resolved, confirm:
(1) service health checks return normal for all affected components,
(2) monitoring dashboards show green for at least 15 minutes (P1) or
30 minutes (P2), (3) no new related alerts during observation, (4)
affected users confirm restoration (sample check for large populations).

Phased restoration — for multi-layer network incidents, restore in
order: core infrastructure → distribution layer → access layer →
end-to-end verification. Verify each phase before proceeding. Do not
restore all layers simultaneously — cascading failures during recovery
are worse than a phased approach.

Back-out plan execution — if the fix causes new issues, execute
the pre-defined rollback. Every remediation action should have a
documented rollback method before execution.

Enhanced monitoring period — maintain heightened monitoring after
resolution: P1 for 24 hours, P2 for 12 hours, P3 through the next
business day. This means reduced alert thresholds on affected systems,
active watch by on-call, and immediate re-escalation if symptoms recur.

Incident closure — send closure notification to all stakeholders
(template in references/communication-templates.md). Update the
ticket with resolution summary, total duration, and final impact.
Schedule the post-incident review.

Step 6: Post-Incident Review

Conduct a blameless post-incident review to identify root cause,
contributing factors, and improvement actions. See
references/rca-framework.md for the full methodology.

Scheduling — hold the post-mortem within 72 hours of incident
resolution while details are fresh. Invite all incident participants
plus relevant stakeholders. Send the invitation using the template in
references/communication-templates.md.

5-whys root cause analysis — apply iteratively: for each "why"
answer, ask "why" again until reaching a systemic root cause (typically
3–5 iterations). See references/rca-framework.md for worked
examples and facilitation guidance.

Contributing factor categorization — classify each contributing
factor as process (missing runbook, unclear escalation path),
people (training gap, staffing shortage), or technology (monitoring
gap, single point of failure, software defect). This categorization
guides the type of remediation action needed.

Action item classification — assign each action item one of four
dispositions: fix (eliminate the root cause), mitigate (reduce
likelihood or impact), accept (risk is within tolerance, document
rationale), or transfer (assign to another team or vendor). Every
fix or mitigate action must have an owner, due date, and verification
method.

Incident metrics — collect and record: Mean Time to Detect (MTTD),
Mean Time to Investigate (MTTI), Mean Time to Resolve (MTTR), total
incident duration, number of customers affected, and whether this is
a recurrence of a previous incident. Track these metrics over time to
measure improvement trends.

Threshold Tables

Severity Classification Matrix

Severity	User Impact	Service Impact	Data Risk	Response SLA
P1 Critical	>50% of users or all VIP users	Complete outage of revenue-generating service	Confirmed data breach or loss	Bridge in 15 min, updates every 30 min
P2 High

10–50% of users affected | Major degradation or redundancy loss on critical path | Suspected data exposure | Bridge in 30 min, updates every 2 hr | | P3 Medium | <10% of users, workaround exists | Partial degradation, non-critical service | No data risk identified | Assessment in 4 hr, updates daily | | P4 Low | Minimal or no user impact | Cosmetic, non-production, or fully redundant | None | Ack in 1 business day |

Escalation and Role Matrix

Severity	Incident Commander	Technical Lead	Comms Lead	Scribe	Management	Executive
P1	Required	Required	Required	Required	Immediate	Within 30 min
P2

Required | Required | Required | Optional | Within 1 hr | If SLA breached | | P3 | Combined with Tech Lead | Required | Optional | No | Within 4 hr | No | | P4 | No | Assigned engineer | No | No | Normal reporting | No |

Enhanced Monitoring Duration

Severity	Monitoring Period	Alert Threshold	Re-escalation Trigger
P1	24 hours	Reduced by 20%	Any recurrence symptom
P2

12 hours | Reduced by 10% | Same failure signature | | P3 | Next business day | Normal thresholds | Identical alert | | P4 | None | Normal | Normal process |

Decision Trees

Incident Severity Assignment

CODEBLOCK0

Escalation Decision

CODEBLOCK1

Report Template

CODEBLOCK2

Troubleshooting

Severity Disagreement Between Teams

Symptom: Teams classify the same incident at different severity
levels, causing confusion about response urgency.

Resolution: The IC makes the final determination using the
Threshold Tables criteria. The highest applicable severity governs.
Document rationale in the ticket. If the IC is not yet assigned, the
first responder sets initial severity and the IC may adjust.

Escalation Fatigue and Alert Noise

Symptom: Frequent P1/P2 declarations for issues that resolve
quickly, eroding trust in severity classification.

Resolution: Review severity criteria quarterly. Track the
false-positive rate (incidents downgraded after initial classification).
If P1 downgrade rate exceeds 30%, tighten P1 criteria. Ensure P3/P4
incidents are not over-classified.

Post-Mortem Action Items Not Completed

Symptom: Action items accumulate but are not completed, leading
to recurring incidents from known causes.

Resolution: Assign every action item an owner and due date at the
review. Track completion in the incident system, not separate documents.
Review open items in weekly standups. Escalate overdue items to
management and report completion rates alongside MTTD/MTTR.

Communication Gaps During Extended Incidents

Symptom: Status updates become infrequent during long incidents
(>4 hours), leaving stakeholders uninformed.

Resolution: The Communications Lead maintains cadence regardless of
investigation progress. If no new findings exist, state that explicitly
in the update. For incidents exceeding 8 hours, rotate the Comms Lead
role to prevent fatigue.

Incident Recurrence After Resolution

Symptom: The same incident recurs after being marked resolved.

Resolution: Check whether prior post-mortem action items were
completed. If yes, the root cause analysis was incomplete — reconvene
with broader scope. If not, escalate the completion failure. Tag the
new incident as a recurrence and increase severity by one level to
reflect accumulated impact.