Overview
The Emergency Change Process is a controlled procedure for implementing critical system changes outside normal change windows to resolve active incidents or prevent significant business impact. It ensures emergency changes under SOC 2 CC8.1 are authorized, documented, tested as feasible, and retrospectively reviewed.
Step-by-Step Process
Identify emergency condition
An on-call engineer or Engineering Lead confirms that an incident requires an immediate change due to service outage, security risk, or data integrity issue. The decision is based on real-time monitoring, alerts, or incident impact. The output is a documented determination that the change qualifies as an emergency.
Role: Engineering Lead
Open emergency change record
The engineer creates an emergency change ticket documenting the issue, scope of change, and systems affected. The ticket must be clearly labeled as an emergency and linked to the incident record. The output is a uniquely identified emergency change record.
Role: Engineer
Obtain expedited approval
The Engineering Lead or delegated approver reviews the proposed change for risk and necessity and provides approval verbally or in-system. Approval must be captured in writing as soon as practicable. The output is recorded approval tied to the change ticket.
Role: Engineering Lead
Implement emergency change
The engineer deploys the change following established technical procedures while minimizing scope and duration. Any deviations from standard testing are noted. The output is a completed change deployed to production.
Role: Engineer
Verify system stability
Post-implementation checks are performed to confirm the issue is resolved and no new issues are introduced. Monitoring and logs are reviewed for anomalies. The output is confirmation of system stability.
Role: Engineer
Document post-change details
The engineer updates the change record with implementation time, results, and any rollback actions taken. Evidence such as logs or screenshots is attached. The output is a fully documented emergency change record.
Role: Engineer
Conduct retrospective review
Within the next business day, the Engineering Lead reviews the emergency change for appropriateness and lessons learned. Required follow-up actions are documented and assigned. The output is a completed post-implementation review.
Role: Engineering Lead
What You Need Before Starting
- Access to production systems requiring emergency changes
- Incident alert or outage notification with timestamp
- Change management tool access (e.g., Jira)
- Approval authority from Engineering Lead
- Communication tools (PagerDuty, Slack)
Evidence Your Auditor Expects
- Emergency change ticket showing creation date/time and emergency flag
- Incident record linked to the change with timestamps
- Approval comment or approval field showing approver name and date/time
- Deployment or system logs showing change execution timestamp
- Post-implementation review notes dated within one business day
How This Looks In Your Tools
Jira
Navigate to Projects > Select the relevant engineering project > Create Issue. Choose issue type “Change” or “Emergency Change” and add “Emergency” in the summary and labels field. Complete required fields including description, impacted systems, and risk, then link the issue to the related Incident ticket using the “Linked Issues” option.
After approval, update the issue status to “In Progress” and document deployment details in the Comments section with timestamps. Once complete, transition the issue to “Done” or “Implemented” and attach evidence such as screenshots, log files, or deployment links. Add a final comment for post-implementation review outcomes.
PagerDuty
From the PagerDuty dashboard, go to Incidents > Open the active incident triggering the emergency change. Verify severity and impact, then assign the incident to the appropriate on-call engineer or team.
Use the incident Notes section to document the decision to perform an emergency change, including date/time and approver name. After resolution, ensure the incident is resolved with detailed notes describing the change and link to the Jira emergency change ticket.
Slack
Open the designated incident or on-call channel (e.g., #incidents or #oncall). Announce the emergency change with a brief description, systems affected, and approving authority.
After implementation, post a follow-up message confirming completion and outcome, including timestamps and a link to the Jira change record. Retain the Slack message history as supporting evidence for communication and approval.
Common Audit Findings
- Emergency changes not clearly labeled
- This occurs when teams use standard change tickets during incidents. Require the use of an "Emergency" label or issue type and periodically review tickets for correct classification.
- Missing documented approval
- Verbal approvals are not later recorded, leaving no audit trail. Enforce a rule that approvals must be captured in the change ticket comments or approval field.
- No post-implementation review
- Teams resolve incidents quickly but skip retrospective reviews. Schedule a mandatory next-business-day review and track completion as part of change closure.
- Insufficient linkage between incident and change
- Auditors cannot trace why the emergency change occurred. Require linking incident IDs in the change record and vice versa.