OPS10-BP02 Have a process per alert
Establishing a clear and defined process for each alert in your system is essential for effective and efficient incident management. This practice ensures that every alert leads to a specific, actionable response, improving the reliability and responsiveness of your operations.
Desired outcome: Every alert initiates a specific, well-defined response plan. Where possible, responses are automated, with clear ownership and a defined escalation path. Alerts are linked to an up-to-date knowledge base so that any operator can respond consistently and effectively. Responses are quick and uniform across the board, enhancing operational efficiency and reliability.
Common anti-patterns:
-
Alerts have no predefined response process, leading to makeshift and delayed resolutions.
-
Alert overload causes important alerts to be overlooked.
-
Alerts are inconsistently handled due to lack of clear ownership and responsibility.
Benefits of establishing this best practice:
-
Reduced alert fatigue by only raising actionable alerts.
-
Decreased mean time to resolution (MTTR) for operational issues.
-
Decreased mean time to investigate (MTTI), which helps reduce MTTR.
-
Enhanced ability to scale operational responses.
-
Improved consistency and reliability in handling operational events.
For example, you have a defined process for AWS Health events for critical accounts, including application alarms, operational issues, and planned lifecycle events (like updating Amazon EKS versions before clusters are auto-updated), and you provide the capability for your teams to actively monitor, communicate, and respond to these events. These actions help you prevent service disruptions caused by AWS-side changes or mitigate them faster when unexpected issues occur.
Level of risk exposed if this best practice is not established: High
Implementation guidance
Having a process per alert involves establishing a clear response plan for each alert, automating responses where possible, and continually refining these processes based on operational feedback and evolving requirements.
Implementation steps
The following diagram illustrates the incident management workflow within AWS Systems Manager Incident Manager
-
Use composite alarms: Create composite alarms in CloudWatch to group related alarms, reducing noise and allowing for more meaningful responses.
-
Monitor AWS Health events with Amazon EventBridge rules: Monitor or integrate programatically with the AWS Health API to automate actions when you receive AWS Health events. These can be general actions, such as sending all planned lifecycle event messages to a chat interface, or specific actions, such as the initiation of a workflow in an IT service management tool.
-
Integrate Amazon CloudWatch alarms with Incident Manager: Configure CloudWatch alarms to automatically create incidents in AWS Systems Manager Incident Manager.
-
Integrate Amazon EventBridge with Incident Manager: Create EventBridge rules to react to events and create incidents using defined response plans.
-
Prepare for incidents in Incident Manager:
-
Establish detailed response plans in Incident Manager for each type of alert.
-
Establish chat channels through Amazon Q Developer in chat applications connected to response plans in Incident Manager, facilitating real-time communication during incidents across platforms like Slack, Microsoft Teams, and Amazon Chime.
-
Incorporate Systems Manager Automation runbooks within Incident Manager to drive automated responses to incidents.
-
Resources
Related best practices:
Related documents:
Related videos:
Related examples:
