Stage 4: Operate

After you've completed the Stage 3: Evaluate and test, you're ready to deploy the application to production. In the Operate stage, you deploy your application to production and manage your customers' experience.  The design and implementation of your application determine many of its resilience outcomes, but this stage focuses on the operational practices your system uses to maintain and improve resilience. Building a culture of operational excellence helps create standards and consistency in these practices.

Observability

The most important part of understanding the customer experience is through monitoring and alarming. You need to instrument your application to understand its state, and you need diverse perspectives, which means that you need to measure from both the server side and the client side, typically with canaries. Your metrics should include data about your application's interactions with its dependencies and dimensions that align to your fault isolation boundaries. You should also produce logs that provide additional details about every unit of work performed by your application. You might consider combining metrics and logs by using a solution such as the Amazon CloudWatch embedded metric format. You'll likely find that you always want more observability, so consider the cost, effort, and complexity trade-offs required to implement your desired level of instrumentation.

The following links provide best practices for instrumenting your application and creating alarms:

Event management

You should have an event management process in place to handle impairments when your alarms (or worse, your customers) tell you that something is going wrong. This process should include engaging an on-call operator, escalating problems, and establishing runbooks for consistent approaches to troubleshooting that help remove human errors. However, impairments typically don't happen in isolation; a single application could impact multiple other applications that depend on it. You can rapidly address issues by understanding all applications that are impacted and bringing operators from multiple teams together on a single conference call. However, depending on your organization's size and structure, this process might require a centralized operations team.

In addition to setting up an event management process, you should regularly review your metrics through dashboards. Regular reviews help you understand the customer experience and longer-term trends in the performance of your application. This helps you identify issues and bottlenecks before they cause significant production impact. Reviewing metrics in a consistent, standardized way provides significant benefits but requires top-down buy-in and an investment of time.

The following links provide best practices on building dashboards and operational metrics reviews:

Continuous resilience

During Stage 2: Design and implement and Stage 3: Evaluate and test, you initiated review and test activities before deploying your application to production. During the operate stage, you should continue iterating on those activities in production. You should periodically review the resilience posture of your application through AWS Well-Architected Framework reviews, Operational Readiness Reviews (ORRs), and the resilience analysis framework. This helps ensure that your application hasn't drifted from established baselines and standards and keeps you up to date with new or updated guidance. These continuous resilience activities help you discover previously unanticipated disruptions and help you come up with new mitigations.

You might also want to consider running game days and chaos engineering experiments in production after you've successfully run them in pre-production environments. Game days simulate known events that you have built resilience mechanisms to mitigate. For example, a game day might simulate an AWS Regional service impairment and implement a multi-Region failover. Although implementing these activities can require a significant level of effort, both practices help you build confidence that your system is resilient to the failure modes that you've designed it to withstand.

By operating your applications, encountering operational events, reviewing metrics, and testing your application, you'll encounter numerous opportunities to respond and learn.