Rotate database credentials without restarting containers

Created by Josh Joy (AWS)

Summary

On the Amazon Web Services (AWS) Cloud, you can use AWS Secrets Manager to rotate, manage, and retrieve database credentials throughout their lifecycle. Users and applications retrieve secrets with a call to the Secrets Manager API, removing the need to hardcode sensitive information in plaintext.

If you’re using containers for microservice workloads, you can securely store credentials in AWS Secrets Manager. To separate out configuration from code, these credentials are commonly injected into the container. However, it's important to rotate your credentials periodically and automatically. It’s also important to support the ability to refresh credentials after revocation. At the same time, applications require the ability to rotate credentials while reducing any potential downstream availability impact.

This pattern describes how to rotate your secrets that are secured with AWS Secrets Manager within your containers without requiring your containers to restart. In addition, this pattern reduces the number of credential lookups to Secrets Manager by using the Secrets Manager client-side caching component. When you use the client-side caching component to refresh the credentials within the application, the container doesn't need to be restarted to fetch a rotated credential.

This approach works for Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon Elastic Container Service (Amazon ECS).

Two scenarios are covered. In the single-user scenario, the database credential is refreshed on secret rotation by detecting the expired credential. The credential cache is instructed to refresh the secret, and then the application re-establishes the database connection. The client-side caching component caches the credential within the application and helps avoid reaching out to Secrets Manager for each credential lookup. The credential is rotated within the application without the need to force the credential refresh by restarting the container.

The second scenario rotates the secret by alternating between two users. Having two active users reduces the potential for downtime, because one user’s credentials are always active. Two-user credential rotation is helpful when you have a large deployment with clusters in which there might be a small propagation delay of credential updates.

Prerequisites and limitations

Prerequisites 

Limitations 

Architecture

Target architecture 

Scenario 1 – Rotation of a credential for a single user

In the first scenario, a single database credential is periodically rotated by Secrets Manager. The application container runs in Fargate. When the first database connection is established, the application container fetches the database credential for Aurora. The Secrets Manager caching component then caches the credential for future connection establishment. When rotation period has elapsed, the credential expires and the database returns an authentication error. The application then fetches the rotated credential, invalidates the cache, and updates the credential cache via the Secrets Manager client-side caching component.

In this scenario, there might be a minimal disruption while the credential is being rotated and stale connections are using the outdated credential. This concern can be addressed by using the two-user scenario.

Scenario 2 – Rotation of credentials for two users

In the second scenario, two database user credentials (Alice’s and Bob’s) are periodically rotated by Secrets Manager. The application container runs in a Fargate cluster. When the first database connection is established, the application container fetches the Aurora database credential for the first user (Alice). The Secrets Manager caching component then caches the credential for future connection establishment.

Although there are two users and credentials, one only active credential is managed by Secrets Manager. In this case, the caching component periodically expires and fetches the latest credential. If the Secrets Manager rotation period is longer than the cache timeout, the caching component picks up the rotated credential for the second user (Bob). For example, if the cache expiration is measured in minutes and the rotation period is measured in days, the caching component fetches the new credential as part of its periodic cache refresh. In this way, the downtime is minimized because each user's credential is active for one Secrets Manager rotation.

Automation and scale

You can use AWS CloudFormation to deploy this pattern by using infrastructure as code. This builds and creates the application container, creates the Fargate task, deploys the container into Fargate, and sets up and configure Secrets Manager with Aurora. For step-by-step deployment instructions, see the readme file.

Tools

Tools

Code 

Example Python code

This pattern uses the Python client-side caching component for Secrets Manager to retrieve the authentication credentials when establishing the database connection. The client-side caching component helps avoid reaching out to Secrets Manager each time.

Now, when the rotation period elapses, the cached credential will be expired, and connecting to the database will result in an authentication error. For MySQL, the authentication error code is 1045. This example uses Amazon Aurora for MySQL, though you could use another engine such as PostgreSQL. Upon the authentication error, the database connection exception handling code catches the error. It then informs the Secrets Manager client-side caching component to refresh the secret, then to reauthenticate and re-establish the database connection. If you are using PostgreSQL or another engine, you must look up the corresponding authentication error code.

The container application can now update the database password with the rotated password without restarting the container.

Place the following code in your application code that handles database connections. This example uses Django, and it subclasses the database backend with a database wrapper for connections. If you are using a different programming language or database connection library, see your database connection library to review how to subclass database connection retrieval.

    def get_new_connection(self, conn_params):
        try:
            logger.info("get connection")
            databasecredentials.get_conn_params_from_secrets_manager(conn_params)
            conn =super(DatabaseWrapper,self).get_new_connection(conn_params)
            return conn
        except MySQLdb.OperationalError as e:
            error_code=e.args[0]
            if error_code!=1045:
                raise e
 
            logger.info("Authentication error. Going to refresh secret and try again.")
            databasecredentials.refresh_now()
            databasecredentials.get_conn_params_from_secrets_manager(conn_params)
            conn=super(DatabaseWrapper,self).get_new_connection(conn_params)
            logger.info("Successfully refreshed secret and established new database connection.")
            return conn

AWS CloudFormation and Python code

Epics

Related resources

Create a Secrets Manager secret

Create an Amazon Aurora cluster

Create the Amazon ECS components

Download and install the Secrets Manager client-side caching component

Attachments

To access additional content that is associated with this document, unzip the following file: attachment.zip