AWS Secrets Manager Agent

The AWS Secrets Manager Agent is a client-side HTTP service that you can use to standardize consumption of secrets from Secrets Manager across environments such as AWS Lambda, Amazon Elastic Container Service, Amazon Elastic Kubernetes Service, and Amazon Elastic Compute Cloud. The Secrets Manager Agent can retrieve and cache secrets in memory so that your applications can consume secrets directly from the cache. That means you can fetch the secrets your application needs from the localhost instead of making calls to Secrets Manager. The Secrets Manager Agent can only make read requests to Secrets Manager - it can't modify secrets.

The Secrets Manager Agent uses the AWS credentials you provide in your environment to make calls to Secrets Manager. The Secrets Manager Agent offers protection against Server Side Request Forgery (SSRF) to help improve secret security. You can configure the Secrets Manager Agent by setting the maximum number of connections, the time to live (TTL), the localhost HTTP port, and the cache size.

Because the Secrets Manager Agent uses an in-memory cache, it resets when the Secrets Manager Agent restarts. The Secrets Manager Agent periodically refreshes the cached secret value. The refresh happens when you try to read a secret from the Secrets Manager Agent after the TTL has expired. The default refresh frequency (TTL) is 300 seconds, and you can change it by using a Configuration file which you pass to the Secrets Manager Agent using the --config command line argument. The Secrets Manager Agent does not include cache invalidation. For example, if a secret rotates before the cache entry expires, the Secrets Manager Agent might return a stale secret value.

The Secrets Manager Agent returns secret values in the same format as the response of GetSecretValue. Secret values are not encrypted in the cache.

To download the source code, see https://github.com/aws/aws-secretsmanager-agent on GitHub.

Step 1: Build the Secrets Manager Agent binary

To build the Secrets Manager Agent binary natively, you need the standard development tools and the Rust tools. Alternatively, you can cross-compile for systems that support it, or you can use Rust cross to cross-compile.

RPM-based systems

1. On RPM-based systems such as AL2023, you can install the development tools by using the Development Tools group.

   sudo yum -y groupinstall "Development Tools"

2. Follow the instructions at Install Rust in the Rust documentation.

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   . "$HOME/.cargo/env"

3. Build the agent using the cargo build command:

   cargo build --release

   You will find the executable under target/release/aws-secrets-manager-agent.

Debian-based systems

1. On Debian-based systems such as Ubuntu, you can install the developer tools using the build-essential package.

   sudo apt install build-essential

2. Follow the instructions at Install Rust in the Rust documentation.

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   . "$HOME/.cargo/env"

3. Build the agent using the cargo build command:

   cargo build --release

   You will find the executable under target/release/aws-secrets-manager-agent.

Windows

To build on Windows, follow the instructions at Set up your dev environment on Windows for Rust in the Microsoft Windows documentation.

Cross-compile natively

On distributions where the mingw-w64 package is available such as Ubuntu, you can cross compile natively.

# Install the cross compile tool chain
sudo add-apt-repository universe
sudo apt install -y mingw-w64
    
# Install the rust build targets
rustup target add x86_64-pc-windows-gnu
    
# Cross compile the agent for Windows
cargo build --release --target x86_64-pc-windows-gnu

You will find the executable at target/x86_64-pc-windows-gnu/release/aws-secrets-manager-agent.exe.

Cross compile with Rust cross

If the cross compile tools are not available natively on the system, you can use the Rust cross project. For more information, see https://github.com/cross-rs/cross.

Important

We recommend 32GB disk space for the build environment.

# Install and start docker
sudo yum -y install docker
sudo systemctl start docker
sudo systemctl enable docker # Make docker start after reboot
    
# Give ourselves permission to run the docker images without sudo
sudo usermod -aG docker $USER
newgrp docker
    
# Install cross and cross compile the executable
cargo install cross
cross build --release --target x86_64-pc-windows-gnu

Step 2: Install the Secrets Manager Agent

Based on the type of compute, you have several options for installing the Secrets Manager Agent.

Amazon EKS, Amazon EC2, and Amazon ECS

To install the Secrets Manager Agent

1. Use the install script provided in the repository.

   The script generates a random SSRF token on startup and stores it in the file /var/run/awssmatoken. The token is readable by the awssmatokenreader group that the install script creates.

2. To allow your application to read the token file, you need to add the user account that your application runs under to the awssmatokenreader group. For example, you can grant permissions for your application to read the token file with the following usermod command, where <APP_USER> is the user ID under which your application runs.

   sudo usermod -aG awssmatokenreader <APP_USER>

Docker

You can run the Secrets Manager Agent as a sidecar container alongside your application by using Docker. Then your application can retrieve secrets from the local HTTP server the Secrets Manager Agent provides. For information about Docker, see the Docker documentation.

To create a sidecar container for the Secrets Manager Agent with Docker

1. Create a Dockerfile for the Secrets Manager Agent sidecar container. The following example creates a Docker container with the Secrets Manager Agent binary.

   # Use the latest Debian image as the base
   FROM debian:latest
   
   # Set the working directory inside the container
   WORKDIR /app 
   
   # Copy the Secrets Manager Agent binary to the container
   COPY secrets-manager-agent . 
   
   # Install any necessary dependencies
   RUN apt-get update && apt-get install -y ca-certificates 
   
   # Set the entry point to run the Secrets Manager Agent binary
   ENTRYPOINT ["./secrets-manager-agent"]

2. Create a Dockerfile for your client application.

3. Create a Docker Compose file to run both containers, being sure that they use the same network interface. This is necessary because the Secrets Manager Agent does not accept requests from outside the localhost interface. The following example shows a Docker Compose file where the network_mode key attaches the secrets-manager-agent container to the network namespace of the client-application container, which allows them to share the same network interface.

   Important

   You must load AWS credentials and the SSRF token for the application to be able to use the Secrets Manager Agent. See the following:

   • Manage access in the Amazon Elastic Kubernetes Service User Guide

   • Amazon ECS task IAM role in the Amazon Elastic Container Service Developer Guide

   version: '3'
   services:
       client-application:
       container_name: client-application
       build:
           context: .
           dockerfile: Dockerfile.client
       command: tail -f /dev/null  # Keep the container running
       
   
       secrets-manager-agent:
       container_name: secrets-manager-agent
       build:
           context: .
           dockerfile: Dockerfile.agent
       network_mode: "container:client-application"  # Attach to the client-application container's network
       depends_on:
           - client-application

4. Copy the secrets-manager-agent binary to the same directory that contains your Dockerfiles and Docker Compose file.

5. Build and run the containers based on the provided Dockerfiles by using the following docker-compose command.

   docker-compose up --build

6. In your client container, you can now use the Secrets Manager Agent to retrieve secrets. For more information, see Step 3: Retrieve secrets with the Secrets Manager Agent.

AWS Lambda

You can package the Secrets Manager Agent as an AWS Lambda extension. Then you can add it to your Lambda function as a layer and call the Secrets Manager Agent from your Lambda function to get secrets.

The following instructions show how to get a secret named MyTest by using the example script secrets-manager-agent-extension.sh in https://github.com/aws/aws-secretsmanager-agent to install the Secrets Manager Agent as a Lambda extension.

Note

The example script uses the curl command, which is included with Amazon Linux 2023 based runtimes such as Python 3.12 and Node.js 20. If you use a runtime environment based on Amazon Linux 2 such as Python 3.11 or Node.js 18, you must first install curl in your Lambda container image. For instructions, see How can I use Amazon Linux 2 AMI native binary packages with Lambda on AWS re:Post.

To create a Lambda extension that packages the Secrets Manager Agent

1. Create a Python Lambda function that queries http://localhost:2773/secretsmanager/get?secretId=MyTest to get the secret. Be sure to implement retry logic in your application code to accommodate delays in initialization and registration of the Lambda extension.

2. From the root of the Secrets Manager Agent code package, run the following commands to test the Lambda extension.

   AWS_ACCOUNT_ID=<AWS_ACCOUNT_ID>
   LAMBDA_ARN=<LAMBDA_ARN>
   
   # Build the release binary 
   cargo build --release --target=x86_64-unknown-linux-gnu
   
   # Copy the release binary into the `bin` folder
   mkdir -p ./bin
   cp ./target/x86_64-unknown-linux-gnu/release/aws_secretsmanager_agent ./bin/secrets-manager-agent
   
   # Copy the `secrets-manager-agent-extension.sh` script into the `extensions` folder.
   mkdir -p ./extensions
   cp aws_secretsmanager_agent/examples/example-lambda-extension/secrets-manager-agent-extension.sh ./extensions
   
   # Zip the extension shell script and the binary 
   zip secrets-manager-agent-extension.zip bin/* extensions/*
   
   # Publish the layer version
   LAYER_VERSION_ARN=$(aws lambda publish-layer-version \
       --layer-name secrets-manager-agent-extension \
       --zip-file "fileb://secrets-manager-agent-extension.zip" | jq -r '.LayerVersionArn')
   
   # Attach the layer version to the Lambda function
   aws lambda update-function-configuration \
       --function-name $LAMBDA_ARN \
       --layers "$LAYER_VERSION_ARN"

3. Invoke the Lambda function to verify that the secret is being correctly fetched.

Step 3: Retrieve secrets with the Secrets Manager Agent

To use the agent, you call the local Secrets Manager Agent endpoint and include the name or ARN of the secret as a query parameter. By default, the Secrets Manager Agent retrieves the AWSCURRENT version of the secret. To retrieve a different version, you can set versionStage or versionId.

To help protect the Secrets Manager Agent, you must include a SSRF token header as part of each request: X-Aws-Parameters-Secrets-Token. The Secrets Manager Agent denies requests that don't have this header or that have an invalid SSRF token. You can customize the SSRF header name in the Configuration file.

The Secrets Manager Agent uses the AWS SDK for Rust, which uses the default credential provider chain. The identity of these IAM credentials determines the permissions the Secrets Manager Agent has to retrieve secrets.

Required permissions:

For more information, see Permissions reference.

curl

The following curl example shows how to get a secret from the Secrets Manager Agent. The example relies on the SSRF being present in a file, which is where it is stored by the install script.

curl -v -H \
    "X-Aws-Parameters-Secrets-Token: $(</var/run/awssmatoken)" \
    'http://localhost:2773/secretsmanager/get?secretId=<YOUR_SECRET_ID>'; \
    echo

Python

The following Python example shows how to get a secret from the Secrets Manager Agent. The example relies on the SSRF being present in a file, which is where it is stored by the install script.

import requests
import json

# Function that fetches the secret from Secrets Manager Agent for the provided secret id. 
def get_secret():
    # Construct the URL for the GET request
    url = f"http://localhost:2773/secretsmanager/get?secretId=<YOUR_SECRET_ID>"

    # Get the SSRF token from the token file
    with open('/var/run/awssmatoken') as fp:
        token = fp.read() 

    headers = {
        "X-Aws-Parameters-Secrets-Token": token.strip()
    }

    try:
        # Send the GET request with headers
        response = requests.get(url, headers=headers)

        # Check if the request was successful
        if response.status_code == 200:
            # Return the secret value
            return response.text
        else:
            # Handle error cases
            raise Exception(f"Status code {response.status_code} - {response.text}")

    except Exception as e:
        # Handle network errors
        raise Exception(f"Error: {e}")

Configure the Secrets Manager Agent

To change the configuration of the Secrets Manager Agent, create a TOML config file, and then call ./aws-secrets-manager-agent --config config.toml.

The following list shows the options you can configure for the Secrets Manager Agent.

Logging

The Secrets Manager Agent logs errors locally to the file logs/secrets_manager_agent.log. When your application calls the Secrets Manager Agent to get a secret, those calls appear in the local log. They do not appear in the CloudTrail logs.

The Secrets Manager Agent creates a new log file when the file reaches 10 MB, and it stores up to five log files total.

The log does not go to Secrets Manager, CloudTrail, or CloudWatch. Requests to get secrets from the Secrets Manager Agent do not appear in those logs. When the Secrets Manager Agent makes a call to Secrets Manager to get a secret, that call is recorded in CloudTrail with a user agent string containing aws-secrets-manager-agent.

You can configure logging in the Configuration file.

Security considerations

For an agent architecture, the domain of trust is where the agent endpoint and SSRF token are accessible, which is usually the entire host. The domain of trust for the Secrets Manager Agent should match the domain where the Secrets Manager credentials are available in order to maintain the same security posture. For example, on Amazon EC2 the domain of trust for the Secrets Manager Agent would be the same as the domain of the credentials when using roles for Amazon EC2.

Security conscious applications that are not already using an agent solution with the Secrets Manager credentials locked down to the application should consider using the language-specific AWS SDKs or caching solutions. For more information, see Get secrets from AWS Secrets Manager.