Data capture for inference on HyperPod

Amazon SageMaker HyperPod inference data capture enables you to record inference request and response data for model monitoring, debugging, and model improvement. Inference requests flow from the SageMaker AI endpoint to the Application Load Balancer and then to the model pod. You can enable capture independently at each level, from the outermost layer (Tier 1, SageMaker AI endpoint) to the deepest layer (Tier 3, model pod):

Tier 1 captures full payloads at the SageMaker AI Runtime API boundary and requires endpoint registration. Use Tier 1 when you need compatibility with SageMaker AI Model Monitor. Tier 3 captures full payloads at the inference container with configurable buffering, sampling, and payload limits, and works without SageMaker AI endpoint registration. Use Tier 3 when you need the deepest visibility closest to the model. Enable any combination — each tier captures at a different point in the request flow.

All tiers write to your Amazon S3 bucket. If you don't specify an s3Uri, data is stored in the TLS certificate bucket under a /data-capture/ prefix by default. Within the bucket, each deployment gets a unique path based on a hash derived from the cluster ARN, namespace, CRD type, and deployment name. The same deployment always generates the same prefix, so data capture artifacts from multiple CRD submissions targeting the same deployment flow to the same Amazon S3 subfolder.

To disable data capture for a tier, set its enabled field to false or remove the tier section from your CRD. To disable all data capture, remove the dataCapture section entirely.

Configuring data capture

Enable data capture by adding a dataCapture section to your InferenceEndpointConfig or JumpStartModel CRD. The dataCapture block contains a single s3Uri (optional) and one or more tier configurations. The following example shows the overall structure with all three tiers enabled:

  dataCapture:
    s3Uri: s3://my-capture-bucket/captures/   # Optional. Defaults to TLS bucket.
    sagemakerEndpoint:
      enabled: true
      # Tier 1 fields...
    loadBalancer:
      enabled: true
    modelPod:
      enabled: true
      # Tier 3 fields...

Tier 1: SageMaker AI endpoint capture

Tier 1 uses SageMaker AI's native DataCaptureConfig to capture inference input and output payloads at the endpoint level. This is the outermost capture point and is compatible with SageMaker AI Model Monitor for automated data quality monitoring. For more information about SageMaker AI data capture, see Capture data from real-time endpoint.

    sagemakerEndpoint:
      enabled: true
      initialSamplingPercentage: 100
      kmsKeyId: arn:aws:kms:us-east-2:123456789012:key/my-key-id
      captureOptions:
        - captureMode: Input
        - captureMode: Output
      captureContentTypeHeader:
        jsonContentTypes:
          - application/json

Tier 2: Load balancer capture

Tier 2 enables ALB access logs, capturing request metadata such as client IPs, request paths, and latencies.

    loadBalancer:
      enabled: true

Tier 3: Model pod capture

Tier 3 captures inference input and output payloads at the model pod level, providing the deepest visibility into inference traffic. Use this tier when you need fine-grained control over buffering, payload size limits, and capture closest to the model.

    modelPod:
      enabled: true
      initialSamplingPercentage: 100
      kmsKeyId: arn:aws:kms:us-east-2:123456789012:key/my-key-id
      captureOptions:
        - captureMode: Input
        - captureMode: Output
      bufferConfig:
        batchSize: 100
        flushIntervalSeconds: 60
      payloadConfig:
        maxPayloadSizeKB: 1024

Update Addon

Prerequisites: Authenticate and connect to your EKS cluster

Authenticate to your AWS account. Before connecting, collect EKS cluster name, region, and HyperPod cluster ARN:

CLUSTER=EKS_CLUSTER_NAME
REGION=REGION
HP_ARN=HYPERPOD_CLUSTER_ARN

Connect to your EKS cluster:

aws eks update-kubeconfig --region REGION --name EKS_CLUSTER_NAME

Update the addon configuration:

VERSION=$(aws eks describe-addon --cluster-name $CLUSTER --addon-name amazon-sagemaker-hyperpod-inference --region $REGION --query 'addon.addonVersion' --output text)
CURRENT_CONFIG=$(aws eks describe-addon --cluster-name $CLUSTER --addon-name amazon-sagemaker-hyperpod-inference --region $REGION --query 'addon.configurationValues' --output text)
NEW_CONFIG=$(echo "$CURRENT_CONFIG" | jq --arg arn "$HP_ARN" '. + {hyperpodClusterArn: $arn}')

aws eks update-addon \
--cluster-name $CLUSTER \
--addon-name amazon-sagemaker-hyperpod-inference \
--addon-version "$VERSION" \
--configuration-values "$NEW_CONFIG" \
--resolve-conflicts OVERWRITE \
--region $REGION

Wait for the Addon to be active and then deploy the models

Updating permissions for existing clusters

To enable data capture on your HyperPod Inference deployments, configure the following IAM permissions.

Best practices