Run an A/B test for agents hosted outside of AgentCore

You can A/B test an agent that runs outside an AgentCore Runtime with an agent hosted anywhere, for example on AWS Lambda, Amazon EKS, or Amazon ECS. For A/B testing agents, AgentCore gateway routes traffic between variants, online evaluation scores each session, and the service computes per-variant statistical significance. When your agent is not on an AgentCore Runtime, you need to instrument it for observability and register it as an AgentCore gateway target yourself, and enable gateway tracing so sessions can be attributed to a variant.

A/B tests supports configuration bundles with a single agent runtime as well as target based routing with two agent endpoints. This page shows how to set up a HTTP reachable agent endpoint as a target on AgentCore gateway with tracing enabled. Then, AgentCore gateway tracing supplies variant attribution, and observability on the agent endpoint supplies the message content for scoring.

This page is an addendum to Run an A/B test with target-based routing. It covers only the extra setup a non-runtime agent needs — using a Lambda-hosted agent (behind a function URL) as the example; the same approach applies to any HTTP-reachable agent endpoint you register as an AgentCore gateway target — then sends you back to that page to create and run the test.

Prerequisites

In addition to the general A/B testing prerequisites, you need:

Step 1: Instrument your agent for AgentCore observability

For a Lambda-hosted agent, attach the AWS Lambda Layer for OpenTelemetry and set the following environment variables:

aws lambda update-function-configuration \
  --function-name <function-name> \
  --region <region> \
  --layers <adot-python-layer-arn> \
  --environment "Variables={
    AGENT_OBSERVABILITY_ENABLED=true,
    OTEL_PROPAGATORS='baggage,xray-lambda,tracecontext',
    OTEL_PYTHON_DISTRO=aws_distro,
    OTEL_PYTHON_CONFIGURATOR=aws_configurator,
    OTEL_LOGS_EXPORTER=otlp,
    OTEL_TRACES_EXPORTER=otlp,
    OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf,
    OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true,
    OTEL_AWS_APPLICATION_SIGNALS_ENABLED=false,
    OTEL_EXPORTER_OTLP_LOGS_HEADERS='x-aws-log-group=/aws/bedrock-agentcore/agents/<function-name>/runtime-logs,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=agentcore',
    OTEL_RESOURCE_ATTRIBUTES='service.name=<function-name>',
    AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
  }"

A minimal Lambda handler that parses the function URL request and sets the session baggage:

import base64, json
from strands import Agent
from strands.models.bedrock import BedrockModel
from opentelemetry import baggage, context

_model = BedrockModel(model_id="global.anthropic.claude-sonnet-4-5-20250929-v1:0")


def lambda_handler(event, _context):
    # Lambda function URL payload format 2.0: JSON body is in event["body"], no httpMethod.
    raw = event.get("body")
    if raw is not None and event.get("isBase64Encoded"):
        raw = base64.b64decode(raw).decode()
    body = json.loads(raw) if isinstance(raw, str) else (raw or event)

    session_id = _session_id(event)
    token = context.attach(baggage.set_baggage("session.id", session_id))
    try:
        agent = Agent(model=_model, system_prompt="You are a helpful assistant. Be concise.")
        result = agent(body.get("prompt", "Hello"))
    finally:
        context.detach(token)
    return {"response": str(result), "sessionId": session_id}


def _session_id(event):
    return (event.get("headers") or {}).get("x-session-id") or "default"

For target-based routing, deploy this twice (control and treatment) with the change you are testing — for example, a different model_id or system prompt.

Step 2: Expose your agent over HTTP

The agent must be reachable at an HTTP endpoint the AgentCore gateway can call. For the Lambda example, create a function URL with IAM authentication for each function:

aws lambda create-function-url-config \
  --function-name cs-agent-control \
  --auth-type AWS_IAM \
  --region us-west-2

Repeat for the treatment function. (For a container or self-hosted agent, expose an HTTPS endpoint instead and skip to Step 3.)

Step 3: Register your agent as an AgentCore gateway target

Register each endpoint as an HTTP passthrough target on an AgentCore gateway. For the details of passthrough targets — including the protocolType, stickiness, and credential options — see HTTP passthrough targets.

To register a Lambda function URL as a target, set protocolType to CUSTOM and provide an iamCredentialProvider. The credential provider authorizes the gateway to sign each outbound request to the function URL as the lambda service:

For target-based routing, repeat for the treatment endpoint (name it customer-support-treatment). Poll each target with get-gateway-target until status is READY.

Step 4: Enable AgentCore gateway tracing

Enable tracing delivery on the AgentCore gateway so that it emits variant-attribution spans to aws/spans. To enable tracing from the console, open the gateway detail page and choose Log deliveries and tracing → Tracing → Enable. For more information, see Configure tracing delivery to CloudWatch.

Once traffic begins to flow, each gateway span carries the aws.agentcore.gateway.routing_experiment_arn and aws.agentcore.gateway.routing_experiment_variant_name (for example, C or T1) attributes, along with the traceId of the request. The online evaluation pipeline joins the gateway span to your agent’s spans by traceId, which is how each scored session is attributed to its variant. Without gateway tracing, sessions are still scored, but they cannot be attributed to a variant, and the A/B test does not produce per-variant results.

Step 5: Create and run the A/B test

Your agent is now a gateway target emitting telemetry. Two steps remain:

When you send traffic through the gateway, direct it to the control variant’s target. The gateway splits all traffic arriving at the control target across the control (C) and treatment (T1) targets according to your A/B test configuration.

Troubleshooting

A/B test shows no results after sending traffic