Inference Gateway for Amazon SageMaker HyperPod Inference

The Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, large language model (LLM)-aware routing and orchestration layer for HyperPod clusters on Amazon EKS. It inspects inference requests, reads the model name from each request, and selects a model-serving pod based on GPU load, so that traffic for multiple models is routed efficiently through a single gateway endpoint.

The gateway routes each request through three layers. The Body-Based Router (BBR), a shared component, reads the model field from the request body, resolves a LoRA adapter name to its base model, and sets the X-Gateway-Model-Name and X-Gateway-Base-Model-Name headers. The gateway then matches those headers to an HTTPRoute and forwards the request to the InferencePool for the requested model. Within that pool, the Endpoint Picker (EPP/scheduler) chooses a model-serving pod by scoring candidates on model-server metrics such as queue depth and KV cache utilization, together with prefix-cache and LoRA adapter affinity. Each entry that you define under spec.schedulers runs its own Endpoint Picker, so this topic uses the terms Endpoint Picker, EPP, and scheduler interchangeably.

The Inference Gateway builds on the HyperPod Inference Operator instead of replacing it. While the operator continues to orchestrate model deployment, the gateway introduces an LLM-aware routing layer in front of the model-serving pods. The gateway does not depend on a specific model server or orchestration layer, and is delivered through the HyperPod Inference Amazon EKS add-on.

You define a gateway with the InferenceGatewayConfig custom resource. A single InferenceGatewayConfig describes one gateway: the shared Body-Based Router, TLS termination, request authentication, and one scheduler for each model the gateway serves. For the full schema, see InferenceGatewayConfig CRD reference.

Prerequisites and deployment

The Inference Gateway is delivered as part of the HyperPod Inference Amazon EKS add-on. Installing the add-on makes the gateway available, so no separate installation is required. You then turn on gateway routing for a model through the spec.inferenceGateway.enabled field on the model's InferenceEndpointConfig resource. For the version in which a capability was introduced, see Amazon SageMaker HyperPod Inference release notes.

Before you route traffic through the gateway, verify the following:

Set up the certificate issuer IAM role

When the gateway auto-issues a TLS certificate, cert-manager generates the certificate in the cluster and the gateway controller imports it into ACM. Because the import is an AWS API call, the controller needs AWS credentials. Provide them by creating an IAM role that the inference-gateway-controller service account in the hyperpod-inference-system namespace assumes through IAM roles for service accounts (IRSA).

Set the following environment variables and retrieve the OIDC issuer for your cluster.

export CLUSTER=EKS_CLUSTER_NAME
export REGION=REGION
export ACCOUNT=AWS_ACCOUNT_ID
export ROLE_NAME=CERT_ISSUER_ROLE_NAME

export OIDC_ID=$(aws eks describe-cluster --name $CLUSTER --region $REGION \
  --query 'cluster.identity.oidc.issuer' --output text | sed 's|https://||')

Create a trust policy that allows the gateway controller service account to assume the role.

cat > trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${ACCOUNT}:oidc-provider/${OIDC_ID}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_ID}:sub": "system:serviceaccount:hyperpod-inference-system:inference-gateway-controller",
        "${OIDC_ID}:aud": "sts.amazonaws.com"
      }
    }
  }]
}
EOF

Create the role and attach the AmazonSageMakerHyperPodInferenceGatewayAccess managed policy. The policy grants the ACM permissions that the controller needs to import, tag, describe, and delete the certificates it creates.

aws iam create-role --role-name $ROLE_NAME --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name $ROLE_NAME --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerHyperPodInferenceGatewayAccess

Provide this role ARN as inferenceGateway.serviceAccount.roleArn when you install the add-on.

Install the add-on

The HyperPod Inference add-on installs both the Inference Operator and the Inference Gateway. Install it with the following command. For inferenceGateway.serviceAccount.roleArn, use the certificate issuer role that you created in the preceding section. Replace the remaining placeholder values with the roles and bucket for your account.

aws eks create-addon \
  --cluster-name $CLUSTER --region $REGION \
  --addon-name amazon-sagemaker-hyperpod-inference \
  --addon-version v2.0.0-eksbuild.2 \
  --configuration-values '{
    "executionRoleArn": "arn:aws:iam::<ACCOUNT>:role/<EXEC_ROLE>",
    "tlsCertificateS3Bucket": "<TLS_BUCKET>",
    "inferenceOperator": { "enabled": true },
    "inferenceGateway": {
      "enabled": true,
      "serviceAccount": { "roleArn": "arn:aws:iam::<ACCOUNT>:role/<CERT_ISSUER_ROLE_NAME>" }
    },
    "keda": { "enabled": true, "auth": { "aws": { "irsa": { "enabled": true, "roleArn": "arn:aws:iam::<ACCOUNT>:role/<KEDA_IRSA_ROLE>" } } } },
    "alb": { "enabled": true, "serviceAccount": { "create": true, "roleArn": "arn:aws:iam::<ACCOUNT>:role/<ALB_IRSA_ROLE>" } },
    "jumpstartGatedModelDownloadRoleArn": "arn:aws:iam::<ACCOUNT>:role/<JUMPSTART_ROLE>"
  }'

If the add-on is already installed on the cluster, replace create-addon with update-addon and add --resolve-conflicts OVERWRITE. The rest of the command is unchanged. OVERWRITE applies the configuration values in the command over the existing add-on configuration.

Confirm that the gateway controller is running and that the gateway resources are registered.

kubectl rollout status deploy/inference-gateway-controller \
  -n hyperpod-inference-system --timeout=150s

kubectl get crd inferencegatewayconfigs.inference.sagemaker.aws.amazon.com

kubectl get gatewayclass inference-gateway

Confirm that the add-on itself is active and reports no health issues.

aws eks describe-addon \
  --cluster-name $CLUSTER --region $REGION \
  --addon-name amazon-sagemaker-hyperpod-inference \
  --query 'addon.{version:addonVersion,status:status,health:health.issues}'

Integration with the HyperPod Inference Operator

The HyperPod Inference Operator and the Inference Gateway have distinct responsibilities. The operator owns model deployment, orchestration, and the wiring that attaches a model to the gateway. The gateway owns request routing and generates the routing resources for each scheduler. For more information about deploying models with the operator, see Deploying models on Amazon SageMaker HyperPod.

You attach a model to the gateway through the operator, on the model's InferenceEndpointConfig (or JumpStartModel) resource, using the spec.inferenceGateway field:

The following snippet shows the inferenceGateway opt-in on an InferenceEndpointConfig resource.

apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
  name: my-model
spec:
  # ... model deployment fields ...
  inferenceGateway:
    enabled: true
    name: my-gateway        # Optional. When empty, the operator generates inf-igw-<uuid>.

The operator mirrors the attachment state on the model resource under status.inferenceGateway, which reports a State of Pending, Ready, or Failed, and the Name of the attached InferenceGatewayConfig. You cannot set inferenceGateway.enabled together with intelligentRoutingSpec.enabled on the same model; these fields are mutually exclusive.

When the gateway is enabled for a model, the operator creates or updates a single InferenceGatewayConfig resource and merges an entry for the model into its spec.schedulers list. The operator sets the scheduler name, modelName, targetPort, and modelSelector, and defaults scheduler to llm-d when it first creates the entry. On later reconciliations, the operator preserves customer-provided values such as scheduler, weights, and loraAdapters. The Body-Based Router is enabled automatically when more than one scheduler is present.

The operator does not delete the InferenceGatewayConfig resource or the downstream routing resources. When you disable the gateway for a model or delete the model, the operator removes only that model's scheduler entry. The gateway controller owns cleanup of the routing resources.

You can author an InferenceGatewayConfig in two ways, and the two paths coexist on the same resource:

The gateway components and the InferenceGatewayConfig CRD must be installed through the HyperPod Inference Amazon EKS add-on before a model resource with the gateway enabled can be reconciled. For operator add-on installation, see Installing the Inference Operator with EKS add-on. For deploying the model-serving pods that a scheduler routes to, see Deploy foundation models and custom fine-tuned models.

InferenceGatewayConfig CRD reference

You configure the Inference Gateway with a single InferenceGatewayConfig custom resource. The resource is a singleton for a given gateway: it holds the shared Body-Based Router configuration and a list of per-model schedulers. The controller generates the underlying InferencePool, Endpoint Picker, HTTPRoute, and EnvoyExtensionPolicy resources for each scheduler; you author only the InferenceGatewayConfig resource.

spec fields

The spec field of the InferenceGatewayConfig resource contains the shared Body-Based Router configuration, TLS settings, the required list of schedulers, and optional observability and pod-default settings.

SchedulerSpec fields

Each entry in spec.schedulers configures routing and endpoint selection for one model. A scheduler names the generated InferencePool, Endpoint Picker, HTTPRoute, and EnvoyExtensionPolicy resources.

Validation rules

The InferenceGatewayConfig resource enforces the following validation rules. A resource that violates any of these rules is rejected.

status fields

The controller reports the observed state of the gateway in the status subresource.

Kubernetes RBAC permissions

The Inference Gateway runs as a single controller under the Kubernetes service account inference-gateway-controller in the hyperpod-inference-system namespace. The Body-Based Router, the per-scheduler Endpoint Pickers, the Gateway controller, and the InferenceGatewayConfig reconciler all run under this one controller. Its cluster-scoped permissions are granted by a ClusterRole named sagemaker-inference-gateway-controller-supplement and a matching ClusterRoleBinding. Together they supplement the service account and permissions that the bundled Gateway chart already provides. These permissions are scoped and least-privilege, and the controller does not run as cluster administrator.

The following table lists the controller's cluster-scoped permissions, grouped by purpose. In this table, full access means the create, get, list, watch, update, patch, and delete verbs.

The controller also creates the following namespaced roles at runtime, depending on your gateway configuration:

Some of these permissions apply only when the corresponding feature is enabled.

Observability

The Inference Gateway emits Prometheus metrics from every gateway component. Both the Body-Based Router and each Endpoint Picker expose a standard Prometheus /metrics endpoint on their pods. Body-Based Router pods run in the hyperpod-inference-system namespace; Endpoint Picker pods run in the same namespace as the InferenceGatewayConfig. Metrics cover per-model request counters and durations, scheduling latency and per-plugin execution time inside the Endpoint Picker, and aggregated pool metrics such as average KV cache utilization and queue depth. Model-server pod metrics (for example, from vLLM or SGLang) are emitted by the model server itself; the Endpoint Picker scrapes them to score candidate pods.

When spec.observability.metrics.enabled is true (the default), the gateway controller injects an OpenTelemetry Collector sidecar into every Body-Based Router and Endpoint Picker pod. The sidecar forwards these metrics to the HyperPod inference observability stack, where the built-in Grafana dashboards surface them alongside model-server and cluster metrics. For setup and dashboard details, see Implementing inference observability on HyperPod clusters. Set the field to false to skip sidecar injection. For the field reference, see observability under spec fields.

To view the gateway metrics in Amazon Managed Grafana, open the Inference Dashboards folder and select the Inference Gateway dashboard. The dashboard reports gateway-wide availability, request rate, and end-to-end latency, per-scheduler throughput, latency, and errors for each model, and the rate at which the Body-Based Router resolves model names from request bodies.

Examples

The following examples show common InferenceGatewayConfig configurations and how to invoke the gateway.

Minimal multi-model configuration

This example routes to an llm-d scheduler with auto-issued TLS. Because tls is set to an empty object, the gateway auto-issues a certificate and imports it into ACM.

apiVersion: inference.sagemaker.aws.amazon.com/v1alpha1
kind: InferenceGatewayConfig
metadata:
  name: inference-gateway-demo
  namespace: inference-gateway
spec:
  bbr:
    enabled: true
  tls: {}                       # Auto-issue a certificate via cert-manager and import to ACM.
  schedulers:
    - name: llama
      modelName: "meta-llama/Llama-3.2-1B-Instruct"
      modelSelector:
        matchLabels:
          app: vllm-llama
      targetPort: 8000
      scheduler: llm-d

SGLang scheduler with explicit weights

This example uses the epp scheduler type with the sglang engine and sets explicit Endpoint Picker scoring weights.

spec:
  bbr:
    enabled: true
  schedulers:
    - name: qwen7b
      modelName: "Qwen/Qwen2.5-7B-Instruct"
      modelSelector:
        matchLabels:
          app: sglang-qwen7b
      targetPort: 8000
      scheduler: epp
      engineType: sglang
      logLevel: 4
      weights:
        kvCache: 2
        prefix: 3
        runningRequests: 2

LoRA adapter ConfigMap

The Body-Based Router discovers LoRA adapters and their base model from a ConfigMap labeled for BBR management. The router uses this mapping to resolve an adapter name in the request body to its base model.

apiVersion: v1
kind: ConfigMap
metadata:
  name: deepseek-adapters
  labels:
    inference.networking.k8s.io/bbr-managed: "true"
data:
  baseModel: deepseek/vllm-deepseek-r1
  adapters: |
    - ski-resorts
    - movie-critique

Invoke the gateway

The gateway serves an OpenAI-compatible inference endpoint. This is the runtime invocation contract for sending inference requests through the gateway; it is not an AWS API operation. Send requests to the gateway endpoint with the model field set to the modelName of the target scheduler. The Body-Based Router reads this field to route the request.

curl https://your-gateway-endpoint/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Hello"}
    ]
  }'