# Inference Gateway troubleshooting guide
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway"></a>

**Overview:** The HyperPod Inference Gateway routes traffic through three layers: the Body-Based Router (BBR), the gateway with `HTTPRoute`, and the Endpoint Picker (EPP). Misconfiguration at any layer can result in failed requests, traffic reaching the wrong model, or uneven load across model-serving pods. This section covers issues with the gateway, BBR, the gateway and `HTTPRoute`, `InferencePool`, and EPP, and any issues that arise from them.

## Diagnose gateway state
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-diagnose"></a>

Use the following commands to inspect the gateway and the resources it manages.

List every `InferenceGatewayConfig` resource across namespaces:

```
kubectl get inferencegatewayconfig -A
```

Show detailed status, per-scheduler rollout state, and condition messages for a specific gateway:

```
kubectl describe inferencegatewayconfig <name> -n <namespace>
```

Check the gateway controller and Body-Based Router pods:

```
kubectl get pods -n hyperpod-inference-system
```

Check the downstream routing resources generated by the controller:

```
kubectl get httproute,inferencepool,securitypolicy -A
```

Inspect `status.conditions` and each scheduler's `rolloutState` (`Pending`, `Progressing`, `Available`, or `Degraded`). Actionable failure reasons are on the corresponding condition message.

## Add-on install issues
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-addon-install"></a>

**Problem:** Gateway resources are missing, or the `GatewayClass` is not accepted after installing the HyperPod Inference Amazon EKS add-on.

**Symptoms and resolution:** `kubectl get gatewayclass inference-gateway` returns `NotFound`, or the resource shows `ACCEPTED=False`. This indicates that the add-on is not installed or that the installation did not complete. Reinstall or update the add-on:

```
aws eks update-addon --cluster-name $CLUSTER --region $REGION \
  --addon-name amazon-sagemaker-hyperpod-inference \
  --resolve-conflicts OVERWRITE
```

Then confirm the gateway controller is running:

```
kubectl rollout status deploy/inference-gateway-controller \
  -n hyperpod-inference-system --timeout=150s
```

## InferenceGatewayConfig not becoming Ready
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-config-not-ready"></a>

**Problem:** An `InferenceGatewayConfig` is created, but its `status.conditions` show `Accepted=False` or `Ready=False`, or the `kubectl apply` is rejected outright by validation.

**Symptoms and resolution:**
+ **`kubectl apply` fails with `bbr must be enabled when more than one scheduler is defined`.** The Body-Based Router is required whenever more than one scheduler is defined. Set `spec.bbr.enabled` to `true`.
+ **`kubectl apply` fails with `modelName must be unique across schedulers`.** Two schedulers declare the same `modelName`. Rename one so that every scheduler has a distinct `modelName`.
+ **`Accepted=False`, `Reason=InvalidLoraAdapters`.** A LoRA adapter name declared under `spec.schedulers[].loraAdapters` is a duplicate across schedulers, or collides with a scheduler's `modelName`. Inspect the condition message for the offending name:

  ```
  kubectl describe inferencegatewayconfig <name> -n <namespace>
  ```
+ **`Accepted=False`, `Reason=ResourceNamingViolation`.** The concatenated name `<config-name>-<scheduler-name>` exceeds the Kubernetes 63-character label limit. Shorten the config or scheduler name.
+ **`Ready=False`, `Reason=GatewayNotProgrammed`.** The gateway has not yet provisioned the load balancer. Inspect the parent Gateway:

  ```
  kubectl get gateway -n hyperpod-inference-system
  kubectl describe gateway <name> -n hyperpod-inference-system
  ```
+ **`AdmissionBlocked=True`, `Reason=WebhookDenied`.** A cluster admission webhook is rejecting the gateway pod. The condition message names the offending webhook. Remove or correct the webhook, then restart the gateway Deployment so that the pod is recreated immediately. The Deployment name is generated, so look it up first:

  ```
  kubectl -n hyperpod-inference-system get deploy \
    -l gateway.envoyproxy.io/owning-gateway-name=<gateway-name>
  ```

  Then restart it:

  ```
  kubectl -n hyperpod-inference-system rollout restart deploy/<gateway-deployment>
  ```

## Per-scheduler failures
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-per-scheduler"></a>

**Problem:** A specific scheduler's condition (`BackendsReady`, `LoraSupported`, or `PoolReady`) indicates that the scheduler is not fully ready.

**Symptoms and resolution:**
+ **`BackendsReady=False`, `Reason=NoModelPods` or `Reason=NoReadyModelPods`.** No pods match `spec.schedulers[].modelSelector`, or matching pods are not yet Ready. Compare labels on the model-serving pods against the scheduler's selector:

  ```
  kubectl get pods -n <namespace> --show-labels
  ```

  Deploy the model-serving pods and wait for them to become Ready before applying the `InferenceGatewayConfig`.
+ **`BackendsReady=False`, `Reason=InvalidModelSelector`.** The `matchLabels` or `matchExpressions` under `modelSelector` is malformed. Correct the selector in the config.
+ **`LoraSupported=False`, `Reason=ModelServerLoraDisabled`.** The model server backing this scheduler was not started with LoRA support enabled. Enable the equivalent flag on the model server (for example, `--enable-lora` for vLLM) and restart the model pods.
+ **`PoolReady=False`, `Reason=NotFound` or `Reason=NotAccepted`.** The `InferencePool` or its `HTTPRoute` has not yet been reconciled or accepted by the gateway. Inspect both:

  ```
  kubectl get inferencepool,httproute -n <namespace>
  ```

  If either is still missing several minutes after the config was applied, describe the parent Gateway to check for admission errors:

  ```
  kubectl describe gateway -n hyperpod-inference-system
  ```

## Scheduler rolloutState is Degraded
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-scheduler-degraded"></a>

**Problem:** The Endpoint Picker Deployment for a scheduler is stuck and does not become Available.

**Symptoms and resolution:** The `EPPReady` condition on the scheduler carries the actionable reason. Common causes include:
+ The container image cannot be pulled.
+ Pods are crash-looping.
+ The container has a configuration error, such as an invalid environment variable, volume mount, or secret reference.
+ The Deployment exceeded its progress deadline.

Use the following commands to identify the failing scheduler and inspect its Deployment:

```
# List the schedulers reporting Degraded
kubectl get inferencegatewayconfig <name> -n <namespace> \
  -o jsonpath='{range .status.schedulers[?(@.rolloutState=="Degraded")]}{.name}{"\n"}{end}'

# Describe the scheduler's Endpoint Picker Deployment for pod events and container errors
kubectl describe deploy -n <namespace> \
  -l inference.sagemaker.aws.amazon.com/scheduler=<scheduler-name>
```

## Stale endpoint after a model pod restart
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-stale-endpoint"></a>

**Problem:** After a model pod is deleted and a replacement pod becomes Ready, the Endpoint Picker continues to route to the deleted pod's IP address. Requests return HTTP 503 or connection refused, and the condition does not recover on its own.

**Resolution:** Add a readiness probe to the model pod so that Kubernetes marks the pod NotReady before its IP is removed from the pool, and only advertises the replacement pod once it is fully serving traffic. Set `port` to the scheduler's `targetPort` and `path` to your model server's health endpoint:

```
readinessProbe:
  httpGet:
    path: /health
    port: 8000
```

**Workaround:** If you cannot redeploy the model pod immediately, restart the scheduler's Endpoint Picker to force it to rebuild its endpoint list from the current pod set:

```
kubectl rollout restart deploy -n <namespace> \
  -l inference.sagemaker.aws.amazon.com/scheduler=<scheduler-name>
```

## JWT authentication returns 401 or 403
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-jwt-auth"></a>

**Problem:** `spec.auth.jwt` is configured and requests are rejected before reaching a model, or the gateway never becomes Ready after JWT authentication is enabled.

**Symptoms and resolution:**
+ **HTTP 401.** The token is missing, expired, malformed, or its `iss` claim does not match the configured provider. Confirm the client sends an `Authorization: Bearer <token>` header, and decode the JWT to compare its `iss` claim against `spec.auth.jwt.provider.issuer`.
+ **HTTP 403.** Signature validation failed, or the token's `aud` or `requiredClaims` do not match the provider configuration. Inspect the provider configuration and confirm the token's `aud` and every entry in `requiredClaims` match:

  ```
  kubectl get inferencegatewayconfig <name> -n <namespace> \
    -o jsonpath='{.spec.auth.jwt.provider}'
  ```
+ **Gateway never becomes Ready with JWT enabled.** A generated `SecurityPolicy` is not Accepted by the gateway. Inspect the SecurityPolicy resources for the failure reason:

  ```
  kubectl get securitypolicy -A
  kubectl describe securitypolicy <name> -n <namespace>
  ```

  A common cause is that `spec.auth.jwt.provider.remoteJWKS.uri` is unreachable from the gateway. Confirm the URI resolves and returns a valid JWKS document.

## Metrics missing from dashboards
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-metrics-missing"></a>

**Problem:** Endpoint Picker or Body-Based Router metrics do not appear in your monitoring dashboard.

**Symptoms and resolution:** Metrics collection is enabled by default, so the OpenTelemetry Collector sidecar is normally present. Confirm whether the sidecar is running on both pod types, and whether metrics were explicitly disabled.

Check the sidecar on Body-Based Router pods:

```
kubectl -n hyperpod-inference-system get pods \
  -o jsonpath='{.items[*].spec.containers[*].name}' | tr ' ' '\n' | grep otel
```

Check the sidecar on Endpoint Picker pods:

```
kubectl -n <namespace> get pods \
  -o jsonpath='{.items[*].spec.containers[*].name}' | tr ' ' '\n' | grep otel
```

Check whether metrics were explicitly disabled:

```
kubectl get inferencegatewayconfig <name> -n <namespace> \
  -o jsonpath='{.spec.observability.metrics.enabled}'
```

Empty output from the last command means the field is not set, and metrics are enabled. Only an explicit `false` disables the sidecar. If the value is `false`, set it to `true` or remove the field, and the controller injects the sidecar on the next reconciliation.

## Request failures
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-request-failures"></a>

**Problem:** The gateway is Ready, but inference requests fail.

**Symptoms and resolution:**
+ **HTTP 404 for a known model.** The `model` value in the request body does not exactly match any scheduler's `modelName`, or the requested model is served through a LoRA adapter that is not declared under `spec.schedulers[].loraAdapters`. If no scheduler matches the requested model and `spec.bbr.defaultBackend` is not set, the gateway returns 404. Verify the configured model names and adapter names:

  ```
  kubectl get inferencegatewayconfig <name> -n <namespace> \
    -o jsonpath='{.spec.schedulers[*].modelName}'
  
  kubectl get inferencegatewayconfig <name> -n <namespace> \
    -o jsonpath='{.spec.schedulers[*].loraAdapters}'
  ```
+ **Requests hang and then time out.** Model-serving pods are still loading model weights, or the `InferencePool` has no Ready endpoints. Wait for the model pods to become Ready before invoking the gateway endpoint. Use the scheduler's `modelSelector` labels from your `InferenceGatewayConfig` as the selector:

  ```
  kubectl get pods -n <namespace> -l <key>=<value>
  kubectl logs <pod> -n <namespace>
  ```

## Debugging endpoint selection
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-debug-scoring"></a>

**Problem:** Traffic skews to a small number of model pods, or a LoRA request is routed to a pod that does not host the adapter.

**Resolution:** Temporarily raise the Endpoint Picker's log verbosity to inspect its scoring decisions. Set `logLevel` on the scheduler:

```
spec:
  schedulers:
    - name: <scheduler-name>
      logLevel: 4
```

Log level meanings:
+ `1` - Request lifecycle events.
+ `2` - Default. Warnings and admission rejections.
+ `3` - Selected endpoint and per-scorer summaries.
+ `4` - Per-endpoint, per-scorer scores and weighted totals.
+ `5` - Protocol-level trace (verbose).

Inspect the Endpoint Picker logs:

```
kubectl logs -n <namespace> -l app=<scheduler-name>-epp -c epp --tail=200 -f
```

Return `logLevel` to its default when the investigation is complete to avoid excess log volume.

## Lifecycle and cleanup
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-lifecycle"></a>

**Problem:** Uninstalling or upgrading the HyperPod Inference Amazon EKS add-on leaves orphaned resources in the cluster, or blocks a subsequent install.

**Resolution:** Always delete every `InferenceGatewayConfig` resource before uninstalling or upgrading the add-on. Uninstalling the add-on while an `InferenceGatewayConfig` is still present removes the controller that owns the resource's finalizers, which leaves those resources stuck in `Terminating`.

```
kubectl delete inferencegatewayconfig --all -A
kubectl get inferencegatewayconfig -A
```

Confirm the second command returns no rows before continuing with the add-on operation.

After reinstalling the add-on, list the resources in the gateway namespace and remove anything that no longer maps to a live `InferenceGatewayConfig`:

```
kubectl get deploy,svc,httproute,inferencepool,gateway,configmap \
  -n hyperpod-inference-system
```

ACM certificates issued by the controller are not deleted by an add-on uninstall. To remove them, filter ACM certificates in the AWS Resource Groups Tagging API by the tag `CreatedBy=HyperPodInference` and delete the certificates you no longer need.

## Collect logs
<a name="sagemaker-hyperpod-model-deployment-ts-inference-gateway-collect-logs"></a>

Use the following commands to retrieve logs from each gateway component:

```
# Gateway controller
kubectl logs -n hyperpod-inference-system deploy/inference-gateway-controller

# Body-Based Router (deployment name is <gateway-name>-bbr)
kubectl logs -n hyperpod-inference-system deploy/<gateway-name>-bbr -c bbr

# Endpoint Picker for a specific scheduler
kubectl logs -n <namespace> -l app=<scheduler-name>-epp -c epp
```