Deploy models from local NVMe storage using kubectl

Set up inference capabilities on your Amazon SageMaker HyperPod clusters. For more information, see Setting up your HyperPod clusters for model deployment.

Installed kubectl utility and configured jq in your terminal.

Pre-populated model weights on the local NVMe storage of your target nodes (see Preload model weights to NVMe for instructions).

A volume named after your modelVolumeMount.name (typically model-weights) — an Amazon S3 CSI fuse mount containing your model

model-weights-copy — a RAM-backed emptyDir where the worker reads from

If the model exists on NVMe — copies from NVMe to RAM (model-weights-copy), skipping the network entirely.

If the model is missing — falls back to copying from the Amazon S3 mount (model-weights) to RAM (model-weights-copy). Optionally copies to NVMe so subsequent pod startups on the same node use the fast local path.

A volume named after your modelVolumeMount.name (typically model-weights) — an Amazon S3 CSI fuse mount containing your model

model-weights-copy — a RAM-backed emptyDir where the worker actually reads from

Update the operator Helm values to enable the feature. If you deployed the operator via Helm, upgrade with the flag:

helm upgrade hyperpod-inference-operator <CHART_PATH> \
  --set enableCustomServiceAccounts=true \
  --reuse-values

If you deployed the operator as an Amazon EKS add-on, update the add-on configuration to include enableCustomServiceAccounts: true in the advanced configuration settings.

Verify the operator pod has the environment variable set:

kubectl get deployment hyperpod-inference-operator-controller-manager \
  -n hyperpod-inference-system \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | jq '.[] | select(.name=="ENABLE_CUSTOM_SERVICE_ACCOUNTS")'

You should see:

{
  "name": "ENABLE_CUSTOM_SERVICE_ACCOUNTS",
  "value": "true"
}

The ServiceAccount does not exist in the namespace — serviceAccountName "X" not found in namespace "Y"

The ServiceAccount is missing the required label — serviceAccountName "X" is not labeled as user-assignable (requires label sagemaker.amazonaws.com/user-assignable=true)

The ServiceAccount is the operator's system ServiceAccount — serviceAccountName must not reference the operator's service account

Create a dedicated ServiceAccount for each inference workload. Do not reuse ServiceAccounts across unrelated workloads.

Bind only the minimum RBAC permissions required. For example, if your init container only needs to read from Amazon S3, the ServiceAccount should not have permissions to list or modify Kubernetes resources.

# Example: minimal Role for an inference workload that only needs S3 access via IRSA
# No Kubernetes API permissions needed — IRSA provides AWS credentials directly
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-inference-sa
  namespace: my-namespace
  labels:
    sagemaker.amazonaws.com/user-assignable: "true"
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<SCOPED_ROLE_NAME>

Avoid granting cluster-wide permissions (ClusterRoleBindings) to ServiceAccounts used by inference pods.

When annotating a ServiceAccount with eks.amazonaws.com/role-arn, ensure the IAM role follows least-privilege principles.

Scope Amazon S3 permissions to the specific bucket and prefix containing your model weights.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::<YOUR_BUCKET>",
      "arn:aws:s3:::<YOUR_BUCKET>/<YOUR_MODEL_PREFIX>/*"
    ]
  }]
}

Do not use broad managed policies such as AmazonS3FullAccess in production. Use AmazonS3ReadOnlyAccess or a custom policy scoped to your model bucket.

Only cluster administrators should add or remove the sagemaker.amazonaws.com/user-assignable=true label. Use Kubernetes RBAC to restrict who can modify ServiceAccount labels in your namespace.

Review the RBAC roles and IAM permissions associated with a ServiceAccount before labeling it as user-assignable.

Periodically audit which ServiceAccounts carry the user-assignable label.

kubectl get serviceaccounts -n <NAMESPACE> -l sagemaker.amazonaws.com/user-assignable=true

Ensure non-admin roles do not include patch, update, or create verbs on ServiceAccount resources. The operator validates the user-assignable label at deployment time, but does not prevent unauthorized users from adding the label to a ServiceAccount. Restricting who can modify ServiceAccounts via RBAC is the primary control for protecting this label. Non-admin users should only have get and list access:

# Example: RBAC Role for non-admin users — read-only access to ServiceAccounts
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sa-read-only
  namespace: <NAMESPACE>
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get", "list"]

Do not use reserved volume names. The operator manages several internal volumes (see Reserved volume names). Using any of these names in kubernetes.volumes results in a KubernetesVolumeValidationFailed condition.

Volume names must match. The operator derives the volume name from modelVolumeMount.name. When using modelSourceType: kubernetesVolume, kubernetes.volumes must contain a volume with that same name.

Mount volumes to the correct location in your initContainer. Ensure that any volume you create is mounted to the correct path in your initContainer.

No custom service account is needed for S3/FSx. If you are unable to create custom service accounts or prefer not to, you can use modelSourceType: s3 or fsx. The operator provisions S3/FSx volumes automatically. You can still add custom initContainers and override volumes on top of the operator-managed storage.

IRSA credentials are injected into all containers. When you set kubernetes.serviceAccountName to a service account with an IRSA annotation, Amazon EKS injects AWS credentials (aws-iam-token volume, AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE) into all containers, including your custom initContainers.

Do not set modelLocation when using kubernetesVolume. The volume path is controlled by kubernetes.volumes. Setting modelLocation when modelSourceType is kubernetesVolume results in a validation error.

Understand how model-weights vs model-weights-copy works with prefetch. When prefetchEnabled: true, the operator creates two model-related volumes:

While you reference model-weights in your config, when prefetchEnabled: true, it is actually model-weights-copy that gets mounted at /opt/ml/model in the worker container. When using a custom initContainer, ensure that you copy the data into the volume called model-weights-copy — that is where the worker expects to find it. When prefetchEnabled: false, there is only one volume (named after your modelVolumeMount.name) and it is mounted directly at /opt/ml/model.

Check the InferenceEndpointConfig status to see the high-level deployment state and any configuration issues.

kubectl describe InferenceEndpointConfig <ENDPOINT_NAME> -n <NAMESPACE>

Check the Kubernetes deployment status.

kubectl describe deployment <ENDPOINT_NAME> -n <NAMESPACE>

Check the status of all Kubernetes objects in your namespace.

kubectl get pods,svc,deployment,InferenceEndpointConfig,sagemakerendpointregistration -n <NAMESPACE>

Check initContainer logs if the model loading step fails.

kubectl logs <POD_NAME> -c smart-loader -n <NAMESPACE>

If the deployment fails with "not found in namespace", verify the ServiceAccount exists:

kubectl get serviceaccount <name> -n <namespace>

If the deployment fails with "not labeled as user-assignable", ask your cluster administrator to add the required label:

kubectl label serviceaccount <name> sagemaker.amazonaws.com/user-assignable=true -n <namespace>

If the deployment fails with "must not reference the operator's service account", create a separate ServiceAccount for your workload. You cannot use the HyperPod inference operator's own ServiceAccount.