Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

Accelerate model loading on Amazon EKS

When you deploy large language models (LLMs) on Amazon EKS, model loading time directly affects how quickly Pods can start serving inference requests. This is especially true during scale up events, when new Pods or nodes must load the model before handling traffic. Model startup has two phases that you can improve with tuning and compile artifact caching:

This topic shows how you can optimize Run:ai Model Streamer performance and torch.compile caching to reduce both phases of the model loading process. For the full procedure of deploying vLLM on Amazon EKS for inference, see Load & Serve Models on Amazon EKS.

Optimize the S3 network path on EKS Auto Mode

If you are running on EKS Auto Mode with GPU nodes in private subnets, we recommend using a Gateway VPC endpoint for S3 to optimize the network path between your nodes and S3. With the Gateway VPC endpoint, traffic to S3 stays on the AWS network and bypasses the NAT Gateway entirely so there is no shared bandwidth ceiling and no per-GB NAT data processing charge. Without the Gateway VPC endpoint, when the traffic traverses through a NAT Gateway, the NAT Gateway becomes a shared bottleneck during scale up when multiple nodes pull the model at the same time.

EKS Auto Mode commonly places nodes in private subnets, so traffic to S3 flows through a NAT Gateway by default. A NAT Gateway provides up to 100 Gbps of bandwidth and 55,000 simultaneous connections per destination. However, that bandwidth is shared across every node in the private subnet. During a scale up event, several nodes downloading the full model at once contend for the same NAT Gateway bandwidth, which can slow model weights loading on all of them.

Run:ai Model Streamer performance tuning

Inference engines like vLLM and SGLang use Run:ai Model Streamer as an alternative mechanism to load weights during inference startup.

By default, Run:ai Model Streamer uses conservative concurrency settings when downloading model weight files from S3. Increasing the download concurrency and chunk size reduces model weights loading time by downloading more data in parallel.

Calculate optimal concurrency

Calculate the optimal concurrency value as:

concurrency = ceil(total_model_size_gb / chunk_size_gb)

Replace the concurrency value you use based on your model size and chunk size. See the following table for a few examples. For example, with a 67 GB model and a 4 GB chunk size: ceil(67 / 4) = 17.

Apply the configuration

Add the following arguments and environment variables to your inference container spec:

containers:
- name: vllm-inference
  image: vllm/vllm-openai:v0.21.0
  command:
    - python3
    - -m
    - vllm.entrypoints.openai.api_server
  args:
  # ... your existing args ...
  - --model=s3://<MODEL_PATH>
  - --tensor-parallel-size=2
  - --load-format=runai_streamer
  - --model-loader-extra-config={"concurrency":17,"distributed":true}
  env:
  - name: RUNAI_STREAMER_CHUNK_BYTESIZE
    value: "4294967296"
  - name: RUNAI_STREAMER_S3_REQUEST_TIMEOUT_MS
    value: "3000"
  - name: RUNAI_STREAMER_S3_LOW_SPEED_LIMIT
    value: "1048576"
         

Optimize torch.compile cold start

The torch.compile step in the inference serving process traces a model’s computation graph (the sequence of math operations) and compiles it into optimized fused CUDA/Triton kernels. It does not compile the model weights — only the operations that transform them.

Inference engines use torch.compile because it provides significant throughput improvements automatically, without custom kernel engineering per model architecture:

The following table shows common inference serving engines that use torch.compile.

The torch.compile cold start problem

The tradeoff of torch.compile is that the first inference Pod must compile before it can serve requests. This compilation can take as long as several minutes depending on model size. The compiled artifacts are small (~15 MB for a 60 GB model) but significantly increase the cold start time. Because the artifacts are small and deterministic for a given configuration, you can cache and reuse them to eliminate the cold start penalty on subsequent Pod and node start.

The artifacts consist of:

The --enforce-eager trade-off

vLLM enables torch.compile and CUDA graph capture by default. The --enforce-eager flag turns both off and runs the model in eager mode, where each operation executes immediately through the Python interpreter. Because eager mode skips both compilation and graph capture, some quick-start guides — including the base deployment in Load & Serve Models on Amazon EKS — use --enforce-eager to start Pods faster.

--enforce-eager is a valid choice for debugging, memory-constrained deployments, or model architectures that do not compile cleanly, and you can use it in production for those reasons. Although it can mitigate the torch.compile cold start penalty, we recommend other approaches, such as those detailed in subsequent sections on this page, to maintain runtime performance in production.

Understand the startup-time versus runtime-performance trade-off of --enforce-eager before deploying in production.

Cache torch.compile artifacts on the same node

This technique applies when you use an inference engine that supports torch.compile, such as vLLM (enabled by default) or SGLang (enabled via --enable-torch-compile). It works only when torch.compile is active, that is, when --enforce-eager is not set.

When an inference engine compiles the model’s computation graph on first startup, you can cache the resulting optimized kernels on the node’s local storage. Subsequent Pods on the same node reuse the cached artifacts and skip the compilation step entirely, which can significantly reduce startup time.

Add cache environment variables

Add the following environment variables to your inference container spec to direct torch.compile and Triton cache to a persistent host path. We recommend using the node’s local NVMe instance store and not the root Amazon Elastic Block Store (Amazon EBS) volume for the hostPath. For examples, see the following section.

containers:
- name: vllm-inference
  env:
  # torch.compile cache
  - name: XDG_CACHE_HOME
    value: "/compile-cache"
  - name: TORCHINDUCTOR_CACHE_DIR
    value: "/compile-cache/inductor"
  - name: TRITON_CACHE_DIR
    value: "/compile-cache/triton"
  volumeMounts:
  - name: compile-cache
    mountPath: /compile-cache
volumes:
- name: compile-cache
  hostPath:
    path: /mnt/k8s-disks/0/compile-cache
    type: DirectoryOrCreate

Set the cache path to NVMe instance store

Compiled artifacts and streamed weights benefit from fast local storage. On GPU instances with NVMe instance store (such as G-family and P-family instances), the instance store delivers roughly 30 GB/s, compared to roughly 1 GB/s for the root Amazon EBS volume. Direct the cache hostPath to the NVMe mount point to optimize throughput.

The NVMe mount point and how you enable it differ between EKS Auto Mode and self-managed nodes:

For EKS Auto Mode, set the hostPath to /mnt/.ephemeral/compile-cache in your container spec:

volumes:
- name: compile-cache
  hostPath:
    path: /mnt/.ephemeral/compile-cache
    type: DirectoryOrCreate

For self-managed Karpenter, set the hostPath to /mnt/k8s-disks/0/compile-cache in your container spec:

volumes:
- name: compile-cache
  hostPath:
    path: /mnt/k8s-disks/0/compile-cache
    type: DirectoryOrCreate

Sample results

The following table shows the improvement with same-node caching for different model sizes:

Pre-warm torch.compile cache on new nodes

This technique applies when you are running multi-node inference with homogeneous GPUs, tensor parallelism, models, and PyTorch versions across nodes. The technique in Cache torch.compile artifacts on the same node focuses on the single node case, but new nodes added during scale up events start with an empty torch.compile cache. To further reduce cold start time on newly initialized nodes, you can implement a caching mechanism that stores the compiled torch.compile artifacts in S3 and pre-downloads them to new nodes when they join the cluster.

The general approach is:

For example, you can implement a DaemonSet that runs on GPU nodes that packages and uploads the torch.compile cache to S3. It can also sync that cache to local node storage before inference Pods are scheduled on new nodes.

Considerations for cross-node caching

When you cache torch.compile artifacts across nodes, the compiled kernels are valid only when these parameters match between the node that generated the cache and the node consuming it. Your caching mechanism must account for all of these. A mismatch on any parameter produces an invalid cache that forces recompilation or causes runtime errors. Your caching tool must differentiate artifacts by these parameters, for example, by incorporating them into the S3 object key or cache directory structure.

Sample results

In directional testing with cross-node cache pre-warming (Qwen3-6-35B-A3B, 67 GB, 2x GPU with TP=2 on p5.48xlarge), the first Pod on newly scaled nodes achieved the same startup time as subsequent Pods on an already-warm node:

Deployment example

The following example combines Run:ai Model Streamer performance tuning and a torch.compile cache into a single vLLM Deployment manifest. Replace the placeholder values with your own configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-inference
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-inference
  template:
    metadata:
      labels:
        app: vllm-inference
    spec:
      serviceAccountName: <SERVICE_ACCOUNT_NAME>
      nodeSelector:
        karpenter.sh/nodepool: <GPU_NODEPOOL>
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.21.0
          command:
            - python3
            - -m
            - vllm.entrypoints.openai.api_server
          args:
            - --model=s3://<BUCKET_NAME>/<MODEL_PATH>
            - --load-format=runai_streamer
            - --model-loader-extra-config={"concurrency":17,"distributed":true}
            - --tensor-parallel-size=2
            - --max-model-len=8192
            - --host=0.0.0.0
            - --port=8000
          ports:
            - containerPort: 8000
              name: http
          env:
            # Run:ai streamer tuning
            - name: RUNAI_STREAMER_CHUNK_BYTESIZE
              value: "4294967296"
            - name: RUNAI_STREAMER_S3_REQUEST_TIMEOUT_MS
              value: "3000"
            - name: RUNAI_STREAMER_S3_LOW_SPEED_LIMIT
              value: "1048576"
            # torch.compile cache
            - name: XDG_CACHE_HOME
              value: "/compile-cache"
            - name: TORCHINDUCTOR_CACHE_DIR
              value: "/compile-cache/inductor"
            - name: TRITON_CACHE_DIR
              value: "/compile-cache/triton"
          resources:
            requests:
              cpu: "12"
              memory: 80Gi
              nvidia.com/gpu: "2"
            limits:
              nvidia.com/gpu: "2"
          volumeMounts:
            - name: compile-cache
              mountPath: /compile-cache
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            failureThreshold: 60
            initialDelaySeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 5
            timeoutSeconds: 3
      volumes:
        - name: compile-cache
          hostPath:
            path: /mnt/k8s-disks/0/compile-cache
            type: DirectoryOrCreate