Getting started with SageMaker HyperPod using the AWS CLI

The following tutorial demonstrates how to create a new SageMaker HyperPod cluster with Slurm through the AWS CLI commands for SageMaker HyperPod. By the end of this tutorial, you will have a working Slurm cluster with a controller node, a login node, and a compute worker group, ready to schedule and run ML workloads. The tutorial covers Slurm topology setup, node lifecycle configuration options, optional FSx shared storage, and how to connect to your cluster.

Before starting, ensure you have completed the Prerequisites for using SageMaker HyperPod (VPC, quotas, FSx) and AWS Identity and Access Management for SageMaker HyperPod (IAM roles, execution role with AmazonSageMakerClusterInstanceRolePolicy).

Key concepts

This section covers the core configuration concepts for creating a SageMaker HyperPod Slurm cluster. Understanding these concepts will help you make informed choices when configuring your cluster, but if you want to get started immediately you can jump directly to Create your cluster and refer back here as needed.

When creating a Slurm-orchestrated cluster, you make two independent configuration choices:

For Slurm topology, this tutorial uses the API-driven configuration approach, where you define node roles and partitions directly in the CreateCluster request using SlurmConfig on each instance group and Orchestrator.Slurm at the cluster level. This is the recommended approach for new clusters. It provides a single source of truth, built-in validation, and partition configuration drift detection with no additional files to manage. Alternatively, you can use a legacy provisioning_parameters.json file stored in Amazon S3 for backward compatibility with existing clusters. For details on the legacy approach, see SageMaker HyperPod Slurm configuration.

For node lifecycle configuration, SageMaker HyperPod supports three options. In the simplest case you omit LifeCycleConfig entirely and HyperPod automatically configures nodes using AMI-based configuration, setting up Slurm and essential packages such as Docker, Enroot, and Pyxis for running ML workloads, with no scripts or Amazon S3 bucket required. If you need customizations on top of the AMI-based configuration, you can provide an extension script via OnInitComplete that runs after the configuration completes. For full control over the entire provisioning sequence, the OnCreate path lets your scripts own everything, including when Slurm starts.

For ML workloads you will typically also need a shared high-performance filesystem for training data, checkpoints, and shared libraries. SageMaker HyperPod supports Amazon FSx for Lustre and FSx for OpenZFS, configured per instance group via InstanceStorageConfigs. FSx configuration is optional for cluster creation but recommended for production workloads.

Configuring Slurm topology via the API

All examples in this tutorial use API-driven Slurm topology configuration, where you define the Slurm cluster structure directly in the CreateCluster API request rather than through a separate configuration file.

A Slurm cluster requires at least a controller node (which runs the slurmctld daemon and coordinates job scheduling) and one or more compute nodes (which execute jobs). Optionally, you can add a login node to provide users with a dedicated access point for submitting and managing jobs without logging into the controller directly. In the API request, you assign each instance group its Slurm role using SlurmConfig, specifying whether the group serves as the controller, login, or compute node. Compute groups are also mapped to one or more Slurm partitions, which act as logical queues that organize how jobs are scheduled across different sets of nodes.

At the cluster level, Orchestrator.Slurm controls how HyperPod manages the partition configuration in slurm.conf. You choose a strategy that determines whether HyperPod is the single source of truth for partition topology, whether it overwrites manual changes, or whether it merges API-defined configuration with any manual edits you've made. Here is a reference for the fields used.

SlurmConfig (per instance group):

"SlurmConfig": {
    "NodeType": "Controller | Login | Compute",
    "PartitionNames": ["partition-name"]
}

Orchestrator.Slurm (cluster level):

"Orchestrator": {
    "Slurm": {
        "SlurmConfigStrategy": "Managed | Overwrite | Merge"
    }
}

SlurmConfigStrategy determines how HyperPod manages the partition-to-node mappings in slurm.conf on the controller node. When you create or update a cluster, HyperPod writes the partition configuration to slurm.conf based on the SlurmConfig you defined on each instance group, mapping compute instance groups to their assigned partitions and registering controller and login nodes with the appropriate Slurm roles.

The strategy you choose controls what happens when the partition configuration in slurm.conf has been modified outside of the API, for example, by an administrator editing the file directly on the controller node. With Managed, HyperPod treats the API as the single source of truth and will detect and block updates if slurm.conf on disk has drifted. With Overwrite, HyperPod forces the API-defined configuration onto the controller, discarding any manual edits to slurm.conf. This is useful for recovering from an unintended change. With Merge, HyperPod preserves manual edits to slurm.conf and merges them with the API configuration, giving advanced users the flexibility to maintain custom slurm.conf settings alongside API-managed partitions.

Node lifecycle configuration options

When creating a SageMaker HyperPod Slurm cluster, you choose how each instance group's nodes are provisioned by configuring the LifeCycleConfig block in the CreateCluster request. SageMaker HyperPod supports three node lifecycle configuration options, each offering a different level of control over the provisioning process.

With AMI-based configuration only, you omit LifeCycleConfig entirely. HyperPod automatically configures nodes using AMI-based configuration, setting up Slurm, installing essential packages, and starting all required services. This is the simplest path and requires no Amazon S3 bucket or scripts.

With the Extension option, you specify OnInitComplete in LifeCycleConfig along with a SourceS3Uri pointing to your extension script in Amazon S3. HyperPod runs the full AMI-based configuration first, then executes your script. This lets you add customizations, such as monitoring agents, LDAP integration, or additional storage mounts, without managing the baseline provisioning.

With the Custom option, you specify OnCreate in LifeCycleConfig along with a SourceS3Uri pointing to your full lifecycle script set in Amazon S3. HyperPod does not run AMI-based configuration and does not start Slurm. Your scripts own the entire provisioning sequence. This gives you complete control over what software is installed, how it is configured, and when Slurm starts.

FSx and VPC configuration

For ML workloads, a shared high-performance filesystem is essential for storing training data, model checkpoints, and shared libraries across cluster nodes. SageMaker HyperPod supports Amazon FSx for Lustre and FSx for OpenZFS, configured per instance group via InstanceStorageConfigs. FSx filesystems reside in your VPC, so a custom VPC configuration (VpcConfig) is required when using FSx.

FSx configuration works with all three node lifecycle configuration options. When using AMI-based configuration or OnInitComplete, HyperPod handles FSx mounting automatically. When using OnCreate, your lifecycle scripts are responsible for mounting.

FSx for Lustre:

"InstanceStorageConfigs": [
    {
        "FsxLustreConfig": {
            "DnsName": "fs-0abc123def456789.fsx.us-west-2.amazonaws.com",
            "MountPath": "/fsx",
            "MountName": "abcdefgh"
        }
    }
]

FSx for OpenZFS:

"InstanceStorageConfigs": [
    {
        "FsxOpenZfsConfig": {
            "DnsName": "fs-0xyz789abc123456.fsx.us-west-2.amazonaws.com",
            "MountPath": "/shared"
        }
    }
]

VPC configuration (required for FSx):

Add VpcConfig at the cluster level in your CreateCluster request:

"VpcConfig": {
    "SecurityGroupIds": ["sg-0abc123def456789a"],
    "Subnets": ["subnet-0abc123def456789a"]
}

For more information on setting up a VPC, see Prerequisites for using SageMaker HyperPod. For more information on FSx setup, see Prerequisites for using SageMaker HyperPod.

Create your cluster

This section walks you through creating a cluster using each of the three node lifecycle configuration options described in Node lifecycle configuration options. For most users, we recommend starting with Option A, AMI-based configuration only. It requires no scripts or Amazon S3 bucket and delivers a fully functional cluster out of the box. Choose Option B if you need to add customizations on top of the AMI-based configuration, or Option C if you need full control over the provisioning process.

For ExecutionRole in all examples, provide the ARN of the IAM role you created with the managed AmazonSageMakerClusterInstanceRolePolicy in Prerequisites for using SageMaker HyperPod.

Option A: AMI-based configuration only (without lifecycle configuration)

This is the simplest path. No Amazon S3 bucket, scripts, or configuration files are needed. SageMaker HyperPod configures nodes automatically using AMI-based configuration, installing essential software and applying configurations so the cluster is ready to run ML workloads out of the box. All software packages are embedded in the AMI, so no internet access is required during provisioning.

The following table lists the capabilities included in AMI-based configuration:

Option B: Extend AMI-based configuration with OnInitComplete

Use this option when you need customizations on top of the AMI-based configuration, such as monitoring agents, LDAP/SSSD integration, or additional storage mounts. SageMaker HyperPod runs AMI-based configuration first, then executes your extension script.

Option C: Full custom control with OnCreate (advanced)

Use this option when you need complete control over provisioning, including installing software, making infrastructure changes, and deciding when to start Slurm. With OnCreate, SageMaker HyperPod does not run AMI-based configuration and does not start Slurm automatically.

Common validation errors

Connect to your cluster

After the cluster status turns to InService (typically 10 to 15 minutes), connect and verify.

For more information about running ML workloads, see Jobs on SageMaker HyperPod clusters.

Delete the cluster and clean resources

After testing, delete the cluster to avoid continued charges:

aws sagemaker delete-cluster --cluster-name my-hyperpod-cluster

If you used node lifecycle scripts (Option B or Option C), clean up the Amazon S3 bucket:

aws s3 rm s3://sagemaker-amzn-s3-demo-bucket/lifecycle/src --recursive

If you used AMI-based configuration only (Option A), no Amazon S3 cleanup is needed for node lifecycle scripts.

If you ran training workloads, also check for data or artifacts in Amazon S3, Amazon FSx for Lustre, or Amazon Elastic File System and delete them to prevent charges.

Related topics