Migrating to API-driven Slurm configuration

If you have an existing SageMaker HyperPod Slurm cluster that uses the legacy provisioning_parameters.json file for Slurm configuration, you can migrate to the API-driven configuration model. With API-driven configuration, you define Slurm node types, partition assignments, and Amazon FSx mounting directly in the CreateCluster and UpdateCluster API payloads, removing the need to manage a separate configuration file in Amazon S3.

Before you begin

Before starting the migration, make sure the following prerequisites are met:

Clusters with multiple controller nodes (multi-head)

If your cluster is configured with multiple controller (head) nodes using the SageMaker HyperPod multi-head node support, you cannot fully migrate to the API-driven configuration model at this time.

The multi-head node architecture relies on the slurm_configurations block in provisioning_parameters.json to configure several components that have no equivalent in the API-driven approach:

"slurm_configurations": {
    "slurm_database_secret_arn": "$SLURM_DB_SECRET_ARN",
    "slurm_database_endpoint": "$SLURM_DB_ENDPOINT_ADDRESS",
    "slurm_shared_directory": "/fsx",
    "slurm_database_user": "$DB_USER_NAME",
    "slurm_sns_arn": "$SLURM_SNS_FAILOVER_TOPIC_ARN"
}

The following table lists the fields in slurm_configurations and their role in the multi-head node architecture.

These fields configure the external Slurm accounting database (slurmdbd), the shared filesystem for controller state replication, and the SNS-based failover notification mechanism. Together, they enable the automatic failover behavior between primary and backup controller nodes.

Because the API-driven SlurmConfig and Orchestrator.Slurm schema does not include parameters for these components, migrating a multi-head cluster to the API-driven approach would result in the loss of:

What you can do

If you have a multi-head node cluster and want to adopt parts of the API-driven approach, you have the following options:

What changes during migration

The following table summarizes what changes when you migrate from the legacy approach to API-driven configuration.

Step 1: Back up your current configuration

Before making any changes, create backups of your existing configuration.

Download provisioning_parameters.json from Amazon S3:

aws s3 cp s3://DOC-EXAMPLE-BUCKET/lifecycle-script-directory/src/provisioning_parameters.json ./backup/

Export the current slurm.conf from the controller node:

aws ssm start-session --target mi-controller-instance-id

sudo cp /opt/slurm/etc/slurm.conf ~/slurm.conf.backup

Record your current instance groups:

aws sagemaker describe-cluster \
    --cluster-name my-hyperpod-cluster \
    --query "InstanceGroups[*].{Name:InstanceGroupName,Type:InstanceType,Count:CurrentCount}"

Keep these backups available throughout the migration process. You need them if you need to recover from a failed migration.

Step 2: Upgrade the cluster software

API-driven Slurm configuration requires a SageMaker HyperPod AMI released in January 2026 or later. If your cluster already runs a qualifying AMI, skip this step. Otherwise, upgrade the cluster software first.

aws sagemaker update-cluster-software \
    --cluster-name my-hyperpod-cluster

Wait for the update to complete:

aws sagemaker describe-cluster \
    --cluster-name my-hyperpod-cluster \
    --query "ClusterStatus"

Proceed to the next step when the cluster status returns to InService.

Step 3: Map your legacy configuration to the API format

Convert the fields in your provisioning_parameters.json file to the corresponding API parameters.

Field mapping reference

The following table maps each legacy provisioning_parameters.json field to its corresponding API parameter.

Example conversion

The following example shows a legacy provisioning_parameters.json file and the equivalent API-driven UpdateCluster request payload.

Legacy provisioning_parameters.json:

{
    "version": "1.0.0",
    "workload_manager": "slurm",
    "controller_group": "controller-machine",
    "login_group": "login-group",
    "worker_groups": [
        {
            "instance_group_name": "gpu-compute",
            "partition_name": "gpu-training"
        },
        {
            "instance_group_name": "cpu-compute",
            "partition_name": "cpu-batch"
        }
    ],
    "fsx_dns_name": "fs-0abc123def456789.fsx.us-west-2.amazonaws.com",
    "fsx_mountname": "abcdefgh"
}

Equivalent API-driven UpdateCluster request (update_cluster_migration.json):

{
    "ClusterName": "my-hyperpod-cluster",
    "InstanceGroups": [
        {
            "InstanceGroupName": "controller-machine",
            "InstanceType": "ml.c5.xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Controller"
            },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://DOC-EXAMPLE-BUCKET/lifecycle-script-directory/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole",
            "InstanceStorageConfigs": [
                {
                    "EbsVolumeConfig": {
                        "VolumeSizeInGB": 500
                    }
                }
            ]
        },
        {
            "InstanceGroupName": "login-group",
            "InstanceType": "ml.m5.xlarge",
            "InstanceCount": 1,
            "SlurmConfig": {
                "NodeType": "Login"
            },
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://DOC-EXAMPLE-BUCKET/lifecycle-script-directory/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        },
        {
            "InstanceGroupName": "gpu-compute",
            "InstanceType": "ml.g5.12xlarge",
            "InstanceCount": 4,
            "SlurmConfig": {
                "NodeType": "Compute",
                "PartitionNames": ["gpu-training"]
            },
            "InstanceStorageConfigs": [
                {
                    "FsxLustreConfig": {
                        "DnsName": "fs-0abc123def456789.fsx.us-west-2.amazonaws.com",
                        "MountPath": "/fsx",
                        "MountName": "abcdefgh"
                    }
                }
            ],
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://DOC-EXAMPLE-BUCKET/lifecycle-script-directory/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        },
        {
            "InstanceGroupName": "cpu-compute",
            "InstanceType": "ml.c5.4xlarge",
            "InstanceCount": 2,
            "SlurmConfig": {
                "NodeType": "Compute",
                "PartitionNames": ["cpu-batch"]
            },
            "InstanceStorageConfigs": [
                {
                    "FsxLustreConfig": {
                        "DnsName": "fs-0abc123def456789.fsx.us-west-2.amazonaws.com",
                        "MountPath": "/fsx",
                        "MountName": "abcdefgh"
                    }
                }
            ],
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://DOC-EXAMPLE-BUCKET/lifecycle-script-directory/src",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodExecutionRole"
        }
    ],
    "Orchestrator": {
        "Slurm": {
            "SlurmConfigStrategy": "Merge"
        }
    }
}

Note the following about this conversion:

Step 4: Remove provisioning_parameters.json from Amazon S3

Removing the provisioning_parameters.json file from Amazon S3 signals HyperPod to use the API-driven configuration instead of the legacy file-based approach.

aws s3 rm s3://DOC-EXAMPLE-BUCKET/lifecycle-script-directory/src/provisioning_parameters.json

Verify the file has been removed:

aws s3 ls s3://DOC-EXAMPLE-BUCKET/lifecycle-script-directory/src/

If your instance groups use different SourceS3Uri locations, remove the file from each of them.

Step 5: Apply the API-driven configuration

Run the UpdateCluster API with the JSON request file you prepared in Step 3: Map your legacy configuration to the API format:

aws sagemaker update-cluster \
    --cli-input-json file://update_cluster_migration.json

Monitor the update progress:

aws sagemaker describe-cluster \
    --cluster-name my-hyperpod-cluster \
    --query "{Status: ClusterStatus, Message: FailureMessage}"

Wait for the cluster status to return to InService before proceeding. The update typically takes several minutes depending on the number of instance groups and nodes in your cluster.

Step 6: Verify the migration

After the cluster status returns to InService, verify that the Slurm configuration was applied correctly.

Check Slurm partitions. Connect to the controller node using SSM Session Manager and verify the partition layout:

aws ssm start-session --target mi-controller-instance-id

sinfo

Expected output:

PARTITION      AVAIL  TIMELIMIT  NODES  STATE NODELIST
dev*           up     infinite   6      idle  gpu-compute-[1-4],cpu-compute-[1-2]
gpu-training   up     infinite   4      idle  gpu-compute-[1-4]
cpu-batch      up     infinite   2      idle  cpu-compute-[1-2]

Check node-to-partition assignments.

scontrol show nodes | grep -E "NodeName|Partitions"

Check Amazon FSx mounts (if configured). On a compute node, verify the filesystem is mounted:

df -h | grep fsx

Expected output:

fs-0abc123def456789.fsx.us-west-2.amazonaws.com@tcp:/abcdefgh  1.2T  12G  1.2T  1% /fsx

Submit a test job.

sbatch --partition=gpu-training --wrap="hostname && nvidia-smi"

Check the job output:

squeue
cat slurm-*.out

Recovering from a failed migration

If the migration does not complete successfully, HyperPod automatically rolls back the cluster to its previous state. The cluster returns to InService and continues to use the legacy configuration.

To recover, restore provisioning_parameters.json to its original Amazon S3 location so that any new nodes provision with the legacy configuration:

aws s3 cp ./backup/provisioning_parameters.json \
    s3://DOC-EXAMPLE-BUCKET/lifecycle-script-directory/src/provisioning_parameters.json

Then review the failure reason, correct your request payload, and retry from Step 4: Remove provisioning_parameters.json from Amazon S3:

aws sagemaker describe-cluster \
    --cluster-name my-hyperpod-cluster \
    --query "{Status: ClusterStatus, Message: FailureMessage}"

Post-migration considerations

After a successful migration, consider the following updates to your environment.

Simplify lifecycle scripts

With API-driven configuration, HyperPod handles Slurm topology setup and Amazon FSx mounting automatically. You can remove the corresponding logic from your lifecycle scripts and keep only custom setup steps such as user creation, package installation, and environment configuration.

The following example shows a minimal on_create.sh lifecycle script for clusters using API-driven configuration:

#!/bin/bash
set -e

echo "=== HyperPod Lifecycle Script ==="
echo "Timestamp: $(date)"
echo "Hostname: $(hostname)"

# Custom setup only - Slurm configuration and FSx mounting
# are handled by HyperPod based on the API payload.

# Example: Install additional packages
# sudo apt-get update && sudo apt-get install -y htop vim

# Example: Create users
# sudo useradd -m -s /bin/bash researcher

# Example: Set environment variables
# echo 'export NCCL_DEBUG=INFO' >> /etc/profile.d/nccl.sh

echo "=== Lifecycle Script Completed ==="

Update automation workflows

If you have automation scripts or CI/CD pipelines that modify provisioning_parameters.json in Amazon S3, update them to use the UpdateCluster API instead. Remove any Amazon S3 file management logic related to provisioning_parameters.json.

Choose a Slurm configuration strategy

After you migrate, you can choose a SlurmConfigStrategy that controls how HyperPod manages the relationship between the API-declared Slurm topology and the actual slurm.conf on the controller node. The strategy determines who owns the Slurm topology—you or the API.

You can change the strategy in a later UpdateCluster request by including Orchestrator.Slurm.SlurmConfigStrategy. Clusters that use continuous provisioning (NodeProvisioningMode set to Continuous) support only the Merge behavior and reject requests that set SlurmConfigStrategy. If you plan to migrate to continuous provisioning, keep Merge. For more information, see Continuous provisioning for enhanced cluster operations with Slurm.

Troubleshooting

"Update required to use SlurmConfig in InstanceGroups"

Your cluster AMI does not include the updated agents required for API-driven configuration. Upgrade the cluster software:

aws sagemaker update-cluster-software \
    --cluster-name my-hyperpod-cluster

Wait for the cluster to return to InService, then retry the migration.

"N Controller Groups found: SlurmConfig for InstanceGroups may only have one Controller Group."

Your API payload specifies more than one instance group with SlurmConfig.NodeType set to "Controller". Make sure exactly one instance group is the controller.

"Cluster X has no InstanceGroup with Controller node type."

Your API payload does not identify a controller instance group. Make sure exactly one instance group has SlurmConfig.NodeType set to "Controller".

"Partitions can only be assigned to Compute node types"

You specified PartitionNames on a controller or login instance group. Remove PartitionNames from any instance group that is not a compute node.

"The LifeCycleConfig cannot include both a SLURM Orchestrator Config and a provisioning_parameters.json file simultaneously."

The update workflow found provisioning_parameters.json in the lifecycle script location while your request included SlurmConfig. The update fails and rolls back, and DescribeCluster reports this message in FailureMessage. Complete Step 4: Remove provisioning_parameters.json from Amazon S3, verify that no copy of the file remains under the SourceS3Uri prefix of any instance group, and retry UpdateCluster.

Slurm partitions not updated after migration

If sinfo does not show the expected partitions:

Amazon FSx not mounted on nodes

Amazon FSx configuration changes only apply to new nodes. Existing nodes retain their original storage configuration. To apply Amazon FSx changes to all nodes in an instance group:

Also verify that your security groups allow the required ports:

"Configuration drift detected."

Drift messages (for example, "Configuration drift detected.", "Partition configuration drift detected.", or "Partition configuration mismatch detected.") appear in the FailureMessage field of DescribeCluster after an UpdateCluster fails and rolls back. They occur when using the Managed strategy and HyperPod detects configuration drift—a mismatch between the API-declared configuration and the actual slurm.conf on the controller node. Drift occurs in the following cases:

To resolve drift, use one of the following options:

Migration checklist

Use this checklist to track your migration progress: