Considering important notes

To migrate to a multi-controller Slurm cluster, complete these steps.

To scale down the number of controller nodes, use the update-cluster CLI command. For each controller instance group, the minimum number of controller nodes you can scale down to is 1. This means that you cannot scale down the number of controller nodes to 0.

The following is an example CLI command to scale down the number of controller nodes.

aws sagemaker update-cluster \
    --cluster-name my_cluster \
    --instance-groups '[{                  
    "InstanceGroupName": "controller_ig_name",
    "InstanceType": "ml.t3.medium",
    "InstanceCount": 3,
    "LifeCycleConfig": {
        "SourceS3Uri": "s3://amzn-s3-demo-bucket1",
        "OnCreate": "on_create.sh"
    },
    "ExecutionRole": "slurm_execution_role_arn",
    "ThreadsPerCore": 1
},
{
    "InstanceGroupName": "compute-ig_name",       
    "InstanceType": "ml.c5.xlarge",
    "InstanceCount": 2,
    "LifeCycleConfig": {
        "SourceS3Uri": "s3://amzn-s3-demo-bucket1",
        "OnCreate": "on_create.sh"
    },
    "ExecutionRole": "compute_node_role_arn",
    "ThreadsPerCore": 1
}]'

To batch delete the controller nodes, use the batch-delete-cluster-nodes CLI command. For each controller instance group, you must keep at least one controller node. If you want to batch delete all the controller nodes, the API operation won't work.

The following is an example CLI command to batch delete the controller nodes.

aws sagemaker batch-delete-cluster-nodes --cluster-name my_cluster --node-ids instance_ids_to_delete

To troubleshoot your cluster creation issues, check the failure message from the cluster details page in your SageMaker AI console. You can also use CloudWatch logs to troubleshoot cluster creation issues. From the CloudWatch console, choose Log groups. Then, search clusters to see the list of log groups related to your cluster creation.