Update the scheduler version of an AWS PCS cluster

Use these steps to update the scheduler version on your cluster. There are two options depending on whether you can tolerate job interruption. For more information about choosing between options, see Updating the scheduler version of a cluster in AWS PCS.

Option 1: Rolling update

The controller is updated while the fleet keeps running. Existing nodes continue using the previous Slurm version until they are drained and replaced. New nodes launched after the update use the target version. Running jobs are not interrupted.

When to use:

Step 0 — Check starting state

Your cluster is running controller version "A" (e.g. 24.11) and you want to migrate to version "B" (e.g. 25.11). Confirm all compute nodes in your fleet run the same major version, using this command from a cluster node:

scontrol show nodes | grep "Version="

# Example output:
#   NodeAddr=compute-1 NodeHostName=compute-1 Version=24.11.7
#   NodeAddr=compute-2 NodeHostName=compute-2 Version=24.11.7

Confirm the AWS PCS agent version on a compute node. Connect to the node with Systems Manager and check the bootstrap log:

grep "PCS Agent version" /var/log/amazon/pcs/bootstrap.log | tail -1

# Example output:
#   /opt/aws/pcs/bin/pcs_bootstrap_init.sh: INFO: Bootstrap starting with PCS Agent version: 1.3.2-1

Rolling updates require AWS PCS agent version 1.4.0 or later on all compute node AMIs. For more information, see AWS PCS agent versions.

Step 1 — Prepare the target AMIs

Build or identify AMIs that include Slurm version B and the latest AWS PCS agent.

Step 2 — Update the cluster controller

Call UpdateCluster with scheduler.version set to version B.

AWS Management Console

1. Open the AWS PCS console at https://console.aws.amazon.com/pcs/.

2. In the navigation pane, choose Clusters.

3. Select the cluster to update and choose Edit.

4. Under Cluster details, select the target scheduler version from the Scheduler dropdown.

5. Choose Update to submit the version update.

6. Monitor the cluster status. The cluster shows as UPDATING during the update and returns to ACTIVE when complete. The update typically completes in 5–15 minutes.

AWS CLI

aws pcs update-cluster \
  --cluster-identifier cluster-id \
  --scheduler version=25.11

Wait for the cluster to return to ACTIVE. The update typically completes in 5–15 minutes.

During this operation the controller is briefly unavailable:

If the cluster does not return to ACTIVE or UPDATE_FAILED within 30 minutes, contact AWS Support for assistance.

Step 3 — Update compute node groups

For each compute node group, set the new AMI with the target Slurm version:

aws pcs update-compute-node-group \
  --cluster-identifier cluster-id \
  --compute-node-group-identifier cng-id \
  --ami-id new-ami-id

AWS PCS sets nodes running the previous version to DRAIN state. After the drained nodes finish their current jobs, AWS PCS terminates the nodes and replaces them with new nodes running Slurm version B.

Step 4 — Verify consistent fleet on Slurm version B

Monitor the fleet transition. From a cluster node, check the version summary across all nodes:

scontrol show nodes | grep "Version=" | awk -F'=' '{print $NF}' | sort | uniq -c

Check nodes in DRAIN state and their version:

scontrol show nodes | awk '/NodeName=/{name=$1; ver=""} /Version=/{ver=$NF} /State=.*DRAIN/{print name, ver}'

Check the version for all active nodes:

scontrol show nodes | awk '/NodeName=/{name=$1; ver=""} /Version=/{ver=$NF} /State=/{if (ver) print name, ver}'

The update is complete when the version summary shows only the target version and no nodes remain in DRAIN or DRAINING state. Nodes in the POWERED_DOWN state do not report a version until AWS PCS launches them.

Option 2: Full-fleet maintenance stop

You terminate the entire fleet before updating the controller, then scale it back up from a new AMI with the target Slurm version. This procedure is simpler, but it terminates all nodes and running jobs.

When to use:

Step 0 — Check starting state

Your cluster is running controller version "A" (e.g. 24.11) and you want to migrate to version "B" (e.g. 25.11). Confirm all compute nodes in your fleet run the same major version, using this command from a cluster node:

scontrol show nodes | grep "Version="

# Example output:
#   NodeAddr=compute-1 NodeHostName=compute-1 Version=24.11.7
#   NodeAddr=compute-2 NodeHostName=compute-2 Version=24.11.7

Confirm the AWS PCS agent version on a compute node. Connect to the node with Systems Manager and check the bootstrap log:

grep "PCS Agent version" /var/log/amazon/pcs/bootstrap.log | tail -1

# Example output:
#   /opt/aws/pcs/bin/pcs_bootstrap_init.sh: INFO: Bootstrap starting with PCS Agent version: 1.3.2-1

Use the latest AWS PCS agent on your target AMIs. For more information, see AWS PCS agent versions.

Step 1 — Prepare the target AMIs

Build or identify AMIs that include Slurm version B and the latest AWS PCS agent.

Step 2 — Scale down the entire fleet

Record the current minNodeCount and maxNodeCount for each compute node group — you will restore these in Step 4.

for cng in $(aws pcs list-compute-node-groups --cluster-identifier cluster-id --query "computeNodeGroups[].id" --output text); do
    aws pcs get-compute-node-group \
      --cluster-identifier cluster-id \
      --compute-node-group-identifier "$cng" \
      --query "computeNodeGroup.{Id:id,AmiId:amiId,Min:scalingConfiguration.minInstanceCount,Max:scalingConfiguration.maxInstanceCount}" \
      --output table
  done

Set minNodeCount and maxNodeCount to 0 on every compute node group:

aws pcs update-compute-node-group \
  --cluster-identifier cluster-id \
  --compute-node-group-identifier cng-id \
  --scaling-configuration '{"minNodeCount": 0, "maxNodeCount": 0}'

Verify no instances tagged aws:pcs:cluster-id matching your cluster are running before continuing:

aws ec2 describe-instances \
    --filters "Name=tag:aws:pcs:cluster-id,Values=cluster-id" \
    --query "Reservations[].Instances[].[InstanceId,ImageId,State.Name]" \
    --output table

Step 3 — Update the cluster controller

AWS Management Console

1. Open the AWS PCS console at https://console.aws.amazon.com/pcs/.

2. In the navigation pane, choose Clusters.

3. Select the cluster to update and choose Edit.

4. Under Cluster details, select the target scheduler version from the Scheduler dropdown.

5. Choose Update to submit the version update.

6. Monitor the cluster status. The cluster shows as UPDATING during the update and returns to ACTIVE when complete. The update typically completes in 5–15 minutes.

AWS CLI

aws pcs update-cluster \
  --cluster-identifier cluster-id \
  --scheduler version=25.11

Wait for the cluster to return to ACTIVE. The update typically completes in 5–15 minutes.

If the cluster does not return to ACTIVE or UPDATE_FAILED within 30 minutes, contact AWS Support for assistance.

Step 4 — Update compute node groups and restore capacity

For each compute node group, set the new AMI and restore the original minimum and maximum capacity limits:

aws pcs update-compute-node-group \
  --cluster-identifier cluster-id \
  --compute-node-group-identifier cng-id \
  --ami-id new-ami-id \
  --scaling-configuration '{"minNodeCount": previous-min, "maxNodeCount": previous-max}'

The cluster scales back up. All new nodes run Slurm version B with the latest AWS PCS agent.

Example: Updating across multiple versions

If the target version is outside the compatibility window of your current version, you must move the controller through one or more intermediate versions, updating it one hop at a time. Each hop must target a supported version within the compatibility window of the current controller version.

Because Option 2: Full-fleet maintenance stop scales the fleet to zero before updating the controller, no compute nodes are running while the controller moves between versions. As a result, your AMIs can use the final target version directly — only the controller update (Step 3) is repeated for each hop.

The following example updates a cluster from 23.11 to 25.11 using the Option 2 procedure. 23.11 is outside the compatibility window of 25.11, so the controller is updated in two hops (23.11 to 25.05, then 25.05 to 25.11). Follow the Option 2 steps, with Step 3 split into one update per hop: