Amazon SageMaker HyperPod release notes

Added the following IAM condition keys for more granular access control in the CreateCluster and UpdateCluster API operations.

Added support for deleting instance groups from your SageMaker HyperPod cluster. For more information, see Delete instance groups from EKS-orchestrated clusters and Scale down a cluster for Slurm-orchestrated clusters.

This release of SageMaker HyperPod incorporates a security update from the Nvidia container toolkit (from version 1.17.3 to version 1.17.4). For more information, see v1.17.4 release note.

Enhanced SageMaker HyperPod multi-AZ support: You can specify different subnets and security groups, cutting across different Availability Zones, for individual instance groups within your cluster. For more information about SageMaker HyperPod multi-AZ support, see Setting up SageMaker HyperPod clusters across multiple AZs.

SageMaker HyperPod AMI releases for Amazon EKS: January 22, 2025

Added IPv6 support: Clusters can use IPv6 addressing when configured with IPv6-enabled VPC and subnets. For more information, see Setting up SageMaker HyperPod with a custom Amazon VPC.

SageMaker HyperPod now supports the following instance types for both Slurm and Amazon EKS clusters.

Enhanced error logging visibility when Slurm interrupts jobs, and prevented unnecessary job step termination during Slurm-initiated job cancellations.

Updated base DLAMI for p5en for both Slurm and Amazon EKS clusters.

SageMaker HyperPod AMI releases for Slurm: December 21, 2024

SageMaker HyperPod AMI releases for Amazon EKS: December 21, 2024

SageMaker HyperPod releases a set of Amazon CloudWatch metrics to monitor the health and performance of SageMaker HyperPod Slurm clusters. These metrics are related to CPU, GPU, memory utilization, and cluster instance information such as node counts and failed nodes. This monitoring feature is enabled by default, and the metrics can be accessed under the /aws/sagemaker/Clusters CloudWatch namespace. You can also set up CloudWatch alarms based on these metrics to proactively detect and address potential issues within their Slurm-based HyperPod clusters. For more information, see Amazon SageMaker HyperPod Slurm metrics.

SageMaker HyperPod AMI releases for Amazon EKS: December 13, 2024

Added support for configuring SageMaker HyperPod clusters across multiple Availability Zones. For more information about SageMaker HyperPod multi-AZ support, see Setting up SageMaker HyperPod clusters across multiple AZs.

SageMaker HyperPod AMI releases for Slurm: November 24, 2024

SageMaker HyperPod AMI releases for Amazon EKS: November 24, 2024

Added support for trn1 and trn1n instance types for both Amazon EKS and Slurm orchestrated clusters.

Improved log management for Slurm clusters:

Adjusted Fluent Bit settings to prevent tracking issues with files containing long lines.

Prevented unintended truncation with Slurm controller node updates in configuration file slurm.config.

SageMaker HyperPod AMI releases for Slurm: November 15, 2024

SageMaker HyperPod AMI releases for Amazon EKS: November 15, 2024

SageMaker HyperPod AMI now supports G6e instance types.

SageMaker HyperPod AMI releases for Slurm: November 11, 2024

SageMaker HyperPod AMI releases for Amazon EKS: November 11, 2024

Added scaling down SageMaker HyperPod clusters at the instance group level and instance level for both Amazon EKS and Slurm orchestrated clusters. For more information about scaling down Amazon EKS clusters, see Scale down a SageMaker HyperPod cluster. For more information about scaling down Slurm clusters, see Scale down a cluster in Using the AWS CLI.

SageMaker HyperPod now supports the P5e instance type for both Amazon EKS and Slurm orchestrated clusters.

SageMaker HyperPod now supports the P5e[n], G6, Gr6, and Trn2[n] instance types for both Slurm and Amazon EKS clusters.

SageMaker HyperPod AMI releases for Slurm: October 21, 2024

SageMaker HyperPod AMI releases for Amazon EKS: October 21, 2024

Added Amazon EKS support in SageMaker HyperPod. To learn more, see Orchestrating SageMaker HyperPod clusters with Amazon EKS.

Added support for managing SageMaker HyperPod clusters through AWS CloudFormation and Terraform. For more information about managing HyperPod clusters through AWS CloudFormation, see CloudFormation documentation for AWS::SageMaker::Cluster. To learn about managing HyperPod clusters through Terraform, see Terraform documentation for awscc_sagemaker_cluster.

SageMaker HyperPod AMI releases for Slurm: September 10, 2024

SageMaker HyperPod AMI releases for Amazon EKS: September 10, 2024

Enhanced the SageMaker HyperPod auto-resume functionality, extending the resiliency capability for Slurm nodes attached with Generic RESources (GRES).

When Generic Resources (GRES) are attached to a Slurm node, Slurm typically doesn't permit changes in the node allocation, such as replacing nodes, and thus doesn’t allow to resume a failed job. Unless explicitly forbidden, the HyperPod auto-resume functionality automatically re-queues any faulty job associated with the GRES-enabled nodes. This process involves stopping the job, placing it back into the job queue, and then restarting the job from the beginning.

Pre-packaged slurmrestd in the SageMaker HyperPod AMI.

Changed the default values for ResumeTimeout and UnkillableStepTimeout from 60 seconds to 300 seconds in slurm.conf to improve system responsiveness and job handling.

Made minor improvements on health checks for NVIDIA Data Center GPU Manager (DCGM) and The NVIDIA System Management Interface (nvidia-smi).

The HyperPod auto-resume plug-in can use idle nodes to resume a job.

Added a new capability of attaching additional storage to SageMaker HyperPod cluster instances. With this capability, you can configure supplementary storage at the instance group configuration level during the cluster creation or update processes, either through the SageMaker HyperPod console or the CreateCluster and UpdateCluster APIs. The additional EBS volume is attached to each instance within a SageMaker HyperPod cluster and mounted to /opt/sagemaker. To learn more about implementing it in your SageMaker HyperPod cluster, see the updated documentation on the following pages.

Note that you need to update the HyperPod cluster software to use this capability. After patching the HyperPod cluster software, you can utilize this capability for existing SageMaker HyperPod clusters created before June 20, 2024 by adding new instance groups. This capability is fully effective for any SageMaker HyperPod clusters created after June 20, 2024.

Run the following command to call the UpdateClusterSoftware API to update your existing HyperPod clusters with the latest HyperPod DLAMI. To find more instructions, see Update the SageMaker HyperPod platform software of a cluster.

 aws sagemaker update-cluster-software --cluster-name your-cluster-name

Fixed a bug with the ThreadsPerCore parameter in the ClusterInstanceGroupSpecification API. With the fix, the CreateCluster and UpdateCluster APIs properly take and apply the user input through ThreadsPerCore. This fix is effective on HyperPod clusters created after April 24, 2024. If you had issues with this bug and want to get this fix applied to your cluster, you need to create a new cluster. Make sure that you back up and restore your work while moving to a new cluster following the instructions at Use the backup script provided by SageMaker HyperPod.

In this release of the HyperPod DLAMI, Slurm is built with REST service (slurmestd) with JSON, YAML, and JWT support.

Upgraded Slurm to v23.11.3.

Increased auto-resume service timeout to 60 minutes.

Improved instance replacement process to not restart the Slurm controller.

Improved error messages from running lifecycle scripts, such as download errors and instance health check errors on instance start-up.

Fixed a bug with chrony service that caused an issue with time synchronization.

Fixed a bug with parsing slurm.conf.

Fixed an issue with NVIDIA go-dcgm library.

HyperPod now properly supports passing partition names provided through provisioning_params.json and creates partitions appropriately based on provided inputs. For more information about provisioning_params.json, see SageMaker HyperPod forms and Customize SageMaker HyperPod clusters using lifecycle scripts.

SageMaker HyperPod AMI releases for Slurm: March 14, 2024

Added a new UpdateClusterSoftware API for SageMaker HyperPod security patching. When security patches become available, we recommend you to update existing SageMaker HyperPod clusters in your account by running aws sagemaker update-cluster-software --cluster-name your-cluster-name. To follow up with future security patches, keep tracking this Amazon SageMaker HyperPod release notes page. To learn how the UpdateClusterSoftware API works, see Update the SageMaker HyperPod platform software of a cluster.

Launched Amazon SageMaker HyperPod at AWS re:Invent 2023.

SageMaker HyperPod AMI release for Slurm: November 29, 2023