Amazon SageMaker Training Compiler Release Notes

Currency Updates

• Added support for PyTorch v1.13.1

Bug Fixes

• Fixed a race condition issue on GPU which was causing NAN loss in some models like vision transformer (ViT) models.

Other Changes

• SageMaker Training Compiler improves performance by letting PyTorch/XLA to automatically override the optimizers (such as SGD, Adam, AdamW) in torch.optim or transformers.optimization with the syncfree versions of them in torch_xla.amp.syncfree (such as torch_xla.amp.syncfree.SGD, torch_xla.amp.syncfree.Adam, torch_xla.amp.syncfree.AdamW). You don't need to change those code lines where you define optimizers in your training script.

Migration to AWS Deep Learning Containers

This release passed benchmark testing and is migrated to the following AWS Deep Learning Container:

• PyTorch v1.13.1

  763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-trcomp-training:1.13.1-gpu-py39-cu117-ubuntu20.04-sagemaker

  To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see Supported Frameworks, AWS Regions, Instance Types, and Tested Models.

• tf.keras.optimizers.Optimizer points to a new optimizer in TensorFlow 2.11.0 and later. The old optimizers are moved to tf.keras.optimizers.legacy. You might encounter job failure due to the breaking change when you do the following.

  • Load checkpoints from an old optimizer. We recommend you to switch to use the legacy optimizers.

  • Use TensorFlow v1. We recommend you to migrate to TensorFlow v2, or switch to the legacy optimizers if you need to continue using TensorFlow v1.

  For more detailed list of breaking changes from the optimizer changes, see the official TensorFlow v2.11.0 release notes in the TensorFlow GitHub repository.

Migration to AWS Deep Learning Containers

This release passed benchmark testing and is migrated to the following AWS Deep Learning Container:

• TensorFlow v2.11.0

  763104351884.dkr.ecr.<region>.amazonaws.com/tensorflow-training:2.11.0-gpu-py39-cu112-ubuntu20.04-sagemaker

  To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see Supported Frameworks, AWS Regions, Instance Types, and Tested Models.

• Fixed the seed for PyTorch training jobs starting PyTorch v1.12 to ensure that there is no discrepancy in model initialization across different processes. See also PyTorch Reproducibility.

• Fixed the issue causing PyTorch distributed training jobs on G4dn and G5 instances to not default to communication through PCIe.

• Improper use of PyTorch/XLA APIs in Hugging Face’s vision transformers might lead to convergence issues.

• When using the Hugging Face Transformers Trainer class, make sure that you use SyncFree optimizers by setting the optim argument to adamw_torch_xla. For more information, see Large Language Models Using the Hugging Face Transformers Trainer Class. See also Optimizer in the Hugging Face Transformers documentation.

Migration to AWS Deep Learning Containers

This release passed benchmark testing and is migrated to the following AWS Deep Learning Container:

• PyTorch v1.12.0

  763104351884.dkr.ecr.<region>.amazonaws.com/pytorch-trcomp-training:1.12.0-gpu-py38-cu113-ubuntu20.04-sagemaker

  To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see Supported Frameworks, AWS Regions, Instance Types, and Tested Models.

Currency Updates

• Added support for TensorFlow v2.10.0.

Other Changes

• Added Hugging Face NLP models using the Transformers library to TensorFlow framework tests. To find the tested Transformer models, see Tested Models.

Migration to AWS Deep Learning Containers

This release passed benchmark testing and is migrated to the following AWS Deep Learning Container:

• TensorFlow v2.10.0

  763104351884.dkr.ecr.<region>.amazonaws.com/tensorflow-training:2.10.0-gpu-py39-cu112-ubuntu20.04-sagemaker

  To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see Supported Frameworks, AWS Regions, Instance Types, and Tested Models.

Currency Updates

• Added support for Hugging Face Transformers v4.21.1 with PyTorch v1.11.0.

Improvements

• Implemented a new distributed training launcher mechanism to activate SageMaker Training Compiler for Hugging Face Transformer models with PyTorch. To learn more, see Run PyTorch Training Jobs with SageMaker Training Compiler for Distributed Training.

• Integrated with EFA to improve the collective communication in distributed training.

• Added support for G5 instances for PyTorch training jobs. For more information, see Supported Frameworks, AWS Regions, Instance Types, and Tested Models.

Migration to AWS Deep Learning Containers

This release passed benchmark testing and is migrated to the following AWS Deep Learning Container:

• HuggingFace v4.21.1 with PyTorch v1.11.0

  763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-trcomp-training:1.11.0-transformers4.21.1-gpu-py38-cu113-ubuntu20.04

  To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see Supported Frameworks, AWS Regions, Instance Types, and Tested Models.

New Features

• Added support for TensorFlow v2.9.1. SageMaker Training Compiler fully supports compiling TensorFlow modules (tf.*) and TensorFlow Keras modules (tf.keras.*).

• Added support for custom containers created by extending AWS Deep Learning Containers for TensorFlow. For more information, see Enable SageMaker Training Compiler Using the SageMaker Python SDK and Extending SageMaker Framework Deep Learning Containers.

• Added support for G5 instances for TensorFlow training jobs.

Migration to AWS Deep Learning Containers

This release passed benchmark testing and is migrated to the following AWS Deep Learning Container:

• TensorFlow 2.9.1

  763104351884.dkr.ecr.<region>.amazonaws.com/tensorflow-training:2.9.1-gpu-py39-cu112-ubuntu20.04-sagemaker

  To find a complete list of the pre-built containers with Amazon SageMaker Training Compiler, see Supported Frameworks, AWS Regions, Instance Types, and Tested Models.

Improvements

• Added support for all of the AWS Regions where AWS Deep Learning Containers are in service except the China regions.

Currency Updates

• Added support for Hugging Face Transformers v4.17.0 with TensorFlow v2.6.3 and PyTorch v1.10.2.

Improvements

• Completed benchmark test and confirmed training speed-ups on the ml.g4dn instance types. To find a complete list of tested ml instances, see Supported Instance Types.

New Features

• Launched Amazon SageMaker Training Compiler at AWS re:Invent 2021.

Migration to AWS Deep Learning Containers

• Amazon SageMaker Training Compiler passed benchmark testing and is migrated to AWS Deep Learning Containers. To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see Supported Frameworks, AWS Regions, Instance Types, and Tested Models.