# Model training
features = 1;
// An optional set of labels for this record.
// Similar to the features field above, the key used for
// generic scalar / vector labels should be 'values'.
map label = 2;
// A unique identifier for this record in the dataset.
//
// Whilst not necessary, this allows better
// debugging where there are data issues.
//
// This is not used by the algorithm directly.
optional string uid = 3;
// Textual metadata describing the record.
//
// This may include JSON-serialized information
// about the source of the record.
//
// This is not used by the algorithm directly.
optional string metadata = 4;
// An optional serialized JSON object that allows per-record
// hyper-parameters/configuration/other information to be set.
//
// The meaning/interpretation of this field is defined by
// the algorithm author and may not be supported.
//
// This is used to pass additional inference configuration
// when batch inference is used (e.g. types of scores to return).
optional string configuration = 5;
}
```
After creating the protocol buffer, store it in an Amazon S3 location that Amazon SageMaker AI can access and that can be passed as part of `InputDataConfig` in `create_training_job`.
**Note**
For all Amazon SageMaker AI algorithms, the `ChannelName` in `InputDataConfig` must be set to `train`. Some algorithms also support a validation or test `input channels`. These are typically used to evaluate the model's performance by using a hold-out dataset. Hold-out datasets are not used in the initial training but can be used to further tune the model.
## Trained Model Deserialization
": dataElement
}
}
]
}
```
You have the following options for specifying the `dataElement`:
**Protocol buffers equivalent**
```
// Has the same format as the protocol buffers implementation described for training.
let dataElement = {
"keys": [],
"values": [],
"shape": []
}
```
**Simple numeric vector **
```
// An array containing numeric values is treated as an instance containing a
// single dense vector.
let dataElement = [1.5, 16.0, 14.0, 23.0]
// It will be converted to the following representation by the SDK.
let converted = {
"features": {
"values": dataElement
}
}
```
**For multiple records**
```
let request = {
"instances": [
// First instance.
{
"features": [ 1.5, 16.0, 14.0, 23.0 ]
},
// Second instance.
{
"features": [ -2.0, 100.2, 15.2, 9.2 ]
}
]
}
```
## Convert data for inference response deserialization
factorization terms model the pairwise interaction between the ith and jth variable.
The global bias and linear terms are the same as in a linear model. The pairwise feature interactions are modeled in the third term as the inner product of the corresponding factors learned for each feature. Learned factors can also be considered as embedding vectors for each feature. For example, in a classification task, if a pair of features tends to co-occur more often in positive labeled samples, then the inner product of their factors would be large. In other words, their embedding vectors would be close to each other in cosine similarity. For more information about the Factorization Machines model, see [Factorization Machines](https://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle2010FM.pdf).
For regression tasks, the model is trained by minimizing the squared error between the model prediction ŷn and the target value yn. This is known as the square loss:
![\[An image containing the equation for square loss.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/FM2.jpg)
For a classification task, the model is trained by minimizing the cross entropy loss, also known as the log loss:
![\[An image containing the equation for log loss.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/FM3.jpg)
where:
![\[An image containing the logistic function of the predicted values.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/FM4.jpg)
For more information about loss functions for classification, see [Loss functions for classification](https://en.wikipedia.org/wiki/Loss_functions_for_classification).
# Factorization Machines Hyperparameters
: :.
For information on instance types and distributed training, see [EC2 instance recommendation for the XGBoost algorithm](xgboost.md#Instance-XGBoost).
For CSV training input mode, the total memory available to the algorithm must be able to hold the training dataset. The total memory available is calculated as `Instance Count * the memory available in the InstanceType`. For libsvm training input mode, it's not required, but we recommend it.
For v1.3-1 and later, SageMaker AI XGBoost saves the model in the XGBoost internal binary format, using `Booster.save_model`. Previous versions use the Python pickle module to serialize/deserialize the model.
**Note**
Be mindful of versions when using an SageMaker AI XGBoost model in open source XGBoost. Versions 1.3-1 and later use the XGBoost internal binary format while previous versions use the Python pickle module.
**To use a model trained with SageMaker AI XGBoost v1.3-1 or later in open source XGBoost**
+ Use the following Python code:
```
import xgboost as xgb
xgb_model = xgb.Booster()
xgb_model.load_model(model_file_path)
xgb_model.predict(dtest)
```
**To use a model trained with previous versions of SageMaker AI XGBoost in open source XGBoost**
+ Use the following Python code:
```
import pickle as pkl
import tarfile
t = tarfile.open('model.tar.gz', 'r:gz')
t.extractall()
model = pkl.load(open(model_file_path, 'rb'))
# prediction with test data
pred = model.predict(dtest)
```
**To differentiate the importance of labelled data points use Instance Weight Supports**
+ SageMaker AI XGBoost allows customers to differentiate the importance of labelled data points by assigning each instance a weight value. For *text/libsvm* input, customers can assign weight values to data instances by attaching them after the labels. For example, `label:weight idx_0:val_0 idx_1:val_1...`. For *text/csv* input, customers need to turn on the `csv_weights` flag in the parameters and attach weight values in the column after labels. For example: `label,weight,val_0,val_1,...`).
# XGBoost sample notebooks
: : ... Inference requests for libsvm may or may not have labels in the libsvm format.
This differs from other SageMaker AI algorithms, which use the protobuf training input format to maintain greater consistency with standard XGBoost data formats.
For CSV training input mode, the total memory available to the algorithm (Instance Count \$1 the memory available in the `InstanceType`) must be able to hold the training dataset. For libsvm training input mode, it's not required, but we recommend it.
SageMaker AI XGBoost uses the Python pickle module to serialize/deserialize the model, which can be used for saving/loading the model.
**To use a model trained with SageMaker AI XGBoost in open source XGBoost**
+ Use the following Python code:
```
import pickle as pkl
import tarfile
import xgboost
t = tarfile.open('model.tar.gz', 'r:gz')
t.extractall()
model = pkl.load(open(model_file_path, 'rb'))
# prediction with test data
pred = model.predict(dtest)
```
**To differentiate the importance of labelled data points use Instance Weight Supports**
+ SageMaker AI XGBoost allows customers to differentiate the importance of labelled data points by assigning each instance a weight value. For *text/libsvm* input, customers can assign weight values to data instances by attaching them after the labels. For example, `label:weight idx_0:val_0 idx_1:val_1...`. For *text/csv* input, customers need to turn on the `csv_weights` flag in the parameters and attach weight values in the column after labels. For example: `label,weight,val_0,val_1,...`).
## EC2 Instance Recommendation for the XGBoost Release 0.72
//model.tar.gz model.tar.gz
#Unzip the model archive
tar -xzf model.tar.gz
#Use the model archive with fastText
fasttext predict ./model.bin test.txt
```
However, the binaries are only supported when training on CPU and single GPU; training on multi-GPU will not produce binaries.
## EC2 Instance Recommendation for the BlazingText Algorithm
,
Where and are integers in the range [1,5000] and define the maximum sequence lengths for the forward and backward encoder.
```
{
"instances" : [
{"in0": [6, 17, 606, 19, 53, 67, 52, 12, 5, 10, 15, 10178, 7, 33, 652, 80, 15, 69, 821, 4]},
{"in0": [22, 1016, 32, 13, 25, 11, 5, 64, 573, 45, 5, 80, 15, 67, 21, 7, 9, 107, 4]},
{"in0": [774, 14, 21, 206]}
]
}
```
Content-type: application/jsonlines; infer\$1max\$1seqlens=,
Where and are integers in the range [1,5000] and define the maximum sequence lengths for the forward and backward encoder.
```
{"in0": [6, 17, 606, 19, 53, 67, 52, 12, 5, 10, 15, 10178, 7, 33, 652, 80, 15, 69, 821, 4]}
{"in0": [22, 1016, 32, 13, 25, 11, 5, 64, 573, 45, 5, 80, 15, 67, 21, 7, 9, 107, 4]}
{"in0": [774, 14, 21, 206]}
```
In both of these formats, you specify only one input type: `“in0”` or `“in1.”` The inference service then invokes the corresponding encoder and outputs the embeddings for each of the instances.
## Output: Encoder Embeddings
/train/class_dog`, `s3:///train/class_cat`, and so on, specify the path for your `train` channel as `s3:///train`, which is the top-level directory for your data. In the `.lst` file, specify the relative path for an individual file named `train_image_dog1.jpg` in the `class_dog` class directory as `class_dog/train_image_dog1.jpg`. You can also store all your image files under one subdirectory inside the `train` directory. In that case, use that subdirectory for the relative path. For example, `s3:///train/your_image_directory`.
### Train with Augmented Manifest Image Format
}
```
If `accept` is set to `application/json`, then the model only outputs normalized boxes, classes, and scores.
## Amazon EC2 instance recommendation for the Object Detection - TensorFlow algorithm
,
instance_type="ml.m5.xlarge",
dependencies='./requirements.txt'
)
@remote(**settings)
def divide(x, y):
return x / y
if __name__ == "__main__":
print(divide(2, 3.0))
```
+ To use dependencies from the active `conda` environment, use the value `auto_capture` for the `dependencies` parameter, as shown in the following.
```
import boto3
import sagemaker
from sagemaker.remote_function import remote
sm_session = sagemaker.Session(boto_session=boto3.session.Session(region_name="us-west-2"))
settings = dict(
sagemaker_session=sm_session,
role=,
instance_type="ml.m5.xlarge",
dependencies="auto_capture"
)
@remote(**settings)
def divide(x, y):
return x / y
if __name__ == "__main__":
print(divide(2, 3.0))
```
**Note**
You can also implement the previous code inside a Jupyter notebook. PyCharm Professional Edition supports Jupyter natively. For more guidance, see [Jupyter notebook support](https://www.jetbrains.com/help/pycharm/ipython-notebook-support.html) in PyCharm's documentation.
# Invoke a remote function
`, in a configuration file. The parameter `job_name` is automatically generated for you.
Under the root directory, SageMaker AI creates a `` folder, which holds your current work directory, serialized function, the arguments for your serialized function, results and any exceptions that arose from invoking the serialized function.
Under ``, the directory `workdir` contains a zipped archive of your current working directory. The zipped archive includes any Python files in your working directory and the `requirements.txt` file, which specifies any dependencies needed to run your remote function.
The following is an example of the folder structure under an S3 bucket that you specify in your configuration file.
```
/ # specified by s3_root_uri or S3RootUri
/ #automatically generated for you
workdir/workspace.zip # archive of the current working directory (workdir)
function/ # serialized function
arguments/ # serialized function arguments
results/ # returned output from the serialized function including the model
exception/ # any exceptions from invoking the serialized function
```
The root directory that you specify in your S3 bucket is not meant for long term storage. The serialized data are tightly tied to the Python version and machine learning (ML) framework version that were used during serialization. If you upgrade the Python version or ML framework, you may not be able to use your serialized data. Instead, do the following.
+ Store your model and model artifacts in a format that is agnostic to your Python version and ML framework.
+ If you upgrade your Python or ML framework, access your model results from your long-term storage.
**Important**
To delete your serialized data after a specified amount of time, set a [lifetime configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html) on your S3 bucket.
**Note**
Files that are serialized with the Python [pickle](https://docs.python.org/3/library/pickle.html) module can be less portable than other data formats including CSV, Parquet and JSON. Be wary of loading pickled files from unknown sources.
For more information about what to include in a configuration file for a remote function, see [Configuration File](https://docs.aws.amazon.com/sagemaker/latest/dg/train-remote-decorator-config.html).
#### Access to your serialized data
.amazonaws.com/tensorflow-training:2.12.0-cpu-py310-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.12.0-gpu-py310-cu118-ubuntu20.04-sagemaker |
| DLC-Tensorflow 2.11.0 for SageMaker training | 3.9(py39) | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.11.0-cpu-py39-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.11.0-gpu-py39-cu112-ubuntu20.04-sagemaker |
| DLC-TensorFlow 2.10.1 for SageMaker training | 3.9(py39) | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.10.1-cpu-py39-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.10.1-gpu-py39-cu112-ubuntu20.04-sagemaker |
| DLC-TensorFlow 2.9.2 for SageMaker training | 3.9(py39) | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.9.2-cpu-py39-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.9.2-gpu-py39-cu112-ubuntu20.04-sagemaker |
| DLC-TensorFlow 2.8.3 for SageMaker training | 3.9(py39) | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.8.3-cpu-py39-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.8.3-gpu-py39-cu112-ubuntu20.04-sagemaker |
| DLC-PyTorch 2.0.0 for SageMaker training | 3.10(py310) | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.0.0-cpu-py310-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.0.0-gpu-py310-cu118-ubuntu20.04-sagemaker |
| DLC-PyTorch 1.13.1 for SageMaker training | 3.9(py39) | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.13.1-cpu-py39-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.13.1-gpu-py39-cu117-ubuntu20.04-sagemaker |
| DLC-PyTorch 1.12.1 for SageMaker training | 3.8(py38) | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.12.1-cpu-py38-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.12.1-gpu-py38-cu113-ubuntu20.04-sagemaker |
| DLC-PyTorch 1.11.0 for SageMaker training | 3.8(py38) | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.11.0-cpu-py38-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.11.0-gpu-py38-cu113-ubuntu20.04-sagemaker |
| DLC-MXNet 1.9.0 for SageMaker training | 3.8(py38) | 763104351884.dkr.ecr..amazonaws.com/mxnet-training:1.9.0-cpu-py38-ubuntu20.04-sagemaker | 763104351884.dkr.ecr..amazonaws.com/mxnet-training:1.9.0-gpu-py38-cu112-ubuntu20.04-sagemaker |
**Note**
To run jobs locally using AWS Deep Learning Containers (DLC) images, use the image URIs found in the [DLC documentation](https://github.com/aws/deep-learning-containers/blob/master/available_images.md). The DLC images do not support the `auto_capture` value for dependencies.
Jobs with [SageMaker AI Distribution in SageMaker Studio](https://github.com/aws/sagemaker-distribution#amazon-sagemaker-studio) run in a container as a non-root user named `sagemaker-user`. This user needs full permission to access `/opt/ml` and `/tmp`. Grant this permission by adding `sudo chmod -R 777 /opt/ml /tmp` to the `pre_execution_commands` list, as shown in the following snippet:
```
@remote(pre_execution_commands=["sudo chmod -R 777 /opt/ml /tmp"])
def func():
pass
```
You can also run remote functions with your custom images. For compatibility with remote functions, custom images should be built with Python version 3.7.x-3.10.x. The following is a minimal Dockerfile example showing you how to use a Docker image with Python 3.10.
```
FROM python:3.10
#... Rest of the Dockerfile
```
To create `conda` environments in your image and use it to run jobs, set the environment variable `SAGEMAKER_JOB_CONDA_ENV` to the `conda` environment name. If your image has the `SAGEMAKER_JOB_CONDA_ENV` value set, the remote function cannot create a new conda environment during the training job runtime. Refer to the following Dockerfile example that uses a `conda` environment with Python version 3.10.
```
FROM continuumio/miniconda3:4.12.0
ENV SHELL=/bin/bash \
CONDA_DIR=/opt/conda \
SAGEMAKER_JOB_CONDA_ENV=sagemaker-job-env
RUN conda create -n $SAGEMAKER_JOB_CONDA_ENV \
&& conda install -n $SAGEMAKER_JOB_CONDA_ENV python=3.10 -y \
&& conda clean --all -f -y \
```
For SageMaker AI to use [mamba](https://mamba.readthedocs.io/en/latest/user_guide/mamba.html) to manage your Python virtual environment in the container image, install the [mamba toolkit from miniforge](https://github.com/conda-forge/miniforge). To use mamba, add the following code example to your Dockerfile. Then, SageMaker AI will detect the `mamba` availability at runtime and use it instead of `conda`.
```
#Mamba Installation
RUN curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh" \
&& bash Mambaforge-Linux-x86_64.sh -b -p "/opt/conda" \
&& /opt/conda/bin/conda init bash
```
Using a custom conda channel on an Amazon S3 bucket is not compatible with mamba when using a remote function. If you choose to use mamba, make sure you are not using a custom conda channel on Amazon S3. For more information, see the **Prerequisites** section under **Custom conda repository using Amazon S3**.
The following is a complete Dockerfile example showing how to create a compatible Docker image.
```
FROM python:3.10
RUN apt-get update -y \
# Needed for awscli to work
# See: https://github.com/aws/aws-cli/issues/1957#issuecomment-687455928
&& apt-get install -y groff unzip curl \
&& pip install --upgrade \
'boto3>1.0<2' \
'awscli>1.0<2' \
'ipykernel>6.0.0<7.0.0' \
#Use ipykernel with --sys-prefix flag, so that the absolute path to
#/usr/local/share/jupyter/kernels/python3/kernel.json python is used
# in kernelspec.json file
&& python -m ipykernel install --sys-prefix
#Install Mamba
RUN curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh" \
&& bash Mambaforge-Linux-x86_64.sh -b -p "/opt/conda" \
&& /opt/conda/bin/conda init bash
#cleanup
RUN apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf ${HOME}/.cache/pip \
&& rm Mambaforge-Linux-x86_64.sh
ENV SHELL=/bin/bash \
PATH=$PATH:/opt/conda/bin
```
The resulting image from running the previous Dockerfile example can also be used as a [SageMaker Studio Classic kernel image](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-byoi.html).
# Logging parameters and metrics with Amazon SageMaker Experiments
/tmp/mlflow-trust-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"sagemaker.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
EOF
```
1. Within your terminal, use the following command to create a file called `custom-policy.json`.
```
cat < /tmp/custom-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:Put*",
"sagemaker:AddTags",
"sagemaker:CreateModelPackageGroup",
"sagemaker:CreateModelPackage",
"sagemaker:DescribeModelPackageGroup",
"sagemaker:UpdateModelPackage",
"s3:List*"
],
"Resource": "*"
}
]
}
EOF
```
1. Use the trust policy file to create a role. Then, attach IAM role policies that allow MLflow to access Amazon S3 and SageMaker Model Registry within your account. MLflow must have access to Amazon S3 for your app's artifact store and SageMaker Model Registry for automatic model registration.
**Note**
If you are updating an existing role, use the following command instead: `aws iam update-assume-role-policy --role-name $role_name --policy-document file:///tmp/mlflow-trust-policy.json`.
```
role_name=role-name
aws iam create-role \
--role-name $role_name \
--assume-role-policy-document file:///tmp/mlflow-trust-policy.json
aws iam put-role-policy \
--role-name $role_name \
--policy-name custom-policy \
--policy-document file:///tmp/custom-policy.json
role_arn=$(aws iam get-role --role-name $role_name --query 'Role.Arn' --output text)
```
## Create MLflow App
/tmp/mlflow-trust-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"sagemaker.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
EOF
```
1. Within your terminal, use the following command to create a file called `custom-policy.json`.
```
cat < /tmp/custom-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:Put*",
"sagemaker:AddTags",
"sagemaker:CreateModelPackageGroup",
"sagemaker:CreateModelPackage",
"sagemaker:DescribeModelPackageGroup",
"sagemaker:UpdateModelPackage",
"s3:List*"
],
"Resource": "*"
}
]
}
EOF
```
1. Use the trust policy file to create a role. Then, attach IAM role policies that allow MLflow to access Amazon S3 and SageMaker Model Registry within your account. MLflow must have access to Amazon S3 for your tracking server's artifact store and SageMaker Model Registry for automatic model registration.
**Note**
If you are updating an existing role, use the following command instead: `aws iam update-assume-role-policy --role-name $role_name --policy-document file:///tmp/mlflow-trust-policy.json`.
```
role_name=role-name
aws iam create-role \
--role-name $role_name \
--assume-role-policy-document file:///tmp/mlflow-trust-policy.json
aws iam put-role-policy \
--role-name $role_name \
--policy-name custom-policy \
--policy-document file:///tmp/custom-policy.json
role_arn=$(aws iam get-role --role-name $role_name --query 'Role.Arn' --output text)
```
## Create MLflow tracking server
-<12digit_account_id>///`. There are two subfolders created by SageMaker Debugger: `debug-output`, and `rule-output`. If you add the `tensorboard_output_config` parameter, you'll also find `tensorboard-output` folder.
See the following topics to find more examples of how to configure the Debugger-specific parameters in detail.
**Topics**
+ [
## Constructing a SageMaker AI Estimator with Debugger-specific parameters
](#debugger-configuration-structure)
+ [
# Configuring SageMaker Debugger to save tensors
](debugger-configure-hook.md)
+ [
# How to configure Debugger built-in rules
](use-debugger-built-in-rules.md)
+ [
# Turn off Debugger
](debugger-turn-off.md)
+ [
# Useful SageMaker AI estimator class methods for Debugger
](debugger-estimator-classmethods.md)
# Configuring SageMaker Debugger to save tensors
-<12digit_account_id>//debug-output/`.
**Topics**
+ [
# Configure tensor collections using the `CollectionConfig` API
](debugger-configure-tensor-collections.md)
+ [
# Configure the `DebuggerHookConfig` API to save tensors
](debugger-configure-tensor-hook.md)
+ [
# Example notebooks and code samples to configure Debugger hook
](debugger-save-tensors.md)
# Configure tensor collections using the `CollectionConfig` API
-<12digit_account_id>//debug-output/.`
If you want to specify an exact S3 bucket URI, use the following code example:
```
from sagemaker.debugger import DebuggerHookConfig
debugger_hook_config=DebuggerHookConfig(
s3_output_path="specify-uri"
collection_configs=collection_configs
)
```
For more information, see [DebuggerHookConfig](https://sagemaker.readthedocs.io/en/stable/api/training/debugger.html#sagemaker.debugger.DebuggerHookConfig) in the [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable).
# Example notebooks and code samples to configure Debugger hook
/debug-output/`, where the `` is the default output path of SageMaker training jobs. For example:
```
"s3://sagemaker-us-east-1-111122223333/sagemaker-debugger-training-YYYY-MM-DD-HH-MM-SS-123/debug-output"
```
```
import sagemaker
from sagemaker.tensorflow import TensorFlow
from sagemaker.debugger import DebuggerHookConfig, CollectionConfig
# use Debugger CollectionConfig to call built-in collections
collection_configs=[
CollectionConfig(name="weights"),
CollectionConfig(name="gradients"),
CollectionConfig(name="losses"),
CollectionConfig(name="biases")
]
# configure Debugger hook
# set a target S3 bucket as you want
sagemaker_session=sagemaker.Session()
BUCKET_NAME=sagemaker_session.default_bucket()
LOCATION_IN_BUCKET='debugger-built-in-collections-hook'
hook_config=DebuggerHookConfig(
s3_output_path='s3://{BUCKET_NAME}/{LOCATION_IN_BUCKET}'.
format(BUCKET_NAME=BUCKET_NAME,
LOCATION_IN_BUCKET=LOCATION_IN_BUCKET),
collection_configs=collection_configs
)
# construct a SageMaker TensorFlow estimator
sagemaker_estimator=TensorFlow(
entry_point='directory/to/your_training_script.py',
role=sm.get_execution_role(),
base_job_name='debugger-demo-job',
instance_count=1,
instance_type="ml.p3.2xlarge",
framework_version="2.9.0",
py_version="py39",
# debugger-specific hook argument below
debugger_hook_config=hook_config
)
sagemaker_estimator.fit()
```
To see a list of Debugger built-in collections, see [Debugger Built-in Collections](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/api.md#collection).
## Save tensors by modifying Debugger built-in collections
-<12digit_account_id>//debug-output/`.
When you construct a SageMaker AI estimator for an XGBoost training job, specify the rule as shown in the following example code.
------
#### [ Using the SageMaker AI generic estimator ]
```
import boto3
import sagemaker
from sagemaker.estimator import Estimator
from sagemaker import image_uris
from sagemaker.debugger import Rule, rule_configs
rules=[
Rule.sagemaker(rule_configs.create_xgboost_report())
]
region = boto3.Session().region_name
xgboost_container=sagemaker.image_uris.retrieve("xgboost", region, "1.2-1")
estimator=Estimator(
role=sagemaker.get_execution_role()
image_uri=xgboost_container,
base_job_name="debugger-xgboost-report-demo",
instance_count=1,
instance_type="ml.m5.2xlarge",
# Add the Debugger XGBoost report rule
rules=rules
)
estimator.fit(wait=False)
```
------
# Download the Debugger XGBoost training report
//rule-output`. Configure the rule output path as follows:
```
rule_output_path = estimator.output_path + "/" + estimator.latest_training_job.job_name + "/rule-output"
```
1. To check if the report is generated, list directories and files recursively under the `rule_output_path` using `aws s3 ls` with the `--recursive` option.
```
! aws s3 ls {rule_output_path} --recursive
```
This should return a complete list of files under autogenerated folders that are named `CreateXgboostReport` and `ProfilerReport-1234567890`. The XGBoost training report is stored in the `CreateXgboostReport`, and the profiling report is stored in the `ProfilerReport-1234567890` folder. To learn more about the profiling report generated by default with the XGBoost training job, see [SageMaker Debugger interactive report](debugger-profiling-report.md).
![\[An example of rule output.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-xgboost-report-ls.png)
The `xgboost_report.html` is an autogenerated XGBoost training report by Debugger. The `xgboost_report.ipynb` is a Jupyter notebook that's used to aggregate training results into the report. You can download all of the files, browse the HTML report file, and modify the report using the notebook.
1. Download the files recursively using `aws s3 cp`. The following command saves all of the rule output files to the `ProfilerReport-1234567890` folder under the current working directory.
```
! aws s3 cp {rule_output_path} ./ --recursive
```
**Tip**
If you are using a Jupyter notebook server, run `!pwd` to verify the current working directory.
1. Under the `/CreateXgboostReport` directory, open `xgboost_report.html`. If you are using JupyterLab, choose **Trust HTML** to see the autogenerated Debugger training report.
![\[An example of rule output.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-xgboost-report-open-trust.png)
1. Open the `xgboost_report.ipynb` file to explore how the report is generated. You can customize and extend the training report using the Jupyter notebook file.
------
#### [ Download using the Amazon S3 console ]
1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/).
1. Search for the base S3 bucket. For example, if you haven't specified any base job name, the base S3 bucket name should be in the following format: `sagemaker--111122223333`. Look up the base S3 bucket through the **Find bucket by name** field.
![\[The Find bucket by name field in the Amazon S3 console.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-report-download-s3console-0.png)
1. In the base S3 bucket, look up the training job name by entering your job name prefix in **Find objects by prefix** and then choosing the training job name.
![\[The Find objects by prefix field in the Amazon S3 console.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-report-download-s3console-1.png)
1. In the training job's S3 bucket, choose **rule-output/** subfolder. There must be three subfolders for training data collected by Debugger: **debug-output/**, **profiler-output/**, and **rule-output/**.
![\[An example to the rule output S3 bucket URI.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-report-download-s3console-2.png)
1. In the **rule-output/** folder, choose the **CreateXgboostReport/** folder. The folder contains **xbgoost\$1report.html** (the autogenerated report in html) and **xbgoost\$1report.ipynb** (a Jupyter notebook with scripts that are used for generating the report).
1. Choose the **xbgoost\$1report.html** file, choose **Download actions**, and then choose **Download**.
![\[An example to the rule output S3 bucket URI.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-xgboost-report-s3-download.png)
1. Open the downloaded **xbgoost\$1report.html** file in a web browser.
------
# Debugger XGBoost training report walkthrough
:111122223333:SMDebugRules`.
1. For **Protocol**, choose **Email** or **SMS**.
1. For **Endpoint**, enter the endpoint value, such as an email address or a phone number that you want to receive notifications.
**Note**
Make sure you type the correct email address and phone number. Phone numbers must include `+`, a country code, and phone number, with no special characters or spaces. For example, the phone number \$11 (222) 333-4444 is formatted as **\$112223334444**.
1. Skip all other optional settings and choose **Create subscription**. If you want to learn more about the optional settings, see [Subscribing to an Amazon SNS topic](https://docs.aws.amazon.com/sns/latest/dg/sns-create-subscribe-endpoint-to-topic.html).
After you subscribe to the **SMDebugRules** topic, you receive the following confirmation message in email or by phone:
![\[A subscription confirmation email message for the Amazon SNS SMDebugRules topic.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-built-in-action-subscription-confirmation.png)
For more information about Amazon SNS, see [Mobile text messaging (SMS)](https://docs.aws.amazon.com/sns/latest/dg/sns-mobile-phone-number-as-subscriber.html) and [Email notifications](https://docs.aws.amazon.com/sns/latest/dg/sns-email-notifications.html) in the *Amazon SNS Developer Guide*.
## Set up your IAM role to attach required policies
",
"collection_names": "losses",
"tensor_regex": ".*"
},
collections_to_save=[
CollectionConfig(
name="losses",
parameters={
"save_interval": "500"
}
)
]
)
]
```
## StalledTrainingRule
"
},
collections_to_save=[
CollectionConfig(
name="losses",
parameters={
"save_interval": "500"
}
)
]
)
]
```
## TensorVariance
//debug-output",
"CollectionConfigurations": [
{
"CollectionName": "gradients",
"CollectionParameters" : {
"save_interval": "500"
}
}
]
}
```
This will make the training job save the tensor collection, `gradients`, every `save_interval` of 500 steps. To find available `CollectionName` values, see [Debugger Built-in Collections](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/api.md#built-in-collections) in the *SMDebug client library documentation*. To find available `CollectionParameters` parameter keys and values, see the [https://sagemaker.readthedocs.io/en/stable/api/training/debugger.html#sagemaker.debugger.CollectionConfig](https://sagemaker.readthedocs.io/en/stable/api/training/debugger.html#sagemaker.debugger.CollectionConfig) class in the *SageMaker Python SDK documentation*.
**To enable Debugger rules for debugging the output tensors**
The following `DebugRuleConfigurations` API example shows how to run the built-in `VanishingGradient` rule on the saved `gradients` collection.
```
"DebugRuleConfigurations": [
{
"RuleConfigurationName": "VanishingGradient",
"RuleEvaluatorImage": "503895931360.dkr.ecr.us-east-1.amazonaws.com/sagemaker-debugger-rules:latest",
"RuleParameters": {
"rule_to_invoke": "VanishingGradient",
"threshold": "20.0"
}
}
]
```
With a configuration like the one in this sample, Debugger starts a rule evaluation job for your training job using the `VanishingGradient` rule on the collection of `gradients` tensor. To find a complete list of available Docker images for using the Debugger rules, see [Docker images for Debugger rules](debugger-reference.md#debugger-docker-images-rules). To find the key-value pairs for `RuleParameters`, see [List of Debugger built-in rules](debugger-built-in-rules.md).
## To configure a Debugger built-in rule for profiling system and framework metrics
//profiler-output",
// Available values for ProfilingIntervalInMilliseconds: 100, 200, 500, 1000 (1 second), 5000 (5 seconds), and 60000 (1 minute) milliseconds.
"ProfilingIntervalInMilliseconds": 500,
"ProfilingParameters": {
"DataloaderProfilingConfig": "{ \"StartStep\": 5, \"NumSteps\": 3, \"MetricsRegex\": \".*\" }",
"DetailedProfilingConfig": "{ \"StartStep\": 5, \"NumSteps\": 3 }",
// For PythonProfilingConfig,
// available ProfilerName options: cProfile, Pyinstrument
// available cProfileTimer options only when using cProfile: cpu, off_cpu, total_time
"PythonProfilingConfig": "{ \"StartStep\": 5, \"NumSteps\": 3, \"ProfilerName\": \"cProfile\", \"cProfileTimer\": \"total_time\" }",
// Optional. Local path for profiling outputs
"LocalPath": "/opt/ml/output/profiler/"
}
}
```
------
#### [ Target Time Duration ]
```
"ProfilerConfig": {
// Optional. Path to an S3 bucket to save profiling outputs
"S3OutputPath": "s3:////profiler-output",
// Available values for ProfilingIntervalInMilliseconds: 100, 200, 500, 1000 (1 second), 5000 (5 seconds), and 60000 (1 minute) milliseconds.
"ProfilingIntervalInMilliseconds": 500,
"ProfilingParameters": {
"DataloaderProfilingConfig": "{ \"StartTimeInSecSinceEpoch\": 12345567789, \"DurationInSeconds\": 10, \"MetricsRegex\": \".*\" }",
"DetailedProfilingConfig": "{ \"StartTimeInSecSinceEpoch\": 12345567789, \"DurationInSeconds\": 10 }",
// For PythonProfilingConfig,
// available ProfilerName options: cProfile, Pyinstrument
// available cProfileTimer options only when using cProfile: cpu, off_cpu, total_time
"PythonProfilingConfig": "{ \"StartTimeInSecSinceEpoch\": 12345567789, \"DurationInSeconds\": 10, \"ProfilerName\": \"cProfile\", \"cProfileTimer\": \"total_time\" }",
// Optional. Local path for profiling outputs
"LocalPath": "/opt/ml/output/profiler/"
}
}
```
------
**To enable Debugger rules for profiling the metrics**
The following example code shows how to configure the `ProfilerReport` rule.
```
"ProfilerRuleConfigurations": [
{
"RuleConfigurationName": "ProfilerReport",
"RuleEvaluatorImage": "895741380848.dkr.ecr.us-west-2.amazonaws.com/sagemaker-debugger-rules:latest",
"RuleParameters": {
"rule_to_invoke": "ProfilerReport",
"CPUBottleneck_cpu_threshold": "90",
"IOBottleneck_threshold": "90"
}
}
]
```
To find a complete list of available Docker images for using the Debugger rules, see [Docker images for Debugger rules](debugger-reference.md#debugger-docker-images-rules). To find the key-value pairs for `RuleParameters`, see [List of Debugger built-in rules](debugger-built-in-rules.md).
## Update Debugger profiling configuration using the `UpdateTrainingJob` API
//debug-output",
"CollectionConfigurations": [
{
"CollectionName": "relu_activations",
"CollectionParameters": {
"include_regex": "relu",
"save_interval": "500",
"end_step": "5000"
}
}
]
},
"DebugRulesConfigurations": [
{
"RuleConfigurationName": "improper_activation_job",
"RuleEvaluatorImage": "552407032007.dkr.ecr.ap-south-1.amazonaws.com/sagemaker-debugger-rule-evaluator:latest",
"InstanceType": "ml.c4.xlarge",
"VolumeSizeInGB": 400,
"RuleParameters": {
"source_s3_uri": "s3://bucket/custom_rules.py",
"rule_to_invoke": "ImproperActivation",
"collection_names": "relu_activations"
}
}
]
```
To find a complete list of available Docker images for using the Debugger rules, see [Docker images for Debugger rules](debugger-reference.md#debugger-docker-images-rules). To find the key-value pairs for `RuleParameters`, see [List of Debugger built-in rules](debugger-built-in-rules.md).
# SDK for Python (Boto3)
//debug-output',
'CollectionConfigurations': [
{
'CollectionName': 'gradients',
'CollectionParameters' : {
'train.save_interval': '500',
'eval.save_interval': '50'
}
}
]
}
```
This will make the training job save a tensor collection, `gradients`, every `save_interval` of 500 steps. To find available `CollectionName` values, see [Debugger Built-in Collections](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/api.md#built-in-collections) in the *SMDebug client library documentation*. To find available `CollectionParameters` parameter keys and values, see the [https://sagemaker.readthedocs.io/en/stable/api/training/debugger.html#sagemaker.debugger.CollectionConfig](https://sagemaker.readthedocs.io/en/stable/api/training/debugger.html#sagemaker.debugger.CollectionConfig) class in the *SageMaker Python SDK documentation*.
**To enable Debugger rules for debugging the output tensors**
The following `DebugRuleConfigurations` API example shows how to run the built-in `VanishingGradient` rule on the saved `gradients` collection.
```
DebugRuleConfigurations=[
{
'RuleConfigurationName': 'VanishingGradient',
'RuleEvaluatorImage': '895741380848.dkr.ecr.us-west-2.amazonaws.com/sagemaker-debugger-rules:latest',
'RuleParameters': {
'rule_to_invoke': 'VanishingGradient',
'threshold': '20.0'
}
}
]
```
With a configuration like the one in this sample, Debugger starts a rule evaluation job for your training job using the `VanishingGradient` rule on the collection of `gradients` tensor. To find a complete list of available Docker images for using the Debugger rules, see [Docker images for Debugger rules](debugger-reference.md#debugger-docker-images-rules). To find the key-value pairs for `RuleParameters`, see [List of Debugger built-in rules](debugger-built-in-rules.md).
## To configure a Debugger built-in rule for profiling system and framework metrics
//profiler-output', # Optional. Path to an S3 bucket to save profiling outputs
# Available values for ProfilingIntervalInMilliseconds: 100, 200, 500, 1000 (1 second), 5000 (5 seconds), and 60000 (1 minute) milliseconds.
'ProfilingIntervalInMilliseconds': 500,
'ProfilingParameters': {
'DataloaderProfilingConfig': '{
"StartStep": 5,
"NumSteps": 3,
"MetricsRegex": ".*"
}',
'DetailedProfilingConfig': '{
"StartStep": 5,
"NumSteps": 3
}',
'PythonProfilingConfig': '{
"StartStep": 5,
"NumSteps": 3,
"ProfilerName": "cprofile", # Available options: cprofile, pyinstrument
"cProfileTimer": "total_time" # Include only when using cprofile. Available options: cpu, off_cpu, total_time
}',
'LocalPath': '/opt/ml/output/profiler/' # Optional. Local path for profiling outputs
}
}
```
------
#### [ Target Time Duration ]
```
ProfilerConfig={
'S3OutputPath': 's3:////profiler-output', # Optional. Path to an S3 bucket to save profiling outputs
# Available values for ProfilingIntervalInMilliseconds: 100, 200, 500, 1000 (1 second), 5000 (5 seconds), and 60000 (1 minute) milliseconds.
'ProfilingIntervalInMilliseconds': 500,
'ProfilingParameters': {
'DataloaderProfilingConfig': '{
"StartTimeInSecSinceEpoch": 12345567789,
"DurationInSeconds": 10,
"MetricsRegex": ".*"
}',
'DetailedProfilingConfig': '{
"StartTimeInSecSinceEpoch": 12345567789,
"DurationInSeconds": 10
}',
'PythonProfilingConfig': '{
"StartTimeInSecSinceEpoch": 12345567789,
"DurationInSeconds": 10,
"ProfilerName": "cprofile", # Available options: cprofile, pyinstrument
"cProfileTimer": "total_time" # Include only when using cprofile. Available options: cpu, off_cpu, total_time
}',
'LocalPath': '/opt/ml/output/profiler/' # Optional. Local path for profiling outputs
}
}
```
------
**To enable Debugger rules for profiling the metrics**
The following example code shows how to configure the `ProfilerReport` rule.
```
ProfilerRuleConfigurations=[
{
'RuleConfigurationName': 'ProfilerReport',
'RuleEvaluatorImage': '895741380848.dkr.ecr.us-west-2.amazonaws.com/sagemaker-debugger-rules:latest',
'RuleParameters': {
'rule_to_invoke': 'ProfilerReport',
'CPUBottleneck_cpu_threshold': '90',
'IOBottleneck_threshold': '90'
}
}
]
```
To find a complete list of available Docker images for using the Debugger rules, see [Docker images for Debugger rules](debugger-reference.md#debugger-docker-images-rules). To find the key-value pairs for `RuleParameters`, see [List of Debugger built-in rules](debugger-built-in-rules.md).
## Update Debugger Profiling Configuration Using the `UpdateTrainingJob` API Operation
//debug-output',
'CollectionConfigurations': [
{
'CollectionName': 'relu_activations',
'CollectionParameters': {
'include_regex': 'relu',
'save_interval': '500',
'end_step': '5000'
}
}
]
},
DebugRulesConfigurations=[
{
'RuleConfigurationName': 'improper_activation_job',
'RuleEvaluatorImage': '552407032007.dkr.ecr.ap-south-1.amazonaws.com/sagemaker-debugger-rule-evaluator:latest',
'InstanceType': 'ml.c4.xlarge',
'VolumeSizeInGB': 400,
'RuleParameters': {
'source_s3_uri': 's3://bucket/custom_rules.py',
'rule_to_invoke': 'ImproperActivation',
'collection_names': 'relu_activations'
}
}
]
```
To find a complete list of available Docker images for using the Debugger rules, see [Docker images for Debugger rules](debugger-reference.md#debugger-docker-images-rules). To find the key-value pairs for `RuleParameters`, see [List of Debugger built-in rules](debugger-built-in-rules.md).
# Amazon SageMaker Debugger references
.dkr.ecr..amazonaws.com/:
```
For the account ID in each AWS Region, the Amazon ECR repository name, and the tag value, see the following topics.
**Topics**
+ [
### Amazon SageMaker Debugger image URIs for built-in rule evaluators
](#debuger-built-in-registry-ids)
+ [
### Amazon SageMaker Debugger image URIs for custom rule evaluators
](#debuger-custom-rule-registry-ids)
### Amazon SageMaker Debugger image URIs for built-in rule evaluators
", #must be owned by your organization or Amazon DLCs
role=role,
instance_type="ml.m5.xlarge",
instance_count=1,
output_path=output_path,
max_run=1800,
enable_remote_debug=True
)
```
**To enable remote debugging by updating a training job**
Using the following estimator class methods, you can enable or disable remote debugging while a training job is running when the `SecondaryStatus` of the job is `Downloading` or `Training`.
```
# Enable RemoteDebug
estimator.enable_remote_debug()
# Disable RemoteDebug
estimator.disable_remote_debug()
```
------
#### [ AWS SDK for Python (Boto3) ]
**To enable remote debugging when you create a training job**
To enable remote debugging when you create a new training job, set the value for the `EnableRemoteDebug` key to `True` in the `RemoteDebugConfig` parameter.
```
import boto3
sm = boto3.Session(region_name=region).client("sagemaker")
# Start a training job
sm.create_training_job(
...,
TrainingJobName=job_name,
AlgorithmSpecification={
// Specify a training Docker container image URI
// (Deep Learning Container or your own training container) to TrainingImage.
"TrainingImage": "",
"TrainingInputMode": "File"
},
RoleArn=iam_role_arn,
OutputDataConfig=output_path,
ResourceConfig={
"InstanceType": "ml.m5.xlarge",
"InstanceCount": 1,
"VolumeSizeInGB": 30
},
StoppingCondition={
"MaxRuntimeInSeconds": 86400
},
RemoteDebugConfig={
"EnableRemoteDebug": True
}
)
```
**To enable remote debugging by updating a training job**
Using the `update_traing_job` API, you can enable or disable remote debugging while a training job is running when the `SecondaryStatus` of the job is `Downloading` or `Training`.
```
# Update a training job
sm.update_training_job(
TrainingJobName=job_name,
RemoteDebugConfig={
"EnableRemoteDebug": True # True | False
}
)
```
------
#### [ AWS Command Line Interface (CLI) ]
**To enable remote debugging when you create a training job**
Prepare a `CreateTrainingJob` request file in JSON format, as follows.
```
// train-with-remote-debug.json
{
"TrainingJobName": job_name,
"RoleArn": iam_role_arn,
"AlgorithmSpecification": {
// Specify a training Docker container image URI (Deep Learning Container or your own training container) to TrainingImage.
"TrainingImage": "",
"TrainingInputMode": "File"
},
"OutputDataConfig": {
"S3OutputPath": output_path
},
"ResourceConfig": {
"InstanceType": "ml.m5.xlarge",
"InstanceCount": 1,
"VolumeSizeInGB": 30
},
"StoppingCondition": {
"MaxRuntimeInSeconds": 86400
},
"RemoteDebugConfig": {
"EnableRemoteDebug": True
}
}
```
After saving the JSON file, run the following command in the terminal where you submit the training job. The following example command assumes that the JSON file is named `train-with-remote-debug.json`. If you run it from a Jupyter notebook, add an exclamation point (`!`) to the beginning of the line.
```
aws sagemaker create-training-job \
--cli-input-json file://train-with-remote-debug.json
```
**To enable remote debugging by updating a training job**
Prepare an `UpdateTrainingJob` request file in JSON format, as follows.
```
// update-training-job-with-remote-debug-config.json
{
"TrainingJobName": job_name,
"RemoteDebugConfig": {
"EnableRemoteDebug": True
}
}
```
After saving the JSON file, run the following command in the terminal where you submit the training job. The following example command assumes that the JSON file is named `train-with-remote-debug.json`. If you run it from a Jupyter notebook, add an exclamation point (`!`) to the beginning of the line.
```
aws sagemaker update-training-job \
--cli-input-json file://update-training-job-with-remote-debug-config.json
```
------
## Access your training container
_algo-`, where `algo-` is the name of the container host. If your job is running on a single instance, the host is always `algo-1`. If you run a distributed training job on multiple instances, SageMaker AI creates an equal number of hosts and log streams. For example, if you use 4 instances, SageMaker AI creates `algo-1`, `algo-2`, `algo-3`, and `algo-4`. You must determine which log stream you want to debug, and its host number. To access log streams that are associated with a training job, do the following.
1. Open the Amazon SageMaker AI console at [https://console.aws.amazon.com/sagemaker/](https://console.aws.amazon.com/sagemaker/).
1. In the left navigation pane, choose **Training**, then choose **Training jobs**.
1. From the **Training jobs** list, choose the training job that you want to debug. The training job details page opens.
1. In the **Monitor** section, choose **View logs**. The related training job log stream list opens in the CloudWatch console.
1. Log stream names appear in `/algo--` format, with `algo-` representing the host name.
To learn more about how SageMaker AI manages configuration information for multi-instance distributed training, see [Distributed Training Configuration](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo-running-container.html#your-algorithms-training-algo-running-container-dist-training).
**To access the training container**
Use the following command in terminal to start the SSM session (`[aws ssm start-session](https://docs.aws.amazon.com/cli/latest/reference/ssm/start-session.html)`) and connect to the training container.
```
aws ssm start-session --target sagemaker-training-job:_algo-
```
For example, if the training job name is `training-job-test-remote-debug` and the host name is `algo-1`, the target ID becomes `sagemaker-training-job:training-job-test-remote-debug_algo-1`. If the output of this command is similar to `Starting session with SessionId:xxxxx`, the connection is successful.
### SSM access with AWS PrivateLink
/algo--/ssm`. For example, if you create a two-node training job named `training-job-test-remote-debug`, the training job log `training-job-test-remote-debug/algo--` and multiple SSM agent error logs `training-job-test-remote-debug/algo--/ssm` are uploaded to your CloudWatch Logs. In this example, you can review the `*/ssm` log streams to troubleshoot SSM issues.
```
training-job-test-remote-debug/algo-1-1680535238
training-job-test-remote-debug/algo-2-1680535238
training-job-test-remote-debug/algo-1-1680535238/ssm
training-job-test-remote-debug/algo-2-1680535238/ssm
```
## Considerations
*.amazonaws.com/pytorch-training:2.2.0-gpu-py310-cu121-ubuntu20.04-sagemaker |
| 2.1.0 | *763104351884*.dkr.ecr.**.amazonaws.com/pytorch-training:2.1.0-gpu-py310-cu121-ubuntu20.04-sagemaker |
| 2.0.1 | *763104351884*.dkr.ecr.**.amazonaws.com/pytorch-training:2.0.1-gpu-py310-cu118-ubuntu20.04-sagemaker *763104351884*.dkr.ecr.**.amazonaws.com/pytorch-training:2.0.1-gpu-py310-cu121-ubuntu20.04-sagemaker |
| 1.13.1 | *763104351884*.dkr.ecr.**.amazonaws.com/pytorch-training:1.13.1-gpu-py39-cu117-ubuntu20.04-sagemaker |
### TensorFlow images
*.amazonaws.com/tensorflow-training:2.13.0-gpu-py310-cu118-ubuntu20.04-sagemaker |
| 2.12.0 | *763104351884*.dkr.ecr.**.amazonaws.com/tensorflow-training:2.12.0-gpu-py310-cu118-ubuntu20.04-sagemaker |
| 2.11.0 | *763104351884*.dkr.ecr.**.amazonaws.com/tensorflow-training:2.11.0-gpu-py39-cu112-ubuntu20.04-sagemaker |
**Important**
Distribution and maintenance of the framework containers in the preceding tables are under the [Framework Support Policy](https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/support-policy.html) managed by the AWS Deep Learning Containers service. We highly recommend you to upgrade to the [currently supported framework versions](https://aws.amazon.com/releasenotes/dlc-support-policy/), if you are using prior framework versions that are no longer supported.
**Note**
If you want to use SageMaker Profiler for other framework images or your own Docker images, you can install SageMaker Profiler using the SageMaker Profiler Python package binary files provided in the following section.
## SageMaker Profiler Python package binary files
-<12digit_account_id>///`. There are three subfolders created by Debugger: `debug-output`, `profiler-output`, and `rule-output`. You can also retrieve the default S3 bucket URIs using the [SageMaker AI estimator classmethods](debugger-estimator-classmethods.md).
See the following topics to find out how to configure the Debugger-specific parameters in detail.
**Topics**
+ [
## Code template for configuring a SageMaker AI estimator object with the SageMaker Debugger Python modules in the SageMaker AI Python SDK
](#debugger-configuration-structure-profiler)
+ [
# Configure settings for basic profiling of system resource utilization
](debugger-configure-system-monitoring.md)
+ [
# Estimator configuration for framework profiling
](debugger-configure-framework-profiling.md)
+ [
# Updating Debugger system monitoring and framework profiling configuration while a training job is running
](debugger-update-monitoring-profiling.md)
+ [
# Turn off Debugger
](debugger-turn-off-profiling.md)
# Configure settings for basic profiling of system resource utilization
-<12digit_account_id>//profiler-output/`.
The following code example shows how to set up the `profiler_config` parameter with a system monitoring time interval of 1000 milliseconds.
```
from sagemaker.debugger import ProfilerConfig
profiler_config=ProfilerConfig(
system_monitor_interval_millis=1000
)
```
+ `system_monitor_interval_millis` (int) – Specify the monitoring intervals in milliseconds to record system metrics. Available values are 100, 200, 500, 1000 (1 second), 5000 (5 seconds), and 60000 (1 minute) milliseconds. The default value is 500 milliseconds.
To see the progress of system monitoring, see [Open the Amazon SageMaker Debugger Insights dashboard](debugger-on-studio-insights.md).
# Estimator configuration for framework profiling
-<12digit_account_id>//profiler-output/`.
**Topics**
+ [
# Default framework profiling
](debugger-configure-framework-profiling-basic.md)
+ [
# Default system monitoring and customized framework profiling for target steps or a target time range
](debugger-configure-framework-profiling-range.md)
+ [
# Default system monitoring and customized framework profiling with different profiling options
](debugger-configure-framework-profiling-options.md)
# Default framework profiling
_ = value
)
)
]
```
If you trigger this ProfilerReport rule without any customized parameter as shown in the following example code, then the ProfilerReport rule triggers all of the built-in rules for monitoring and profiling with their default parameter values.
```
rules=[ProfilerRule.sagemaker(rule_configs.ProfilerReport())]
```
The following example code shows how to specify and adjust the CPUBottleneck rule's `cpu_threshold` parameter and the IOBottleneck rule's `threshold` parameter.
```
rules=[
ProfilerRule.sagemaker(
rule_configs.ProfilerReport(
CPUBottleneck_cpu_threshold = 90,
IOBottleneck_threshold = 90
)
)
]
```
To explore what's in the profiler report, see [SageMaker Debugger Profiling Report](https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-profiling-report.html). Also, because this rule activates all of the profiling rules, you can also check the rule analysis status using the [SageMaker Debugger UI in SageMaker Studio Experiments](https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-on-studio.html).
Parameter Descriptions for the OverallSystemUsage Rule
| Parameter Name | Description |
| --- | --- |
| base\$1trial | The base trial training job name. This parameter is automatically set to the current training job by Amazon SageMaker Debugger. **Required** Valid values: String |
| \$1 | Customizable parameter to adjust thresholds of other built-in monitoring and profiling rules. **Optional** Default value: `None` |
## BatchSize
//rule-output`. Configure the rule output path as follows:
```
rule_output_path = estimator.output_path + estimator.latest_training_job.job_name + "/rule-output"
```
1. To check if the report is generated, list directories and files recursively under the `rule_output_path` using `aws s3 ls` with the `--recursive` option.
```
! aws s3 ls {rule_output_path} --recursive
```
This should return a complete list of files under an autogenerated folder that's named as `ProfilerReport-1234567890`. The folder name is a combination of strings: `ProfilerReport` and a unique 10-digit tag based on the Unix timestamp when the ProfilerReport rule is initiated.
![\[An example of rule output\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-rule-output-ls-example.png)
The `profiler-report.html` is an autogenerated profiling report by Debugger. The remaining files are the built-in rule analysis components stored in JSON and a Jupyter notebook that are used to aggregate them into the report.
1. Download the files recursively using `aws s3 cp`. The following command saves all of the rule output files to the `ProfilerReport-1234567890` folder under the current working directory.
```
! aws s3 cp {rule_output_path} ./ --recursive
```
**Tip**
If using a Jupyter notebook server, run `!pwd` to double check the current working directory.
1. Under the `/ProfilerReport-1234567890/profiler-output` directory, open `profiler-report.html`. If using JupyterLab, choose **Trust HTML** to see the autogenerated Debugger profiling report.
![\[An example of rule output\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-rule-output-open-html.png)
1. Open the `profiler-report.ipynb` file to explore how the report is generated. You can also customize and extend the profiling report using the Jupyter notebook file.
------
#### [ Download using Amazon S3 Console ]
1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/).
1. Search for the base S3 bucket. For example, if you haven't specified any base job name, the base S3 bucket name should be in the following format: `sagemaker--111122223333`. Look up the base S3 bucket through the *Find bucket by name* field.
![\[An example to the rule output S3 bucket URI\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-report-download-s3console-0.png)
1. In the base S3 bucket, look up the training job name by specifying your job name prefix into the *Find objects by prefix* input field. Choose the training job name.
![\[An example to the rule output S3 bucket URI\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-report-download-s3console-1.png)
1. In the training job's S3 bucket, there must be three subfolders for training data collected by Debugger: **debug-output/**, **profiler-output/**, and **rule-output/**. Choose **rule-output/**.
![\[An example to the rule output S3 bucket URI\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-report-download-s3console-2.png)
1. In the **rule-output/** folder, choose **ProfilerReport-1234567890**, and choose **profiler-output/** folder. The **profiler-output/** folder contains **profiler-report.html** (the autogenerated profiling report in html), **profiler-report.ipynb** (a Jupyter notebook with scripts that are used for generating the report), and a **profiler-report/** folder (contains rule analysis JSON files that are used as components of the report).
1. Select the **profiler-report.html** file, choose **Actions**, and **Download**.
![\[An example to the rule output S3 bucket URI\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/debugger/debugger-report-download-s3console-3.png)
1. Open the downloaded **profiler-report.html** file in a web browser.
------
**Note**
If you started your training job without configuring the Debugger-specific parameters, Debugger generates the report based only on the system monitoring rules because the Debugger parameters are not configured to save framework metrics. To enable framework metrics profiling and receive an extended Debugger profiling report, configure the `profiler_config` parameter when constructing or updating SageMaker AI estimators.
To learn how to configure the `profiler_config` parameter before starting a training job, see [Estimator configuration for framework profiling](debugger-configure-framework-profiling.md).
To update the current training job and enable framework metrics profiling, see [Update Debugger Framework Profiling Configuration](debugger-update-monitoring-profiling.md).
# Debugger profiling report walkthrough
.amazonaws.com/smdistributed-modelparallel:2.4.1-gpu-py311-cu121 | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.4.1/cu121/2024-10-09/smdistributed\$1dataparallel-2.5.0-cp311-cp311-linux\$1x86\$164.whl |
| v2.3.0 | smdistributed-dataparallel==v2.3.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121-ubuntu20.04-sagemaker | Currently not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.3.0/cu121/2024-05-23/smdistributed\$1dataparallel-2.3.0-cp311-cp311-linux\$1x86\$164.whl |
| v2.2.0 | smdistributed-dataparallel==v2.2.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.2.0-gpu-py310-cu121-ubuntu20.04-sagemaker | 658645717510.dkr.ecr..amazonaws.com/smdistributed-modelparallel:2.2.0-gpu-py310-cu121 | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.2.0/cu121/2024-03-04/smdistributed\$1dataparallel-2.2.0-cp310-cp310-linux\$1x86\$164.whl |
| v2.1.0 | smdistributed-dataparallel==v2.1.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.1.0-gpu-py310-cu121-ubuntu20.04-sagemaker | 658645717510.dkr.ecr..amazonaws.com/smdistributed-modelparallel:2.1.2-gpu-py310-cu121 | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.1.0/cu121/2024-02-04/smdistributed\$1dataparallel-2.1.0-cp310-cp310-linux\$1x86\$164.whl |
| v2.0.1 | smdistributed-dataparallel==v2.0.1 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.0.1-gpu-py310-cu118-ubuntu20.04-sagemaker | Not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.0.1/cu118/2023-12-07/smdistributed\$1dataparallel-2.0.2-cp310-cp310-linux\$1x86\$164.whl |
| v2.0.0 | smdistributed-dataparallel==v1.8.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.0.0-gpu-py310-cu118-ubuntu20.04-sagemaker | Not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.0.0/cu118/2023-03-20/smdistributed\$1dataparallel-1.8.0-cp310-cp310-linux\$1x86\$164.whl |
| v1.13.1 | smdistributed-dataparallel==v1.7.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.13.1-gpu-py39-cu117-ubuntu20.04-sagemaker | Not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/1.13.1/cu117/2023-01-09/smdistributed\$1dataparallel-1.7.0-cp39-cp39-linux\$1x86\$164.whl |
| v1.12.1 | smdistributed-dataparallel==v1.6.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.12.1-gpu-py38-cu113-ubuntu20.04-sagemaker | Not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/1.12.1/cu113/2022-12-05/smdistributed\$1dataparallel-1.6.0-cp38-cp38-linux\$1x86\$164.whl |
| v1.12.0 | smdistributed-dataparallel==v1.5.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.12.0-gpu-py38-cu113-ubuntu20.04-sagemaker | Not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/1.12.0/cu113/2022-07-01/smdistributed\$1dataparallel-1.5.0-cp38-cp38-linux\$1x86\$164.whl |
| v1.11.0 | smdistributed-dataparallel==v1.4.1 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.11.0-gpu-py38-cu113-ubuntu20.04-sagemaker | Not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/1.11.0/cu113/2022-04-14/smdistributed\$1dataparallel-1.4.1-cp38-cp38-linux\$1x86\$164.whl |
\$1\$1 The URLs of the binary files are for installing the SMDDP library in custom containers. For more information, see [Create your own Docker container with the SageMaker AI distributed data parallel library](data-parallel-bring-your-own-container.md).
**Note**
The SMDDP library is available in AWS Regions where the [SageMaker AI Framework Containers](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#sagemaker-framework-containers-sm-support-only) and the [SMP Docker images](distributed-model-parallel-support-v2.md) are in service.
**Note**
The SMDDP library v1.4.0 and later works as a backend of PyTorch distributed (torch.distributed) data parallelism (torch.parallel.DistributedDataParallel). In accordance with the change, the following [smdistributed APIs](https://sagemaker.readthedocs.io/en/stable/api/training/sdp_versions/latest/smd_data_parallel_pytorch.html#pytorch-api) for the PyTorch distributed package have been deprecated.
`smdistributed.dataparallel.torch.distributed` is deprecated. Use the [torch.distributed](https://pytorch.org/docs/stable/distributed.html) package instead.
`smdistributed.dataparallel.torch.parallel.DistributedDataParallel` is deprecated. Use the [torch.nn.parallel.DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) API instead.
If you need to use the previous versions of the library (v1.3.0 or before), see the [archived SageMaker AI distributed data parallelism documentation](https://sagemaker.readthedocs.io/en/stable/api/training/sdp_versions/latest.html#documentation-archive) in the *SageMaker AI Python SDK documentation*.
### PyTorch Lightning
.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121-ubuntu20.04-sagemaker | Currently not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.3.0/cu121/2024-05-23/smdistributed\$1dataparallel-2.3.0-cp311-cp311-linux\$1x86\$164.whl |
| 2.2.0 | 2.2.0 | smdistributed-dataparallel==v2.2.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.2.0-gpu-py310-cu121-ubuntu20.04-sagemaker | 658645717510.dkr.ecr..amazonaws.com/smdistributed-modelparallel:2.2.0-gpu-py310-cu121 | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.2.0/cu121/2024-03-04/smdistributed\$1dataparallel-2.2.0-cp310-cp310-linux\$1x86\$164.whl |
| 2.1.2 | 2.1.0 | smdistributed-dataparallel==v2.1.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.1.0-gpu-py310-cu121-ubuntu20.04-sagemaker | 658645717510.dkr.ecr..amazonaws.com/smdistributed-modelparallel:2.1.2-gpu-py310-cu121 | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.1.0/cu121/2024-02-04/smdistributed\$1dataparallel-2.1.0-cp310-cp310-linux\$1x86\$164.whl |
| 2.1.0 | 2.0.1 | smdistributed-dataparallel==v2.0.1 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:2.0.1-gpu-py310-cu118-ubuntu20.04-sagemaker | Not available | https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.0.1/cu118/2023-12-07/smdistributed\$1dataparallel-2.0.2-cp310-cp310-linux\$1x86\$164.whl |
**PyTorch Lightning v1**
| PyTorch Lightning version | PyTorch version | SMDDP library version | SageMaker AI Framework Container images pre-installed with SMDDP | URL of the binary file\$1\$1 |
| --- | --- | --- | --- | --- |
| 1.7.2 1.7.0 1.6.4 1.6.3 1.5.10 | 1.12.0 | smdistributed-dataparallel==v1.5.0 | 763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.12.0-gpu-py38-cu113-ubuntu20.04-sagemaker | https://smdataparallel.s3.amazonaws.com/binary/pytorch/1.12.0/cu113/2022-07-01/smdistributed\$1dataparallel-1.5.0-cp38-cp38-linux\$1x86\$164.whl |
\$1\$1 The URLs of the binary files are for installing the SMDDP library in custom containers. For more information, see [Create your own Docker container with the SageMaker AI distributed data parallel library](data-parallel-bring-your-own-container.md).
**Note**
PyTorch Lightning and its utility libraries such as Lightning Bolts are not preinstalled in the PyTorch DLCs. When you construct a SageMaker AI PyTorch estimator and submit a training job request in [Step 2](https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-use-api.html#data-parallel-framework-estimator), you need to provide `requirements.txt` to install `pytorch-lightning` and `lightning-bolts` in the SageMaker AI PyTorch training container.
```
# requirements.txt
pytorch-lightning
lightning-bolts
```
For more information about specifying the source directory to place the `requirements.txt` file along with your training script and a job submission, see [Using third-party libraries](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#id12) in the *Amazon SageMaker AI Python SDK documentation*.
### Hugging Face Transformers
.amazonaws.com/pytorch-training:
ENV PATH="/opt/ml/code:${PATH}"
# this environment variable is used by the SageMaker AI PyTorch container to determine our user code directory.
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code
# /opt/ml and all subdirectories are utilized by SageMaker AI, use the /code subdirectory to store your user code.
COPY train.py /opt/ml/code/train.py
# Defines cifar10.py as script entrypoint
ENV SAGEMAKER_PROGRAM train.py
```
You can further customize your own Docker container to work with SageMaker AI using the [SageMaker Training toolkit](https://github.com/aws/sagemaker-training-toolkit) and the binary file of the SageMaker AI distributed data parallel library. To learn more, see the instructions in the following section.
# Create your own Docker container with the SageMaker AI distributed data parallel library
.amazonaws.com/smdistributed-modelparallel:2.4.1-gpu-py311-cu121
```
For Regions where the SMP Docker images are available, see [AWS Regions](distributed-model-parallel-support-v2.md#distributed-model-parallel-availablity-zone-v2).
**Binary file of this release**
You can download or install the library using the following URL.
```
https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.4.1/cu121/2024-10-09/smdistributed_dataparallel-2.5.0-cp311-cp311-linux_x86_64.whl
```
## The SageMaker AI distributed data parallelism library v2.3.0
.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121-ubuntu20.04-sagemaker
```
For a complete list of versions of the SMDDP library and the pre-built containers, see [Supported frameworks, AWS Regions, and instances types](distributed-data-parallel-support.md).
**Binary file of this release**
You can download or install the library using the following URL.
```
https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.3.0/cu121/2024-05-23/smdistributed_dataparallel-2.3.0-cp311-cp311-linux_x86_64.whl
```
**Other changes**
+ The SMDDP library v2.2.0 is integrated into the SageMaker AI framework container for PyTorch v2.2.0.
## The SageMaker AI distributed data parallelism library v2.2.0
.amazonaws.com/smdistributed-modelparallel:2.2.0-gpu-py310-cu121
```
For Regions where the SMP Docker images are available, see [AWS Regions](distributed-model-parallel-support-v2.md#distributed-model-parallel-availablity-zone-v2).
**Binary file of this release**
You can download or install the library using the following URL.
```
https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.2.0/cu121/2024-03-04/smdistributed_dataparallel-2.2.0-cp310-cp310-linux_x86_64.whl
```
## The SageMaker AI distributed data parallelism library v2.1.0
.amazonaws.com/pytorch-training:2.1.0-gpu-py310-cu121-ubuntu20.04-sagemaker
```
**Integration into Docker containers distributed by the SageMaker AI model parallelism (SMP) library**
This version of the SMDDP library is migrated to [The SageMaker model parallelism library v2.1.0](model-parallel-release-notes.md#model-parallel-release-notes-20240206).
```
658645717510.dkr.ecr..amazonaws.com/smdistributed-modelparallel:2.1.2-gpu-py310-cu121
```
For Regions where the SMP Docker images are available, see [AWS Regions](distributed-model-parallel-support-v2.md#distributed-model-parallel-availablity-zone-v2).
**Binary file of this release**
You can download or install the library using the following URL.
```
https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.1.0/cu121/2024-02-04/smdistributed_dataparallel-2.1.0-cp310-cp310-linux_x86_64.whl
```
## The SageMaker AI distributed data parallelism library v2.0.1
.amazonaws.com/pytorch-training:2.0.1-gpu-py310-cu118-ubuntu20.04-sagemaker
```
**Binary file of this release**
You can download or install the library using the following URL.
```
https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.0.1/cu118/2023-12-07/smdistributed_dataparallel-2.0.2-cp310-cp310-linux_x86_64.whl
```
**Other changes**
+ Starting from this release, documentation for the SMDDP library is fully available in this *Amazon SageMaker AI Developer Guide*. In favor of the complete developer guide for SMDDP v2 housed in the *Amazon SageMaker AI Developer Guide*, documentation for the [additional reference for SMDDP v1.x](https://sagemaker.readthedocs.io/en/stable/api/training/smd_data_parallel.html) in the *SageMaker AI Python SDK documentation* is no longer supported. If you still need SMP v1.x documentation, see the following snapshot of the documentation at [SageMaker Python SDK v2.212.0 documentation](https://sagemaker.readthedocs.io/en/v2.212.0/api/training/distributed.html#the-sagemaker-distributed-data-parallel-library).
# SageMaker model parallelism library v2
" # For using prior versions, specify the SMP image URI directly.
entry_point="your-training-script.py", # Pass the training script you adapted with SMP from Step 1.
... # Configure other required and optional parameters
distribution={
"torch_distributed": { "enabled": True },
"smdistributed": {
"modelparallel": {
"enabled": True,
"parameters": {
"hybrid_shard_degree": Integer,
"sm_activation_offloading": Boolean,
"activation_loading_horizon": Integer,
"fsdp_cache_flush_warnings": Boolean,
"allow_empty_shards": Boolean,
"tensor_parallel_degree": Integer,
"expert_parallel_degree": Integer,
"random_seed": Integer
}
}
}
}
)
```
**Important**
For using one of the prior versions of PyTorch or SMP instead of the latest, you need to specify the SMP Docker image directly using the `image_uri` argument instead of the `framework_version` and `py_version` pair. The following is an example of
```
estimator = PyTorch(
...,
image_uri="658645717510.dkr.ecr.us-west-2.amazonaws.com/smdistributed-modelparallel:2.2.0-gpu-py310-cu121"
)
```
To find SMP Docker image URIs, see [Supported frameworks](distributed-model-parallel-support-v2.md#distributed-model-parallel-supported-frameworks-v2).
------
#### [ SageMaker HyperPod ]
Before you start, make sure if the following prerequisites are met.
+ An Amazon FSx shared directory mounted (`/fsx`) to your HyperPod cluster.
+ Conda installed in the FSx shared directory. To learn how to install Conda, use the instructions at [Installing on Linux](https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html) in the *Conda User Guide*.
+ `cuda11.8` or `cuda12.1` installed on the head and compute nodes of your HyperPod cluster.
If the prerequisites are all met, proceed to the following instructions on launching a workload with SMP v2 on a HyperPod cluster.
1. Prepare an `smp_config.json` file that contains a dictionary of [SMP v2 core feature configuration parameters](distributed-model-parallel-v2-reference.md#distributed-model-parallel-v2-reference-init-config). Make sure that you upload this JSON file to where you store your training script, or the path you specified to the `torch.sagemaker.init()` module in [Step 1](#model-parallel-adapt-pytorch-script-v2). If you’ve already passed the configuration dictionary to the `torch.sagemaker.init()` module in the training script in [Step 1](#model-parallel-adapt-pytorch-script-v2), you can skip this step.
```
// smp_config.json
{
"hybrid_shard_degree": Integer,
"sm_activation_offloading": Boolean,
"activation_loading_horizon": Integer,
"fsdp_cache_flush_warnings": Boolean,
"allow_empty_shards": Boolean,
"tensor_parallel_degree": Integer,
"expert_parallel_degree": Integer,
"random_seed": Integer
}
```
1. Upload the `smp_config.json` file to a directory in your file system. The directory path must match with the path you specified in [Step 1](#model-parallel-adapt-pytorch-script-v2). If you’ve already passed the configuration dictionary to the `torch.sagemaker.init()` module in the training script, you can skip this step.
1. On the compute nodes of your cluster, start a terminal session with the following command.
```
sudo su -l ubuntu
```
1. Create a Conda environment on the compute nodes. The following code is an example script of creating a Conda environment and installing SMP, [SMDDP](data-parallel.md), CUDA, and other dependencies.
```
# Run on compute nodes
SMP_CUDA_VER=<11.8 or 12.1>
source /fsx//miniconda3/bin/activate
export ENV_PATH=/fsx//miniconda3/envs/
conda create -p ${ENV_PATH} python=3.10
conda activate ${ENV_PATH}
# Verify aws-cli is installed: Expect something like "aws-cli/2.15.0*"
aws ‐‐version
# Install aws-cli if not already installed
# https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#cliv2-linux-install
# Install the SMP library
conda install pytorch="2.0.1=sm_py3.10_cuda${SMP_CUDA_VER}*" packaging ‐‐override-channels \
-c https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/smp-2.0.0-pt-2.0.1/2023-12-11/smp-v2/ \
-c pytorch -c numba/label/dev \
-c nvidia -c conda-forge
# Install dependencies of the script as below
python -m pip install packaging transformers==4.31.0 accelerate ninja tensorboard h5py datasets \
&& python -m pip install expecttest hypothesis \
&& python -m pip install "flash-attn>=2.0.4" ‐‐no-build-isolation
# Install the SMDDP wheel
SMDDP_WHL="smdistributed_dataparallel-2.0.2-cp310-cp310-linux_x86_64.whl" \
&& wget -q https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.0.1/cu118/2023-12-07/${SMDDP_WHL} \
&& pip install ‐‐force ${SMDDP_WHL} \
&& rm ${SMDDP_WHL}
# cuDNN installation for Transformer Engine installation for CUDA 11.8
# Please download from below link, you need to agree to terms
# https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.5/local_installers/11.x/cudnn-linux-x86_64-8.9.5.30_cuda11-archive.tar.xz
tar xf cudnn-linux-x86_64-8.9.5.30_cuda11-archive.tar.xz \
&& rm -rf /usr/local/cuda-$SMP_CUDA_VER/include/cudnn* /usr/local/cuda-$SMP_CUDA_VER/lib/cudnn* \
&& cp ./cudnn-linux-x86_64-8.9.5.30_cuda11-archive/include/* /usr/local/cuda-$SMP_CUDA_VER/include/ \
&& cp ./cudnn-linux-x86_64-8.9.5.30_cuda11-archive/lib/* /usr/local/cuda-$SMP_CUDA_VER/lib/ \
&& rm -rf cudnn-linux-x86_64-8.9.5.30_cuda11-archive.tar.xz \
&& rm -rf cudnn-linux-x86_64-8.9.5.30_cuda11-archive/
# Please download from below link, you need to agree to terms
# https://developer.download.nvidia.com/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz \
# cuDNN installation for TransformerEngine installation for cuda12.1
tar xf cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz \
&& rm -rf /usr/local/cuda-$SMP_CUDA_VER/include/cudnn* /usr/local/cuda-$SMP_CUDA_VER/lib/cudnn* \
&& cp ./cudnn-linux-x86_64-8.9.7.29_cuda12-archive/include/* /usr/local/cuda-$SMP_CUDA_VER/include/ \
&& cp ./cudnn-linux-x86_64-8.9.7.29_cuda12-archive/lib/* /usr/local/cuda-$SMP_CUDA_VER/lib/ \
&& rm -rf cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz \
&& rm -rf cudnn-linux-x86_64-8.9.7.29_cuda12-archive/
# TransformerEngine installation
export CUDA_HOME=/usr/local/cuda-$SMP_CUDA_VER
export CUDNN_PATH=/usr/local/cuda-$SMP_CUDA_VER/lib
export CUDNN_LIBRARY=/usr/local/cuda-$SMP_CUDA_VER/lib
export CUDNN_INCLUDE_DIR=/usr/local/cuda-$SMP_CUDA_VER/include
export PATH=/usr/local/cuda-$SMP_CUDA_VER/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-$SMP_CUDA_VER/lib
python -m pip install ‐‐no-build-isolation git+https://github.com/NVIDIA/TransformerEngine.git@v1.0
```
1. Run a test training job.
1. In the shared file system (`/fsx`), clone the [Awsome Distributed Training GitHub repository](https://github.com/aws-samples/awsome-distributed-training/), and go to the `3.test_cases/11.modelparallel` folder.
```
git clone https://github.com/aws-samples/awsome-distributed-training/
cd awsome-distributed-training/3.test_cases/11.modelparallel
```
1. Submit a job using `sbatch` as follows.
```
conda activate
sbatch -N 16 conda_launch.sh
```
If the job submission is successful, the output message of this `sbatch` command should be similar to `Submitted batch job ABCDEF`.
1. Check the log file in the current directory under `logs/`.
```
tail -f ./logs/fsdp_smp_ABCDEF.out
```
------
# Core features of the SageMaker model parallelism library v2
.amazonaws.com/smdistributed-modelparallel:2.5.1-gpu-py311-cu124
```
+ SMP Enroot container for PyTorch v2.5.1 with CUDA v12.4
```
https://sagemaker-distributed-model-parallel.s3..amazonaws.com/enroot/2.5.1-gpu-py311-cu124.sqsh
```
+ Pre-installed packages
+ The SMP library v2.8.0
+ The SMDDP library v2.6.0
+ CUDNN v9.4.0
+ FlashAttention v2.5.8
+ TransformerEngine v1.10
+ Megatron v0.8.0
+ Hugging Face Transformers v4.44.2
+ Hugging Face Datasets library v2.19.0
+ EFA v1.36.0
+ NCCL v2.23.4
+ AWS-OFI-NCCL v1.13.2
### SMP Conda channel
.smdistributed-modelparallel:2.4.1-gpu-py311-cu121
```
+ SMP Enroot container for PyTorch v2.4.1 with CUDA v12.1
```
https://sagemaker-distributed-model-parallel.s3..amazonaws.com/enroot/2.4.1-gpu-py311-cu121.sqsh
```
+ Pre-installed packages
+ The SMP library v2.7.0
+ The SMDDP library v2.5.0
+ CUDNN v9.4.0
+ FlashAttention v2.5.8
+ TransformerEngine v1.10
+ Megatron v0.8.0
+ Hugging Face Transformers v4.44.2
+ Hugging Face Datasets library v2.19.0
+ EFA v1.32.0
+ NCCL v2.21.5
### SMP Conda channel
.amazonaws.com/smdistributed-modelparallel:2.4.1-gpu-py311-cu121
```
+ Pre-installed packages
+ The SMP library v2.6.1
+ The SMDDP library v2.5.0
+ CUDNN v9.4.0
+ FlashAttention v2.5.8
+ TransformerEngine v1.10
+ Megatron v0.8.0
+ Hugging Face Transformers v4.44.2
+ Hugging Face Datasets library v2.19.0
+ EFA v1.32.0
+ NCCL v2.21.5
### SMP Conda channel
.amazonaws.com/smdistributed-modelparallel:2.4.1-gpu-py311-cu121
```
+ Pre-installed packages
+ The SMP library v2.6.0
+ The SMDDP library v2.5.0
+ CUDNN v9.4.0
+ FlashAttention v2.5.8
+ TransformerEngine v1.10
+ Megatron v0.8.0
+ Hugging Face Transformers v4.44.2
+ Hugging Face Datasets library v2.19.0
+ EFA v1.32.0
+ NCCL v2.21.5
### SMP Conda channel
.amazonaws.com/smdistributed-modelparallel:2.3.1-gpu-py311-cu121
```
For a complete list of supported regions, see [AWS Regions](distributed-data-parallel-support.md#distributed-data-parallel-availablity-zone).
+ Pre-installed packages
+ The SMP library v2.5.0
+ The SMDDP library v2.3.0
+ CUDNN v8.9.7.29
+ FlashAttention v2.5.8
+ TransformerEngine v1.8
+ Megatron v0.7.0
+ Hugging Face Transformers v4.40.1
+ Hugging Face Datasets library v2.19.0
+ EFA v1.32.0
+ NCCL v2.21.5
### SMP Conda channel
.amazonaws.com/pytorch-training:2.0.0-gpu-py310-cu118-ubuntu20.04-sagemaker` | https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-2.0.0/build-artifacts/2023-04-14-20-14/smdistributed\$1modelparallel-1.15.0-cp310-cp310-linux\$1x86\$164.whl |
| v1.13.1 | smdistributed-modelparallel==v1.15.0 | `763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.13.1-gpu-py39-cu117-ubuntu20.04-sagemaker` | https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-1.13.1/build-artifacts/2023-04-17-15-49/smdistributed\$1modelparallel-1.15.0-cp39-cp39-linux\$1x86\$164.whl |
| v1.12.1 | smdistributed-modelparallel==v1.13.0 | `763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.12.1-gpu-py38-cu113-ubuntu20.04-sagemaker` | https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-1.12.1/build-artifacts/2022-12-08-21-34/smdistributed\$1modelparallel-1.13.0-cp38-cp38-linux\$1x86\$164.whl |
| v1.12.0 | smdistributed-modelparallel==v1.11.0 | `763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.12.0-gpu-py38-cu113-ubuntu20.04-sagemaker` | https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-1.12.0/build-artifacts/2022-08-12-16-58/smdistributed\$1modelparallel-1.11.0-cp38-cp38-linux\$1x86\$164.whl |
| v1.11.0 | smdistributed-modelparallel==v1.10.0 | `763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.11.0-gpu-py38-cu113-ubuntu20.04-sagemaker` | https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-1.11.0/build-artifacts/2022-07-11-19-23/smdistributed\$1modelparallel-1.10.0-cp38-cp38-linux\$1x86\$164.whl |
| v1.10.2 | smdistributed-modelparallel==v1.7.0 | `763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.10.2-gpu-py38-cu113-ubuntu20.04-sagemaker` | - |
| v1.10.0 | smdistributed-modelparallel==v1.5.0 | `763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.10.0-gpu-py38-cu113-ubuntu20.04-sagemaker` | - |
| v1.9.1 | smdistributed-modelparallel==v1.4.0 | `763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.9.1-gpu-py38-cu111-ubuntu20.04` | - |
| v1.8.1\$1 | smdistributed-modelparallel==v1.6.0 | `763104351884.dkr.ecr..amazonaws.com/pytorch-training:1.8.1-gpu-py36-cu111-ubuntu18.04` | - |
**Note**
The SageMaker model parallelism library v1.6.0 and later provides extended features for PyTorch. For more information, see [Core Features of the SageMaker Model Parallelism Library](model-parallel-core-features.md).
\$1\$1 The URLs of the binary files are for installing the SageMaker model parallelism library in custom containers. For more information, see [Create Your Own Docker Container with the SageMaker Distributed Model Parallel Library](model-parallel-sm-sdk.md#model-parallel-bring-your-own-container).
TensorFlow versions supported by SageMaker AI and the SageMaker model parallelism library
| TensorFlow version | SageMaker model parallelism library version | `smdistributed-modelparallel` integrated DLC image URI |
| --- | --- | --- |
| v2.6.0 | smdistributed-modelparallel==v1.4.0 | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.6.0-gpu-py38-cu112-ubuntu20.04 |
| v2.5.1 | smdistributed-modelparallel==v1.4.0 | 763104351884.dkr.ecr..amazonaws.com/tensorflow-training:2.5.1-gpu-py37-cu112-ubuntu18.04 |
**Hugging Face Transformers versions supported by SageMaker AI and the SageMaker distributed data parallel library**
The AWS Deep Learning Containers for Hugging Face use the SageMaker Training Containers for PyTorch and TensorFlow as their base images. To look up the Hugging Face Transformers library versions and paired PyTorch and TensorFlow versions, see the latest [Hugging Face Containers](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#huggingface-training-containers) and the [Prior Hugging Face Container Versions](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#prior-hugging-face-container-versions).
## AWS Regions
To extend a pre-built container and use SageMaker's model parallelism library, you must use one of the available AWS Deep Learning Containers (DLC) images for PyTorch or TensorFlow. The SageMaker model parallelism library is included in the TensorFlow (2.3.0 and later) and PyTorch (1.6.0 and later) DLC images with CUDA (`cuxyz`). For a complete list of DLC images, see [Available Deep Learning Containers Images](https://github.com/aws/deep-learning-containers/blob/master/available_images.md) in the *AWS Deep Learning Containers GitHub repository*.
**Tip**
We recommend that you use the image that contains the latest version of TensorFlow or PyTorch to access the most up-to-date version of the SageMaker model parallelism library.
For example, your Dockerfile should contain a `FROM` statement similar to the following:
```
# Use the SageMaker DLC image URI for TensorFlow or PyTorch
FROM aws-dlc-account-id.dkr.ecr.aws-region.amazonaws.com/framework-training:{framework-version-tag}
# Add your dependencies here
RUN ...
ENV PATH="/opt/ml/code:${PATH}"
# this environment variable is used by the SageMaker AI container to determine our user code directory.
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code
```
Additionally, when you define a PyTorch or TensorFlow estimator, you must specify that the `entry_point` for your training script. This should be the same path identified with `ENV SAGEMAKER_SUBMIT_DIRECTORY` in your Dockerfile.
**Tip**
You must push this Docker container to Amazon Elastic Container Registry (Amazon ECR) and use the image URI (`image_uri`) to define a SageMaker estimator for training. For more information, see [Extend a Pre-built Container](prebuilt-containers-extend.md).
After you finish hosting the Docker container and retrieving the image URI of the container, create a SageMaker `PyTorch` estimator object as follows. This example assumes that you have already defined `smp_options` and `mpi_options`.
```
smd_mp_estimator = Estimator(
entry_point="your_training_script.py",
role=sagemaker.get_execution_role(),
instance_type='ml.p3.16xlarge',
sagemaker_session=sagemaker_session,
image_uri='your_aws_account_id.dkr.ecr.region.amazonaws.com/name:tag'
instance_count=1,
distribution={
"smdistributed": smp_options,
"mpi": mpi_options
},
base_job_name="SMD-MP-demo",
)
smd_mp_estimator.fit('s3://my_bucket/my_training_data/')
```
## Create Your Own Docker Container with the SageMaker Distributed Model Parallel Library
```
**Tip**
The official AWS Deep Learning Container (DLC) images are built from the [NVIDIA CUDA base images](https://hub.docker.com/r/nvidia/cuda). We recommend you look into the [official Dockerfiles of AWS Deep Learning Container for PyTorch](https://github.com/aws/deep-learning-containers/tree/master/pytorch/training/docker) to find which versions of the libraries you need to install and how to configure them. The official Dockerfiles are complete, benchmark tested, and managed by the SageMaker and Deep Learning Container service teams. In the provided link, choose the PyTorch version you use, choose the CUDA (`cuxyz`) folder, and choose the Dockerfile ending with `.gpu` or `.sagemaker.gpu`.
1. To set up a distributed training environment, you need to install software for communication and network devices, such as [Elastic Fabric Adapter (EFA)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html), [NVIDIA Collective Communications Library (NCCL)](https://developer.nvidia.com/nccl), and [Open MPI](https://www.open-mpi.org/). Depending on the PyTorch and CUDA versions you choose, you must install compatible versions of the libraries.
**Important**
Because the SageMaker model parallel library requires the SageMaker data parallel library in the subsequent steps, we highly recommend that you follow the instructions at [Create your own Docker container with the SageMaker AI distributed data parallel library](data-parallel-bring-your-own-container.md) to properly set up a SageMaker training environment for distributed training.
For more information about setting up EFA with NCCL and Open MPI, see [Get started with EFA and MPI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html) and [Get started with EFA and NCCL](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start-nccl.html).
1. Add the following arguments to specify the URLs of the SageMaker distributed training packages for PyTorch. The SageMaker model parallel library requires the SageMaker data parallel library to use the cross-node Remote Direct Memory Access (RDMA).
```
ARG SMD_MODEL_PARALLEL_URL=https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-1.10.0/build-artifacts/2022-02-21-19-26/smdistributed_modelparallel-1.7.0-cp38-cp38-linux_x86_64.whl
ARG SMDATAPARALLEL_BINARY=https://smdataparallel.s3.amazonaws.com/binary/pytorch/1.10.2/cu113/2022-02-18/smdistributed_dataparallel-1.4.0-cp38-cp38-linux_x86_64.whl
```
1. Install dependencies that the SageMaker model parallel library requires.
1. Install the [METIS](http://glaros.dtc.umn.edu/gkhome/metis/metis/overview) library.
```
ARG METIS=metis-5.1.0
RUN rm /etc/apt/sources.list.d/* \
&& wget -nv http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/${METIS}.tar.gz \
&& gunzip -f ${METIS}.tar.gz \
&& tar -xvf ${METIS}.tar \
&& cd ${METIS} \
&& apt-get update \
&& make config shared=1 \
&& make install \
&& cd .. \
&& rm -rf ${METIS}.tar* \
&& rm -rf ${METIS} \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get clean
```
1. Install the [RAPIDS Memory Manager library](https://github.com/rapidsai/rmm#rmm-rapids-memory-manager). This requires [CMake](https://cmake.org/) 3.14 or later.
```
ARG RMM_VERSION=0.15.0
RUN wget -nv https://github.com/rapidsai/rmm/archive/v${RMM_VERSION}.tar.gz \
&& tar -xvf v${RMM_VERSION}.tar.gz \
&& cd rmm-${RMM_VERSION} \
&& INSTALL_PREFIX=/usr/local ./build.sh librmm \
&& cd .. \
&& rm -rf v${RMM_VERSION}.tar* \
&& rm -rf rmm-${RMM_VERSION}
```
1. Install the SageMaker model parallel library.
```
RUN pip install --no-cache-dir -U ${SMD_MODEL_PARALLEL_URL}
```
1. Install the SageMaker data parallel library.
```
RUN SMDATAPARALLEL_PT=1 pip install --no-cache-dir ${SMDATAPARALLEL_BINARY}
```
1. Install the [sagemaker-training toolkit](https://github.com/aws/sagemaker-training-toolkit). The toolkit contains the common functionality that's necessary to create a container compatible with the SageMaker training platform and the SageMaker Python SDK.
```
RUN pip install sagemaker-training
```
1. After you finish creating the Dockerfile, see [Adapting Your Own Training Container](https://docs.aws.amazon.com/sagemaker/latest/dg/adapt-training-container.html) to learn how to build the Docker container and host it in Amazon ECR.
**Tip**
For more general information about creating a custom Dockerfile for training in SageMaker AI, see [Use Your Own Training Algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html).
# Checkpointing and Fine-Tuning a Model with Model Parallelism
"
},
"HyperParameters": {
"sagemaker_training_compiler_enabled": "true",
"sagemaker_training_compiler_debug_mode": "false",
"sagemaker_pytorch_xla_multi_worker_enabled": "false" // set to "true" for distributed training
}
```
To find a complete list of deep learning container image URIs that have SageMaker Training Compiler implemented, see [Supported Frameworks](training-compiler-support.md#training-compiler-supported-frameworks).
# Run TensorFlow Training Jobs with SageMaker Training Compiler
.amazonaws.com/tensorflow-training:
ENV PATH="/opt/ml/code:${PATH}"
# This environment variable is used by the SageMaker AI container
# to determine user code directory.
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code
# Add more code lines to customize for your use-case
...
```
For more information, see [Step 2: Create and upload the Dockerfile and Python training scripts](https://docs.aws.amazon.com/sagemaker/latest/dg/adapt-training-container.html#byoc-training-step2).
Consider the following pitfalls when extending SageMaker AI Framework DLCs:
+ Do not explicitly uninstall or change the version of TensorFlow packages in SageMaker AI containers. Doing so causes the AWS optimized TensorFlow packages to be overwritten by open-source TensorFlow packages, which might result in performance degradation.
+ Watch out for packages that have a particular TensorFlow version or flavor as a dependency. These packages might implicitly uninstall the AWS optimized TensorFlow and install open-source TensorFlow packages.
For example, there’s a known issue that the [tensorflow/models](https://github.com/tensorflow/models) and [tensorflow/text](https://github.com/tensorflow/text) libraries always attempt to [reinstall open source TensorFlow](https://github.com/tensorflow/models/issues/9267). If you need to install these libraries to choose a specific version for your use case, we recommend that you look into the SageMaker AI TensorFlow DLC Dockerfiles for v2.9 or later. The paths to the Dockerfiles are typically in the following format: `tensorflow/training/docker//py3//Dockerfile.gpu`. In the Dockerfiles, you should find the code lines to reinstall AWS managed TensorFlow binary (specified to the `TF_URL` environment variable) and other dependencies in order. The reinstallation section should look like the following example:
```
# tf-models does not respect existing installations of TensorFlow
# and always installs open source TensorFlow
RUN pip3 install --no-cache-dir -U \
tf-models-official==x.y.z
RUN pip3 uninstall -y tensorflow tensorflow-gpu \
; pip3 install --no-cache-dir -U \
${TF_URL} \
tensorflow-io==x.y.z \
tensorflow-datasets==x.y.z
```
### Build and push to ECR
"
},
"HyperParameters": {
"sagemaker_training_compiler_enabled": "true",
"sagemaker_training_compiler_debug_mode": "false"
}
```
To find a complete list of deep learning container image URIs that have SageMaker Training Compiler implemented, see [Supported Frameworks](training-compiler-support.md#training-compiler-supported-frameworks).
# SageMaker Training Compiler Example Notebooks and Blogs
.amazonaws.com/tensorflow-training:2.11.0-gpu-py39-cu112-ubuntu20.04-sagemaker
```
To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see [Supported Frameworks, AWS Regions, Instance Types, and Tested Models](training-compiler-support.md).
## SageMaker Training Compiler Release Notes: December 8, 2022
.amazonaws.com/pytorch-trcomp-training:1.12.0-gpu-py38-cu113-ubuntu20.04-sagemaker
```
To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see [Supported Frameworks, AWS Regions, Instance Types, and Tested Models](training-compiler-support.md).
## SageMaker Training Compiler Release Notes: October 4, 2022
.amazonaws.com/tensorflow-training:2.10.0-gpu-py39-cu112-ubuntu20.04-sagemaker
```
To find a complete list of the prebuilt containers with Amazon SageMaker Training Compiler, see [Supported Frameworks, AWS Regions, Instance Types, and Tested Models](training-compiler-support.md).
## SageMaker Training Compiler Release Notes: September 1, 2022
.amazonaws.com/tensorflow-training:2.9.1-gpu-py39-cu112-ubuntu20.04-sagemaker
```
To find a complete list of the pre-built containers with Amazon SageMaker Training Compiler, see [Supported Frameworks, AWS Regions, Instance Types, and Tested Models](training-compiler-support.md).
## SageMaker Training Compiler Release Notes: April 26, 2022
:role/",
RoleSessionName="SessionName",
Tags=tags)
credentials = response['Credentials']
# Create a client with your job creation role (which was assumed with tags)
sagemaker_client = boto3.client(
'sagemaker',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken']
)
sagemaker_session = sagemaker.Session(sagemaker_client=sagemaker_client)
```
When appending the tags `"tenant-id=example-tenant"` to the job creation role, these tags are extracted by the execution role to use the following policy:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::your-input-s3-bucket/example-tenant/*"
],
"Effect": "Allow"
},
{
"Action": [
"s3:PutObject"
],
"Resource": "arn:aws:s3:::your-output-s3-bucket/example-tenant/*",
"Effect": "Allow"
},
{
"Action": "s3:ListBucket",
"Resource": "*",
"Effect": "Allow"
}
]
}
```
------
1. Define an estimator to create a training job using the SageMaker Python SDK. Set `enable_session_tag_chaining` to `True` to allow your SageMaker AI training execution role to retrieve the tags from your job creation role.
```
# Specify your training input
trainingInput = TrainingInput(
s3_data='s3:///example-tenant',
distribution='ShardedByS3Key',
s3_data_type='S3Prefix'
)
# Specify your training job execution role
execution_role_arn = "arn:aws:iam:::role/"
# Define your esimator with session tag chaining enabled
estimator = Estimator(
image_uri="",
role=execution_role_arn,
instance_count=1,
instance_type='ml.m4.xlarge',
volume_size=20,
max_run=3600,
sagemaker_session=sagemaker_session,
output_path="s3:///example-tenant",
enable_session_tag_chaining=True
)
estimator.fit(inputs=trainingInput, job_name="abac-demo")
```
SageMaker AI can only read tags provided in the training job request and does not add any tags to resources on your behalf.
ABAC for SageMaker training is compatible with SageMaker AI managed warm pools. To use ABAC with warm pools, matching training jobs must have identical session tags. For more information, see [Matching training jobs](train-warm-pools.md#train-warm-pools-matching-criteria).
# Mapping of training storage paths managed by Amazon SageMaker AI
/:" # Specify to use built-in algorithms
output_path=bucket,
base_job_name=base_job_name,
# Parameters required to enable checkpointing
checkpoint_s3_uri=checkpoint_s3_bucket,
checkpoint_local_path=checkpoint_local_path
)
```
The following two parameters specify paths for checkpointing:
+ `checkpoint_local_path` – Specify the local path where the model saves the checkpoints periodically in a training container. The default path is set to `'/opt/ml/checkpoints'`. If you are using other frameworks or bringing your own training container, ensure that your training script's checkpoint configuration specifies the path to `'/opt/ml/checkpoints'`.
**Note**
We recommend specifying the local paths as `'/opt/ml/checkpoints'` to be consistent with the default SageMaker AI checkpoint settings. If you prefer to specify your own local path, make sure you match the checkpoint saving path in your training script and the `checkpoint_local_path` parameter of the SageMaker AI estimators.
+ `checkpoint_s3_uri` – The URI to an S3 bucket where the checkpoints are stored in real time. You can specify either an S3 general purpose or S3 directory bucket to store your checkpoints. For more information on S3 directory buckets, see [Directory buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html) in the *Amazon Simple Storage Service User Guide*.
To find a complete list of SageMaker AI estimator parameters, see the [Estimator API](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html#sagemaker.estimator.Estimator) in the *[Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable) documentation*.
# Browse checkpoint files