View a markdown version of this page

Instance preference lists for training jobs - Amazon SageMaker AI
How it worksService quotasWaiting for capacityConsiderationsTraining plansExamplesSelected instance typePython SDKConsoleRelated resources

Instance preference lists for training jobs

A training job can accept an ordered list of up to 5 instance types instead of one. SageMaker AI launches the job on the first type in the list that has capacity, so the job starts sooner when your preferred type is constrained. It then runs, is monitored, and is billed exactly as if you had requested that type directly.

Instance preferences are available through the CreateTrainingJob API, the AWS SDKs, the AWS CLI, the SageMaker Python SDK, and the SageMaker AI console.

How it works

Provide InstancePreferences in ResourceConfig instead of InstanceType, and set the instance count in one of two ways:

  • Uniform – one InstanceCount in ResourceConfig, used with whichever type is selected.

  • Per preference – an InstanceCount on every entry, for types with different accelerator counts. For example, ml.g5.12xlarge has four GPUs and ml.g5.16xlarge has one, so you can give the second a higher count.

A request that sets counts in both places, in neither, or on only some entries is rejected.

Service quotas

SageMaker AI checks your service quota for every type in the list, not just the one it selects. Each type's quota must cover the instance count you request for it. If any type falls short, the request is rejected, even when the types ahead of it would have succeeded.

Request quota increases for every type in the list before you submit the job. For more information, see Step 2: Check and request service quotas.

Waiting for capacity

If no type in the list has capacity, the job waits and retries as capacity is released. MaxPendingTimeInSeconds sets how long it keeps retrying, as described in On-Demand Instances. With a list:

  • The limit covers the total time spent across the whole list, not each type separately.

  • Some Pending time on a training-plan-backed type doesn't count toward the limit. For more information, see MaxPendingTimeInSeconds.

Note

The job waits only if the list includes at least one accelerated computing instance type, such as ml.p, ml.g, or ml.trn. A CPU-only list is evaluated once: the job starts if one of the types has capacity, and otherwise fails with a capacity error without being re-evaluated.

Considerations

  • InstancePreferences can't be combined with InstanceType, InstanceGroups, InstancePlacementConfig, or EnableManagedSpotTraining.

  • Each instance type can appear only once.

  • If you use a built-in algorithm, every listed type must be one it supports; otherwise the request is rejected.

  • SageMaker AI selects by capacity only. It doesn't compare GPU architecture, accelerator memory, Elastic Fabric Adapter (EFA) support, or driver versions, so make sure your container works with every type in the list. A job that launches on an incompatible type can fail.

  • VolumeSizeInGB, VolumeKmsKeyId, and KeepAlivePeriodInSeconds apply to whichever type is selected.

Use instance preferences with training plans

Add TrainingPlanArns to an entry to fill it from a training plan. The entry's instance type must match the plan's. Entries without a plan use On-Demand capacity. Put the plan-backed entry first so that your reserved capacity is tried before the other types. For more information about plans, see Reserve Flexible Training Plans for ML workloads.

If the plan has no capacity when the job is evaluated, SageMaker AI moves on to the next entry rather than waiting for the reservation, so the job can start on On-Demand capacity right away.

Alternatively, set the job-level TrainingPlanArn. It applies to the entry whose instance type matches the plan and is rejected if none does. You can't set both the job-level and per-entry fields.

Examples

The examples show the ResourceConfig and StoppingCondition parts of a CreateTrainingJob request.

Two types with a uniform count. The job launches on ml.g6.48xlarge, or on ml.g5.48xlarge if that has no capacity:

"ResourceConfig": {
  "InstanceCount": 2,
  "VolumeSizeInGB": 500,
  "InstancePreferences": [
    { "InstanceType": "ml.g6.48xlarge" },
    { "InstanceType": "ml.g5.48xlarge" }
  ]
}

A count on each entry:

"ResourceConfig": {
  "VolumeSizeInGB": 500,
  "InstancePreferences": [
    { "InstanceType": "ml.g6.48xlarge", "InstanceCount": 2 },
    { "InstanceType": "ml.g5.48xlarge", "InstanceCount": 4 }
  ]
}

A 30-minute limit on waiting for capacity:

"StoppingCondition": {
  "MaxRuntimeInSeconds": 86400,
  "MaxPendingTimeInSeconds": 1800
}

A training plan on the first entry and On-Demand capacity on the second:

"ResourceConfig": {
  "InstanceCount": 4,
  "VolumeSizeInGB": 500,
  "InstancePreferences": [
    {
      "InstanceType": "ml.p5.48xlarge",
      "TrainingPlanArns": ["arn:aws:sagemaker:us-west-2:111122223333:training-plan/p5-plan"]
    },
    { "InstanceType": "ml.p4d.24xlarge" }
  ]
}

The same job using the job-level plan field:

"ResourceConfig": {
  "InstanceCount": 4,
  "VolumeSizeInGB": 500,
  "TrainingPlanArn": "arn:aws:sagemaker:us-west-2:111122223333:training-plan/p5-plan",
  "InstancePreferences": [
    { "InstanceType": "ml.p5.48xlarge" },
    { "InstanceType": "ml.p4d.24xlarge" }
  ]
}

Find the selected instance type

DescribeTrainingJob returns the list you submitted plus the read-only SelectedInstanceType and SelectedInstanceCount. These two fields appear only after a type is selected, so they are absent while the job waits for capacity. The top-level InstanceType isn't returned for these jobs:

"ResourceConfig": {
  "VolumeSizeInGB": 500,
  "InstancePreferences": [
    { "InstanceType": "ml.g6.48xlarge", "InstanceCount": 2 },
    { "InstanceType": "ml.g5.48xlarge", "InstanceCount": 4 }
  ],
  "SelectedInstanceType": "ml.g5.48xlarge",
  "SelectedInstanceCount": 4
}

Use instance preferences with the SageMaker Python SDK

Pass instance_preferences in the Compute configuration of ModelTrainer. Set instance_count on Compute for a uniform count, or on each InstancePreference instead. Add training_plan_arns to an entry to fill it from a training plan.

from sagemaker.core.shapes import InstancePreference
from sagemaker.core.training.configs import Compute
from sagemaker.train.model_trainer import ModelTrainer

compute = Compute(
    volume_size_in_gb=500,
    instance_preferences=[
        InstancePreference(
            instance_type="ml.p5.48xlarge",
            instance_count=4,
            training_plan_arns=["arn:aws:sagemaker:us-west-2:111122223333:training-plan/p5-plan"],
        ),
        InstancePreference(instance_type="ml.p4d.24xlarge", instance_count=8),
    ],
)

trainer = ModelTrainer(
    training_image="<training-image-uri>",
    compute=compute,
)
trainer.train()

For more information, see the SageMaker Python SDK documentation on the Read the Docs website and the SageMaker Python SDK example notebook on the GitHub website.

Use instance preferences in the SageMaker AI console

  1. On the Create training job page, under Instance types, choose the Instance type you prefer most. Then choose its Instance capacity (On-Demand Instances or a training plan) and enter its Instance count.

  2. Choose Add another instance type. The form becomes a list, with your first choice as Priority 1 and a new row below it. Fill in the new row, and repeat for up to 5 types.

  3. To reorder, change a row's Priority; the row swaps places with the one that held that number. Choose Remove to drop a row.

After launch, the job details page shows an Instance type preferences table with the chosen type marked Selected. Until a type is chosen, the instance type shows Pending selection.