InferenceComponentComputeResourceRequirements
Defines the compute resources to allocate to run a model, plus any adapter models, that you assign to an inference component. These resources include CPU cores, accelerators, and memory.
Contents
- MinMemoryRequiredInMb
-
The minimum MB of memory to allocate to run a model that you assign to an inference component.
Type: Integer
Valid Range: Minimum value of 128.
Required: Yes
- AcceleratorPartitionType
-
The accelerator partition profile this inference component occupies; on NVIDIA GPUs this is a Multi-Instance GPU (MIG) profile such as
mig-3g.40gb. It must be one of the partition profiles declared in the target endpoint's accelerator partition layout, and when set,NumberOfAcceleratorDevicesRequiredmust be1. This field is required when the endpoint's production variant has an accelerator partition layout. For endpoints without a layout, omit this field to place the inference component on a whole accelerator.Type: String
Length Constraints: Minimum length of 0. Maximum length of 32.
Pattern:
mig-[0-9]+g\.[0-9]+gb([+-][a-z0-9.]+)*Required: No
- MaxMemoryRequiredInMb
-
The maximum MB of memory to allocate to run a model that you assign to an inference component.
Type: Integer
Valid Range: Minimum value of 128.
Required: No
- NumberOfAcceleratorDevicesRequired
-
The number of accelerators to allocate to run a model that you assign to an inference component. Accelerators include GPUs and AWS Inferentia.
Type: Float
Valid Range: Minimum value of 1.
Required: No
- NumberOfCpuCoresRequired
-
The number of CPU cores to allocate to run a model that you assign to an inference component.
Type: Float
Valid Range: Minimum value of 0.25.
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
