PrefixAwareRoutingConfig
The configuration for prefix-aware routing on a SageMaker real-time inference endpoint.
Specify PrefixLength and ConcurrencyThreshold to control
routing behavior.
Contents
- ConcurrencyThreshold
-
The maximum number of in-flight requests on the target instance before the endpoint routes to another instance. Required when
RoutingStrategyisPREFIX_AWARE. When in-flight requests on the prefix-selected instance reach this threshold, the endpoint routes the request to an instance with more available capacity.Type: Integer
Valid Range: Minimum value of 1. Maximum value of 1024.
Required: No
- PrefixLength
-
The maximum length of the prefix used for routing decisions. Required when
RoutingStrategyisPREFIX_AWARE.-
For the SageMaker Runtime
InvokeEndpointandInvokeEndpointWithResponseStreamAPIs, this value specifies the number of bytes from the beginning of the request body. -
For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.
The endpoint routes requests that share the same prefix to the same instance. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances.
Type: Integer
Valid Range: Minimum value of 1024. Maximum value of 65536.
Required: No
-
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
