Class PrefixAwareRoutingConfig

java.lang.Object
software.amazon.awssdk.services.sagemaker.model.PrefixAwareRoutingConfig
All Implemented Interfaces:
Serializable, SdkPojo, ToCopyableBuilder<PrefixAwareRoutingConfig.Builder,PrefixAwareRoutingConfig>
@Generated("software.amazon.awssdk:codegen") public final class PrefixAwareRoutingConfig extends Object implements SdkPojo, Serializable, ToCopyableBuilder<PrefixAwareRoutingConfig.Builder,PrefixAwareRoutingConfig>

The configuration for prefix-aware routing on a SageMaker real-time inference endpoint. Specify PrefixLength and ConcurrencyThreshold to control routing behavior.

See Also:

  • Method Details

    • prefixLength

      public final Integer prefixLength()

      The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

      • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

      • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

      The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

      Returns:
      The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

      • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

      • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

      The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

    • concurrencyThreshold

      public final Integer concurrencyThreshold()

      The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

      On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

      On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

      The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

      Returns:
      The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

      On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

      On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

      The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

    • toBuilder

      public PrefixAwareRoutingConfig.Builder toBuilder()
      Description copied from interface: ToCopyableBuilder
      Take this object and create a builder that contains all of the current property values of this object.
      Specified by:
      toBuilder in interface ToCopyableBuilder<PrefixAwareRoutingConfig.Builder,PrefixAwareRoutingConfig>
      Returns:
      a builder for type T

    • builder

      public static PrefixAwareRoutingConfig.Builder builder()

    • serializableBuilderClass

      public static Class<? extends PrefixAwareRoutingConfig.Builder> serializableBuilderClass()

    • hashCode

      public final int hashCode()
      Overrides:
      hashCode in class Object

    • equals

      public final boolean equals(Object obj)
      Overrides:
      equals in class Object

    • equalsBySdkFields

      public final boolean equalsBySdkFields(Object obj)
      Description copied from interface: SdkPojo
      Indicates whether some other object is "equal to" this one by SDK fields. An SDK field is a modeled, non-inherited field in an SdkPojo class, and is generated based on a service model.

      If an SdkPojo class does not have any inherited fields, equalsBySdkFields and equals are essentially the same.

      Specified by:
      equalsBySdkFields in interface SdkPojo
      Parameters:
      obj - the object to be compared with
      Returns:
      true if the other object equals to this object by sdk fields, false otherwise.

    • toString

      public final String toString()
      Returns a string representation of this object. This is useful for testing and debugging. Sensitive data will be redacted from this string using a placeholder value.
      Overrides:
      toString in class Object

    • getValueForField

      public final <T> Optional<T> getValueForField(String fieldName, Class<T> clazz)

    • sdkFields

      public final List<SdkField<?>> sdkFields()
      Specified by:
      sdkFields in interface SdkPojo
      Returns:
      List of SdkField in this POJO. May be empty list but should never be null.

    • sdkFieldNameToField

      public final Map<String,SdkField<?>> sdkFieldNameToField()
      Specified by:
      sdkFieldNameToField in interface SdkPojo
      Returns:
      The mapping between the field name and its corresponding field.