Interface ProductionVariantRoutingConfig.Builder

All Superinterfaces:
Buildable, CopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>, SdkBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>, SdkPojo
Enclosing class:
ProductionVariantRoutingConfig
@Mutable @NotThreadSafe public static interface ProductionVariantRoutingConfig.Builder extends SdkPojo, CopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>

  • Method Details

    • routingStrategy

      ProductionVariantRoutingConfig.Builder routingStrategy(String routingStrategy)

      Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

      Parameters:
      routingStrategy - Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

      Returns:
      Returns a reference to this object so that method calls can be chained together.
      See Also:

    • routingStrategy

      ProductionVariantRoutingConfig.Builder routingStrategy(RoutingStrategy routingStrategy)

      Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

      Parameters:
      routingStrategy - Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

      Returns:
      Returns a reference to this object so that method calls can be chained together.
      See Also:

    • prefixAwareRoutingConfig

      ProductionVariantRoutingConfig.Builder prefixAwareRoutingConfig(PrefixAwareRoutingConfig prefixAwareRoutingConfig)

      The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

      Parameters:
      prefixAwareRoutingConfig - The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.
      Returns:
      Returns a reference to this object so that method calls can be chained together.

    • prefixAwareRoutingConfig

      default ProductionVariantRoutingConfig.Builder prefixAwareRoutingConfig(Consumer<PrefixAwareRoutingConfig.Builder> prefixAwareRoutingConfig)

      The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

      This is a convenience method that creates an instance of the PrefixAwareRoutingConfig.Builder avoiding the need to create one manually via PrefixAwareRoutingConfig.builder().

      When the Consumer completes, SdkBuilder.build() is called immediately and its result is passed to prefixAwareRoutingConfig(PrefixAwareRoutingConfig).

      Parameters:
      prefixAwareRoutingConfig - a consumer that will call methods on PrefixAwareRoutingConfig.Builder
      Returns:
      Returns a reference to this object so that method calls can be chained together.
      See Also: