Interface ProductionVariantRoutingConfig.Builder

All Superinterfaces:
Buildable, CopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>, SdkBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>, SdkPojo
Enclosing class:
ProductionVariantRoutingConfig
@Mutable @NotThreadSafe public static interface ProductionVariantRoutingConfig.Builder extends SdkPojo, CopyableBuilder<ProductionVariantRoutingConfig.Builder,ProductionVariantRoutingConfig>

  • Method Details

    • routingStrategy

      ProductionVariantRoutingConfig.Builder routingStrategy(String routingStrategy)

      Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance. When the number of in-flight requests on the selected instance reaches the configured threshold, the endpoint routes the request to an instance with more available capacity.

      Parameters:
      routingStrategy - Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance. When the number of in-flight requests on the selected instance reaches the configured threshold, the endpoint routes the request to an instance with more available capacity.

      Returns:
      Returns a reference to this object so that method calls can be chained together.
      See Also:

    • routingStrategy

      ProductionVariantRoutingConfig.Builder routingStrategy(RoutingStrategy routingStrategy)

      Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance. When the number of in-flight requests on the selected instance reaches the configured threshold, the endpoint routes the request to an instance with more available capacity.

      Parameters:
      routingStrategy - Sets how the endpoint routes incoming traffic:

      • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

      • RANDOM: The endpoint routes each request to a randomly chosen instance.

      • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance. When the number of in-flight requests on the selected instance reaches the configured threshold, the endpoint routes the request to an instance with more available capacity.

      Returns:
      Returns a reference to this object so that method calls can be chained together.
      See Also:

    • prefixAwareRoutingConfig

      ProductionVariantRoutingConfig.Builder prefixAwareRoutingConfig(PrefixAwareRoutingConfig prefixAwareRoutingConfig)

      The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

      Parameters:
      prefixAwareRoutingConfig - The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.
      Returns:
      Returns a reference to this object so that method calls can be chained together.

    • prefixAwareRoutingConfig

      default ProductionVariantRoutingConfig.Builder prefixAwareRoutingConfig(Consumer<PrefixAwareRoutingConfig.Builder> prefixAwareRoutingConfig)

      The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

      This is a convenience method that creates an instance of the PrefixAwareRoutingConfig.Builder avoiding the need to create one manually via PrefixAwareRoutingConfig.builder().

      When the Consumer completes, SdkBuilder.build() is called immediately and its result is passed to prefixAwareRoutingConfig(PrefixAwareRoutingConfig).

      Parameters:
      prefixAwareRoutingConfig - a consumer that will call methods on PrefixAwareRoutingConfig.Builder
      Returns:
      Returns a reference to this object so that method calls can be chained together.
      See Also: