ProductionVariantRoutingConfig

Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

RoutingStrategy

Sets how the endpoint routes incoming traffic:

LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.
RANDOM: The endpoint routes each request to a randomly chosen instance.
PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance. When the number of in-flight requests on the selected instance reaches the configured threshold, the endpoint routes the request to an instance with more available capacity.

Type: String

Valid Values: LEAST_OUTSTANDING_REQUESTS | RANDOM | PREFIX_AWARE

Required: Yes

PrefixAwareRoutingConfig

The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

Required: No

Contents