InferenceConfiguration¶

Structure Class¶

The configuration parameters that control how the foundation model behaves during evaluation, including response generation settings.

max_tokens: int | None = None

The maximum number of tokens to generate in the model response during evaluation.

stop_sequences: list[str] | None = None

The list of sequences that will cause the model to stop generating tokens when encountered.

temperature: float | None = None

The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.

top_p: float | None = None

The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.