InferenceConfiguration¶
Structure Class¶
dataclass
¶
Base inference parameters to pass to a model. For more information, see Inference parameters for foundation models.
Attributes¶
class-attribute
instance-attribute
¶
max_tokens: int | None = None
The maximum number of tokens to allow in the generated response. The default value is the maximum allowed value for the model that you are using.
class-attribute
instance-attribute
¶
stop_sequences: list[str] | None = None
A list of stop sequences. A stop sequence is a sequence of characters that causes the model to stop generating the response.
class-attribute
instance-attribute
¶
temperature: float | None = None
The likelihood of the model selecting higher-probability options while generating a response. A lower value makes the model more likely to choose higher-probability options, while a higher value makes the model more likely to choose lower-probability options.
class-attribute
instance-attribute
¶
top_p: float | None = None
The percentage of most-likely candidates that the model considers for
the next token. For example, if you choose a value of 0.8 for topP,
the model selects from the top 80% of the probability distribution of
tokens that could be next in the sequence.