create_evaluator¶
Operation¶
async
¶
create_evaluator(input: CreateEvaluatorInput, plugins: list[Plugin] | None = None) -> CreateEvaluatorOutput
Creates a custom evaluator for agent quality assessment. Custom evaluators can use either LLM-as-a-Judge configurations with user-defined prompts, rating scales, and model settings, or code-based configurations with customer-managed Lambda functions to evaluate agent performance at tool call, trace, or session levels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input
|
CreateEvaluatorInput
|
An instance of |
required |
plugins
|
list[Plugin] | None
|
A list of callables that modify the configuration dynamically. Changes made by these plugins only apply for the duration of the operation execution and will not affect any other operation invocations. |
None
|
Returns:
| Type | Description |
|---|---|
CreateEvaluatorOutput
|
An instance of |
Input¶
dataclass
¶
Dataclass for CreateEvaluatorInput structure.
Attributes¶
class-attribute
instance-attribute
¶
client_token: str | None = None
A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.
class-attribute
instance-attribute
¶
description: str | None = field(repr=False, default=None)
The description of the evaluator that explains its purpose and evaluation criteria.
class-attribute
instance-attribute
¶
evaluator_config: EvaluatorConfig | None = None
The configuration for the evaluator. Specify either LLM-as-a-Judge settings with instructions, rating scale, and model configuration, or code-based settings with a customer-managed Lambda function.
class-attribute
instance-attribute
¶
evaluator_name: str | None = None
The name of the evaluator. Must be unique within your account.
class-attribute
instance-attribute
¶
kms_key_arn: str | None = None
The Amazon Resource Name (ARN) of a customer managed KMS key to use for encrypting sensitive evaluator data, including instructions and rating scale. If you don't specify a KMS key, the evaluator data is encrypted with an Amazon Web Services owned key. Only symmetric encryption KMS keys are supported. For more information, see Encryption at rest for AgentCore Evaluations.
class-attribute
instance-attribute
¶
level: EvaluatorLevel | None = None
The evaluation level that determines the scope of evaluation. Valid
values are TOOL_CALL for individual tool invocations, TRACE for
single request-response interactions, or SESSION for entire
conversation sessions.
class-attribute
instance-attribute
¶
tags: dict[str, str] | None = None
A map of tag keys and values to assign to an AgentCore Evaluator. Tags enable you to categorize your resources in different ways, for example, by purpose, owner, or environment.