Request Syntax URI Request Parameters Request Body Response Syntax Response Elements Errors See Also

CreateEvaluator

Creates a custom evaluator for agent quality assessment. Custom evaluators use LLM-as-a-Judge configurations with user-defined prompts, rating scales, and model settings to evaluate agent performance at tool call, trace, or session levels.

Request Syntax


POST /evaluators/create HTTP/1.1
Content-type: application/json

{
   "clientToken": "string",
   "description": "string",
   "evaluatorConfig": { ... },
   "evaluatorName": "string",
   "level": "string"
}

URI Request Parameters

The request does not use any URI parameters.

Request Body

The request accepts the following data in JSON format.

clientToken

A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.

Type: String

Length Constraints: Minimum length of 33. Maximum length of 256.

Pattern: [a-zA-Z0-9](-*[a-zA-Z0-9]){0,256}

Required: No

description

The description of the evaluator that explains its purpose and evaluation criteria.

Type: String

Length Constraints: Minimum length of 1. Maximum length of 200.

Required: No

evaluatorConfig

The configuration for the evaluator, including LLM-as-a-Judge settings with instructions, rating scale, and model configuration.

Type: EvaluatorConfig object

Note: This object is a Union. Only one member of this object can be specified or returned.

Required: Yes

evaluatorName

The name of the evaluator. Must be unique within your account.

Type: String

Pattern: [a-zA-Z][a-zA-Z0-9_]{0,47}

Required: Yes

level

The evaluation level that determines the scope of evaluation. Valid values are TOOL_CALL for individual tool invocations, TRACE for single request-response interactions, or SESSION for entire conversation sessions.

Type: String

Valid Values: TOOL_CALL | TRACE | SESSION

Required: Yes

Response Syntax


HTTP/1.1 202
Content-type: application/json

{
   "createdAt": number,
   "evaluatorArn": "string",
   "evaluatorId": "string",
   "status": "string"
}

Response Elements

If the action is successful, the service sends back an HTTP 202 response.

The following data is returned in JSON format by the service.

createdAt

The timestamp when the evaluator was created.

Type: Timestamp

evaluatorArn

The Amazon Resource Name (ARN) of the created evaluator.

Type: String

Pattern: arn:aws:bedrock-agentcore:[a-z0-9-]+:[0-9]{12}:evaluator\/[a-zA-Z][a-zA-Z0-9-_]{0,99}-[a-zA-Z0-9]{10}

evaluatorId

The unique identifier of the created evaluator.

Type: String

Pattern: (Builtin.[a-zA-Z0-9_-]+|[a-zA-Z][a-zA-Z0-9-_]{0,99}-[a-zA-Z0-9]{10})

status

The status of the evaluator creation operation.

Type: String

Errors

For information about the errors that are common to all actions, see Common Errors.

AccessDeniedException

This exception is thrown when a request is denied per access permissions

HTTP Status Code: 403

ConflictException

This exception is thrown when there is a conflict performing an operation

HTTP Status Code: 409

InternalServerException

This exception is thrown if there was an unexpected error during processing of request

HTTP Status Code: 500

ServiceQuotaExceededException

This exception is thrown when a request is made beyond the service quota

HTTP Status Code: 402

ThrottlingException

This exception is thrown when the number of requests exceeds the limit

HTTP Status Code: 429

ValidationException

The input fails to satisfy the constraints specified by the service.

HTTP Status Code: 400