EvaluationDatasetMetricConfig

Defines the built-in prompt datasets, built-in metric names and custom metric names, and the task type.

dataset

Specifies the prompt dataset.

Required: Yes

metricNames

The names of the metrics you want to use for your evaluation job.

For automated model evaluation jobs, valid values are "Builtin.Accuracy", "Builtin.Robustness", and "Builtin.Toxicity".

For human-based model evaluation jobs, the list of strings must match the name parameter specified in HumanEvaluationCustomMetric.

Type: Array of strings

Array Members: Minimum number of 1 item. Maximum number of 10 items.

Length Constraints: Minimum length of 1. Maximum length of 63.

Pattern: ^[0-9a-zA-Z-_.]+$

Required: Yes

taskType

TThe the type of task you want to evaluate for your evaluation job.

Type: String

Length Constraints: Minimum length of 1. Maximum length of 63.

Pattern: ^[A-Za-z0-9]+$

Valid Values: Summarization | Classification | QuestionAndAnswer | Generation | Custom

Required: Yes

Contents