在 Amazon Bedrock 中建立使用判斷模型的模型評估任務

下列範例示範如何使用 提出create-evaluation-job請求 AWS CLI。使用模型指標中找到的資料表做為判斷，以查看如何正確格式化 metricNames。確定 evaluationContext 指定為 Model。

aws bedrock create-evaluation-job

{
	"jobName": "model-eval-llmaj",
	"roleArn": "arn:aws:iam::111122223333:role/Amazon-Bedrock-ModelAsAJudgeTest",
	"evaluationContext": "Model",
	"evaluationConfig": {
		"automated": {
			"datasetMetricConfigs": [
	{
	"taskType": "General",
	"dataset": {
	"name": "text_dataset",
	"datasetLocation": {
	"s3Uri": "s3://bedrock-model-as-a-judge-test-1/input_datasets/text_dataset_input.jsonl"
	}
	},
	"metricNames": [
	"Builtin.Correctness",
	"Builtin.Completeness"
	]
	}
	],
	"evaluatorModelConfig": {
	"bedrockEvaluatorModels": [
	{
	"modelIdentifier": "anthropic.claude-3-haiku-20240307-v1:0"
	}
	]
	}
	}
	},
	"inferenceConfig": {
	"models": [
	{
	"bedrockModel": {
	"modelIdentifier": "anthropic.claude-v2",
	"inferenceParams": "{}"
	}
	}
	]
	},
	"outputDataConfig": {
	"s3Uri": "s3://bedrock-model-as-a-judge-test-1/output_data/"
	}
	}

當您在 Amazon Bedrock 主控台之外建立以人類為基礎的模型評估任務時，您需要建立 Amazon SageMaker AI 流程定義 ARN。

流程定義 ARN 是定義模型評估任務工作流程的地方。流程定義用於定義工作者界面，以及您想要指派給任務的工作團隊，以及連線到 Amazon Bedrock。

對於開始使用 Amazon Bedrock API 操作的模型評估任務，您必須使用 AWS CLI 或支援的 AWS SDK 建立流程定義 ARN。若要進一步了解流程定義的運作方式，並以程式設計方式建立流程，請參閱 SageMaker AI 開發人員指南中的建立人工審核工作流程 (API)。

在 中CreateFlowDefinition，您必須指定 AWS/Bedrock/Evaluation做為 的輸入AwsManagedHumanLoopRequestSource。Amazon Bedrock 服務角色也必須具有存取流程定義輸出儲存貯體的許可。

以下是使用 AWS CLI的範例請求。在請求中， HumanTaskUiArn 是 SageMaker AI 擁有的 ARN。在 ARN 中，您只能修改 AWS 區域。

aws sagemaker create-flow-definition --cli-input-json '
{
	"FlowDefinitionName": "human-evaluation-task01",
	"HumanLoopRequestSource": {
        "AwsManagedHumanLoopRequestSource": "AWS/Bedrock/Evaluation"
    },
    "HumanLoopConfig": {
		"WorkteamArn": "arn:aws:sagemaker:AWS 區域:111122223333:workteam/private-crowd/my-workteam",
		## The Task UI ARN is provided by the service team, you can only modify the AWS Region.
		"HumanTaskUiArn":"arn:aws:sagemaker:AWS Region:394669845002:human-task-ui/Evaluation"
		"TaskTitle": "Human review tasks",
		"TaskDescription": "Provide a real good answer",
		"TaskCount": 1,
		"TaskAvailabilityLifetimeInSeconds": 864000,
		"TaskTimeLimitInSeconds": 3600,
		"TaskKeywords": [
		    "foo"
        ]
    },
    "OutputConfig": {
        "S3OutputPath": "s3://your-output-bucket"
    },
    "RoleArn": "arn:aws:iam::111122223333:role/SageMakerCustomerRoleArn"
}'

下列程式碼範例示範如何建立模型評估任務，透過適用於 Python 的 SDK 使用人力工作者。

import boto3
client = boto3.client('bedrock')

job_request = client.create_evaluation_job(
    jobName="111122223333-job-01",
    jobDescription="two different task types",
    roleArn="arn:aws:iam::111122223333:role/example-human-eval-api-role",
    inferenceConfig={
        ## You must specify and array of models
        "models": [
            {
                "bedrockModel": {
                    "modelIdentifier":"arn:aws:bedrock:us-west-2::foundation-model/amazon.titan-text-lite-v1",
                    "inferenceParams":"{\"temperature\":\"0.0\", \"topP\":\"1\", \"maxTokenCount\":\"512\"}"
                }

            },
            {
                "bedrockModel": {
                    "modelIdentifier": "anthropic.claude-v2",
                    "inferenceParams": "{\"temperature\":\"0.25\",\"top_p\":\"0.25\",\"max_tokens_to_sample\":\"256\",\"top_k\":\"1\"}"
                }
            }
        ]

    },
    outputDataConfig={
        "s3Uri":"s3://job-bucket/outputs/"
    },
    evaluationConfig={
        "human": {
        "humanWorkflowConfig": {
            "flowDefinitionArn": "arn:aws:sagemaker:us-west-2:111122223333:flow-definition/example-workflow-arn",
            "instructions": "some human eval instruction"
        },
        "customMetrics": [
            {
                "name": "IndividualLikertScale",
                "description": "testing",
                "ratingMethod": "IndividualLikertScale"
            }
        ],
        "datasetMetricConfigs": [
            {
                "taskType": "Summarization",
                "dataset": {
                    "name": "Custom_Dataset1",
                    "datasetLocation": {
                        "s3Uri": "s3://job-bucket/custom-datasets/custom-trex.jsonl"
                    }
                },
                "metricNames": [
                  "IndividualLikertScale"
                ]
            }
        ]
      }

    }
)

print(job_request)