Human-based model evaluation jobs

To create a model evaluation job that uses human workers

1. Open the Amazon Bedrock console: https://console.aws.amazon.com/bedrock/

2. In the navigation pane, choose Model evaluation.

3. In the Build an evaluation card, under Human: bring your own team choose Create human-based evaluation.

4. On the Specify job details page provide the following.

   1. Evaluation name — Give the model evaluation job a name that describes the job. This name is shown in your model evaluation job list. The name must be unique in your account in an AWS Region.

   2. Description (Optional) — Provide an optional description.

5. Then, choose Next.

6. On the Set up evaluation page provide the following.

   1. Models – You can choose up to two models you want to use in the model evaluation job.

      To learn more about available models in Amazon Bedrock, see Access Amazon Bedrock foundation models.

   2. (Optional) To change the inference configuration for the selected models choose update.

      Changing the inference configuration changes the responses generated by the selected models. To learn more about the available inferences parameters, see Inference request parameters and response fields for foundation models.

   3. Task type – Choose the type of task you want the model to attempt to perform during the model evaluation job. All instructions for the model must be included in the prompts themselves. The task type does not control the model's responses.

   4. Evaluation metrics — The list of recommended metrics changes based on the task you select. For each recommended metric, you must select a Rating method. You can have a maximum of 10 evaluation metrics per model evaluation job.

   5. (Optional) Choose Add metric to add a metric. You must define the Metric, Description, and Rating method.

   6. In the Datasets card you must provide the following.

      1. Choose a prompt dataset – Specify the S3 URI of your prompt dataset file or choose Browse S3 to see available S3 buckets. You can have a maximum of 1000 prompts in a custom prompt dataset.

      2. Evaluation results destination – You must specify the S3 URI of the directory where you want the results of your model evaluation job saved, or choose Browse S3 to see available S3 buckets.

   7. (Optional) AWS KMS key – Provide the ARN of the customer managed key you want to use to encrypt your model evaluation job.

   8. In the Amazon Bedrock IAM role – Permissions card, you must do the following. To learn more about the required permissions for model evaluations, see Service role requirements for model evaluation jobs.

      1. To use an existing Amazon Bedrock service role, choose Use an existing role. Otherwise, use Create a new role to specify the details of your new IAM service role.

      2. In Service role name, specify the name of your IAM service role.

      3. When ready, choose Create role to create the new IAM service role.

7. Then, choose Next.

8. In the Permissions card, specify the following. To learn more about the required permissions for model evaluations, see Service role requirements for model evaluation jobs.

9. Human workflow IAM role – Specify a SageMaker AI service role that has the required permissions.

10. In the Work team card, specify the following.

    Human worker notification requirements

    When you add a new human worker to a model evaluation job, they automatically receive an email inviting them to participate in the model evaluation job. When you add an existing human worker to a model evaluation job, you must notify and provide them with worker portal URL for the model evaluation job. The existing worker will not receive an automated email notification that they are added to the new model evaluation job.

    1. Using the Select team dropdown, specify either Create a new work team or the name of an existing work team.

    2. (Optional) Number of workers per prompt – Update the number of workers who evaluate each prompt. After the responses for each prompt have been reviewed by the number of workers you selected, the prompt and its responses will be taken out of circulation from the work team. The final results report will include all ratings from each worker.

    3. (Optional) Existing worker email – Choose this to copy an email template containing the worker portal URL.

    4. (Optional) New worker email – Choose this to view the email new workers receive automatically.

       Important

       Large language models are known to occasionally hallucinate and produce toxic or offensive content. Your workers may be shown toxic or offensive material during this evaluation. Ensure you take proper steps to train and notify them before they work on the evaluation. They can decline and release tasks or take breaks during the evaluation while accessing the human evaluation tool.

11. Then, choose Next.

12. On the Provide instruction page use the text editor to provide instructions for completing the task. You can preview the evaluation UI that your work team uses to evaluate the responses, including the metrics, rating methods, and your instructions. This preview is based on the configuration you have created for this job.

13. Then, choose Next.

14. On the Review and create page, you can view a summary of the options you've selected in the previous steps.

15. To start your model evaluation job, choose Create.

    Note

    Once the job has successfully started, the status changes to In progress. When the job has finished, the status changes to Completed. While a model evaluation job is still In progress, you can choose to the stop the job before all the models' responses have been evaluated by your work team. To do so, choose Stop evaluation on the model evaluation landing page. This will change the Status of the model evaluation job to Stopping. Once the model evaluation job has successfully stopped, you can delete the model evaluation job.