Analyze the results of a model customization job - Amazon Bedrock

Analyze the results of a model customization job

After a model customization job completes, you can analyze the results of the training process by looking at the files in the output S3 folder that you specified when you submitted the job or view details about the model. Amazon Bedrock stores your customized models in AWS-managed storage scoped to your account.

You can also evaluate your model by running a model evaluation job. For more information, see Evaluate the performance of Amazon Bedrock resources.

The S3 output for a model customization job contains the following output files in your S3 folder. The validation artifacts only appear if you included a validation dataset.

- model-customization-job-training-job-id/
    - training_artifacts/
        - step_wise_training_metrics.csv
    - validation_artifacts/
        - post_fine_tuning_validation/
            - validation_metrics.csv

Use the step_wise_training_metrics.csv and the validation_metrics.csv files to analyze the model customization job and to help you adjust the model as necessary.

The columns in the step_wise_training_metrics.csv file are as follows.

  • step_number – The step in the training process. Starts from 0.

  • epoch_number – The epoch in the training process.

  • training_loss – Indicates how well the model fits the training data. A lower value indicates a better fit.

  • perplexity – Indicates how well the model can predict a sequence of tokens. A lower value indicates better predictive ability.

The columns in the validation_metrics.csv file are the same as the training file, except that validation_loss (how well the model fits the validation data) appears in place of training_loss.

You can find the output files by opening up the https://console.aws.amazon.com/s3 directly or by finding the link to the output folder within your model details. Choose the tab for your preferred method, and then follow the steps:

Console

  1. Sign in to the AWS Management Console using an IAM role with Amazon Bedrock permissions, and open the Amazon Bedrock console at https://console.aws.amazon.com/bedrock/.

  2. From the left navigation pane, choose Custom models under Foundation models.

  3. In the Models tab, select a model to view its details. The Job name can be found in the Model details section.

  4. To view the output S3 files, select the S3 location in the Output data section.

  5. Find the training and validation metrics files in the folder whose name matches the Job name for the model.

API

To list information about all your custom models, send a ListCustomModels (see link for request and response formats and field details) request with an Amazon Bedrock control plane endpoint. Refer to ListCustomModels for filters that you can use.

To list all the tags for a custom model, send a ListTagsForResource request with an Amazon Bedrock control plane endpoint and include the Amazon Resource Name (ARN) of the custom model.

To monitor the status of a model customization job, send a GetCustomModel (see link for request and response formats and field details) request with an Amazon Bedrock control plane endpoint with the modelIdentifier, which is either of the following.

  • The name that you gave the model.

  • The ARN of the model.

You can see trainingMetrics and validationMetrics for a model customization job in either the GetModelCustomizationJob or GetCustomModel response.

To download the training and validation metrics files, follow the steps at Downloading objects. Use the S3 URI you provided in the outputDataConfig.

See code examples