Guidelines for model customization - Amazon Bedrock
Amazon Titan Text Premier

Guidelines for model customization

The ideal parameters for customizing a model depend on the dataset and the task for which the model is intended. You should experiment with values to determine which parameters work best for your specific case. To help, evaluate your model by running a model evaluation job. For more information, see Model evaluation.

This topic provides guidelines and recommended values as a baseline for customization of the Amazon Titan Text Premier model. For other models, check the provider's documentation.

Use the training and validation metrics from the output files generated when you submit a fine-tuning job to help you adjust your parameters. Find these files in the Amazon S3 bucket to which you wrote the output, or use the GetCustomModel operation.

Amazon Titan Text Premier

The following guidelines are for the Titan Text Premier text-to-text model model. For information about the hyperparameters that you can set, see Amazon Titan text model customization hyperparameters.

Impact on other tasks types

In general, the larger the training dataset, the better the performance for a specific task. However, training for a specific task might make the model perform worse on different tasks, especially if you use a lot of examples. For example, if the training dataset for a summarization task contains 100,000 samples, the model might perform worse on a classification task).

Model size

In general, the larger the model, the better the task performs given limited training data.

If you are using the model for a classification task, you might see relatively small gains for few-shot fine-tuning (less than 100 samples), especially if the number of classes is relatively small (less than 100).

Epochs

We recommend using the following metrics to determine the number of epochs to set:

  1. Validation output accuracy – Set the number of epochs to one that yields a high accuracy.

  2. Training and validation loss – Determine the number of epochs after which the training and validation loss becomes stable. This corresponds to when the model converges. Find the training loss values in the step_wise_training_metrics.csv and validation_metrics.csv files.

Batch size

When you change the batch size, we recommend that you change the learning rate using the following formula:

newLearningRate = oldLearningRate x newBatchSize / oldBatchSize

Titan Text Premier model currently only supports mini-batch size of 1 for customer finetuning.

Learning rate

To get the best results from finetuning capabilities, we recommend using a learning rate between 1.00E-07 and 1.00E-05. A good starting point is the recommended default value of 1.00E-06. A larger learning rate may help training converge faster, however, it may adversely impact core model capabilities.

Validate your training data with small sub-sample - To validate the quality of your training data, we recommend experimenting with a smaller dataset (~100s of samples) and monitoring the validation metrics, before submitting the training job with larger training dataset.

Learning warmup steps

We recommend the default value of 5.