# Increase model invocation capacity with Provisioned Throughput in Amazon Bedrock **Throughput** refers to the number and rate of inputs and outputs that a model processes and returns. You can purchase **Provisioned Throughput** to provision a higher level of throughput for a model at a fixed cost. If you customized a model, you must purchase Provisioned Throughput to be able to use it. You're billed hourly for a Provisioned Throughput that you purchase. For detailed information about pricing, see [Amazon Bedrock Pricing](https://aws.amazon.com/bedrock/pricing). The price per hour depends on the following factors: 1. The model that you choose (for custom models, pricing is the same as the base model that it was customized from). 1. The number of Model Units (MUs) that you specify for the Provisioned Throughput. An MU delivers a specific throughput level for the specified model. The throughput level of an MU specifies the following: + The number of input tokens that an MU can process across all requests within a span of one minute. + The number of output tokens that an MU can generate across all requests within a span of one minute. **Note** For more information about what an MU specifies, pricing per MU, and to request limit increases, contact your AWS account manager. 1. The duration of time you commit to keeping the Provisioned Throughput. The longer the commitment duration, the more discounted the hourly price becomes. You can choose between the following levels of commitment: + No commitment – You can delete the Provisioned Throughput at any time. + 1 month – You can't delete the Provisioned Throughput until the one month commitment term is over. + 6 months – You can't delete the Provisioned Throughput until the six month commitment term is over. **Note** Billing continues until you delete the Provisioned Throughput. The following steps outline the process of setting up and using Provisioned Throughput. 1. Determine the number of MUs you wish to purchase for a Provisioned Throughput and the amount of time for which you want to commit to using the Provisioned Throughput. 1. Purchase Provisioned Throughput for a base or custom model. 1. After the provisioned model is created, you can use it to [run model inference](inference.md). **Topics** + [Supported Region and models for Provisioned Throughput](prov-thru-supported.md) + [Prerequisites for Provisioned Throughput](prov-thru-prereq.md) + [Purchase a Provisioned Throughput for an Amazon Bedrock model](prov-thru-purchase.md) + [View information about a Provisioned Throughput](prov-thru-info.md) + [Modify a Provisioned Throughput](prov-thru-edit.md) + [Use a Provisioned Throughput with an Amazon Bedrock resource](prov-thru-use.md) + [Delete a Provisioned Throughput or cancel auto renew](prov-thru-delete.md) + [Code examples for Provisioned Throughput](prov-thru-code-examples.md) # Supported Region and models for Provisioned Throughput If you purchase Provisioned Throughput through the Amazon Bedrock API, you must specify a contextual variant of Amazon Bedrock FMs for the model ID. **Note** Provisioned Throughput is supported in AWS GovCloud (US-West) only for custom models with a no-commitment purchase. Use the ID of a custom model when purchasing Provisioned Throughput for it. The following table shows the models for which you can purchase Provisioned Throughput, the model ID to use when purchasing Provisioned Throughput, and the AWS Regions in which you can purchase Provisioned Throughput for the model. | Provider | Model | Model ID | Single-region model support | | --- | --- | --- | --- | | Amazon | Nova 2 Lite | amazon.nova-2-lite-v1:0:256k | us-east-1 | | Amazon | Nova Canvas | amazon.nova-canvas-v1:0 | us-east-1 | | Amazon | Nova Lite | amazon.nova-lite-v1:0:24k | us-east-1 | | Amazon | Nova Lite | amazon.nova-lite-v1:0:300k | us-east-1 | | Amazon | Nova Micro | amazon.nova-micro-v1:0:128k | us-east-1 | | Amazon | Nova Micro | amazon.nova-micro-v1:0:24k | us-east-1 | | Amazon | Nova Pro | amazon.nova-pro-v1:0:24k | us-east-1 | | Amazon | Nova Pro | amazon.nova-pro-v1:0:300k | us-east-1 | | Amazon | Titan Embeddings G1 - Text | amazon.titan-embed-text-v1:2:8k | us-east-1 us-west-2 | | Amazon | Titan Image Generator G1 v2 | amazon.titan-image-generator-v2:0 | us-east-1 us-west-2 | | Amazon | Titan Multimodal Embeddings G1 | amazon.titan-embed-image-v1:0 | ap-south-1 ap-southeast-2 ca-central-1 eu-central-1 eu-west-1 eu-west-2 eu-west-3 sa-east-1 us-east-1 us-west-2 | | Anthropic | Claude | anthropic.claude-v2:0:100k | us-east-1 us-west-2 | | Anthropic | Claude | anthropic.claude-v2:0:18k | us-east-1 us-west-2 | | Anthropic | Claude | anthropic.claude-v2:1:18k | eu-central-1 us-east-1 us-west-2 | | Anthropic | Claude | anthropic.claude-v2:1:200k | eu-central-1 us-east-1 us-west-2 | | Anthropic | Claude 3 Haiku | anthropic.claude-3-haiku-20240307-v1:0:200k | ap-southeast-2 eu-west-3 us-east-1 us-west-2 | | Anthropic | Claude 3 Haiku | anthropic.claude-3-haiku-20240307-v1:0:48k | ap-south-1 ap-southeast-2 eu-west-1 eu-west-3 us-east-1 us-west-2 | | Anthropic | Claude 3 Sonnet | anthropic.claude-3-sonnet-20240229-v1:0:200k | ap-southeast-2 eu-west-1 eu-west-3 us-east-1 us-west-2 | | Anthropic | Claude 3 Sonnet | anthropic.claude-3-sonnet-20240229-v1:0:28k | ap-south-1 ap-southeast-2 eu-west-1 eu-west-3 us-east-1 us-west-2 | | Anthropic | Claude 3.5 Sonnet | anthropic.claude-3-5-sonnet-20240620-v1:0:18k | us-west-2 | | Anthropic | Claude 3.5 Sonnet | anthropic.claude-3-5-sonnet-20240620-v1:0:200k | us-west-2 | | Anthropic | Claude 3.5 Sonnet | anthropic.claude-3-5-sonnet-20240620-v1:0:51k | us-west-2 | | Anthropic | Claude 3.5 Sonnet v2 | anthropic.claude-3-5-sonnet-20241022-v2:0:18k | us-west-2 | | Anthropic | Claude 3.5 Sonnet v2 | anthropic.claude-3-5-sonnet-20241022-v2:0:200k | us-west-2 | | Anthropic | Claude 3.5 Sonnet v2 | anthropic.claude-3-5-sonnet-20241022-v2:0:51k | us-west-2 | | Anthropic | Claude Instant | anthropic.claude-instant-v1:2:100k | us-east-1 us-west-2 | | Cohere | Embed English | cohere.embed-english-v3:0:512 | ca-central-1 eu-west-2 eu-west-3 sa-east-1 us-east-1 us-west-2 | | Cohere | Embed Multilingual | cohere.embed-multilingual-v3:0:512 | ca-central-1 eu-west-2 eu-west-3 sa-east-1 us-east-1 us-west-2 | | Meta | Llama 3.1 70B Instruct | meta.llama3-1-70b-instruct-v1:0:128k | us-west-2 | | Meta | Llama 3.1 8B Instruct | meta.llama3-1-8b-instruct-v1:0:128k | us-west-2 | | Meta | Llama 3.2 11B Instruct | meta.llama3-2-11b-instruct-v1:0:128k | us-west-2 | | Meta | Llama 3.2 1B Instruct | meta.llama3-2-1b-instruct-v1:0:128k | us-west-2 | | Meta | Llama 3.2 3B Instruct | meta.llama3-2-3b-instruct-v1:0:128k | us-west-2 | | Meta | Llama 3.2 90B Instruct | meta.llama3-2-90b-instruct-v1:0:128k | us-west-2 | **Note** The following models don't support no-commitment purchases for the base model: Titan Image Generator G1 V1 Titan Image Generator G1 V2 # Prerequisites for Provisioned Throughput Before you can purchase and manage Provisioned Throughput, you need to fulfill the following prerequisites: 1. [Request access to the model or models](model-access.md) that you want to purchase Provisioned Throughput for. After access has been granted, you can purchase Provisioned Throughput for the base model and any models customized from it. 1. Ensure that your IAM role has access to the Provisioned Throughput API actions. If your role has the [AmazonBedrockFullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonBedrockFullAccess) AWS-managed policy attached, you can skip this step. Otherwise, do the following: 1. Follow the steps at [Creating IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) and create the following policy, which allows a role to create a Provisioned Throughput for all foundation and custom models. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "PermissionsForProvisionedThroughput", "Effect": "Allow", "Action": [ "bedrock:GetFoundationModel", "bedrock:ListFoundationModels", "bedrock:GetCustomModel", "bedrock:ListCustomModels", "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream", "bedrock:ListTagsForResource", "bedrock:UntagResource", "bedrock:TagResource", "bedrock:CreateProvisionedModelThroughput", "bedrock:GetProvisionedModelThroughput", "bedrock:ListProvisionedModelThroughputs", "bedrock:UpdateProvisionedModelThroughput", "bedrock:DeleteProvisionedModelThroughput" ], "Resource": "*" } ] } ``` ------ **Note** If you're using Provisioned Throughput with cross-Region inference, you may need additional permissions. See [Increase throughput with cross-Region inference](cross-region-inference.md) to learn more. (Optional) You can restrict the role's access in the following ways: + To restrict the API actions that the role can make, modify the list in the `Action` field to contain only the [API operations](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonbedrock.html#amazonbedrock-actions-as-permissions) that you want to allow access to. + After creating a provisioned model, you can restrict the role's ability to perform an API request with the provisioned model by modifying the `Resource` list to contain only the [provisioned models](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonbedrock.html#amazonbedrock-resources-for-iam-policies) that you want to allow access to. For an example, see [Allow users to invoke a provisioned model](security_iam_id-based-policy-examples.md#security_iam_id-based-policy-examples-perform-actions-pt). + To restrict a role's ability to create provisioned models from specific foundation or custom models, modify the `Resource` list to contain only the [foundation and custom models](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonbedrock.html#amazonbedrock-resources-for-iam-policies) that you want to allow access to. 1. Follow the steps at [Adding and removing IAM identity permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) to attach the policy to a role to grant the role permissions. 1. If you're purchasing Provisioned Throughput for a custom model that's encrypted with a customer-managed AWS KMS key, your IAM role must have permissions to decrypt the key. You can use the template at [Understand how to create a customer managed key and how to attach a key policy to it](encryption-custom-job.md#encryption-key-policy). For minimal permissions, you can use only the *Permissions for custom model users* policy statement. # Purchase a Provisioned Throughput for an Amazon Bedrock model Amazon Bedrock offers two types of Provisioned Throughput - by Tokens and by Model Units. Refer to the following instructions for the type of Provisioned Throughput you wish to to purchase. To learn more about the differences between the two types of Provisioned Throughput, see [Increase model invocation capacity with Provisioned Throughput in Amazon Bedrock](prov-throughput.md). ## Provisioned Throughput by Model Units When you purchase a Provisioned Throughput by Model Units for a model, you specify the level of commitment for it and the number of model units (MUs) to allot. For MU quotas, see [Amazon Bedrock endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/bedrock.html) in the AWS General Reference. Before you can purchase a Provisioned Throughput (with commitment or no commitment), you must first visit the [AWS support center](https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase) to request MUs for your account to be distributed between the Provisioned Throughputs. After your request has been granted, you can purchase a Provisioned Throughput. **Note** After you purchase the Provisioned Throughput, if it's associated with a custom model, you can change the model by specifying one of the following options: The base model from which the custom model was customized Another custom model that was customized from the same base model as the custom model You can only change the associated model for Provisioned Throughputs associated with a custom model. To learn how to purchase Provisioned Throughput for a model, choose the tab for your preferred method, and then follow the steps: ------ #### [ Console ] 1. Sign in to the AWS Management Console with an IAM identity that has permissions to use the Amazon Bedrock console. Then, open the Amazon Bedrock console at [https://console.aws.amazon.com/bedrock](https://console.aws.amazon.com/bedrock). 1. Select **Provisioned Throughput** from the left navigation pane. 1. In the **Provisioned Throughput** section, choose **Purchase Provisioned Throughput**. 1. For the **Provisioned Throughput details** section, do the following: 1. In the **Provisioned Throughput name** field, enter a name for the Provisioned Throughput. 1. Under **Select model**, select a base model provider or a custom model category. Then select the model for which to provision throughput. **Note** To see the base models for which you can purchase Provisioned Throughput without commitment, see the supported models documentation. In the AWS GovCloud (US) Region, you can only purchase Provisioned Throughput for custom models with no commitment. 1. (Optional) To associate tags with your Provisioned Throughput, expand the **Tags** section and choose **Add new tag**. For more information, see [Tagging Amazon Bedrock resources](tagging.md). 1. For **Provisioning mode**, select **By Model Units** 1. For the **Commitment term & model units** section, do the following: 1. In the **Select commitment term** section, select the amount of time for which you want to commit to using the Provisioned Throughput. 1. In the **Model units** field, enter the desired number of model units (MUs). If you are provisioning a model with commitment, you must first visit the [AWS support center](https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase) to request an increase in the number of MUs that you can purchase. 1. Choose **Purchase Provisioned Throughput**. 1. Review the note that appears and acknowledge the commitment duration and price by selecting the checkbox. Then choose **Confirm purchase**. 1. The console displays the **Provisioned Throughput** overview page. The **Status** of the Provisioned Throughput in the Provisioned Throughput table becomes **Creating**. When the Provisioned Throughput is finished being created, the **Status** becomes **In service**. If the update fails, the **Status** becomes **Failed**. ------ #### [ API ] To purchase a Provisioned Throughput, send a [CreateProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateProvisionedModelThroughput.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp). To learn more about the contents of the request body and the parameters you need to supply to create a Provisioned Throughput by Model Units, see [CreateProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateProvisionedModelThroughput.html) in the *Amazon Bedrock API Reference*. **Note** To see the base models for which you can purchase Provisioned Throughput without commitment, see the supported models documentation. In the AWS GovCloud (US) Region, you can only purchase Provisioned Throughput for custom models with no commitment. The response returns a `provisionedModelArn` that you can use as a `modelId` in [model inference](inference.md). To check when the Provisioned Throughput is ready for use, send a [GetProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_GetProvisionedModelThroughput.html) request and check that the status is `InService`. If the update fails, its status will be `Failed`, and the [GetProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_GetProvisionedModelThroughput.html) response will contain a `failureMessage`. [See code examples](prov-thru-code-examples.md) ------ # View information about a Provisioned Throughput To learn how to view information about a Provisioned Throughput that you've purchased, choose the tab for your preferred method, and then follow the steps: ------ #### [ Console ] **To view information about a Provisioned Throughput** 1. Sign in to the AWS Management Console with an IAM identity that has permissions to use the Amazon Bedrock console. Then, open the Amazon Bedrock console at [https://console.aws.amazon.com/bedrock](https://console.aws.amazon.com/bedrock). 1. Select **Provisioned Throughput** from the left navigation pane. 1. From the **Provisioned Throughput** section, select a Provisioned Throughput. 1. View the details for the Provisioned Throughput in the **Provisioned Throughput overview** section and the tags associated with your Provisioned Throughput in the **Tags** section. ------ #### [ API ] To retrieve information about a specific Provisioned Throughput, send a [GetProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_GetProvisionedModelThroughput.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp). Specify either the name of the Provisioned Throughput or its ARN as the `provisionedModelId`. To list information about all the Provisioned Throughputs in an account, send a [ListProvisionedModelThroughputs](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListProvisionedModelThroughputs.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp). To control the number of results that are returned, you can specify the following optional parameters: **** | Field | Short description | | --- | --- | | maxResults | The maximum number of results to return in a response. | | nextToken | If there are more results than the number you specified in the maxResults field, the response returns a nextToken value. To see the next batch of results, send the nextToken value in another request. | For other optional parameters that you can specify to sort and filter the results, see [ListProvisionedModelThroughputs](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListProvisionedModelThroughputs.html). To list all the tags for a Provisioned Throughput, send a [ListTagsForResource](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListTagsForResource.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp) and include the Amazon Resource Name (ARN) of the Provisioned Throughput. [See code examples](prov-thru-code-examples.md) ------ # Modify a Provisioned Throughput The aspects of a Provisioned Throughput you can edit after purchase depend on the provisioning mode. For Provisioned Throughputs by Model Units, you can edit only the name and tags of your Provisioned Throughput, and the model if it's a custom model. With Provisioned Throughputs by Tokens, you have more options, including modifying the number of input and output tokens per minute for your Provisioned Throughput. Refer to the following sections to learn more about editing the type of Provisioned Throughput you want to modify. ## Modify a Provisioned Throughput by Model Units You can edit the name or tags of an existing Provisioned Throughput. The following restrictions apply to changing the model that the Provisioned Throughput is associated with: + You can't change the model for a Provisioned Throughput associated with a base model. + If the Provisioned Throughput is associated with a custom model, you can change the association to the base model that it's customized from, or to another custom model that was derived from the same base model. While a Provisioned Throughput is updating, you can run inference using the Provisioned Throughput without disrupting the on-going traffic from your end customers. If you changed the model that the Provisioned Throughput is associated with, you might receive output from the old model until the update is fully deployed. To learn how to edit a Provisioned Throughput, choose the tab for your preferred method, and then follow the steps: ------ #### [ Console ] 1. Sign in to the AWS Management Console with an IAM identity that has permissions to use the Amazon Bedrock console. Then, open the Amazon Bedrock console at [https://console.aws.amazon.com/bedrock](https://console.aws.amazon.com/bedrock). 1. Select **Provisioned Throughput** from the left navigation pane. 1. From the **Provisioned Throughput** section, select a Provisioned Throughput. 1. Choose **Edit**. You can edit the following fields: + **Provisioned Throughput name** – Change the name of the Provisioned Throughput. + **Select model** – If the Provisioned Throughput is associated with a custom model, you can change the associated model. 1. You can edit the tags associated with your Provisioned Throughput in the **Tags** section. For more information, see [Tagging Amazon Bedrock resources](tagging.md). 1. To save your changes, choose **Save edits**. 1. The console displays the **Provisioned Throughput** overview page. The **Status** of the Provisioned Throughput in the Provisioned Throughput table becomes **Updating**. When the Provisioned Throughput is finished being update, the **Status** becomes **In service**. If the update fails, the **Status** becomes **Failed**. ------ #### [ API ] To edit a Provisioned Throughput, send an [UpdateProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_UpdateProvisionedModelThroughput.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp). To learn more about the request body and the parameters you need to supply, see [UpdateProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_UpdateProvisionedModelThroughput.html) in the *Amazon Bedrock API Reference*. If the action is successful, the response returns an HTTP 200 status response. To check when the Provisioned Throughput is ready for use, send a [GetProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_GetProvisionedModelThroughput.html) request and check that the status is `InService`. You can't update or delete a Provisioned Throughput while its status is `Updating`. If the update fails, its status will be `Failed`, and the [GetProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_GetProvisionedModelThroughput.html) response will contain a `failureMessage`. To add tags to a Provisioned Throughput, send a [TagResource](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_TagResource.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp) and include the Amazon Resource Name (ARN) of the Provisioned Throughput. The request body contains a `tags` field, which is an object containing a key-value pair that you specify for each tag. To remove tags from a Provisioned Throughput, send an [UntagResource](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_UntagResource.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp) and include the Amazon Resource Name (ARN) of the Provisioned Throughput. The `tagKeys` request parameter is a list containing the keys for the tags that you want to remove. [See code examples](prov-thru-code-examples.md) ------ # Use a Provisioned Throughput with an Amazon Bedrock resource After you purchase a Provisioned Throughput, you can use it with the following features: + **Model inference** – You can test the Provisioned Throughput in an Amazon Bedrock console playground. When you're ready to deploy the Provisioned Throughput, set up your application to invoke the provisioned model. Choose the tab for your preferred method, and then follow the steps: ------ #### [ Console ] **To use a Provisioned Throughput in the Amazon Bedrock console playground** 1. Sign in to the AWS Management Console with an IAM identity that has permissions to use the Amazon Bedrock console. Then, open the Amazon Bedrock console at [https://console.aws.amazon.com/bedrock](https://console.aws.amazon.com/bedrock). 1. From the left navigation pane, select **Chat**, **Text**, or **Image** under **Playgrounds**, depending your use case. 1. Choose **Select model**. 1. In the **1. Category** column, select a provider or custom model category. Then, in the **2. Model** column, select the model that your Provisioned Throughput is associated with. 1. In the **3. Throughput** column, select your Provisioned Throughput. 1. Choose **Apply**. To learn how to use the Amazon Bedrock playgrounds, see [Generate responses in the console using playgrounds](playgrounds.md). ------ #### [ API ] To run inference using a Provisioned Throughput, send an [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html), [InvokeModelWithResponseStream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModelWithResponseStream.html), [Converse](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html), or [ConverseStream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html) request with an [Amazon Bedrock runtime endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-rt). Specify the provisioned model ARN as the `modelId` parameter. To see requirements for the request body for different models, see [Inference request parameters and response fields for foundation models](model-parameters.md). [See code examples](prov-thru-code-examples.md) ------ + **Associate a Provisioned Throughput with an agent alias** – You can associate a Provisioned Throughput when you [create](agents-deploy.md) or [update](agents-alias-edit.md) an agent alias. In the Amazon Bedrock console, you choose the Provisioned Throughput when setting up the alias or editing it. In the Amazon Bedrock API, you specify the `provisionedThroughput` in the `routingConfiguration` when you send a [CreateAgentAlias](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent_CreateAgentAlias.html) or [UpdateAgentAlias](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent_UpdateAgentAlias.html); request. # Delete a Provisioned Throughput or cancel auto renew Your Provisioned Throughput will automatically renew at the end of each commitment term, maintaining your current input and output tokens configurations. If you don't want to keep your Provisioned Throughput, you can delete it or, for Provisioned Throughput by Tokens, cancel auto renew to prevent it renewing when the current term ends. ## Deleting a Provisioned Throughput When you delete a Provisioned Throughput, you'll no longer be able to invoke the model at the throughput level that you purchased it for. If you delete a Provisioned Throughput associated with a custom model, the custom model isn't deleted. To learn how to delete a custom model, see [Delete a custom model](model-customization-delete.md). **Note** You can't delete a Provisioned Throughput by Model Units with commitment before the commitment term is complete. To learn how to delete a Provisioned Throughput, choose the tab for your preferred method, and then follow the steps: ------ #### [ Console ] 1. Sign in to the AWS Management Console with an IAM identity that has permissions to use the Amazon Bedrock console. Then, open the Amazon Bedrock console at [https://console.aws.amazon.com/bedrock](https://console.aws.amazon.com/bedrock). 1. Select **Provisioned Throughput** from the left navigation pane. 1. From the **Provisioned Throughput** section, select a Provisioned Throughput. 1. Choose **Delete** from the **Actions** dropdown menu. 1. The console displays a modal form to warn you that deletion is permanent. Choose **Confirm** to proceed. 1. The Provisioned Throughput is immediately deleted. ------ #### [ API ] To delete a Provisioned Throughput, send a [DeleteProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_DeleteProvisionedModelThroughput.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp). Specify either the name of the Provisioned Throughput or its ARN as the `provisionedModelId`. If deletion is successful, the response returns an HTTP 200 status code. [See code examples](prov-thru-code-examples.md) ------ ## Canceling auto renew for a Provisioned Throughput For Provisioned Throughput by Tokens, you can cancel auto renew at any point before your commitment term ends to prevent a Provisioned Throughput from automatically renewing. If you cancel auto renew, your Provisioned Throughput will remain in service until the end of your commitment term. You will still be charged the full provision fee for your current term, whether you run inference or not. After you cancel auto renew for a Provisioned Throughput, you can't make any further modifications to your Provisioned Throughput for the remainder of the commitment term. **Note** Auto renew cannot be re-enabled once cancelled. If you need Provisioned Throughput after your current term expires, you will need to purchase a new Provisioned Throughput. To learn how to cancel auto renew for a Provisioned Throughput by Tokens, choose the tab for your preferred method, and then follow the steps: ------ #### [ Console ] 1. Sign in to the AWS Management Console with an IAM identity that has permissions to use the Amazon Bedrock console. Then, open the Amazon Bedrock console at [https://console.aws.amazon.com/bedrock](https://console.aws.amazon.com/bedrock). 1. Select **Provisioned Throughput** from the left navigation pane. 1. From the **Provisioned Throughput** section, select a Provisioned Throughput. 1. Choose **Cancel auto renew** from the **Actions** dropdown menu. 1. The console displays a modal form to warn you that this action cannot be undone. Choose **Confirm** to proceed. 1. The Provisioned Throughput will remain active until the end of the current commitment term, after which it will be automatically deleted. ------ #### [ API ] To cancel auto renew for a Provisioned Throughput, send an [UpdateProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_UpdateProvisionedModelThroughput.html) request with an [Amazon Bedrock control plane endpoint](https://docs.aws.amazon.com/general/latest/gr/bedrock.html#br-cp) with the `disableAutoRenew` parameter set to `true`. The Provisioned Throughput will remain active until the end of the current commitment term. [See code examples](prov-thru-code-examples.md) ------ # Code examples for Provisioned Throughput The following code examples demonstrate how to create a Provisioned Throughput and how to manage and invoke it, using the AWS CLI and the Python SDK. You can create a Provisioned Throughput from a foundation model or from a model that you've already customized. Before you get started, carry out the following prerequisites: **Prerequisites** The following examples use the Amazon Nova Lite model, whose model ID is `amazon.nova-lite-v1:0:24k`. If you haven't already, request access to the Amazon Nova Lite by following the steps at [Manage model access using SDK and CLI](model-access.md#model-access-modify). If you want to purchase Provisioned Throughput for a different foundation model or a custom model, you'll have to do the following: 1. Find the model's ID (for foundation models), name (for custom models), or ARN (for either) by doing one of the following: + If you're purchasing a Provisioned Throughput for a foundation model, find the ID or Amazon Resource Name (ARN) of a model that supports provisioning in one of the following ways: + Look up the value in the table. + Send a [ListFoundationModels](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListFoundationModels.html) request and specify the `byInferenceType` value as `PROVISIONED` to see a list of models that support provisioning. Find the value in the `modelId` or `modelArn` field. + If you're purchasing a Provisioned Throughput for a custom model, find the name or Amazon Resource Name (ARN) of the model that you customized in one of the following ways: + In the Amazon Bedrock console, choose **Custom models** from the left navigation pane. Find the name of your customized model in the **Models** list or select it and find the **Model ARN** in the **Model details**. + Send a [ListCustomModels](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListCustomModels.html) request and find the `modelName` or `modelArn` value of your custom model in the response. 1. Modify the `body` of the [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) request in the examples below to match the format of the body of the model by finding it in [Inference request parameters and response fields for foundation models](model-parameters.md). Choose the tab for your preferred method, and then follow the steps: ------ #### [ AWS CLI ] 1. Send a [CreateProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateProvisionedModelThroughput.html) request to create a no-commitment Provisioned Throughput called *MyPT*, by running the following command in a terminal: ``` aws bedrock create-provisioned-model-throughput \ --model-units 1 \ --provisioned-model-name MyPT \ --model-id amazon.nova-lite-v1:0:24k ``` 1. The response returns a `provisioned-model-arn`. Allow some time for the creation to complete. To check its status, send a [GetProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_GetProvisionedModelThroughput.html) request and provide the name or ARN of the provisioned model as the `provisioned-model-id`, by running the following command: ``` aws bedrock get-provisioned-model-throughput \ --provisioned-model-id ${provisioned-model-arn} ``` 1. Run inference with your provisioned model by sending an [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) request. Provide the ARN of the provisioned model that was returned in the `CreateProvisionedModelThroughput` response, as the `model-id`. The output is written to a file named *output.txt* in your current folder. ``` aws bedrock-runtime invoke-model \ --model-id ${provisioned-model-arn} \ --body '{ "messages": [{ "role": "user", "content": [{ "text": "Hello" }] }], "inferenceConfig": { "temperature":0.7 } }' \ --cli-binary-format raw-in-base64-out \ output.txt ``` 1. Send a [DeleteProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_DeleteProvisionedModelThroughput.html) request to delete the Provisioned Throughput using the following command. You'll no longer be charged for the Provisioned Throughput. ``` aws bedrock delete-provisioned-model-throughput --provisioned-model-id MyPT ``` ------ #### [ Python (Boto) ] The following code snippets walk you through creating a Provisioned Throughput getting information about it, and invoking the Provisioned Throughput. 1. To create a no-commitment Provisioned Throughput called *MyPT* and assign the ARN of the Provisioned Throughput to a variable called *provisioned\$1model\$1arn*, send the following [CreateProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateProvisionedModelThroughput.html) request: ``` import boto3 provisioned_model_name = 'MyPT' bedrock = boto3.client(service_name='bedrock') response = bedrock.create_provisioned_model_throughput( modelUnits=1, provisionedModelName=provisioned_model_name, modelId='amazon.nova-lite-v1:0:24k' ) provisioned_model_arn = response['provisionedModelArn'] ``` 1. Allow some time for the creation to complete. You can check its status with the following code snippet. You can provide either the name of the Provisioned Throughput or the ARN returned from the [CreateProvisionedModelThroughput](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateProvisionedModelThroughput.html) response as the `provisionedModelId`. ``` bedrock.get_provisioned_model_throughput(provisionedModelId=provisioned_model_name) ``` 1. Run inference with your updated provisioned model with the following command and using the ARN of the provisioned model as the `modelId`. ``` import json import logging import boto3 from botocore.exceptions import ClientError class ImageError(Exception): "Custom exception for errors returned by the model" def __init__(self, message): self.message = message logger = logging.getLogger(__name__) logging.basicConfig(level=logging.INFO) def generate_text(model_id, body): """ Generate text using your provisioned custom model. Args: model_id (str): The model ID to use. body (str) : The request body to use. Returns: response (json): The response from the model. """ logger.info( "Generating text with your provisioned custom model %s", model_id) brt = boto3.client(service_name='bedrock-runtime') accept = "application/json" content_type = "application/json" response = brt.invoke_model( body=body, modelId=model_id, accept=accept, contentType=content_type ) response_body = json.loads(response.get("body").read()) finish_reason = response_body.get("error") if finish_reason is not None: raise ImageError(f"Text generation error. Error is {finish_reason}") logger.info( "Successfully generated text with provisioned custom model %s", model_id) return response_body def main(): """ Entrypoint for example. """ try: logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") model_id = provisioned-model-arn body = json.dumps({ "inputText": "what is AWS?" }) response_body = generate_text(model_id, body) print(f"Input token count: {response_body['inputTextTokenCount']}") for result in response_body['results']: print(f"Token count: {result['tokenCount']}") print(f"Output text: {result['outputText']}") print(f"Completion reason: {result['completionReason']}") except ClientError as err: message = err.response["Error"]["Message"] logger.error("A client error occurred: %s", message) print("A client error occured: " + format(message)) except ImageError as err: logger.error(err.message) print(err.message) else: print( f"Finished generating text with your provisioned custom model {model_id}.") if __name__ == "__main__": main() ``` 1. Delete the Provisioned Throughput with the following code snippet. You'll no longer be charged for the Provisioned Throughput. ``` bedrock.delete_provisioned_model_throughput(provisionedModelId=provisioned_model_name) ``` ------